Project

General

Profile

Feature #7148

New question type: Audio frequency analysis

Added by Yannick Warnier about 5 years ago. Updated about 3 years ago.

Status:
Assigned
Priority:
Low
Assignee:
Category:
Exercises
Target version:
Start date:
07/05/2014
Due date:
% Done:

0%

Estimated time:
150.00 h
Spent time:
Complexity:
Wizard-level
SCRUM pts - complexity:
100

Description

With the advent of open audio formats and the availability of audio frameworks in HTML5, it is now feasable to analyse the audio frequency of a recorded voice and compare it to another recorded voice, or even a live audio stream being pronounced by a user.

This makes it an ideal case for exercises of type "Audio pronounciation".

The concept would be simple:
  • the teacher records an audio and puts it into an audio-type question, then sets a minimum "pass" percentage for the question (+ a score received if succeeded)
  • the student takes the test
  • the question shows as an audio player with a frequency signal below
  • while "playing" the sound, a vertical bar moves through the frequency
  • the student click "record" and uses his/her microphone through HTML5 + WebAudio API
  • while recording, the frequency of the sound appears to the student as another box below the first one
  • the student presses the "stop" button
  • the differences in frequency are analyzed and a score is given
  • if the score is passed, the student frequency appears on green background and the student receives the success score (0/1) when moving to the next question
The algorithm of comparison wouldn't be that easy, as we need to include aspects like:
  • frequency
  • pitch
  • speed
  • strength
  • starting time (recording might start only a certain time after the student pressed "record")

These parameters, considered as a whole, should provide a good overall result of how close the student was from the recording.
Obviously, these settings (thresholds) should be easily configurable by the developers or ideally by the platform administrator and/or the course teacher.

The visual representation doesn't have to be "complete" or "precise". It just needs to be a visual representation of how wrong the student might have been.

Project planning

  • Identify audio-analysis JS libraries
  • Identify customizable audio-recording and -playing JS libraries
  • Implement recorder (HTML5-based, OGG format)
  • Implement player of response audio (with debug to console.log) (HTML5-based, OGG format)
  • Implement comparison algorithm (with debug to console.log)
  • Tune variables of algorithm
  • Integrate into exercises flow of Chamilo

Possible sources of inspiration



Files

frequency-match.jpeg (14.3 KB) frequency-match.jpeg Yannick Warnier, 07/05/2014 19:52
frequency-match2.jpeg (10.8 KB) frequency-match2.jpeg Yannick Warnier, 07/05/2014 19:52

History

#1

Updated by Yannick Warnier about 5 years ago

  • Description updated (diff)
#2

Updated by Francis Gonzales about 5 years ago

Some days ago I found these examples that can help:

https://webaudiodemos.appspot.com

https://github.com/mattdiamond/Recorderjs

#4

Updated by Yannick Warnier about 5 years ago

Francis Gonzales wrote:

Some days ago I found these examples that can help:

https://webaudiodemos.appspot.com

https://github.com/mattdiamond/Recorderjs

The last one has been forked and advanced further here: https://github.com/itsjoesullivan/recorderjs

#5

Updated by Yannick Warnier almost 5 years ago

A nice tutorial to show live audio frequency analysis:
http://www.developphp.com/view.php?tid=1348

<!doctype html>
<html>
 <head>
  <style type="text/css">
   div#mp3_player{ width:500px; height:60px; background:#000; padding:5px; margin:50px auto; }
   div#mp3_player > div > audio{ width:500px; background:#000; float:left; }
   div#mp3_player > canvas{ width:500px; height:30px; background:#002D3C; float:left; }
  </style>
  <script>
   // Create a new instance of an audio object and adjust some of its properties
   var audio = new Audio();
   audio.src = 'track1.mp3';
   audio.controls = true;
   audio.loop = true;
   audio.autoplay = true;
   // Establish all variables that your Analyser will use
   var canvas, ctx, source, context, analyser, fbc_array, bars, bar_x, bar_width, bar_height;
   // Initialize the MP3 player after the page loads all of its HTML into the window
   window.addEventListener("load", initMp3Player, false);
   function initMp3Player(){
    document.getElementById('audio_box').appendChild(audio);
    context = new webkitAudioContext();
    // AudioContext object instance
    analyser = context.createAnalyser();
    // AnalyserNode method
    canvas = document.getElementById('analyser_render');
    ctx = canvas.getContext('2d');
    // Re-route audio playback into the processing graph of the AudioContext
    source = context.createMediaElementSource(audio);
    source.connect(analyser);
    analyser.connect(context.destination);
    frameLooper();
   }
   // frameLooper() animates any style of graphics you wish to the audio frequency
   // Looping at the default frame rate that the browser provides(approx. 60 FPS)
   function frameLooper(){
    window.webkitRequestAnimationFrame(frameLooper);
    fbc_array = new Uint8Array(analyser.frequencyBinCount);
    analyser.getByteFrequencyData(fbc_array);
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    // Clear the canvas ctx.fillStyle = '#00CCFF';
    // Color of the bars bars = 100;
    for (var i = 0; i < bars; i++) {
     bar_x = i * 3;
     bar_width = 2;
     bar_height = -(fbc_array[i] / 2);
     //fillRect( x, y, width, height )
     // Explanation of the parameters below
     ctx.fillRect(bar_x, canvas.height, bar_width, bar_height);
    }
   }
  </script>
 </head>
 <body>
  <div id="mp3_player">
   <div id="audio_box"></div>
   <canvas id="analyser_render"></canvas>
  </div>
 </body>
</html>

Technically, we should see the framework differently here, and divide the frequency canvas into sections that are proportional to the sound length and progressive (moving X axis) with only one bar that shows (mainly a human voice frequency or something like that).
I'm not sure exactly which frequency is shown when it is used to represent only one bar for a whole split-second... I will have to research that.

#6

Updated by Yannick Warnier almost 5 years ago

  • Assignee set to anibal copitan
#7

Updated by Yannick Warnier over 4 years ago

  • Assignee changed from anibal copitan to Imanol Losada
#8

Updated by Yannick Warnier over 3 years ago

  • Target version changed from 2.0 to 3.0
#9

Updated by Yannick Warnier about 3 years ago

There's a basic implementation here, but it requires forking and integrating into Chamilo. This work was made for BeezNest but was not finished, so we still need to evaluate its quality.

https://github.com/amandic007/speech-comparison

#10

Updated by Yannick Warnier about 3 years ago

  • Status changed from New to Assigned
  • Assignee changed from Imanol Losada to Angel Quiroz
#11

Updated by Yannick Warnier about 3 years ago

  • Target version changed from 3.0 to 2.0

Also available in: Atom PDF