Feature #7148
New question type: Audio frequency analysis
Description
With the advent of open audio formats and the availability of audio frameworks in HTML5, it is now feasable to analyse the audio frequency of a recorded voice and compare it to another recorded voice, or even a live audio stream being pronounced by a user.
This makes it an ideal case for exercises of type "Audio pronounciation".
The concept would be simple:- the teacher records an audio and puts it into an audio-type question, then sets a minimum "pass" percentage for the question (+ a score received if succeeded)
- the student takes the test
- the question shows as an audio player with a frequency signal below
- while "playing" the sound, a vertical bar moves through the frequency
- the student click "record" and uses his/her microphone through HTML5 + WebAudio API
- while recording, the frequency of the sound appears to the student as another box below the first one
- the student presses the "stop" button
- the differences in frequency are analyzed and a score is given
- if the score is passed, the student frequency appears on green background and the student receives the success score (0/1) when moving to the next question
- frequency
- pitch
- speed
- strength
- starting time (recording might start only a certain time after the student pressed "record")
These parameters, considered as a whole, should provide a good overall result of how close the student was from the recording.
Obviously, these settings (thresholds) should be easily configurable by the developers or ideally by the platform administrator and/or the course teacher.
The visual representation doesn't have to be "complete" or "precise". It just needs to be a visual representation of how wrong the student might have been.
Project planning¶
- Identify audio-analysis JS libraries
- Identify customizable audio-recording and -playing JS libraries
- Implement recorder (HTML5-based, OGG format)
- Implement player of response audio (with debug to console.log) (HTML5-based, OGG format)
- Implement comparison algorithm (with debug to console.log)
- Tune variables of algorithm
- Integrate into exercises flow of Chamilo
Possible sources of inspiration¶
- http://stackoverflow.com/questions/4503181/birdsong-audio-analysis-finding-how-well-two-clips-match
- Duolingo app (it doesn't show a frequency, but it gets the same result)
- Any advanced desktop-based language-learning application
- The images attached
Files
| frequency-match.jpeg (14.3 KB) frequency-match.jpeg | |||
| frequency-match2.jpeg (10.8 KB) frequency-match2.jpeg |
History
Updated by
Updated by Francis Gonzales over 6 years ago
Updated by Francis Gonzales Some days ago I found these examples that can help:
Updated by Yannick Warnier over 6 years ago
- File frequency-match.jpeg frequency-match.jpeg added
- File frequency-match2.jpeg frequency-match2.jpeg added
- Description updated (diff)
Updated by Yannick Warnier over 6 years ago
Francis Gonzales wrote:
Some days ago I found these examples that can help:
The last one has been forked and advanced further here: https://github.com/itsjoesullivan/recorderjs
Updated by Yannick Warnier about 6 years ago
A nice tutorial to show live audio frequency analysis:
http://www.developphp.com/view.php?tid=1348
<!doctype html>
<html>
<head>
<style type="text/css">
div#mp3_player{ width:500px; height:60px; background:#000; padding:5px; margin:50px auto; }
div#mp3_player > div > audio{ width:500px; background:#000; float:left; }
div#mp3_player > canvas{ width:500px; height:30px; background:#002D3C; float:left; }
</style>
<script>
// Create a new instance of an audio object and adjust some of its properties
var audio = new Audio();
audio.src = 'track1.mp3';
audio.controls = true;
audio.loop = true;
audio.autoplay = true;
// Establish all variables that your Analyser will use
var canvas, ctx, source, context, analyser, fbc_array, bars, bar_x, bar_width, bar_height;
// Initialize the MP3 player after the page loads all of its HTML into the window
window.addEventListener("load", initMp3Player, false);
function initMp3Player(){
document.getElementById('audio_box').appendChild(audio);
context = new webkitAudioContext();
// AudioContext object instance
analyser = context.createAnalyser();
// AnalyserNode method
canvas = document.getElementById('analyser_render');
ctx = canvas.getContext('2d');
// Re-route audio playback into the processing graph of the AudioContext
source = context.createMediaElementSource(audio);
source.connect(analyser);
analyser.connect(context.destination);
frameLooper();
}
// frameLooper() animates any style of graphics you wish to the audio frequency
// Looping at the default frame rate that the browser provides(approx. 60 FPS)
function frameLooper(){
window.webkitRequestAnimationFrame(frameLooper);
fbc_array = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(fbc_array);
ctx.clearRect(0, 0, canvas.width, canvas.height);
// Clear the canvas ctx.fillStyle = '#00CCFF';
// Color of the bars bars = 100;
for (var i = 0; i < bars; i++) {
bar_x = i * 3;
bar_width = 2;
bar_height = -(fbc_array[i] / 2);
//fillRect( x, y, width, height )
// Explanation of the parameters below
ctx.fillRect(bar_x, canvas.height, bar_width, bar_height);
}
}
</script>
</head>
<body>
<div id="mp3_player">
<div id="audio_box"></div>
<canvas id="analyser_render"></canvas>
</div>
</body>
</html>
Technically, we should see the framework differently here, and divide the frequency canvas into sections that are proportional to the sound length and progressive (moving X axis) with only one bar that shows (mainly a human voice frequency or something like that).
I'm not sure exactly which frequency is shown when it is used to represent only one bar for a whole split-second... I will have to research that.
Updated by Yannick Warnier almost 6 years ago
- Assignee changed from anibal copitan to Imanol Losada
Updated by Yannick Warnier over 4 years ago
There's a basic implementation here, but it requires forking and integrating into Chamilo. This work was made for BeezNest but was not finished, so we still need to evaluate its quality.
Updated by Yannick Warnier over 4 years ago
- Status changed from New to Assigned
- Assignee changed from Imanol Losada to Angel Quiroz