This week my main objectives were to integrate my code onto the pi, gather more video from the pianists, and prepare for the interim demo next week.
For integration, I ported my pre-existing pipeline to rely on an accelerated blaze-based version of Mediapipe. I used code from this repo as the base to add my pipeline to. Currently, I’ve included our old tension algorithm as a placeholder for the typical processing workload we’d be doing with each frame. This will allow us to get a realistic estimate for our anticipated framerate (20 fps woo!). The modularity of the code will allow me to update the tension algorithm independently of the system running on the RPi, making future independent work straightforward. A video of this fully integrated with buzzer is linked here.
During our last session with Professor Dueck’s students, I gathered more clips of the tense/ non-tense technique playing. I also gathered some videos of students playing familiar & unfamiliar pieces and split them into 10 second clips—Jessie will send out a form this weekend for the students to label these clips as tense/ nonsense. This will become the data I’ll use to verify that the tension algorithm uses.
Lastly, now that the system’s been integrated more, I’ll also start focusing on verifying my algorithm using the data gathered last week. My current plan is to run the system on the labeled clips, then comparing the output to label. I’ll record the number of times we match the label, and the number of times we give false positives & false negatives for tension. If some results are inaccurate, I’ll adjust some of the parameters for the algorithm and try the tests again. This would involve adding more span-based conditionals when looking for tension indicators, and adjusting the existing window size & convolution parameters from before.
Additionally, I’ll use the technique clips to verify that output from the RPI matches the output from the Mediapipe-based pipeline running on my laptop. This means running the same test video on both systems, writing output information on tension to a text file, and comparing the textfiles to check that they are consistent. If they are, I’ll be able to safely assume that any work on the tension algorithm on my laptop will produce the same result on the RPi, making integrated testing much easier as we’ll only need to test the algorithm with my laptop.