Pareekshith’s Status Report for 4/4

This past week, I mostly completed the work on the speech track of the project, and I was able to show my progress in the demo where Ted and I played Nora and Torvald from A Doll’s House. I also added a manual override button in the UI, allowing the director to automatically trigger a cue in case, for some reason, the actor wasn’t heard correctly by the system. To complete this track, I need to simply add the specific motor controller instructions to the Python logic rather than only displaying to the UI. On the opera track, I also made progress, meeting with Professor Dannenberg to debug early setup issues I was facing with the off-the-shelf solution, Accomplice. After our call, I was able to correctly open the music projects and play the music through the SimpleSynth synthesizer. Our team was able to get access to the studio theater where we regularly meet Dr. Dueck and the School of Music, and so I was able to play around with the MIDI keyboard and connect it to my Mac. While most of the system works, I was facing issues with receiving the OSC messages that represent the triggers in the piano script, and it turns out that it’s an issue with the Accomplice software that Professor Dannenberg will resolve soon. I’m currently on progress with my schedule, and I hope to test and also complete the opera track by the end of the week once Accomplice is fixed. Regarding the verification testing, I’ve already been testing with scripts and seeing the success rate of each trigger. I’ve noticed that on modern English, Vosk does extremely well, with a 95+% accuracy, but with Middle English or Early modern English (e.g. Shakespeare, Marlowe, etc.), certain words like “thy” and “thine” aren’t translated correctly, and thus reduce the accuracy of the model. By using fuzzy matching and hacks like changing the trigger from “thy” to “by,” I’m able to improve the accuracy of the system, but I need to explore more permanent solutions that may help with WER and trigger precision. The speech track will rely mostly on these kinds of tests, using actual plays and scenes as references and counting the number of times the triggers don’t engage, and the results will be used to modify the triggers, increase the range of the fuzzy matching, and improve the system itself. For both the speech and opera tracks, I will be running latency tests to determine how long it takes for the system to acknowledge the trigger after the conclusion of the performer’s dialogue. We want to minimize the latency to a time that won’t be noticeable for the audience.