Category: Pareekshith’s Status Reports

This week, I focused mainly on re-testing my speech technology setup to ensure that nothing horrifically wrong had happened or will happen in the next two weeks, and I also met with Dr. Dueck and the School of Music students to troubleshoot our Accomplice setup and the cueing. One good thing is that the software now correctly follows the MIDI keyboard and is able to output cues properly, although I need the exact cue positions so that I can pinpoint exactly where and when our camera needs to turn. I also needed to order a new BLDC motor driver after one of the motor drivers we originally got through the School of Music was faulty, and so we received that today and look to complete the setup and fully integrate. My progress is on schedule, and since my portion is mostly complete (apart from getting the exact cue positions for the “Ah Perdona” piece), I’ll be helping out my team with any other necessary tasks this week as we look to complete the project and demo it at the TechSpark demo as well as at the public demo. I’ll also be working on the poster, the final video, and the final report this week so we won’t be pressed for time as we conclude the semester.

This week, I mostly worked on the opera portion of our collaboration with the School of Music and was debugging issues with the tempo of the pieces. After several attempts, I figured out the problem and was able to obtain the correct tempo for the piece that Dr. Dueck will be playing for our demo. There’s one more piece that I need to work on that is more difficult because of certain changes in tempo throughout the piece, but I’ll try to get that to work using specific cuing mechanisms. On the other hand, I am also working on testing the speech portion and using the WER metric to see the accuracy of the ASR. I don’t need the WER of the entire dialogue by an actor, simply the trigger phrase that’s used to pivot between actors. This shouldn’t take too long to do, and I should have it completed in about a day or so. This upcoming week, I plan to complete the validation and testing for the speech side, set up the cues in Accomplice, and help my team with any final debugging that needs to be done. While working on this project, I learned a lot about speech technology, fuzzy matching, and music technology. Interestingly, I learned a majority of the first two from my Speech Technology for Conversational AI course that I’m taking this semester, and that’s where I learned about WER, Vosk, and the other components of the speech side. On the other hand, learning about Accomplice, how it works, and how to use it needed me to get on a call with Professor Dannenberg, who was extremely kind and helpful. I was able to get most of the information I needed through our chat, and he’s been nice enough to be available for any questions and concerns that I might have.

This past week, I mostly completed the work on the speech track of the project, and I was able to show my progress in the demo where Ted and I played Nora and Torvald from A Doll’s House. I also added a manual override button in the UI, allowing the director to automatically trigger a cue in case, for some reason, the actor wasn’t heard correctly by the system. To complete this track, I need to simply add the specific motor controller instructions to the Python logic rather than only displaying to the UI. On the opera track, I also made progress, meeting with Professor Dannenberg to debug early setup issues I was facing with the off-the-shelf solution, Accomplice. After our call, I was able to correctly open the music projects and play the music through the SimpleSynth synthesizer. Our team was able to get access to the studio theater where we regularly meet Dr. Dueck and the School of Music, and so I was able to play around with the MIDI keyboard and connect it to my Mac. While most of the system works, I was facing issues with receiving the OSC messages that represent the triggers in the piano script, and it turns out that it’s an issue with the Accomplice software that Professor Dannenberg will resolve soon. I’m currently on progress with my schedule, and I hope to test and also complete the opera track by the end of the week once Accomplice is fixed. Regarding the verification testing, I’ve already been testing with scripts and seeing the success rate of each trigger. I’ve noticed that on modern English, Vosk does extremely well, with a 95+% accuracy, but with Middle English or Early modern English (e.g. Shakespeare, Marlowe, etc.), certain words like “thy” and “thine” aren’t translated correctly, and thus reduce the accuracy of the model. By using fuzzy matching and hacks like changing the trigger from “thy” to “by,” I’m able to improve the accuracy of the system, but I need to explore more permanent solutions that may help with WER and trigger precision. The speech track will rely mostly on these kinds of tests, using actual plays and scenes as references and counting the number of times the triggers don’t engage, and the results will be used to modify the triggers, increase the range of the fuzzy matching, and improve the system itself. For both the speech and opera tracks, I will be running latency tests to determine how long it takes for the system to acknowledge the trigger after the conclusion of the performer’s dialogue. We want to minimize the latency to a time that won’t be noticeable for the audience.

This week, I made progress on the speech as well as the musical portion of the project. I’ve decided to follow 2 tracks: one which focuses on opera and getting the piano music (as part of the collaboration with the School of Music) and the other which is based on plays and uses Vosk for ASR. For the demo, I’ll be showing the latter track and display the Lavalier microphone connected to the wearable node that’ll indicate the parsing of the speech and the switching of the pivots on a UI. Regarding the other path, we had a call with Professor Dannenberg during our meeting with Dr. Dueck and the School of Music, and he introduced an off-the-shelf solution that relies on MIDI files and software; this solution may be exactly what we need for our project. This solution has the ability to send information and commands through a robust network protocol developed by Professor Dannenberg, so we’d be able to send information to the central Pi and make it send the commands over to the motor controller. I’m currently on schedule with my work; I’ll be meeting with Professor Dannenberg next week to set up his solution locally and make sure it’s all working, and I’ll be further enhancing the speech solution and verifying that it works accurately when there are multiple microphones involved in a close setting. I hope to be able to get both systems to work for our final project presentation, as it’d be cool to demonstrate the different capabilities our system has and the various use cases we support.

This week, I did research into open-source libraries that we could use to do real-time music tracking. The one library that I found that works really well with our use case was Matchmaker, which was released last year and is able to output the current position in the script based on the sheet music and the audio data. With this, we might have to pivot from our Lavalier microphones to a Samson microphone that isn’t omnidirectional and can focus properly on the piano music and not the surrounding operatic dialogue. The only concern I have with this is that it works specifically for piano; other instruments and orchestras would most likely mess up the alignment. Because of this, I want to check with Dr. Dueck if using this library is feasible. We might need to slightly pivot with this, especially because I am unable to find any other library that works on orchestral music. During our group’s work session this Saturday, I also helped with the taping of the anchors and the battery packs. I’m currently on schedule, and I plan to experiment with the Samson microphone that one of my partners might have. There are some technical details that need to be honed out, such as the conversion of the sheet music into specific MusicXML files and the exact audio input format. These will be covered and explored next week, and I plan on doing on-device testing and integration with the Pis.

This past week, I received the Lavalier microphones from the ECE Receiving office and thus, got a better idea of how to integrate them into our system. When we met with Dr. Dueck and her students, we discussed placements of the wearable nodes and the mics under the costumes, and we will be in contact figuring out how exactly to hide them discreetly. The good news is that the microphones are 6 meters long, so if there is a safe spot on an actor, the microphone can easily clip on and work properly. I also brought up the dilemma that we have with Vox, namely that it won’t work as well as expected with an operatic voice. Because of this, we are considering switching gears and using the sheet music as the script and then indicating where the cues should happen. Although our original plan was to use speech technology, given the collaboration with the School of Music, it makes sense to go with this approach; if we have time near the end, we might also try to incorporate speech technology into our system for drama productions, which typically have more conversational speech than operas and musicals. I’m still on track with my schedule, although I might need some more time to research libraries that can track music in real time; we’ll also have to discuss with Dr. Dueck the situation regarding an orchestra, where multiple instruments will be playing and could cause interference. This upcoming week, I hope to do just that, and I also want to find a way to keep the microphones incorporated so that we don’t waste our budget for no reason; we might end up using the microphones as microphones, and these can then be used to simply store the audio from each actor and be used for post-production processing.

The week before spring break, my group and I were focused heavily on completing the design report. Along with detailed audio subsystem explanations and certain design tradeoffs, I specifically took charge of the Design Requirements and Project Management portions of the report. I also ordered the Lavalier microphones that were critical for the audio component. Along with this, the team also met Dr. Dueck and her students in the 57584 course from the School of Music, and we had a great conversation about the potential role of our project in opera productions. We pored over different details, explored optimizations, and voiced our opinions about potential challenges that could arise through the course of the collaboration. I’m in charge of keeping the group updated with our team’s schedule, and we plan to meet weekly to show our latest progress, get feedback, and have actual testers of our product. I’m currently on track with my schedule, and next week, I hope to receive my microphones (along with our team’s parts) and start setting up our wearable node and its corresponding software. The only thing I’m a bit worried about with the School of Music collaboration is that I don’t know if Vosk, our offline ASR library, will be able to handle opera voices that aren’t spoken English. We’ll have to see if it’s viable for that, and if not, we’d either need to search for a model that is compatible or abandon the audio component specifically for opera (it would still work for stage plays though).

This week, I was the speaker for our group’s design presentation, so I had to prepare and review the content of the slides and our design to make sure I was ready for the presentation as well as any further questions. During the Q&A portion, Tamal mentioned that using Raspberry Pi 5s for the wearable node component could be risky due to potential inconsistency, so he recommended using earlier models like the Raspberry Pi 4. In light of this, my group and I decided to look further into the differences and the causes for concern, and we plan to discuss this as well with our TA Rhea and Professor Qing Li. Also, as the speech lead, I’m looking into finalizing the design and pipeline for the audio component, making sure the right data is getting processed by the right libraries to maximize accuracy and minimize latency. Next week, I plan to finish the design report with my team, order the necessary components (especially for the audio system), and begin the software implementation of the system.

This week, I was working heavily on the final design and the design presentation due next week, which I’ll be presenting for my group. I also requested a USB 3.0 video camera that functions in low light, which is perfect in our scenario of a stage production. I’ve been doing more research on the auditory portion of the project and was able to get through what I was hoping I would be able to accomplish from last week. Specifically, I found certain modules/libraries that come with the Raspberry Pi 5 such as WebRTC VAD for the voice activity detection component and Vosk for the automatic speech recognition portion. During our group meetings, I was also looking into the UWB sensors and their compatibility with the Raspberry Pi 5, and my team and I discussed different options for synchronization of the anchors and the tags to the Pi. My progress is on schedule, and next week, I hope to deliver a good and explanatory presentation and also find the exact microphones that our group should be using. I’m looking at Lavalier microphones over Bluetooth microphones because of their better range, but I need to look into transceiver compatibility with our Raspberry Pi 5 ports.

This week, I worked with my group to complete the proposal presentation slides and discuss components that would be required. After some discussion, we ordered a Raspberry Pi 5 and a motor driver, and I’m also thinking of ordering the USB 3.0 video camera that’s available on the parts list. I’ve also started taking a look at the auditory component and the technology stack that would be needed to accomplish VAD (voice activity detection) and ASR (automatic speech recognition) reliably. Currently, I’m on schedule and next week, I look to finalize the design of the entire system and its integration with my group, and on my own end, I want to figure out exactly how the audio system would work in tandem with the Raspberry Pi 5 that we ordered. Specifically, I want to see if there are any online libraries or in-built features that come with the Pi that would be helpful for transcribing the speech from the actors’ microphones into usable text.