Posted on

Team’s Status Report for 4/25

The most significant risks that could jeopardize the success of this project would be our inability to tune the motor controllers correctly so that they can properly give the right instructions to the motors themselves. We’re currently working on this portion of the project, and once that’s done, we should mostly be done with the project, apart from incorporating the audio component to handle pivoting to the designated actor after the previous actor’s dialogue is completed. Once that is done, we’ll be ready for the TechSpark demo along with the final demo, and we’ll also be able to meet on Thursday with Dr. Dueck and the School of Music for our demo with them for the “Ah Perdona” piece. Apart from that, we will of course also be working on the final report and the final video so that we can submit all of them in time when they are due.

On the audio side, the main tests that we performed were calculating the WER for the ASR system, specifically for the trigger phrases rather than the whole dialogue. We realized that it didn’t make sense to track the whole dialogue by each speaker, rather it would improve processing and accuracy if we only focused on the few words before the next actor’s part began. What we found was that the system was very accurate, with a 7% WER (10% is high-quality) on these trigger phrases, and that the trigger was almost always correctly initiated. However, the fact that it wasn’t perfect concerned us slightly, which is why we included a manual override button in the UI that allowed the director to control the cueing themselves in case the trigger didn’t work. On the tracking side, the main tests we performed were calculating the difference between the ground truth and the predictions by the UI, and we discovered that the accuracy was even better than we expected, allowing us to stick with our current design. On the overall system side, we tested the reliability of the entire system, and we confirmed that the nodes and other components all ran for at least 4 hours, which was what we were hoping for.

Posted on

Ted’s Status Report for 04/25/2026

This week, I’ve mainly been working on ironing out any small design issues with the physical rail system and tuning/calibrating the motor system. We had an issue where the replacement motor driver for the previously broken motor driver also arrived broken (hooray) but we managed to procure a replacement as fast as possible and am working on tuning right now. I’ve also started working on the poster and final report just to save time, and will hopefully be meeting with our SOM collaborators to record a demo with our working product. The main thing to focus on now is the motor tuning so we can capture accurate video for our video demo and our collaboration. The main concern we have now is finishing our motor movement fine tuning so we can have a presentable project before both demo days. At this point schedule isn’t really relevant, we’re just focusing on getting it done as fast as possible.

Posted on

Pareekshith’s Status Report for 4/25

This week, I focused mainly on re-testing my speech technology setup to ensure that nothing horrifically wrong had happened or will happen in the next two weeks, and I also met with Dr. Dueck and the School of Music students to troubleshoot our Accomplice setup and the cueing. One good thing is that the software now correctly follows the MIDI keyboard and is able to output cues properly, although I need the exact cue positions so that I can pinpoint exactly where and when our camera needs to turn. I also needed to order a new BLDC motor driver after one of the motor drivers we originally got through the School of Music was faulty, and so we received that today and look to complete the setup and fully integrate. My progress is on schedule, and since my portion is mostly complete (apart from getting the exact cue positions for the “Ah Perdona” piece), I’ll be helping out my team with any other necessary tasks this week as we look to complete the project and demo it at the TechSpark demo as well as at the public demo. I’ll also be working on the poster, the final video, and the final report this week so we won’t be pressed for time as we conclude the semester.

Posted on

Team Status Report for 04/18/2026

The only obstacle/risk that’s present on the audio side has to do with the opera portion and the different tempos present in one of the pieces. To mitigate this, we’ll have to contact Professor Dannenberg and get his advice on how to deal with the multiple tempos using Accomplice. Apart from that, the only other obstacle is going through the blocking to figure out where the cues need to be set in the .gro file, but we’ll again communicate with Dr. Dueck and her students figured this out. The biggest obstacle regarding the physical tracking system for Autocam seems to be regarding the feedback loop for the rail. Accurately measuring the position of the trucking could be simultaneously affected by slipping belts, added friction, or even a bearing slightly losing contact with the aluminium pipe. These minute changes in variables could cause the system to be inaccurate when measuring the distance traveled across the x-axis. We aim to minimize these risks by consolidating the rail system and fine tuning and physical issues we encounter. Our mitigation methods include rebalancing the physical rail, use locktite for bearing connections, and re-tightening our trucking belt.  We got the trucking part of the Autocam working and live tracking a node this week! Please see our github for more details which we hope and are designing to allow any one in the future to improve upon and create our system with detailed instructions, (GitHub).

Posted on

Ted’s Status Report for 04/18/2026

This week, I mainly focus on getting the rail system up and running now that we had the replacement parts needed. The design has been slightly modified as we are now using two motor drivers instead of one VESC, and I had to adjust the design of the pan camera mount accordingly. We’re still working on getting the motor tracking up and running, and hopefully by next week we can start implementing the panning motor tracking in tandem with our trucking motor movement. I also began some validation testing, such as measuring noise levels during operation and recording our trucking motor accuracy. Our project is currently behind schedule obviously, but after the whole fiasco with the broken motor controller we’ve made a pretty good pace on catching back up once our replacement parts arrived. For next week, I plan on helping Ahmad get the motor movement calibrated and finish implementing the subsystems together with Pareek’s accomplice and stageplay audio features, as well as testing and validating for our use case and design requirements. Overall, this project has been pretty fun to work on compared to other classes I’ve taken before at CMU. I’ve learned a lot about reading through documentation and ensuring that system parts are compatible before actual ordering and testing. I’ve also learned a lot about the VESC software systems, even though we couldn’t end up using the VESC motor controller in the end. A lot of this knowledge was acquired through learning about similar motor systems or projects people did in the past and learning from their mistakes and combined experience through online forums or write-ups. I also learned a lot about communication protocols such as UART, GPIO, and MODBUS, through similar methods of youtube videos, tutorials, and online forum posts. Learning more about my partner’s project parts was also very interesting, as I had never actually realized how UWB sensors or audio libraries like VOSK or word error rate functioned either. Most of the things I learned from these topics, I learned through talking with my group members and asking questions. Also, while not technical, our collaboration with Dr. Dueck and the School of Music also exposed me to a lot of musical and technical knowledge and terminology about opera and other music performances that I hadn’t known before, which I learned mainly through our collaboration meetings and the questions I was able to ask these talented and experienced performers.

Posted on

Pareekshith’s Status Report for 4/18

This week, I mostly worked on the opera portion of our collaboration with the School of Music and was debugging issues with the tempo of the pieces. After several attempts, I figured out the problem and was able to obtain the correct tempo for the piece that Dr. Dueck will be playing for our demo. There’s one more piece that I need to work on that is more difficult because of certain changes in tempo throughout the piece, but I’ll try to get that to work using specific cuing mechanisms. On the other hand, I am also working on testing the speech portion and using the WER metric to see the accuracy of the ASR. I don’t need the WER of the entire dialogue by an actor, simply the trigger phrase that’s used to pivot between actors. This shouldn’t take too long to do, and I should have it completed in about a day or so. This upcoming week, I plan to complete the validation and testing for the speech side, set up the cues in Accomplice, and help my team with any final debugging that needs to be done. While working on this project, I learned a lot about speech technology, fuzzy matching, and music technology. Interestingly, I learned a majority of the first two from my Speech Technology for Conversational AI course that I’m taking this semester, and that’s where I learned about WER, Vosk, and the other components of the speech side. On the other hand, learning about Accomplice, how it works, and how to use it needed me to get on a call with Professor Dannenberg, who was extremely kind and helpful. I was able to get most of the information I needed through our chat, and he’s been nice enough to be available for any questions and concerns that I might have.

Posted on

Team Status Report for 04/04/2026

The most significant risk related to the audio portion of the project is the consistency and reliability of Accomplice. While it’s been working for the most part through my testing, there’s been an issue with receiving OSC messages (mentioned more in individual reports) that should be fixed soon by Professor Dannenberg. Although this challenge should be addressed presently, there’s a worry that other issues could pop up in a variety of circumstances, and we’d have to prepare and solve each of them. This shouldn’t be impossible, but it might be difficult and annoying, especially because we wouldn’t know if it’s an issue on our side or in the Accomplice software unless we check in with Professor Dannenberg continuously. Another possible risk we have is that the VESC motor controller we’re currently using had one of its micro-usb ports for communication break off, so we’ll try to solder it back on. If that fails, we plan on trying to use the UART communication ports with the GPIO pins to control. We also had one of our printed parts for the rail system break recently, which delayed our assembly by a bit. To counteract this, we plan on printing extra parts for high strain areas of our design to ensure that if any future parts break, we’ll have a quick way to replace them and get back to testing/building.

 

One minor change that was made to the design of the audio system was the usage of a USB-B to USB-C adapter instead of the predetermined MIDI interface for connecting the MIDI keyboard to the Mac. Our idea is to have a RPi 5 connect to the keyboard and send cues to the central Pi, and so we will check that the USB-C port on the RPi keyboard can be used for a non-powering purpose. If this doesn’t work, we’d have to purchase an adapter from USB-B to a different USB type that’s available on the RPi. This causes one of the nodes to be used as a middle communication device as it’ll provide key information for the AutoCam system to change scenes. This upcoming week we hope to finalize the hardware integration of the AutoCam. As we finalize the hardware integration, we can bring all the systems together. From the audio cue system to the UWB tracking system. We will utilize both systems to finalize the motor controller software. This will be specifically done through the mathematical work of homography. For validation, we look to confirm the durability and the consistency of our product. We will be running the entire system at expected workloads to determine that it can work accurately for the full length of an opera or a theater performance. Each subsystem’s accuracy will be vetted thoroughly, and descriptions have been included in the individual reports. Validation testing will be done to measure the latency between the completion of the performer’s dialogue, the triggering of the cue, and the actual movement of the motors, which can then be used to optimize and minimize the time required. Additionally, final UWB accuracy testing will be performed from ground truth location to the filtered estimated location of the node. Finally, we will be testing the waterproofing nature of our wearable components and ensure that they are wearable and durable based on survey feedback from the performers and stress testing with water at different intensities. Noise levels will also be measured to ensure that our design won’t be overpowering the important audio in either the opera performances or stageplays. We’ll be spending the next couple weeks working with the opera students to ensure that our system meets their needs and fine tuning our subsystem integration.

Posted on

Ahmad’s Status Report for 04/04/2026

Over the past week, I have been heavily focused on the UWB tracking system and its physical integration. Significant progress has been made through continued refinement, and I successfully mapped out the opera rehearsal room to support more accurate testing and tracking. From there, I concentrated on improving the code and debugging the system to ensure the highest level of accuracy possible. For instance, one of the key improvements was the addition of height calibration to the software implementation. This enhancement allows the AutoCam system to achieve considerably better accuracy in a wider range of situations, especially when tracking movement in a three dimensional space. This is vital when the height position of the anchors do not match the height of the node.  In addition, the AutoCam system has entered the final stages of hardware integration. This has involved wiring the motors, assembling the pulley system, and connecting the central Raspberry Pi processor. One key component is that a microusb port is broken on our motor controller. I was able to find a replacement part for the port and began soldering. We were also able to meet with the opera group, further surveying questions, issues, and movement. I took the opportunity to note their movements during the opera, which provided useful insights into blocking patterns, transitions, and the areas where the tracking system will need to respond effectively. These observations will help guide further calibration and camera movement tuning so that the AutoCam can better match real performance capabilities. With both the software and hardware coming together, the system is moving closer to a fully integrated and functional state. This next week we hope to have the full system operational to begin our final testing stages.

Posted on

Pareekshith’s Status Report for 4/4

This past week, I mostly completed the work on the speech track of the project, and I was able to show my progress in the demo where Ted and I played Nora and Torvald from A Doll’s House. I also added a manual override button in the UI, allowing the director to automatically trigger a cue in case, for some reason, the actor wasn’t heard correctly by the system. To complete this track, I need to simply add the specific motor controller instructions to the Python logic rather than only displaying to the UI. On the opera track, I also made progress, meeting with Professor Dannenberg to debug early setup issues I was facing with the off-the-shelf solution, Accomplice. After our call, I was able to correctly open the music projects and play the music through the SimpleSynth synthesizer. Our team was able to get access to the studio theater where we regularly meet Dr. Dueck and the School of Music, and so I was able to play around with the MIDI keyboard and connect it to my Mac. While most of the system works, I was facing issues with receiving the OSC messages that represent the triggers in the piano script, and it turns out that it’s an issue with the Accomplice software that Professor Dannenberg will resolve soon. I’m currently on progress with my schedule, and I hope to test and also complete the opera track by the end of the week once Accomplice is fixed. Regarding the verification testing, I’ve already been testing with scripts and seeing the success rate of each trigger. I’ve noticed that on modern English, Vosk does extremely well, with a 95+% accuracy, but with Middle English or Early modern English (e.g. Shakespeare, Marlowe, etc.), certain words like “thy” and “thine” aren’t translated correctly, and thus reduce the accuracy of the model. By using fuzzy matching and hacks like changing the trigger from “thy” to “by,” I’m able to improve the accuracy of the system, but I need to explore more permanent solutions that may help with WER and trigger precision. The speech track will rely mostly on these kinds of tests, using actual plays and scenes as references and counting the number of times the triggers don’t engage, and the results will be used to modify the triggers, increase the range of the fuzzy matching, and improve the system itself. For both the speech and opera tracks, I will be running latency tests to determine how long it takes for the system to acknowledge the trigger after the conclusion of the performer’s dialogue. We want to minimize the latency to a time that won’t be noticeable for the audience.

Posted on

Team Status Report for 03/28/2026

On the audio side, there are 2 significant risks, one related to the opera (music-based) portion and one related to the play (speech-based) portion. On the opera side, we need to ensure that the equipment that we’ll be using to obtain the MIDI recording is compatible with the off-the-shelf software that Professor Dannenberg designed. On the speech side, while Vosk is working pretty well with a single microphone, it remains to be seen how the performance will be with multiple actors, especially performers who are close to each other. There’s a worry that the interference caused by the audio being heard on multiple microphones could confuse Vosk and thus, mess up the processing and cue engine. This week allowed for major improvements to the tracking system overall. New firmware, updated software, and increased polling rate to the UWB sensors has allowed considerable system improvements. The central RPI 5 has also been configured to handle all processing needs offloading load from the four wearable nodes. The centralized system will allow seamless integration with our motors, CV camera, and audio cues. Communication from the wearable nodes has also been configured which will allow us to quickly send audio cues to the central pi to adjust the camera as necessary. Having this centralized system all communication goes to reliably initiates our full project implementation. Regarding the opera portion, a significant change has been made to the design, in that we’re now using an off-the-shelf system that was created by Professor Dannenberg. This software is based on MIDI files, and it enables high performance regardless of the presence of other performers and their vocalization. Our group is currently on schedule, as we look to complete the rail design by the end of the week and finish the audio and tracking subsystems too. We’ll look to integrate these different systems together in the coming weeks as our entire project comes together. There haven’t been any changes to our schedule.

 

 

4 Node Tracking