Over the past week, I have been heavily focused on the UWB tracking system and its physical integration. Significant progress has been made through continued refinement, and I successfully mapped out the opera rehearsal room to support more accurate testing and tracking. From there, I concentrated on improving the code and debugging the system to ensure the highest level of accuracy possible. For instance, one of the key improvements was the addition of height calibration to the software implementation. This enhancement allows the AutoCam system to achieve considerably better accuracy in a wider range of situations, especially when tracking movement in a three dimensional space. This is vital when the height position of the anchors do not match the height of the node. In addition, the AutoCam system has entered the final stages of hardware integration. This has involved wiring the motors, assembling the pulley system, and connecting the central Raspberry Pi processor. One key component is that a microusb port is broken on our motor controller. I was able to find a replacement part for the port and began soldering. We were also able to meet with the opera group, further surveying questions, issues, and movement. I took the opportunity to note their movements during the opera, which provided useful insights into blocking patterns, transitions, and the areas where the tracking system will need to respond effectively. These observations will help guide further calibration and camera movement tuning so that the AutoCam can better match real performance capabilities. With both the software and hardware coming together, the system is moving closer to a fully integrated and functional state. This next week we hope to have the full system operational to begin our final testing stages.
Pareekshith’s Status Report for 4/4
This past week, I mostly completed the work on the speech track of the project, and I was able to show my progress in the demo where Ted and I played Nora and Torvald from A Doll’s House. I also added a manual override button in the UI, allowing the director to automatically trigger a cue in case, for some reason, the actor wasn’t heard correctly by the system. To complete this track, I need to simply add the specific motor controller instructions to the Python logic rather than only displaying to the UI. On the opera track, I also made progress, meeting with Professor Dannenberg to debug early setup issues I was facing with the off-the-shelf solution, Accomplice. After our call, I was able to correctly open the music projects and play the music through the SimpleSynth synthesizer. Our team was able to get access to the studio theater where we regularly meet Dr. Dueck and the School of Music, and so I was able to play around with the MIDI keyboard and connect it to my Mac. While most of the system works, I was facing issues with receiving the OSC messages that represent the triggers in the piano script, and it turns out that it’s an issue with the Accomplice software that Professor Dannenberg will resolve soon. I’m currently on progress with my schedule, and I hope to test and also complete the opera track by the end of the week once Accomplice is fixed. Regarding the verification testing, I’ve already been testing with scripts and seeing the success rate of each trigger. I’ve noticed that on modern English, Vosk does extremely well, with a 95+% accuracy, but with Middle English or Early modern English (e.g. Shakespeare, Marlowe, etc.), certain words like “thy” and “thine” aren’t translated correctly, and thus reduce the accuracy of the model. By using fuzzy matching and hacks like changing the trigger from “thy” to “by,” I’m able to improve the accuracy of the system, but I need to explore more permanent solutions that may help with WER and trigger precision. The speech track will rely mostly on these kinds of tests, using actual plays and scenes as references and counting the number of times the triggers don’t engage, and the results will be used to modify the triggers, increase the range of the fuzzy matching, and improve the system itself. For both the speech and opera tracks, I will be running latency tests to determine how long it takes for the system to acknowledge the trigger after the conclusion of the performer’s dialogue. We want to minimize the latency to a time that won’t be noticeable for the audience.
Updated Schedule
Ted’s Status Report for 03/28/2026
This week I focused on getting the rail system up and running for testing and coordination with our other subsystems. We met with Dr. Dueck and we went over various parts of our design, such as rail length and integration with the audio system to meet requirements for working with the opera students. She introduced us to Dr. Dannenberg, who’s given us a lot of helpful input on how the MIDI audio input interaction might work and what we might need to do.I finished using Solidworks to design all the parts needed, cut them all out at TechSpark, and have been working on assembling the rail system. I’ve had to redesign various parts of the rail such as the motor pulley system as the original design isn’t within tolerances for our belts and camera mount assembly. I’m currently working on getting the VESC and RPi set up and communicating. By next week, I hope to get motor control code working so we can begin testing with our different subsystems together. Our design is still on schedule, and I’m hoping to get the motor code working in tandem with our UWB sensor code on the central processing RPi so we can determine if it will bottleneck processing or not. I also helped Ahmad with testing the UWB tracking system. We are currently still on schedule, and hope to have a basic full implementation consisting of audio, rail, and UWB tracking soon.
Team Status Report for 03/28/2026
On the audio side, there are 2 significant risks, one related to the opera (music-based) portion and one related to the play (speech-based) portion. On the opera side, we need to ensure that the equipment that we’ll be using to obtain the MIDI recording is compatible with the off-the-shelf software that Professor Dannenberg designed. On the speech side, while Vosk is working pretty well with a single microphone, it remains to be seen how the performance will be with multiple actors, especially performers who are close to each other. There’s a worry that the interference caused by the audio being heard on multiple microphones could confuse Vosk and thus, mess up the processing and cue engine. This week allowed for major improvements to the tracking system overall. New firmware, updated software, and increased polling rate to the UWB sensors has allowed considerable system improvements. The central RPI 5 has also been configured to handle all processing needs offloading load from the four wearable nodes. The centralized system will allow seamless integration with our motors, CV camera, and audio cues. Communication from the wearable nodes has also been configured which will allow us to quickly send audio cues to the central pi to adjust the camera as necessary. Having this centralized system all communication goes to reliably initiates our full project implementation. Regarding the opera portion, a significant change has been made to the design, in that we’re now using an off-the-shelf system that was created by Professor Dannenberg. This software is based on MIDI files, and it enables high performance regardless of the presence of other performers and their vocalization. Our group is currently on schedule, as we look to complete the rail design by the end of the week and finish the audio and tracking subsystems too. We’ll look to integrate these different systems together in the coming weeks as our entire project comes together. There haven’t been any changes to our schedule.
Pareekshith’s Status Report for 3/28
This week, I made progress on the speech as well as the musical portion of the project. I’ve decided to follow 2 tracks: one which focuses on opera and getting the piano music (as part of the collaboration with the School of Music) and the other which is based on plays and uses Vosk for ASR. For the demo, I’ll be showing the latter track and display the Lavalier microphone connected to the wearable node that’ll indicate the parsing of the speech and the switching of the pivots on a UI. Regarding the other path, we had a call with Professor Dannenberg during our meeting with Dr. Dueck and the School of Music, and he introduced an off-the-shelf solution that relies on MIDI files and software; this solution may be exactly what we need for our project. This solution has the ability to send information and commands through a robust network protocol developed by Professor Dannenberg, so we’d be able to send information to the central Pi and make it send the commands over to the motor controller. I’m currently on schedule with my work; I’ll be meeting with Professor Dannenberg next week to set up his solution locally and make sure it’s all working, and I’ll be further enhancing the speech solution and verifying that it works accurately when there are multiple microphones involved in a close setting. I hope to be able to get both systems to work for our final project presentation, as it’d be cool to demonstrate the different capabilities our system has and the various use cases we support.
Ahmad’s Status Report 3/28
This week I focused on the communication and coordination pipeline for our UWB localization system. I set up MQTT (Mosquitto) based communication between the central Raspberry Pi and the four node Raspberry Pis. Now we can transfer raw distance measurements to the central Pi for processing and filtering. I also reworked the node sender and central logger architecture so that all filtering and localization logic now runs on the central Pi, which made the system much easier to manage and debug. Another note is that centralizing our system allows our nodes to have a much stronger battery life as there is now minimal processing done by the wearable nodes. The centralized system is additionally crucial when integrating our motors, CV camera, and audio cues later on. Having this centralized system all communication goes to reliably initiates our full project implementation. I also fixed several issues during integration, one in which was correcting the node shutdown and restart behavior. Originally I would send “stop” to each node when switching nodes to ping. This caused tons of overhead as it was costly to restart a node. Now UWB sessions properly stop and can be restarted cleanly without breaking serial communication. This was achieved through a custom firmware update, which I introduced “pause” and “activate” key words to allow our anchors to quickly ping node by node. Furthermore, I was able to manually configure the UWB nodes to reliably poll at 20Hz, up from the original, slow 5Hz (4x improvement!). At this point, single and multi node tracking is working reliably and I was able to confirm that the communication pipeline between nodes and the central Pi is functioning correctly. In addition to the software work, I also continued physical system implementation, and I assisted my teammate Ted with laser cutting, material searching, and verification at TechSpark. We are on schedule.
Ahmad’s Status Report 3/21
This week I completed the hardware preparation needed for full anchor deployment by soldering and rewiring all five anchors so they could reliably operate with their battery packs. I validated the setup through over five hours of continuous operation to confirm that the system can support the required runtime. I also flashed and configured the operating systems on all of the Raspberry Pi 5 units, completed device setup, and connected them to the campus network so that our team can remotely access and manage the devices when needed. Additionally, I implemented a real time UWB tracking pipeline to begin evaluating node accuracy and movement across the anchors which now utilize a polygon system to create the 2D space upon configuration of locations. Moreover, I developed per anchor range filtering using a rolling median and adaptive EWMA approach to reduce noise while still preserving responsiveness, then using bounded Gauss Newton multilateration to estimate node position from the anchor measurements while keeping the solution within the valid anchor footprint. Next up I hope to properly calibrate the anchors, and continue fine tuning accuracy. In parallel, I built a live UWB visualizer that displays the anchor layout, current node position, and movement history, along with playback controls for reviewing recorded tracking data. I have also began software preparation on our multi-node configuration, which will be finalized in the next couple of days, maintaining our schedule.
Ted’s Status Report for 03/21/2026
This week, I worked on connecting the VESC motor controller to the BLDC motors and power source. I mainly focused on CADing (is that the right acronym?) the 3d driver pulleys, idler pulleys, motor plates, and other miscellaneous things and printed them before our meeting Saturday. Then, I hooked up the motors to the VESC, power source, and RPi and tested them to see if all the connections had been soldered correctly and were functioning. Bearings have been delivered, but I wasn’t able to pick them up before the ECE office closed. I’m hoping to pick them up and finish constructing the rail monday, and then begin testing with our completed UWB sensor/node system. In terms of schedule, we’re developing most of our subsystems at the same time, and once the rail is finished and all parts are received, we should be at a good pace. I need to reprint some of the pulleys for a better fit on the timing belt, and I hope to get that done by next week as well. I want to reduce wear and tear as much as possible. The open source configuration software for the motor controller is a bit finicky to work with, and I’m trying to get experienced with it as fast as possible.
Team Status Report for 03/21/2026
The ability of the UWB anchors to work continuously over the duration of the play is something we have to test thoroughly; during our Saturday work session, we’ve had problems with the anchors turning off due to slightly incorrect resistor values. Although we’ve now gotten them all to run properly, we’ll have to do more stress testing to ensure that they are capable of recording a full play with no problems. Another risk that we have regarding the audio portion of the project is the presence of multiple instruments and the operatic dialogue that could cause interference with the piano music that is necessary for the Matchmaker library. We’ll have to discuss the scope of the project with Dr. Dueck and understand the exact use case, ensuring that we’re using the right tools. The plan to mitigate this is to use a powerful Samson microphone solely for the piano, as opposed to the Lavalier microphones we initially were using for each actor’s dialogue. Another risk we have is the motor VESC wiring. We’re currently connecting the motor phase wires to the VESC through just solder and e tape, and we run the risk of the connections breaking/shorting if they become a stress point during rail movement. We’re planning to mitigate these risks by later replacing the E tape with heat shrink for a more secure connection. In terms of schedule, we’re working on most systems concurrently. The UWB sensors/anchors should be up and running very soon, and once the ECE pick-up center is open, every part needed will be in our possession.