Over the past week, I have been focused on fine-tuning the new motor controller that we received so that the panning would work accurately regardless of where the performer is on stage. Software wise I refactored our implementation to simultaneously handle two types of motor controller. Much of my time was taken by reading the documentation of the new controller and properly configuring the different controllers to work together. I had to begin this new type of integration mostly due to the fact that I had discovered that the replacement motor controller that we ordered was faulty, and so we had to scramble to order a new one off Amazon. I will take this opportunity to complete the rest of the software integration so that both the tracking and the audio components are perfectly synchronized with motor movement. At this point, fine-tuning is the last obstacle I have to overcome, and we are currently reaching the final stage of our product! Super excited to see this completely in motion in a couple of days.
Team’s Status Report for 4/25
The most significant risks that could jeopardize the success of this project would be our inability to tune the motor controllers correctly so that they can properly give the right instructions to the motors themselves. We’re currently working on this portion of the project, and once that’s done, we should mostly be done with the project, apart from incorporating the audio component to handle pivoting to the designated actor after the previous actor’s dialogue is completed. Once that is done, we’ll be ready for the TechSpark demo along with the final demo, and we’ll also be able to meet on Thursday with Dr. Dueck and the School of Music for our demo with them for the “Ah Perdona” piece. Apart from that, we will of course also be working on the final report and the final video so that we can submit all of them in time when they are due.
On the audio side, the main tests that we performed were calculating the WER for the ASR system, specifically for the trigger phrases rather than the whole dialogue. We realized that it didn’t make sense to track the whole dialogue by each speaker, rather it would improve processing and accuracy if we only focused on the few words before the next actor’s part began. What we found was that the system was very accurate, with a 7% WER (10% is high-quality) on these trigger phrases, and that the trigger was almost always correctly initiated. However, the fact that it wasn’t perfect concerned us slightly, which is why we included a manual override button in the UI that allowed the director to control the cueing themselves in case the trigger didn’t work. On the tracking side, the main tests we performed were calculating the difference between the ground truth and the predictions by the UI, and we discovered that the accuracy was even better than we expected, allowing us to stick with our current design. On the overall system side, we tested the reliability of the entire system, and we confirmed that the nodes and other components all ran for at least 4 hours, which was what we were hoping for.
Ted’s Status Report for 04/25/2026
This week, I’ve mainly been working on ironing out any small design issues with the physical rail system and tuning/calibrating the motor system. We had an issue where the replacement motor driver for the previously broken motor driver also arrived broken (hooray) but we managed to procure a replacement as fast as possible and am working on tuning right now. I’ve also started working on the poster and final report just to save time, and will hopefully be meeting with our SOM collaborators to record a demo with our working product. The main thing to focus on now is the motor tuning so we can capture accurate video for our video demo and our collaboration. The main concern we have now is finishing our motor movement fine tuning so we can have a presentable project before both demo days. At this point schedule isn’t really relevant, we’re just focusing on getting it done as fast as possible.
Pareekshith’s Status Report for 4/25
This week, I focused mainly on re-testing my speech technology setup to ensure that nothing horrifically wrong had happened or will happen in the next two weeks, and I also met with Dr. Dueck and the School of Music students to troubleshoot our Accomplice setup and the cueing. One good thing is that the software now correctly follows the MIDI keyboard and is able to output cues properly, although I need the exact cue positions so that I can pinpoint exactly where and when our camera needs to turn. I also needed to order a new BLDC motor driver after one of the motor drivers we originally got through the School of Music was faulty, and so we received that today and look to complete the setup and fully integrate. My progress is on schedule, and since my portion is mostly complete (apart from getting the exact cue positions for the “Ah Perdona” piece), I’ll be helping out my team with any other necessary tasks this week as we look to complete the project and demo it at the TechSpark demo as well as at the public demo. I’ll also be working on the poster, the final video, and the final report this week so we won’t be pressed for time as we conclude the semester.
Team Status Report for 04/18/2026
The only obstacle/risk that’s present on the audio side has to do with the opera portion and the different tempos present in one of the pieces. To mitigate this, we’ll have to contact Professor Dannenberg and get his advice on how to deal with the multiple tempos using Accomplice. Apart from that, the only other obstacle is going through the blocking to figure out where the cues need to be set in the .gro file, but we’ll again communicate with Dr. Dueck and her students figured this out. The biggest obstacle regarding the physical tracking system for Autocam seems to be regarding the feedback loop for the rail. Accurately measuring the position of the trucking could be simultaneously affected by slipping belts, added friction, or even a bearing slightly losing contact with the aluminium pipe. These minute changes in variables could cause the system to be inaccurate when measuring the distance traveled across the x-axis. We aim to minimize these risks by consolidating the rail system and fine tuning and physical issues we encounter. Our mitigation methods include rebalancing the physical rail, use locktite for bearing connections, and re-tightening our trucking belt. We got the trucking part of the Autocam working and live tracking a node this week! Please see our github for more details which we hope and are designing to allow any one in the future to improve upon and create our system with detailed instructions, (GitHub).
Ted’s Status Report for 04/18/2026
This week, I mainly focus on getting the rail system up and running now that we had the replacement parts needed. The design has been slightly modified as we are now using two motor drivers instead of one VESC, and I had to adjust the design of the pan camera mount accordingly. We’re still working on getting the motor tracking up and running, and hopefully by next week we can start implementing the panning motor tracking in tandem with our trucking motor movement. I also began some validation testing, such as measuring noise levels during operation and recording our trucking motor accuracy. Our project is currently behind schedule obviously, but after the whole fiasco with the broken motor controller we’ve made a pretty good pace on catching back up once our replacement parts arrived. For next week, I plan on helping Ahmad get the motor movement calibrated and finish implementing the subsystems together with Pareek’s accomplice and stageplay audio features, as well as testing and validating for our use case and design requirements. Overall, this project has been pretty fun to work on compared to other classes I’ve taken before at CMU. I’ve learned a lot about reading through documentation and ensuring that system parts are compatible before actual ordering and testing. I’ve also learned a lot about the VESC software systems, even though we couldn’t end up using the VESC motor controller in the end. A lot of this knowledge was acquired through learning about similar motor systems or projects people did in the past and learning from their mistakes and combined experience through online forums or write-ups. I also learned a lot about communication protocols such as UART, GPIO, and MODBUS, through similar methods of youtube videos, tutorials, and online forum posts. Learning more about my partner’s project parts was also very interesting, as I had never actually realized how UWB sensors or audio libraries like VOSK or word error rate functioned either. Most of the things I learned from these topics, I learned through talking with my group members and asking questions. Also, while not technical, our collaboration with Dr. Dueck and the School of Music also exposed me to a lot of musical and technical knowledge and terminology about opera and other music performances that I hadn’t known before, which I learned mainly through our collaboration meetings and the questions I was able to ask these talented and experienced performers.
Ahmad’s Status Report 4/18
This week, I made considerable progress on the central Pi’s motion control and tracking stack. I built the homography-based motor control foundation early so software development could continue before the final pan motor driver arrives. I implemented the initial camera geometry model, environment-driven motor configuration, live Modbus motor control, and the first complete auto-tracking pipeline. I then refined the homography, camera placement, startup pose, and projection behavior to improve real-world tracking performance and better align the UWB pose stream, image-space projection, and motor commands so subject tracking behaves more reliably during live motion. Furthermore, I expanded the system into a practical manual calibration with jog controls, mark-left, mark-right, and mark-center tools, and various other commands. Additionally, I added a seamless manual-to-auto handoff which allows calibration to automation to be quite smooth. This work brings Autocam closer to the next phase of integration, where the second motor can be brought online for full pan-and-truck tracking once the remaining motor controller arrives. On the hardware side, I completed the physical setup of the motor system, including wiring, power, and bringing the core rail-drive hardware online, which enabled extensive live testing throughout the rest of the week. Those tests drove a long series of fixes and refinements across manual calibration, startup-state handoff, soft-limit behavior, control responsiveness, live status polling, left-right recovery, and safety polling to make the system behave more reliably under real motion. Now lastly, because leaving the central Pi unnecessarily exposed on a public network felt ironic while taking 18-330 Computer Security, I set up firewall protections to reduce unnecessary exposure and restrict access to only needed connections and not everyone at Carnegie Mellon.
To address the final point, I had to learn practical tools and concepts like Modbus/RS485 motor control, re learn homography and camera geometry which was a nice refresher on Computer Vision, firewall hardening for the central Pi. I learned them through documentation, hardware testing, debugging logs, and a lot of iterative trial and error on the rail system in lab.
Please check out the github for all the progress!
https://github.com/ahmadmla/autocam
Pareekshith’s Status Report for 4/18
This week, I mostly worked on the opera portion of our collaboration with the School of Music and was debugging issues with the tempo of the pieces. After several attempts, I figured out the problem and was able to obtain the correct tempo for the piece that Dr. Dueck will be playing for our demo. There’s one more piece that I need to work on that is more difficult because of certain changes in tempo throughout the piece, but I’ll try to get that to work using specific cuing mechanisms. On the other hand, I am also working on testing the speech portion and using the WER metric to see the accuracy of the ASR. I don’t need the WER of the entire dialogue by an actor, simply the trigger phrase that’s used to pivot between actors. This shouldn’t take too long to do, and I should have it completed in about a day or so. This upcoming week, I plan to complete the validation and testing for the speech side, set up the cues in Accomplice, and help my team with any final debugging that needs to be done. While working on this project, I learned a lot about speech technology, fuzzy matching, and music technology. Interestingly, I learned a majority of the first two from my Speech Technology for Conversational AI course that I’m taking this semester, and that’s where I learned about WER, Vosk, and the other components of the speech side. On the other hand, learning about Accomplice, how it works, and how to use it needed me to get on a call with Professor Dannenberg, who was extremely kind and helpful. I was able to get most of the information I needed through our chat, and he’s been nice enough to be available for any questions and concerns that I might have.
Team Status Report for 04/04/2026
The most significant risk related to the audio portion of the project is the consistency and reliability of Accomplice. While it’s been working for the most part through my testing, there’s been an issue with receiving OSC messages (mentioned more in individual reports) that should be fixed soon by Professor Dannenberg. Although this challenge should be addressed presently, there’s a worry that other issues could pop up in a variety of circumstances, and we’d have to prepare and solve each of them. This shouldn’t be impossible, but it might be difficult and annoying, especially because we wouldn’t know if it’s an issue on our side or in the Accomplice software unless we check in with Professor Dannenberg continuously. Another possible risk we have is that the VESC motor controller we’re currently using had one of its micro-usb ports for communication break off, so we’ll try to solder it back on. If that fails, we plan on trying to use the UART communication ports with the GPIO pins to control. We also had one of our printed parts for the rail system break recently, which delayed our assembly by a bit. To counteract this, we plan on printing extra parts for high strain areas of our design to ensure that if any future parts break, we’ll have a quick way to replace them and get back to testing/building.
One minor change that was made to the design of the audio system was the usage of a USB-B to USB-C adapter instead of the predetermined MIDI interface for connecting the MIDI keyboard to the Mac. Our idea is to have a RPi 5 connect to the keyboard and send cues to the central Pi, and so we will check that the USB-C port on the RPi keyboard can be used for a non-powering purpose. If this doesn’t work, we’d have to purchase an adapter from USB-B to a different USB type that’s available on the RPi. This causes one of the nodes to be used as a middle communication device as it’ll provide key information for the AutoCam system to change scenes. This upcoming week we hope to finalize the hardware integration of the AutoCam. As we finalize the hardware integration, we can bring all the systems together. From the audio cue system to the UWB tracking system. We will utilize both systems to finalize the motor controller software. This will be specifically done through the mathematical work of homography. For validation, we look to confirm the durability and the consistency of our product. We will be running the entire system at expected workloads to determine that it can work accurately for the full length of an opera or a theater performance. Each subsystem’s accuracy will be vetted thoroughly, and descriptions have been included in the individual reports. Validation testing will be done to measure the latency between the completion of the performer’s dialogue, the triggering of the cue, and the actual movement of the motors, which can then be used to optimize and minimize the time required. Additionally, final UWB accuracy testing will be performed from ground truth location to the filtered estimated location of the node. Finally, we will be testing the waterproofing nature of our wearable components and ensure that they are wearable and durable based on survey feedback from the performers and stress testing with water at different intensities. Noise levels will also be measured to ensure that our design won’t be overpowering the important audio in either the opera performances or stageplays. We’ll be spending the next couple weeks working with the opera students to ensure that our system meets their needs and fine tuning our subsystem integration.
Ted’s Status Report for 04/04/2026
This past week, I’ve been working on finalizing the rail design. Unfortunately, one of our gt2 pulleys broke during assembly, so we’ve been a little delayed in getting the pivoting motor movement up and functional. I spent this week mainly focusing on getting the rail system up and running, by finishing any possible parts we would need cut/soldered/printed. We also had a micro-usb port break on the VESC motor controller while soldering, and I’m currently working on a backup plan for firmware installation and communication in case we can’t get that fixed. I’ve been working on creating some jst-ph style connections for communication from the VESC to our GPIO pins using female dupont connectors on our RPi. I’ve also begun working on some rudimentary motor control code using a GPIo communication library with our motor controller for simple lateral movement to ensure that all our connections have been soldered correctly with no major issues that’d require disassembly down the road, which would require a lot of time and wasted effort. For verification testing, as mentioned above, I plan on first writing some motor movement code to test that our VESC to RPi communication protocols are working correctly, and then using test inputs to simulate UWB sensor communication and ensure that the rail system is moving how it’s supposed to. Noise level requirements will be tested using a decibel meter that will positioned at different locations during motor testing to ensure that our noise level is below our specified amount. Once replacement parts have been procured and our VESC repaired, I hope to get our full system up and running so we can start testing with our other subsystems and the opera students. For now, I plan to write some simple test code and fix any issues that might arise from that and hopefully have a complete system before next week.