Category: Team Status Reports

The most significant risks that could jeopardize the success of this project would be our inability to tune the motor controllers correctly so that they can properly give the right instructions to the motors themselves. We’re currently working on this portion of the project, and once that’s done, we should mostly be done with the project, apart from incorporating the audio component to handle pivoting to the designated actor after the previous actor’s dialogue is completed. Once that is done, we’ll be ready for the TechSpark demo along with the final demo, and we’ll also be able to meet on Thursday with Dr. Dueck and the School of Music for our demo with them for the “Ah Perdona” piece. Apart from that, we will of course also be working on the final report and the final video so that we can submit all of them in time when they are due.

On the audio side, the main tests that we performed were calculating the WER for the ASR system, specifically for the trigger phrases rather than the whole dialogue. We realized that it didn’t make sense to track the whole dialogue by each speaker, rather it would improve processing and accuracy if we only focused on the few words before the next actor’s part began. What we found was that the system was very accurate, with a 7% WER (10% is high-quality) on these trigger phrases, and that the trigger was almost always correctly initiated. However, the fact that it wasn’t perfect concerned us slightly, which is why we included a manual override button in the UI that allowed the director to control the cueing themselves in case the trigger didn’t work. On the tracking side, the main tests we performed were calculating the difference between the ground truth and the predictions by the UI, and we discovered that the accuracy was even better than we expected, allowing us to stick with our current design. On the overall system side, we tested the reliability of the entire system, and we confirmed that the nodes and other components all ran for at least 4 hours, which was what we were hoping for.

The only obstacle/risk that’s present on the audio side has to do with the opera portion and the different tempos present in one of the pieces. To mitigate this, we’ll have to contact Professor Dannenberg and get his advice on how to deal with the multiple tempos using Accomplice. Apart from that, the only other obstacle is going through the blocking to figure out where the cues need to be set in the .gro file, but we’ll again communicate with Dr. Dueck and her students figured this out. The biggest obstacle regarding the physical tracking system for Autocam seems to be regarding the feedback loop for the rail. Accurately measuring the position of the trucking could be simultaneously affected by slipping belts, added friction, or even a bearing slightly losing contact with the aluminium pipe. These minute changes in variables could cause the system to be inaccurate when measuring the distance traveled across the x-axis. We aim to minimize these risks by consolidating the rail system and fine tuning and physical issues we encounter. Our mitigation methods include rebalancing the physical rail, use locktite for bearing connections, and re-tightening our trucking belt.  We got the trucking part of the Autocam working and live tracking a node this week! Please see our github for more details which we hope and are designing to allow any one in the future to improve upon and create our system with detailed instructions, (GitHub).

On the audio side, there are 2 significant risks, one related to the opera (music-based) portion and one related to the play (speech-based) portion. On the opera side, we need to ensure that the equipment that we’ll be using to obtain the MIDI recording is compatible with the off-the-shelf software that Professor Dannenberg designed. On the speech side, while Vosk is working pretty well with a single microphone, it remains to be seen how the performance will be with multiple actors, especially performers who are close to each other. There’s a worry that the interference caused by the audio being heard on multiple microphones could confuse Vosk and thus, mess up the processing and cue engine. This week allowed for major improvements to the tracking system overall. New firmware, updated software, and increased polling rate to the UWB sensors has allowed considerable system improvements. The central RPI 5 has also been configured to handle all processing needs offloading load from the four wearable nodes. The centralized system will allow seamless integration with our motors, CV camera, and audio cues. Communication from the wearable nodes has also been configured which will allow us to quickly send audio cues to the central pi to adjust the camera as necessary. Having this centralized system all communication goes to reliably initiates our full project implementation. Regarding the opera portion, a significant change has been made to the design, in that we’re now using an off-the-shelf system that was created by Professor Dannenberg. This software is based on MIDI files, and it enables high performance regardless of the presence of other performers and their vocalization. Our group is currently on schedule, as we look to complete the rail design by the end of the week and finish the audio and tracking subsystems too. We’ll look to integrate these different systems together in the coming weeks as our entire project comes together. There haven’t been any changes to our schedule.

 

 

 

 

Some risks we noticed were the RPis we were planning to use with our UWB node weren’t fully compatible with our design. Additionally, a risk that also came to light was the possibility that RPi 5s wouldn’t be consistent in their compatibility with other aspects of our design, such as the UWB nodes and Microphones due to port configuration. Furthermore, what we expect for our project is the fact that we have so many moving parts and modules that depend and interact with each other. If we get slightly delayed in development of one part, the delay could cascade down our schedule and cause pretty bad problems in terms of development down the line. We plan on mitigating these risks by first testing with the RPi 5 to ensure that they work with our modules, as well as including slack time in our schedule to ensure that if we are delayed, we have sufficient time to catch up and finish before any deadlines. As of right now, the only potential change to our design might be the shift from using the RPi 5s to the RPi 4s because of the powering/communication issue that could happen with the Pi 5s and the UWB tags. We do want to test both and see if there is genuinely an issue with the Pi 5. If the Pi 5 works, we’ll stick with that because we already got several from the inventory, and we wouldn’t have to use a portion of our budget on purchasing 4 different Pi 4s. Additionally, we switched to using BLDC motors instead of stepper motors, as we realized the VESC motor controller we’re borrowing from inventory doesn’t support stepper motor operation. Not many costs are incurred, as connections all stay the same.We slightly delayed our ordering schedule to wait for the design review from our TA and Professor Qing. This way we can adjust the bill of materials accordingly within specification. We are now planning to order the necessary parts before spring break starts on 02/28. Doing so will allow us to have the necessary parts and initialize our first prototype.

During the week we had the opportunity to meet up multiple times and assign each other roles. For instance, Ahmad is tasked to handle compatibility of UWB sensors and power, Ted is handling motor configuration, and Pareekshith is responsible for the microphone system. We are actively working and communicating to make sure each one of our products are compatible with one another. Some significant risks we have are related to module compatibility. We need certain requirements from modules. For example, the Lavalier mics that we now plan to use need transceivers to connect to the RPi, and we’ll need to find a way to ensure that multiple transceivers, the RPi, and the mics are all compatible with each other. Each transceiver is only compatible with 2 Lavalier microphones, so for scenes with more actors, we’ll have to ensure compatibility among the different microphones and increased transceivers. If we order parts that aren’t compatible, and we only realize that after the fact, we’ll have to wait a decent amount of time in order to research and order a new part. Another issue we might have is that the UWB sensors we plan on using don’t necessarily tend to be waterproof. If these devices are going to be placed directly on the actors, there’s always the chance that sweat or other foreign substances could affect our tag setups. To mitigate these effects, we plan on and are currently doing more extensive research to ensure everything we order has both a way to interface with other modules. This ensures that any parts we might need will arrive in an acceptable timeframe. To solve the UWB sensor issue, we additionally plan on designing some sort of water-proof enclosure that remains ergonomic for the actors and won’t interfere with any design requirements we might have. We are all currently still on track, with no changes being made to the schedule. For changes made to the design, we’ve dropped the original Arduinos and replaced them with RPis, because we will need more processing power for audio and visual. This will cause a slight increase in price, as RPis tend to be more expensive, but our budget is still under 600 dollars for now.

A was written by Pareekshith, B was written by Ted, and C was written by Ahmad.

Part A: Actors utilizing  our system will wear lightweight UWB sensors that prevent them from facing any injury risks; having a light sensor will also reduce any discomfort and interference during movements. Also, our system will move at adjustable speeds which can be varied depending on the script and the play, and it can be slowed for greater safety precautions. We plan on putting foam on both sides of the cart in case someone decides to put their finger or hand on the rail system, creating greater safety. This system also reduces psychological stress on camera operators, who often have to make rapid adjustments based on the script and have little to no room for error. Finally, this project gives smaller productions the opportunity to have a better cinematographic experience and to create professional quality films based on their plays.
Part B:

AutoCam might have some issues with privacy/biometric data, as we’re currently planning on using some form of facial recognition/computer vision to track actors on stage in tandem with the UWB tag. If actors/singers are concerned about their biometric data being stored, we plan to only implement realtime processing of data so there’s no long term storage, and because all tracking will occur locally, there will be no possible cloud upload.  Recordings will also only be scored locally so that there’s no possible concern of data leakage. Another issue is that audience members who don’t want to be recorded could accidentally be captured.  We plan on strictly tracking individuals within predefined UWB anchor zones so that it won’t be an issue.

Part C: AutoCam has several economic implications across production. On the cost perspective, teams save money by reducing dependence on dedicated camera operators for routine tracking tasks, which would lower labor costs for small productions.  However, this does not eliminate human roles. This would result in the redistribution of their role from camera operators to script operators. This way they are controlling the camera production through script commands and will have the opportunity to fine tune camera angles, timing, and even zoom.  Thus, shifting work to less labor requires extensive tasks like shot supervision, system monitoring, and cueing.

During the week we presented our proposal to faculty and classmates.The presentation proposed some questions that may jeopardize the success of the project with respect to interference from the audience. We are working on mitigating the potential intrusion of our system to the audience and the actors and will have to discuss more details with Professor Dueck if possible. We are currently still on schedule, and aim to finalize a preliminary design by the end of next week, which will specify the exact components we’ll need as well as how they’ll be integrated with each other. This also pushes to minimize potential risks of order delays by expediting our final bill of materials. Delays would be harmful to our ability to follow the schedule, especially given how our software needs to interact with physical hardware components like cameras, microphones, and sensors.