Weekly Status Reports

Siying’s Status Report for 4/4

This week, a mid-term demo was presented to the TA and professor, and we showed the current state of the TrashDash system including voice activation, gesture detection, and chassis motion. Based on their feedback, we discussed several improvements. For example, the obstacle detection threshold for the ToF sensors should be increased to give the robot more time to decelerate before reaching an obstacle. Also, to reduce the risk of unintentional gesture triggers, switching from Thumbs Up to a less commonly made gesture such as Thumbs Down was considered, as it is less likely to be accidentally activated in everyday scenarios.

I also researched on the next step of how to make the robot actively move toward the user rather than executing a fixed motion sequence of moving forward. For example, the hand pixel x-coordinate from the OAK-D camera can be used to determine whether the user is to the left or right of the robot, and the depth value z to estimate how far away the user is, allowing the robot to steer accordingly. Also, I considered the potential issues with depth accuracy, and one mitigation strategy is to use depth readings from several consecutive frames and average them to smooth out noise and reduce the impact of individual bad readings.

For this week, the progress aligns with our current schedule. In the next step, we will need to work on making the robot approach the user and testing. 

Verification for subsystems: 

For Voice Command Recognition, we will perform evaluation at varying distances (1m, 2.5m, 4m) under both quiet and moderate noise levels, with a target of 90 percent accuracy with 50 tests. We will also test latency by recording time from spoken keyword to system response, and we expect immediate response from the users perspective. For right now, subsystem tests for hand gesture detection can also be performed, and after we finish the last part of approaching the user, we will also perform testing for localization accuracy. 

 

Yilu‘s Status Report for 4/4

This week, our team completed the interim demo, where we showcased the integration of the voice, camera, and motor subsystems, demonstrating that the system can respond to voice commands and trigger movement based on camera input. Since the ToF sensors were not yet mounted, we presented the obstacle detection functionality separately using an Arduino setup for better clarity. In addition to the demo, we ordered additional components including a UPS for the Raspberry Pi and camera mounts to securely attach the camera on top of the trash bin for improved stability.

As we move into the verification and validation phase, I will be focusing on the motor subsystem by testing basic motion commands (forward, backward, lateral movement, and rotation) to ensure correct response to control inputs, as well as evaluating consistency and responsiveness under repeated trials. For verification, I plan to measure latency from command to actuation and compare actual vs. expected movement (e.g., direction accuracy and speed consistency) to ensure the motor controller meets our design requirements. For validation, we will test the fully integrated system by evaluating whether the robot can reliably move toward a detected hand within the required time and accuracy constraints in realistic environments. There were no major changes to the system design, but integration and mounting remain potential risks, which we plan to address through early assembly and incremental testing.

Next week, we will focus on mounting all components onto the chassis and begin implementing the follow-human logic as the next key feature.

Qimeng’s Status Report for 3/28

This week was a major milestone as we brought all subsystems together into a working integrated demo. Yilu and I set up the Raspberry Pi 5 and installed ROS 2 Jazzy. We ran into several internet connectivity and package dependency issues during setup, but after multiple attempts everything was running. We then tested the chassis motion together and confirmed the mecanum wheel control was working correctly.

I migrated the computer vision pipeline from my personal computer to the Raspberry Pi. I created a ROS 2 gesture detection node using the OAK-D Pro camera with MediaPipe’s Gesture Recognizer to detect hand gestures (Thumbs Up to start, Closed Fist to stop) and estimate the 3D hand position using stereo depth.

We integrated the subsystems step by step. First, we confirmed that hand gestures could reliably activate and stop the motors. Then we added ToF-based obstacle avoidance by creating a node that reads four VL53L1X sensors via Arduino over USB serial, with control logic that automatically steers the robot away from nearby obstacles. Finally, we integrated the voice detection module, allowing users to activate the gesture control system through voice commands. This forms a complete workflow: voice activation, gesture-triggered motion, and ToF obstacle avoidance, which is what we will demonstrate at the interim demo.

For next week, we plan to mount all components onto the chassis and use an aluminum alloy frame for the trash bin so we can test on the ground. We will also refine the control logic. Currently we use translational strafing at low speed on a tabletop, but we may add turning maneuvers based on real-world testing needs.

Siying’s Status Report for 3/28

This week the Raspberry Pi and Ros 2 system were first configured and set up correctly, then I also participated in integrating our system pipeline together. Right now, there are 3 stages in total, the first stage is voice trigger, the second stage is the hand gesture detection, and the third stage is the motor control. I also worked on connecting the voice recognition module into the ROS 2 system. I wrote the SpeechControlNode that polls the ASR module via ROS timer. Upon detecting a certain command, the node launches the gesture detection pipeline as a subprocess. Also, the I2C address conflict between the ASR module and the motor controller (both at 0x34) is addressed by routing the ASR module to a secondary I2C bus (i2c-3) using a GPIO software overlay.

For this week, the progress aligns with our current schedule. In the next step, we will need to work on locating the user with the camera, and let the robot be able to reach the user based on the position.

Team Status Report for 3/28

System Progress
This week we successfully tested all major subsystems on the Raspberry Pi and achieved initial system integration. The motor driver is fully functional, supporting forward/backward movement, lateral shifting, rotation, and turning. The OAK-D depth camera can reliably detect different hand gestures, and the voice recognition module successfully detects the wake word “hi trashdash.” We integrated the car chassis, camera, and voice module into a working pipeline: after the wake word and confirmation (“throw trash”), the system activates, moves forward upon detecting a thumbs-up gesture, and stops when a fist gesture is recognized. Additionally, preliminary obstacle avoidance logic using ToF sensors has been validated in a simulated setup.

System Design & Changes
There are no major changes to the overall system architecture. We continue to use the Raspberry Pi as the central controller coordinating voice, vision, and motor subsystems. One practical adjustment is prioritizing early integration of gesture-based control (thumbs-up to move, fist to stop) to demonstrate an end-to-end workflow for the interim demo, before fully implementing continuous hand tracking and navigation.

Risk & Mitigation
The main risks currently are hardware integration and system latency. Mounting all components (camera, ToF sensors, and wiring) onto the chassis may introduce instability or connection issues; this will be mitigated through secure mounting and incremental testing after each component is installed. Another risk is latency between voice input, gesture recognition, and motor response, which could affect responsiveness; we plan to optimize processing pipelines and reduce unnecessary computation. Finally, obstacle avoidance is not fully validated yet since ToF sensors are not mounted; we will mitigate this by conducting real-world testing once installation is complete and refining thresholds for reliable detection.

Next Week’s Plan
Next week, we will focus on physically mounting and securing all components onto the chassis and ensuring stable system operation. After full hardware integration, we will implement and test hand-following behavior so the robot can continuously track and move toward the user’s hand instead of only responding to discrete gestures. In parallel, we will prepare for the interim demo: Siying will cover the introduction and voice module, Qimeng will present the camera and ToF sensing, and Yilu will present the motor system and outline plans for the next three weeks.

Yilu’s Status Report for 3/28

This week I made some solid progress on the car subsystem. At the beginning, we encountered several setup issues with the Raspberry Pi since none of us had prior experience, but we were able to resolve these within a day through troubleshooting and documentation. On Thursday, Aislinn and I tested the chassis and confirmed that the motor control driver is working reliably. We also began initial integration of the voice wake-up system with the camera and motor control, establishing a basic pipeline for interaction.

For the upcoming interim demo, I will be responsible for presenting our progress on the motor system and outlining our next steps, including further integration and system-level testing.

Qimeng’s Status Report for 3/21

This week I continued working on the perception subsystem for the TrashDash project. I focused on setting up the camera pipeline and running MediaPipe for real-time hand detection. I was able to successfully run the MediaPipe hand tracking model and verify that the system can detect hand landmarks from the camera stream. 

In addition to the vision pipeline, I also started testing the Time-of-Flight (ToF) sensor (VL53L1X) using Arduino. I worked on setting up the sensor and running example code to read distance measurements. While I was able to initialize the sensor and communicate with it, I encountered some instability in getting consistent distance outputs, which I suspect is related to hardware connection issues. I am currently debugging the wiring and setup to ensure reliable readings.

One challenge this week is ensuring stable and accurate perception data from both the camera and ToF sensor. For the vision subsystem, lighting conditions and hand poses can affect detection quality. For the ToF sensor, hardware reliability and communication stability need to be improved before integration.

Next week, I plan to continue improving the robustness of the MediaPipe hand detection pipeline and begin integrating the 3D hand position into the control logic. I will also fix the ToF sensor setup and verify that it can provide consistent distance measurements for obstacle detection.

Siying’s Status Report for 3/21

This week I participated in the ethics lecture and discussed with our team about the ethics problem that our project could potentially have, and we also discussed the possible ways to mitigate these issues. I also explored how to integrate a comprehensive voice-interaction and hardware control system for our robot. By using Python and I2C communication, I unified the speech recognition and broadcast capabilities into a single control framework. I also utilized multithreading. This allows the OpenCV-based camera feed to run in the background without blocking the voice-recognition loop, enabling the robot to “listen” and “see” simultaneously. 

For this week, the progress aligns with our current schedule. In the next step, we will need to set up the sd card for our Rpi, and configure the network connection for our Rpi, so that all our individual modules can be connected to Rpi and tested together. 

Team Status Report for 3/21

System Progress

This week, we worked on both software and hardware development. The vision subsystem can now run MediaPipe for hand detection, and the voice module has been integrated with hardware control using Python and I2C. We also started assembling the motor chassis and testing basic motor control and ToF sensing.

System Design & Changes

There are no major changes to our system design this week. We continue to use the Raspberry Pi as the main controller and keep STM32 as a backup option. The system is still developed in separate modules before integration.

Risk & Mitigation

One risk is hardware integration delay, since some components arrived later and testing is still in progress. We also observed some instability in sensors like the ToF module. To address this, we are testing each subsystem independently and debugging hardware connections before full integration.

Next Week’s Plan

Next week, we will set up the Raspberry Pi environment and connect all subsystems together. We will also continue improving the stability of the vision and sensing modules. Our goal is to start system-level integration and prepare for the interim demo.

Yilu’s Status Report for 3/21

This week, I assembled the car chassis and researched how to interface it with both the Raspberry Pi and STM32 board. I also set up the development environment and wrote some preliminary motor control code to test basic functionality. In addition, our team attended an ethics lecture, which prompted us to think more carefully about potential issues such as user safety, privacy, and reliability. For example, since our system involves autonomous movement and sensing, we need to ensure it avoids collisions reliably and does not unintentionally collect sensitive data from users or their surroundings.

Overall, there are no major blockers at the moment, but we are slightly tight on time with the interim demo approaching. The car chassis arrived later than expected, which caused a small delay in hardware testing. To stay on track, we plan to increase our workload next week and focus on accelerating integration and testing progress.