Month: March 2026

Qimeng’s Status Report for 3/28

This week was a major milestone as we brought all subsystems together into a working integrated demo. Yilu and I set up the Raspberry Pi 5 and installed ROS 2 Jazzy. We ran into several internet connectivity and package dependency issues during setup, but after multiple attempts everything was running. We then tested the chassis motion together and confirmed the mecanum wheel control was working correctly.

I migrated the computer vision pipeline from my personal computer to the Raspberry Pi. I created a ROS 2 gesture detection node using the OAK-D Pro camera with MediaPipe’s Gesture Recognizer to detect hand gestures (Thumbs Up to start, Closed Fist to stop) and estimate the 3D hand position using stereo depth.

We integrated the subsystems step by step. First, we confirmed that hand gestures could reliably activate and stop the motors. Then we added ToF-based obstacle avoidance by creating a node that reads four VL53L1X sensors via Arduino over USB serial, with control logic that automatically steers the robot away from nearby obstacles. Finally, we integrated the voice detection module, allowing users to activate the gesture control system through voice commands. This forms a complete workflow: voice activation, gesture-triggered motion, and ToF obstacle avoidance, which is what we will demonstrate at the interim demo.

For next week, we plan to mount all components onto the chassis and use an aluminum alloy frame for the trash bin so we can test on the ground. We will also refine the control logic. Currently we use translational strafing at low speed on a tabletop, but we may add turning maneuvers based on real-world testing needs.

Siying’s Status Report for 3/28

This week the Raspberry Pi and Ros 2 system were first configured and set up correctly, then I also participated in integrating our system pipeline together. Right now, there are 3 stages in total, the first stage is voice trigger, the second stage is the hand gesture detection, and the third stage is the motor control. I also worked on connecting the voice recognition module into the ROS 2 system. I wrote the SpeechControlNode that polls the ASR module via ROS timer. Upon detecting a certain command, the node launches the gesture detection pipeline as a subprocess. Also, the I2C address conflict between the ASR module and the motor controller (both at 0x34) is addressed by routing the ASR module to a secondary I2C bus (i2c-3) using a GPIO software overlay.

For this week, the progress aligns with our current schedule. In the next step, we will need to work on locating the user with the camera, and let the robot be able to reach the user based on the position.

Team Status Report for 3/28

System Progress
This week we successfully tested all major subsystems on the Raspberry Pi and achieved initial system integration. The motor driver is fully functional, supporting forward/backward movement, lateral shifting, rotation, and turning. The OAK-D depth camera can reliably detect different hand gestures, and the voice recognition module successfully detects the wake word “hi trashdash.” We integrated the car chassis, camera, and voice module into a working pipeline: after the wake word and confirmation (“throw trash”), the system activates, moves forward upon detecting a thumbs-up gesture, and stops when a fist gesture is recognized. Additionally, preliminary obstacle avoidance logic using ToF sensors has been validated in a simulated setup.

System Design & Changes
There are no major changes to the overall system architecture. We continue to use the Raspberry Pi as the central controller coordinating voice, vision, and motor subsystems. One practical adjustment is prioritizing early integration of gesture-based control (thumbs-up to move, fist to stop) to demonstrate an end-to-end workflow for the interim demo, before fully implementing continuous hand tracking and navigation.

Risk & Mitigation
The main risks currently are hardware integration and system latency. Mounting all components (camera, ToF sensors, and wiring) onto the chassis may introduce instability or connection issues; this will be mitigated through secure mounting and incremental testing after each component is installed. Another risk is latency between voice input, gesture recognition, and motor response, which could affect responsiveness; we plan to optimize processing pipelines and reduce unnecessary computation. Finally, obstacle avoidance is not fully validated yet since ToF sensors are not mounted; we will mitigate this by conducting real-world testing once installation is complete and refining thresholds for reliable detection.

Next Week’s Plan
Next week, we will focus on physically mounting and securing all components onto the chassis and ensuring stable system operation. After full hardware integration, we will implement and test hand-following behavior so the robot can continuously track and move toward the user’s hand instead of only responding to discrete gestures. In parallel, we will prepare for the interim demo: Siying will cover the introduction and voice module, Qimeng will present the camera and ToF sensing, and Yilu will present the motor system and outline plans for the next three weeks.

Yilu’s Status Report for 3/28

This week I made some solid progress on the car subsystem. At the beginning, we encountered several setup issues with the Raspberry Pi since none of us had prior experience, but we were able to resolve these within a day through troubleshooting and documentation. On Thursday, Aislinn and I tested the chassis and confirmed that the motor control driver is working reliably. We also began initial integration of the voice wake-up system with the camera and motor control, establishing a basic pipeline for interaction.

For the upcoming interim demo, I will be responsible for presenting our progress on the motor system and outlining our next steps, including further integration and system-level testing.

Qimeng’s Status Report for 3/21

This week I continued working on the perception subsystem for the TrashDash project. I focused on setting up the camera pipeline and running MediaPipe for real-time hand detection. I was able to successfully run the MediaPipe hand tracking model and verify that the system can detect hand landmarks from the camera stream. 

In addition to the vision pipeline, I also started testing the Time-of-Flight (ToF) sensor (VL53L1X) using Arduino. I worked on setting up the sensor and running example code to read distance measurements. While I was able to initialize the sensor and communicate with it, I encountered some instability in getting consistent distance outputs, which I suspect is related to hardware connection issues. I am currently debugging the wiring and setup to ensure reliable readings.

One challenge this week is ensuring stable and accurate perception data from both the camera and ToF sensor. For the vision subsystem, lighting conditions and hand poses can affect detection quality. For the ToF sensor, hardware reliability and communication stability need to be improved before integration.

Next week, I plan to continue improving the robustness of the MediaPipe hand detection pipeline and begin integrating the 3D hand position into the control logic. I will also fix the ToF sensor setup and verify that it can provide consistent distance measurements for obstacle detection.

Siying’s Status Report for 3/21

This week I participated in the ethics lecture and discussed with our team about the ethics problem that our project could potentially have, and we also discussed the possible ways to mitigate these issues. I also explored how to integrate a comprehensive voice-interaction and hardware control system for our robot. By using Python and I2C communication, I unified the speech recognition and broadcast capabilities into a single control framework. I also utilized multithreading. This allows the OpenCV-based camera feed to run in the background without blocking the voice-recognition loop, enabling the robot to “listen” and “see” simultaneously. 

For this week, the progress aligns with our current schedule. In the next step, we will need to set up the sd card for our Rpi, and configure the network connection for our Rpi, so that all our individual modules can be connected to Rpi and tested together. 

Team Status Report for 3/21

System Progress

This week, we worked on both software and hardware development. The vision subsystem can now run MediaPipe for hand detection, and the voice module has been integrated with hardware control using Python and I2C. We also started assembling the motor chassis and testing basic motor control and ToF sensing.

System Design & Changes

There are no major changes to our system design this week. We continue to use the Raspberry Pi as the main controller and keep STM32 as a backup option. The system is still developed in separate modules before integration.

Risk & Mitigation

One risk is hardware integration delay, since some components arrived later and testing is still in progress. We also observed some instability in sensors like the ToF module. To address this, we are testing each subsystem independently and debugging hardware connections before full integration.

Next Week’s Plan

Next week, we will set up the Raspberry Pi environment and connect all subsystems together. We will also continue improving the stability of the vision and sensing modules. Our goal is to start system-level integration and prepare for the interim demo.

Yilu’s Status Report for 3/21

This week, I assembled the car chassis and researched how to interface it with both the Raspberry Pi and STM32 board. I also set up the development environment and wrote some preliminary motor control code to test basic functionality. In addition, our team attended an ethics lecture, which prompted us to think more carefully about potential issues such as user safety, privacy, and reliability. For example, since our system involves autonomous movement and sensing, we need to ensure it avoids collisions reliably and does not unintentionally collect sensitive data from users or their surroundings.

Overall, there are no major blockers at the moment, but we are slightly tight on time with the interim demo approaching. The car chassis arrived later than expected, which caused a small delay in hardware testing. To stay on track, we plan to increase our workload next week and focus on accelerating integration and testing progress.

Qimeng’s Status Report for 3/14

This week I started working on the computer vision subsystem for the TrashDash project. I set up the camera pipeline and began experimenting with MediaPipe for hand detection. The goal of this subsystem is to detect the user’s hand in real time and estimate its position so that the robot can navigate toward the user for trash collection. I installed the necessary libraries on my computer and ran initial tests to verify that MediaPipe can successfully detect hand landmarks from the camera stream.

In addition to basic hand detection, I also started exploring how to combine the RGB image with depth information from the camera in order to estimate the 3D position of the user’s hand. This information will later be used by the control logic to compute the robot’s motion commands. Initial tests confirmed that the camera can stream frames reliably and that MediaPipe can detect the hand landmarks in real time.

One challenge I am currently working on is ensuring stable detection and reducing noise in the hand position estimation. Variations in lighting conditions and hand poses can sometimes affect the detection confidence. To address this, I plan to experiment with filtering and parameter tuning in the vision pipeline.

Next week I plan to continue improving the vision subsystem and begin integrating the hand position output with the control logic.

Siying’s Status Report for 3/14

This week I read the articles and watched the videos related to the ethnic assignment, and I also analyzed our own project from different aspects related to ethnic such as the potential safety concerns and public welfare aspect. I also focused on setting up and experimenting with the voice recognition module used in our robot system. I followed the official tutorial to configure the module and successfully modified both the wake-up word and the command words so that the system can respond to our customized voice inputs instead of the default ones. After updating these settings, the module was able to detect and recognize the newly defined words and output the corresponding command IDs. In addition, I studied how to enable communication between the voice recognition module and the Raspberry Pi. Specifically, I explored how the Raspberry Pi reads recognition results from the module and processes them in Python. 

For this week, the progress aligns with our current schedule. In the next step, we will also need to modify the starter program to receive the recognized command IDs and trigger corresponding actions to start the work flow of the robot, and interact with the overall system.