Category: Qimeng’s Status Reports

This week, I focused on adapting our vision system to support fully wireless operation. Previously, the OAK-D Pro’s stereo depth pipeline required high USB bandwidth and stable power, which tied the robot to a wall outlet. To enable battery-powered operation, I replaced the stereo depth estimation with a hand-size-based distance estimation approach using only the RGB camera. This significantly reduces USB bandwidth and power consumption, allowing reliable operation on battery power. The system calculates distance from the pixel size of detected hand landmarks using a calibration lookup table with linear interpolation across multiple reference distances (0.5m to 2.5m). I also resolved a camera firmware issue by clearing the device flash using the DepthAI DeviceBootloader API, and switched from the V3 pipeline API to the V2 API for better stability.

I also delivered the final presentation this week, presenting our complete solution, component tradeoffs, and verification results. After the presentation, I continued calibrating the hand-size distance estimation by measuring hand landmark pixel sizes at known distances and iteratively tuning the interpolation table to minimize error within our 1–2.5m operating range.

This week, I tested the follow-hand navigation logic that I implemented last week in the gesture controller, which uses the OAK-D Pro’s stereo depth to estimate the user’s distance, calibrates across 15 frames with outlier trimming, and drives the robot forward until the target distance is reached. I also built the lateral-shift obstacle avoidance system: when the front ToF sensor detects an obstacle, the robot stops and shifts sideways until the path clears, then continues shifting for a buffer duration to ensure the full chassis width passes the obstacle. A significant portion of my time was spent debugging ToF hardware issues, including Arduino serial port changes between ttyACM0/ttyACM1 on each reboot, EMI-corrupted serial data during motor operation, and intermittent sensor disconnections. I resolved these by adding strict line-level validation in tof_node that only accepts correctly formatted data with all four labeled fields.

For the final presentation, I was responsible for updating the use-case requirements to be more user-focused based on professor feedback, redesigning the architecture block diagram to reflect our as-built system (including the Arduino for ToF and software I2C for the voice module), writing the complete solution and component tradeoff slides, and working with teammates to compile a comprehensive testing plan with 27 tests mapped to our five use-case requirements.

In terms of new knowledge, I strengthened my Python skills through extensive ROS2 development, learning to structure multi-node systems with publishers, subscribers, and state machines, as well as coordinating nodes across separate processes using topic-based communication. I also deepened my understanding of ROS2 Jazzy specifically, including writing launch files, declaring parameters, and debugging inter-node message flow. On the hardware side, I learned I2C protocol debugging using i2cdetect and i2cget to resolve bus conflicts between the voice module and motor controller, which both shared address 0x34, and learned to read firmware source code to identify correct register addresses when the voice module stopped responding after a reflash. For serial communication, I learned that motor EMI can corrupt USB data and that strict validation is more robust than simple parsing. These were all learned through hands-on debugging, reading ROS2 and DepthAI documentation, and examining vendor-provided source code.

This week, our team completed the interim demo, successfully demonstrating the full workflow of voice activation, gesture-triggered motor control, and ToF-based obstacle avoidance. Based on feedback from the TA and professor, we identified several improvements: switching the activation gesture from Thumbs Up to Thumbs Down to reduce unintentional triggers, and increasing the ToF obstacle detection threshold to give the robot more time to decelerate. I also began researching the follow-user logic, which will use the hand’s pixel x-coordinate to determine the user’s lateral position and the stereo depth value to estimate distance, allowing the robot to steer toward the user rather than moving in a fixed direction. To handle depth noise, I plan to average readings across consecutive frames.

For verification, I am planning tests for both of my subsystems. For gesture detection, I will test recognition accuracy across varying distances, lighting levels, and hand orientations, targeting at least 90 percent accuracy over 50 trials per condition, along with measuring detection latency. For the ToF subsystem, I will compare sensor readings against known distances and test whether the robot correctly chooses the appropriate avoidance direction when obstacles are detected. Next week, I will focus on implementing the follow-user navigation logic and mounting the ToF sensors onto the chassis.

This week was a major milestone as we brought all subsystems together into a working integrated demo. Yilu and I set up the Raspberry Pi 5 and installed ROS 2 Jazzy. We ran into several internet connectivity and package dependency issues during setup, but after multiple attempts everything was running. We then tested the chassis motion together and confirmed the mecanum wheel control was working correctly.

I migrated the computer vision pipeline from my personal computer to the Raspberry Pi. I created a ROS 2 gesture detection node using the OAK-D Pro camera with MediaPipe’s Gesture Recognizer to detect hand gestures (Thumbs Up to start, Closed Fist to stop) and estimate the 3D hand position using stereo depth.

We integrated the subsystems step by step. First, we confirmed that hand gestures could reliably activate and stop the motors. Then we added ToF-based obstacle avoidance by creating a node that reads four VL53L1X sensors via Arduino over USB serial, with control logic that automatically steers the robot away from nearby obstacles. Finally, we integrated the voice detection module, allowing users to activate the gesture control system through voice commands. This forms a complete workflow: voice activation, gesture-triggered motion, and ToF obstacle avoidance, which is what we will demonstrate at the interim demo.

For next week, we plan to mount all components onto the chassis and use an aluminum alloy frame for the trash bin so we can test on the ground. We will also refine the control logic. Currently we use translational strafing at low speed on a tabletop, but we may add turning maneuvers based on real-world testing needs.

This week I continued working on the perception subsystem for the TrashDash project. I focused on setting up the camera pipeline and running MediaPipe for real-time hand detection. I was able to successfully run the MediaPipe hand tracking model and verify that the system can detect hand landmarks from the camera stream. 

In addition to the vision pipeline, I also started testing the Time-of-Flight (ToF) sensor (VL53L1X) using Arduino. I worked on setting up the sensor and running example code to read distance measurements. While I was able to initialize the sensor and communicate with it, I encountered some instability in getting consistent distance outputs, which I suspect is related to hardware connection issues. I am currently debugging the wiring and setup to ensure reliable readings.

One challenge this week is ensuring stable and accurate perception data from both the camera and ToF sensor. For the vision subsystem, lighting conditions and hand poses can affect detection quality. For the ToF sensor, hardware reliability and communication stability need to be improved before integration.

Next week, I plan to continue improving the robustness of the MediaPipe hand detection pipeline and begin integrating the 3D hand position into the control logic. I will also fix the ToF sensor setup and verify that it can provide consistent distance measurements for obstacle detection.

This week I started working on the computer vision subsystem for the TrashDash project. I set up the camera pipeline and began experimenting with MediaPipe for hand detection. The goal of this subsystem is to detect the user’s hand in real time and estimate its position so that the robot can navigate toward the user for trash collection. I installed the necessary libraries on my computer and ran initial tests to verify that MediaPipe can successfully detect hand landmarks from the camera stream.

In addition to basic hand detection, I also started exploring how to combine the RGB image with depth information from the camera in order to estimate the 3D position of the user’s hand. This information will later be used by the control logic to compute the robot’s motion commands. Initial tests confirmed that the camera can stream frames reliably and that MediaPipe can detect the hand landmarks in real time.

One challenge I am currently working on is ensuring stable detection and reducing noise in the hand position estimation. Variations in lighting conditions and hand poses can sometimes affect the detection confidence. To address this, I plan to experiment with filtering and parameter tuning in the vision pipeline.

Next week I plan to continue improving the vision subsystem and begin integrating the hand position output with the control logic.

This week we completed the design presentation and finalized the design report for the TrashDash project. I contributed to several sections of the report, including the Architecture and Principle of Operation, Design Trade Studies, System Implementation, and Summary. In addition, I also helped revise the Use-case Requirements, Design Requirements, and Testing sections based on feedback from the design presentation.

After receiving feedback from the design presentation, I spent additional time refining parts of the system that were not emphasized enough previously, especially the software architecture and control logic development. I worked on clarifying how the executive finite state machine coordinates system behavior and how sensor data are converted into motion commands through the control pipeline. These improvements helped make the system design and implementation more clearly documented in the report.

At the moment, the main concern is hardware integration. Since some hardware components have not arrived yet, we cannot fully test the integrated system. Once the hardware is delivered, I plan to begin testing the perception pipeline and control logic with the motor platform to verify that the system can correctly track the user’s hand and generate stable motion commands.

This week, I focused on sensor selection and early design documentation. I compared several Time-of-Flight (ToF) sensors, including the VL53L1X, VL53L0X, and VL6180X, by reviewing their detection range, sampling frequency, power consumption, and interface compatibility. After evaluating the trade-offs, I selected the VL53L1X due to its longer detection range and higher performance capability, which better supports our system’s object detection requirements. I also verified that the sensor is compatible with our Raspberry Pi 5 and planned power regulation setup to ensure smooth integration.

In addition, our team divided responsibilities for the design report, and I am currently writing the System Implementation, Design Trade Studies, Architecture, and Summary sections. I am documenting the technical rationale behind our key design decisions, including sensor and processor selection, and refining the system block diagram to reflect our finalized BOM. My progress is on schedule, and next week I plan to complete full drafts of my sections and begin initial hardware testing once the components arrive.

This week I worked together with my teammates on selecting key hardware components for our system. In particular, I helped evaluate different motor chassis options and AI voice modules to ensure they meet our load capacity, control compatibility, and integration requirements. After comparing several platforms, we aligned on a chassis that supports encoder feedback and is compatible with both Raspberry Pi and STM32 control, which gives us flexibility in our system architecture.

In addition, I focused on figuring out how our different components should connect to the Raspberry Pi. I mapped out the communication interfaces and corresponding ports for each module, including the OAK-D Pro camera, ToF sensors, AI voice module, and motor control interface. Beyond hardware integration, I also worked on refining our presentation materials. Specifically, I contributed to the use case section, quantitative design requirements, system specifications, and implementation plan. 

Overall, our progress remains aligned with the planned schedule, and we have not encountered any major blockers. The next phase will focus on beginning early-stage integration once components arrive.

This week I worked on use case requirements, testing, verification and metrics, and solution approach sections in the proposal presentation slides. I researched potential implementations for obstacle avoidance and return-to-home behavior. For obstacle avoidance, I explored using four discrete ToF sensors mounted at the four corners of the mobile base to provide close-range distance sensing and collision prevention. The current plan is to interface these ToF sensors with an Arduino for low-level distance monitoring and safety checks, although directly connecting them to the Raspberry Pi is also a feasible alternative and remains under consideration. For return-to-home localization, I investigated UWB-based solutions and proposed using two DWM1001-DEV modules, configured as one anchor and one tag, to provide a reliable reference for navigating back to the initial standby position.

Overall, my progress this week aligns well with our project schedule. For next week, I plan to focus on developing a more detailed system design, including drawing clear design and integration diagrams that illustrate how different components communicate and work together. I will also begin working on the design review presentation as well.