Weekly Status Reports

This week, I focused on adapting our vision system to support fully wireless operation. Previously, the OAK-D Pro’s stereo depth pipeline required high USB bandwidth and stable power, which tied the robot to a wall outlet. To enable battery-powered operation, I replaced the stereo depth estimation with a hand-size-based distance estimation approach using only the RGB camera. This significantly reduces USB bandwidth and power consumption, allowing reliable operation on battery power. The system calculates distance from the pixel size of detected hand landmarks using a calibration lookup table with linear interpolation across multiple reference distances (0.5m to 2.5m). I also resolved a camera firmware issue by clearing the device flash using the DepthAI DeviceBootloader API, and switched from the V3 pipeline API to the V2 API for better stability.

I also delivered the final presentation this week, presenting our complete solution, component tradeoffs, and verification results. After the presentation, I continued calibrating the hand-size distance estimation by measuring hand landmark pixel sizes at known distances and iteratively tuning the interpolation table to minimize error within our 1–2.5m operating range.

This week, I tested the follow-hand navigation logic that I implemented last week in the gesture controller, which uses the OAK-D Pro’s stereo depth to estimate the user’s distance, calibrates across 15 frames with outlier trimming, and drives the robot forward until the target distance is reached. I also built the lateral-shift obstacle avoidance system: when the front ToF sensor detects an obstacle, the robot stops and shifts sideways until the path clears, then continues shifting for a buffer duration to ensure the full chassis width passes the obstacle. A significant portion of my time was spent debugging ToF hardware issues, including Arduino serial port changes between ttyACM0/ttyACM1 on each reboot, EMI-corrupted serial data during motor operation, and intermittent sensor disconnections. I resolved these by adding strict line-level validation in tof_node that only accepts correctly formatted data with all four labeled fields.

For the final presentation, I was responsible for updating the use-case requirements to be more user-focused based on professor feedback, redesigning the architecture block diagram to reflect our as-built system (including the Arduino for ToF and software I2C for the voice module), writing the complete solution and component tradeoff slides, and working with teammates to compile a comprehensive testing plan with 27 tests mapped to our five use-case requirements.

In terms of new knowledge, I strengthened my Python skills through extensive ROS2 development, learning to structure multi-node systems with publishers, subscribers, and state machines, as well as coordinating nodes across separate processes using topic-based communication. I also deepened my understanding of ROS2 Jazzy specifically, including writing launch files, declaring parameters, and debugging inter-node message flow. On the hardware side, I learned I2C protocol debugging using i2cdetect and i2cget to resolve bus conflicts between the voice module and motor controller, which both shared address 0x34, and learned to read firmware source code to identify correct register addresses when the voice module stopped responding after a reflash. For serial communication, I learned that motor EMI can corrupt USB data and that strict validation is more robust than simple parsing. These were all learned through hands-on debugging, reading ROS2 and DepthAI documentation, and examining vendor-provided source code.

This week, our team completed the interim demo, successfully demonstrating the full workflow of voice activation, gesture-triggered motor control, and ToF-based obstacle avoidance. Based on feedback from the TA and professor, we identified several improvements: switching the activation gesture from Thumbs Up to Thumbs Down to reduce unintentional triggers, and increasing the ToF obstacle detection threshold to give the robot more time to decelerate. I also began researching the follow-user logic, which will use the hand’s pixel x-coordinate to determine the user’s lateral position and the stereo depth value to estimate distance, allowing the robot to steer toward the user rather than moving in a fixed direction. To handle depth noise, I plan to average readings across consecutive frames.

For verification, I am planning tests for both of my subsystems. For gesture detection, I will test recognition accuracy across varying distances, lighting levels, and hand orientations, targeting at least 90 percent accuracy over 50 trials per condition, along with measuring detection latency. For the ToF subsystem, I will compare sensor readings against known distances and test whether the robot correctly chooses the appropriate avoidance direction when obstacles are detected. Next week, I will focus on implementing the follow-user navigation logic and mounting the ToF sensors onto the chassis.