Category: Siying’s Status Reports

This week in order to make the robot to move more freely without being restricted by charging cables, we switched to using wireless power supply, then full integration testing was conducted on the system and identified several issues: the OAK-D camera repeatedly dropping and reconnecting with XLink communication errors, and unstable ToF serial data reads. And all the above issues were identified and resolved. Firstly, to make the system become more stable, we switched to a power bank instead of batteries. Then to solve the problem of the oak-d camera, I helped replace stereo depth with RGB-based distance estimation. Due to power constraints, running all three camera streams simultaneously — RGB plus both stereo monochrome cameras — placed too much demand on USB bandwidth and power, causing frequent system crashes. The depth estimation approach was reworked to derive hand distance from RGB pixel data alone, using the apparent size of the hand in the frame to estimate real-world distance by a linear lookup table on multiple reference distances. Then to adjust the estimation to make it more accurate, I performed several actual measurements (the number of pixels occupied by the hand, as well as the actual distance), and the data in the table was adjusted accordingly to make the estimates more accurate. This method also removes the short-range limitation inherent to stereo depth sensing, effectively increasing the usable hand detection range. 

This week I worked together with my teammates on system integration and testing. All components were assembled into a complete working pipeline, including adding hand localization logic to the gesture detection module so that the robot actively moves toward the user’s hand position rather than executing a fixed motion sequence. As well as adding the function of first finding the user’s position by turning around. The full end-to-end flow was completed and verified as a unified system.  I also helped conduct testing on all individual parts and the overall system. Unit-level tests were performed on the voice recognition module, the OAK-D camera gesture recognition, and the ToF obstacle avoidance, each verified independently before being tested together. The end-to-end tests were then run on the complete pipeline to validate the full interaction flow. Test data was collected and analyzed to evaluate whether the current system’s performance meets the defined user requirements.

In the process of working on this project, since my background is primarily in software engineering and computer systems, it required me to learn a significant amount of knowledge about hardware integration and low-level hardware communication. For example, I learned a lot about RPi usage, including how to set it up from scratch, configure the operating system, and correctly use its GPIO pins for connecting sensors and other components. This was quite different from typical software-only development and required understanding both the physical layout and electrical considerations. I also learned a lot about Ros2 middleware (publish/subscribe model), which was essential for implementing features like voice activation, gesture detection, and navigation. In terms of learning strategies, I found watching online videos and tutorials very useful. Also, reading the official instruction documents is also useful for setting up the different modules that we use, and hands-on experimentation is also helpful for making concepts clear.

This week, a mid-term demo was presented to the TA and professor, and we showed the current state of the TrashDash system including voice activation, gesture detection, and chassis motion. Based on their feedback, we discussed several improvements. For example, the obstacle detection threshold for the ToF sensors should be increased to give the robot more time to decelerate before reaching an obstacle. Also, to reduce the risk of unintentional gesture triggers, switching from Thumbs Up to a less commonly made gesture such as Thumbs Down was considered, as it is less likely to be accidentally activated in everyday scenarios.

I also researched on the next step of how to make the robot actively move toward the user rather than executing a fixed motion sequence of moving forward. For example, the hand pixel x-coordinate from the OAK-D camera can be used to determine whether the user is to the left or right of the robot, and the depth value z to estimate how far away the user is, allowing the robot to steer accordingly. Also, I considered the potential issues with depth accuracy, and one mitigation strategy is to use depth readings from several consecutive frames and average them to smooth out noise and reduce the impact of individual bad readings.

For this week, the progress aligns with our current schedule. In the next step, we will need to work on making the robot approach the user and testing. 

Verification for subsystems: 

For Voice Command Recognition, we will perform evaluation at varying distances (1m, 2.5m, 4m) under both quiet and moderate noise levels, with a target of 90 percent accuracy with 50 tests. We will also test latency by recording time from spoken keyword to system response, and we expect immediate response from the users perspective. For right now, subsystem tests for hand gesture detection can also be performed, and after we finish the last part of approaching the user, we will also perform testing for localization accuracy. 

 

This week the Raspberry Pi and Ros 2 system were first configured and set up correctly, then I also participated in integrating our system pipeline together. Right now, there are 3 stages in total, the first stage is voice trigger, the second stage is the hand gesture detection, and the third stage is the motor control. I also worked on connecting the voice recognition module into the ROS 2 system. I wrote the SpeechControlNode that polls the ASR module via ROS timer. Upon detecting a certain command, the node launches the gesture detection pipeline as a subprocess. Also, the I2C address conflict between the ASR module and the motor controller (both at 0x34) is addressed by routing the ASR module to a secondary I2C bus (i2c-3) using a GPIO software overlay.

For this week, the progress aligns with our current schedule. In the next step, we will need to work on locating the user with the camera, and let the robot be able to reach the user based on the position.

This week I participated in the ethics lecture and discussed with our team about the ethics problem that our project could potentially have, and we also discussed the possible ways to mitigate these issues. I also explored how to integrate a comprehensive voice-interaction and hardware control system for our robot. By using Python and I2C communication, I unified the speech recognition and broadcast capabilities into a single control framework. I also utilized multithreading. This allows the OpenCV-based camera feed to run in the background without blocking the voice-recognition loop, enabling the robot to “listen” and “see” simultaneously. 

For this week, the progress aligns with our current schedule. In the next step, we will need to set up the sd card for our Rpi, and configure the network connection for our Rpi, so that all our individual modules can be connected to Rpi and tested together. 

This week I read the articles and watched the videos related to the ethnic assignment, and I also analyzed our own project from different aspects related to ethnic such as the potential safety concerns and public welfare aspect. I also focused on setting up and experimenting with the voice recognition module used in our robot system. I followed the official tutorial to configure the module and successfully modified both the wake-up word and the command words so that the system can respond to our customized voice inputs instead of the default ones. After updating these settings, the module was able to detect and recognize the newly defined words and output the corresponding command IDs. In addition, I studied how to enable communication between the voice recognition module and the Raspberry Pi. Specifically, I explored how the Raspberry Pi reads recognition results from the module and processes them in Python. 

For this week, the progress aligns with our current schedule. In the next step, we will also need to modify the starter program to receive the recognized command IDs and trigger corresponding actions to start the work flow of the robot, and interact with the overall system. 

 

This week I mainly worked on researching obstacle avoidance algorithms that can be integrated with the ToF sensor in our system. I reviewed several possible approaches and evaluated their feasibility based on the characteristics of ToF distance measurements. Since the ToF sensor provides accurate real-time distance data, I focused on algorithms that can effectively utilize this information for obstacle detection and avoidance. For example, I looked into methods such as simple threshold-based avoidance, potential field methods, and local path adjustment algorithms. I also compared the algorithms in terms of response speed, computational complexity, and ease of implementation on our current hardware platform. Some methods provide better navigation but require more computation, while simpler methods can respond quickly and are easier to implement with embedded systems. 

For this week, the progress aligns with our current schedule. In the next step, we will need to consider how these algorithms could be adjusted to better fit with our need, as well as how to integrate the tof obstacle avoidance with our camera sensing detection to accomplish the whole work flow of detection hand gesture and then navigate to the user’s position. 

This week I worked on the design presentation, and I practiced the presentation and then carried out the speech during class time. I also searched on the tof sensors’ working principle (measures distance by calculating the time taken by a light signal to travel to an object and return back to the sensor) and how it would be helpful for obstacle avoidance (by continuously measuring distance in front of it, and then detects if distance is smaller than predefined threshold, which then enables controller to take action).

Furthermore,  our team divided responsibilities for the design report due next Friday. I will be working on the use-case requirements, design requirements, test & verification & validation, and system implementation. Our progress is on schedule this week. For next week, the main focus would be motor bring up and initial system testing.

This week I mainly worked on the design for our voice recognition module. I compared a few different options that I found last week, and selected the most feasible one that we could use based on our current design. For example, when selecting the microphone, I realized that the 4 microphone array would have a more precise voice recognition function than the 2 microphone array, and it can also know the direction of the sound source. For the voice recognition engine, I focused on offline modules, and compared the recognition accuracy as well as the feasibility of using it such as whether it’s open source or not. What’s more, the delivery time for the products is also considered. Therefore, with all these factors taken into account, my teammates and I finally decided to use the WonderEcho AI Voice Recognition Module, where it has a builtin neural network processor for offline voice recognition, a noise reducing microphone, and runs on the CI1302 voice chip model. I also worked on the design presentation, where I mainly focused on the solution approach, implementation plan, and quantitative design requirements. 

For this week, the progress aligns with our current schedule, and the required components are all settled and ordered from the inventory. Then, after the components arrive, we will work on testing the components by parts, and make sure the motor for the trash bin works fine. 

This week I worked together with my teammates on preparing the presentation slides, including working on the use case and technical challenges. I also discussed with my teammates about the hardware items that we need to buy and order from the inventory list as well as other vendors, such as items required for the motor platform, visual detection model, and voice recognition model. What’s more, I also researched how we could implement the voice recognition part. Including two types of microphone arrays that we could potentially use (ReSpeaker 2-Mic HAT, ReSpeaker 4-Mic Array), as well as several offline keyword spotting engines that are suitable for running Raspberry Pi (vosk, Picovoice Rhino, Porcupine). 

This week my progress aligns with our current schedule, which is to finish proposal presentation, prepare for hardware items, and research on voice recognition modules implementation. 

For next week, I’m planning to further decide on how we will carry out the voice module. And I’m also going to work on the design presentation, and prepare for the upcoming deadline for our design plan.