Author: xinyul3

This week I focused on final preparation and full system integration. We completed the hardware integration, bringing together the Raspberry Pi, camera, ultrasonic sensors, and robot base into a single working system. I worked on assembling the sensing components onto a breadboard and connecting them to the Raspberry Pi to ensure stable operation for the final demo setup. This required verifying wiring, signal reliability, and making sure all sensors could be read consistently within the control loop.

On the testing side, I conducted validation trials of the integrated system, focusing on key metrics such as following distance and maximum speed. I measured how well the robot maintains the target distance during motion and evaluated its responsiveness under different speeds. Based on these results, we further tuned the PID controller to improve reaction speed and reduce lag in following behavior. The system now responds more quickly to changes in user position while remaining stable.

My progress is on schedule, as these tasks align with our final integration and validation phase. Next week, I plan to continue refining the system and perform more comprehensive testing to ensure reliability and consistency before the final demo.

This week, Xinyu focused heavily on end-to-end integration and testing of the system on the Raspberry Pi. He worked on combining multiple components, including PID-based motor control, vision-based bounding box detection, stereo depth estimation, and ultrasonic distance sensing to design a more complete path-following behavior. In particular, he integrated these signals into a unified control loop where the robot adjusts its motion based on estimated user position and distance while also reacting to nearby obstacles. A significant portion of time was spent testing the system under real conditions on the resource-constrained Raspberry Pi, identifying performance bottlenecks and ensuring that the perception and control modules can run together reliably.

During testing, Xinyu evaluated how different components interact, such as how depth estimation and bounding box tracking affect PID stability, and how ultrasonic signals should override motion commands for safety. He also experimented with different parameter settings to improve responsiveness and reduce oscillations in motion. Overall, the system is functioning at a basic level, but still requires further tuning to achieve consistent and stable behavior. Progress is on schedule, and next week he plans to continue refining control parameters and conducting more extensive real-world testing.

Additional section: New tools, knowledge, and learning strategies

During this stage, Xinyu needed to learn more about real-time system integration and control under constrained hardware. This included gaining a deeper understanding of PID control tuning in a robotics context, how to fuse multiple sensing modalities such as vision and ultrasonic sensors, and how to manage performance trade-offs on the Raspberry Pi. He also became more familiar with debugging system-level issues where perception and control interact, rather than isolated modules.

To acquire this knowledge, Xinyu primarily used informal learning strategies. He read documentation and online resources related to PID control and embedded vision systems, explored GitHub repositories and forum discussions to understand common implementation patterns, and watched tutorial videos for practical insights. In addition, he relied heavily on iterative experimentation, repeatedly testing the system, adjusting parameters, and observing behavior to build an intuitive understanding of how different components interact in real time.

This week, Xinyu was unable to attend class due to family matters, which slightly impacted overall progress and put his individual tasks slightly behind schedule. Despite this, he continued working on the vision pipeline, with a focus on improving its stability and robustness under real deployment conditions on the Raspberry Pi. In particular, he implemented fallback mechanisms such as dynamically adjusting input image resolution to balance detection accuracy and frame rate, as well as adapting the update frequency of bounding boxes so that the system can maintain smoother tracking when full detection cannot be performed at every frame. He also explored how these adaptive strategies interact with tracking logic to maintain consistent user identification under constrained compute.

Due to the reduced availability this week, progress is slightly behind schedule. However, the work done on improving system robustness is valuable for later integration and testing. Next week, Xinyu plans to dedicate more time to the project, focusing on completing integration with the control pipeline and conducting end-to-end testing on hardware to recover the schedule.

Additional section: New tools, knowledge, and learning strategies

During this phase of the project, Xinyu needed to learn several new tools and concepts related to embedded vision deployment and system integration. This included understanding how to optimize deep learning inference on resource-constrained devices like the Raspberry Pi, working with camera drivers and system-level configurations, and tuning parameters for real-time performance such as resolution scaling and frame scheduling. He also gained experience with stereo vision processing, including generating and interpreting disparity maps for depth estimation.

To acquire this knowledge, Xinyu primarily used informal learning strategies. He referred to official documentation for libraries such as OpenCV and YOLO, read technical discussions on forums and GitHub issues to resolve compatibility and performance problems, and watched online tutorials to better understand deployment workflows. In addition, he relied heavily on iterative experimentation, testing different configurations and observing system behavior to develop a practical understanding of performance trade-offs and system limitations.

This week I mainly focused on preparing for the interim demo, especially on the vision side. I recorded videos to demonstrate the current vision pipeline and its performance, and worked on designing a color frequency analysis method to help distinguish the target user across different bounding boxes. This is intended to improve person identification in multi-person scenarios by adding an additional signal beyond detection and tracking. I also trained our vision model using both our homegrown dataset and the COCO dataset, and evaluated it on a COCO test split to obtain baseline accuracy results for person detection.

In addition, we attempted to test the robot’s motion behavior, including speed and responsiveness, but encountered issues with the battery running out during testing. This limited the amount of hardware validation we could complete this week. As a result, my progress is slightly behind schedule. However, we believe this delay is manageable and plan to make up for it by continuing testing and integration early next week once the hardware is fully operational again.

Additional section: For verification of my vision and perception subsystem, I have started evaluating both detection accuracy and runtime performance. I trained the model on our homegrown dataset and the COCO dataset, and evaluated it on a COCO test set to obtain baseline accuracy metrics such as detection confidence and qualitative correctness. In addition, I plan to measure frame rate (FPS) on the Raspberry Pi under different input resolutions to verify that the system meets the real-time requirement. For the color-based person identification component, I will compare consistency of predicted identities across frames in multi-person scenarios. The results will be analyzed by checking whether detection accuracy remains high and whether the system maintains stable tracking at or near the required frame rate. These tests directly verify that the perception pipeline satisfies both accuracy and real-time constraints.

This week, I focused on integrating the robot platform with the vision pipeline on the Raspberry Pi. A significant portion of time was spent dealing with system-level issues, particularly version mismatches across drivers, camera libraries, and dependencies required to interface both the camera and the robot base. These compatibility issues made the setup more time-consuming than expected and required debugging at the OS and library level to ensure all components could work together properly.

In parallel, I worked on improving the depth estimation component using disparity maps from the stereo camera. I experimented with generating and refining disparity outputs to better capture relative depth differences in the scene, as shown in the attached results. This helps provide more stable distance estimation compared to simpler heuristics, especially in cluttered environments. My progress is slightly behind schedule due to the integration overhead, but resolving these system issues is critical for enabling full pipeline testing. Next week, I plan to complete the integration and begin testing the combined perception and control loop on hardware.

This week I focused on the design and implementation of the state machine for path finding and user following behavior. In particular, I worked on defining how the robot should transition between different states when navigating around obstacles while maintaining a stable following distance. This includes handling cases where the robot needs to temporarily deviate from a direct path to the user and then re-align once the path is clear. I refined the logic to make sure the behavior remains predictable and safe, especially when combined with the existing idle, searching, following, and stop states.

In addition, I worked on improving the vision pipeline by exploring ways to better distinguish the target user in multi-person scenarios. I experimented with training a lightweight CNN-based component to capture color features, which can help differentiate between individuals when detection alone is ambiguous. This is intended to complement the existing tracking approach and improve robustness in cluttered environments.

My progress is currently on schedule, as these tasks align with our planned work on perception and system behavior. Next week, I plan to continue integrating the state machine with the rest of the system and begin testing how the color-based features interact with the tracking pipeline in real scenarios.

This week I focused on the design of the vision and sensing pipeline, particularly considering edge cases that could affect the reliability of the system. One scenario I examined was when the user suddenly performs a U-turn in a narrow hallway or walkway. In such cases the robot must avoid overshooting or losing track of the user, so I began designing logic that can handle rapid direction changes while maintaining safe motion. I also worked on refining the high-level state diagram of the system, including the idle, searching, following, and stop states, to ensure that transitions between these states are well defined and safe during unexpected situations.

In addition, I started thinking more carefully about how the ultrasonic sensing component should be integrated with the vision pipeline. The ultrasonic sensors are intended to provide a fast safety signal independent of the vision system, so I explored how they should interact with the perception and control loop. This includes determining when ultrasonic readings should override motion commands and how to structure the sensing pipeline so that obstacle detection remains responsive even if the vision processing slows down.

My progress is currently on schedule, as these design tasks align with our planned work on perception and sensing logic. Next week, I plan to continue implementing parts of the vision pipeline and begin integrating ultrasonic sensing with the rest of the perception system so that both modules can operate together in the control loop.

This week I focused on getting the YOLO object detection model running on our Raspberry Pi so that we can start testing the vision component of SafeFollow. I worked on setting up the runtime environment, installing the necessary dependencies, and running initial inference with the camera input. One challenge I encountered was the frame processing rate. Because the Raspberry Pi does not have a dedicated AI accelerator, running YOLO on the CPU can result in very low frame rates, sometimes only a few frames per second, which makes real-time tracking difficult.

To address this, I experimented with reducing the input image resolution and adjusting the camera capture settings to balance accuracy and speed. I also explored using lighter YOLO model variants and optimizing the inference pipeline. These adjustments helped improve the processing speed somewhat, although performance tuning is still ongoing. Next week, I plan to continue optimizing the pipeline and integrate the detection output with our person tracking module.