Posted on

Team’s Status Report for 4/12

The most significant risks that could jeopardize the success of our project is the error while overlaying the coordinate system given by the UWB sensors and the coordinate system that we have from the segmentation model. Since obstacles are in one coordinate system and user localization is in another, incorrect overlaying could cause our user to bump into obstacles or never be guided to the right location. This could definitely jeopardize our project working, and is the last main milestone we have to accomplish. To manage these risks, we came up with a calibration system that could recalibrate the UWB coordinate system and scale it to match the occupancy matrix in which we run path planning on. We would have to work on it further to ensure it is fully robust. Another slight risk is that the Jetson does not have enough processing power to run the segmentation model that identifies our environment as obstacles or free space. To manage this risk, we could offload the compute to the cloud or on a separate laptop. Since the SAM model is not exactly tiny or meant for edge devices, the cloud or a working laptop is definitely a better alternative.

Thus, the only possible change that we might make to our existing design is offloading the compute from the Jetson to the cloud or a laptop. This change is necessary because it is quite evident that the Jetson does not have enough processing power or RAM to run segmentation models. We could try troubleshooting this a bit more or consider alternatives, but this is a possible change we might make.

To verify our project starting from the beginning of the pipeline, we would first test the accuracy of the segmentation model. So far we have inputted around 5-6 example images from the bird’s eye view and analyzed how well it was able to segment free space and obstacles. It has done an accurate job so far, but once we are in the verification stage we would empirically test it on 10 different environments and make sure the accuracy is above 95%. For latency of the pipeline on segmentation, we already know it will be approximately 30 seconds. Thus, we would have to slightly adjust our use case requirements to a fixed environment but still monitor the user in real time. The rest of the pipeline has minimal latency (downsampling algorithm, A* path finding algorithm, socket communication between Raspberry Pi and laptop, etc), so the segmentation model is the only bottleneck. We would run it in different environments and note the latency to verify our use case requirements. For correct navigation of the user to the destination, the main concern is the overlay between the coordinate system given by the UWB sensors and the coordinate system given by the segmentation model. Getting the haptics to vibrate in the right direction should not be a huge issue, but the main issue is most likely having the person be at the correct location indicated by the UWB sensors. We can measure this empirically by noting the error between where the person is located and where our system thinks he / she is located. This metric could help guide us to the navigation accuracy metric.

Posted on

Talay’s Status Report for 4/12

This week, I worked on setting up the Arducam wide-lens camera on the Jetson, setting up the keypad to work on the Raspberry Pi, setting up the haptics sensors to work on the Raspberry Pi, and setting up socket communication under the same wifi subnet between the Raspberry Pi and my laptop. The first task I did was setting up the Arducam wide-lens camera by installing the camera drivers on the Jetson. The process was quite smooth compared to the OV2311 Stereo Camera since this wide-lens camera relied on the USB protocol which is a lot more robust.

From the image you can see the quality is decent and the camera has a 102 FOV.

Next I worked on communication between the Raspberry Pi and the laptop using sockets. I set up a virtual environment on the Raspberry Pi and installed all the necessary dependencies to set this up. After this I set up the keypad on the Raspberry Pi using GPIO pins. Once the Raspberry Pi detects that a key is pressed, it sends data to the laptop which runs the A* path planning module and has the occupancy matrix visualized. The keypad is able to select different destinations on the occupancy matrix via network connection.

Once the laptop runs path-planning and determines the next move for the user, it sends the directions back to the user (Raspberry Pi) through the same socket connection. I then set up the haptic vibration motors which is also on the Raspberry Pi. I had to use different GPIO pins to power the vibration motors since some pins were already used for the keypad.

The Python code on the Raspberry Pi was able to detect which direction the user had to move in next and vibrate the corresponding haptic vibration motors. Our haptic vibration motors had slightly short wires, so we might need an extension for the final belt design.

I believe my progress is on schedule. Currently, the next main milestone for the team is to integrate the UWB sensor localization with the existing occupancy matrix. This is our last moving part in terms of project functionality. The last milestone for this project would be to integrate the Arducam wide-lens camera instead of the phone camera.

Next week, I hope to work with Kevin on integrating the UWB sensor localization with the occupancy matrix generated from the segmentation model and down-sampling. Currently, the user location is simulated through mouse presses, but we want it to be updated on our program through data received from the UWB tags. We would have to think about the scale factor of the camera frame and how much it has been downsampled, and apply the same scale to the UWB reference frame.

To verify my components of the subsystem, I would first test the downsampling algorithm and the A* path-planning module on different environments. I would feed the pipeline around 10-15 bird’s eye view images of indoor spaces. After the segmentation model, I would note if the downsampling does a good job of retaining the required resolution and identifying obstacles and free space. I would empirically measure the occupancy matrix on whether 95% of obstacles are actually still labeled as obstacles. To meet our use case requirement of not bumping into obstacles, I would come up with a metric that shoots for over-detection of obstacles rather than under-detection. Our current downsampling algorithm creates a halo effect around the obstacles, which would meet this requirement. For the communication requirements between the Raspberry Pi (the user’s belt) and the laptop, there is already minimal processing time. The A* path planning algorithm also has minimal processing time, so the user will have real-time updates on which direction to go through haptic sensors that meets our timing requirements. For percentage of navigation to the right location, I would check whether the haptic sensors always vibrate in the right direction out of 10 tries. If there is some calibration required by a compass, that would be taken into account as well.

Posted on

Charles’s Status Report for 4/12/2025

This week I spent my time working with the Jetson to try to get our image processing code to work on it. This mostly involved a lot of package building and library downloading which ended up being much more involved than I initially thought it was going to be.

We ordered a WiFi adapter for the Jetson that way we could use WiFi instead of Ethernet. This made it so that we could use the Jetson at home and start working on getting all the necessary packages installed.

Installing the packages actually ended up being pretty complex. All the simple libraries like numpy were relatively painless to install. However, all the ML libraries ended up being much more of a hassle. PyTorch and Torchvision both took an extremely long time to install and build. This is because the normal libraries for these that are hosted through pip are not compatible with the Jetson architecture. Using this link, I was able install and build a PyTorch and Torchvision that was compatible with the jetson architecture. This did end up taking multiple hours to build and compile, so I had to leave it running overnight. After finally getting all the necessary libraries downloaded to finally compile our image processing code, we ran into another problem.

As we can see here, we have the 2GB of RAM that our board has, but we also have a 9GB swap file. This 9GB swapfile is essentially virtual RAM for the board because the base amount of RAM is very small. The problem with this is that, even when loading in smaller models, the system runs out of memory on almost any input.

Because the process runs out of memory, it ends up being killed by the OS. Resulting in this message:
[Sat Apr 12 19:12:14 2025] Killed process 10399 (python3) total-vm:9650160kB, anon-rss:0kB, file-rss:537696kB, shmem-rss:0kB

This essentially means that we have completely ran out of memory and thus unable to continue being run.

For the upcoming week, I am going to figure out if there is anything that we can do to run the image processing on the Jetson, and if that’s not possible, if there is any other alternatives.

For verification and validation, I am going to start testing the camera and the image processing in combination. We will do multiple room layouts to test the robustness of the mask generation. We will most likely still do the processing on our laptops because it will be significantly faster, but we will use the camera that we have recently bought.

Posted on

Kevin’s Status Report for 4/12

This week I received all the DWM1001 UWB anchors. I was able to set them all up to communicate with the tag. I configured the UWB API to read 3 distances simultaneously.

I was able to use the UWB hardware to triangulate to position of the tag on a smaller scale. An image of this setup is attached below, but I was able to successfully triangulate the tag on a table by placing anchors on the corners of the table.

I also helped with the GPIO communication with the haptic vibrators. We were able to trigger vibration motors, and Talay adapted them to our path planning communication.

I think we made good progress this week but our schedule is still tight. For the upcoming week, I think we are on a good track if we can have the entire system integrated; this involves setting up the UWB to be calibrated with the occupancy matrix in the full room. We also need to consider how to set this system up for demo day; I am thinking to buy some tall stands, and use them to hang our overhead camera and install our UWB anchors.

For verification, I want to measure the precision between the sensors. What is interesting is that the absolute distance doesn’t really matter since our distances are arbitrarily scaled. But we can verify that the measurements are consistent across each anchor. Then, we should verify the accuracy of the actual localization system by comparing the actual vs measured location.