Posted on

Team Status Report for 4/19/2025

Last week, we managed to get both the object detection model and walk sign model onto the board. Due to make sure that the object detection model could use the GPU and decrease the inference time, we had to create a docker container for it to run in. However, due to Python dependency issues with tensorflow and pytorch, we are currently trying to change the walk sign image classification model to only use pytorch. We have tried a variety of methods such as converting a .h5 tensorflow model to a .pth pyotrch model, and also just rewriting everything using pytorch, but both have had issues. We are still currently exploring solutions to this problem.

Regarding hardware, we have finally mounted all our components to the chest mount. The power bank also arrived, which fits nicely into a fanny pack. We have tested running our system fully attached to a person, and it works as expected. The device is comfortable, and does not impede user motion or weigh too much. We are still improving the mount, but it’s in a very good state right now.

Update on 4/20/2025: Regarding the issue with converting the model from tensorflow to pytorch, we have successfully recreated the model in pytorch. We now have 3 different models trained on the much larger dataset collected from last week, using resnet 34, 101, and 152. The performance and confusion matrix for each are pretty similar. This is the confusion matrix for the resnet 152 model:

The test accuracy is 90.54%, but it’s possible that the class imbalance is skewing this accuracy. In any case, it’s better for the model to be more cautious when predicting “walk” than “don’t walk”, and we see that the error rate for the “don’t walk” class is very low.

Posted on

William Shaw’s Status Report for 04/19/25

This week, I focused more on fine-tuning the chest mount. We also continued working on integration and testing all our components. Last week’s changes to the chestmount were well received. We tested the chest mount of a few women, all of whom stated that the mount was secure and comfortable. This resolved our worries about non-flat chests being incompatible with our mounting design. The standoff components arrived, and I had to make some tweaks to the mounting dimensions to get them to work properly. However, after the modifications, everything works perfectly. Currently, the Jetson Orin Nano, camera, and IMU are nicely and securely mounted onto the chestpiece. One small issue was that I didn’t consider mounting the WiFi module, along with its antennas. These were connected to the original case. I had assumed that we would have our stuff running offline, so I thought I wouldn’t need to mount these. I will look into fully mounting these, but for now, they are safely tucked into the IMU standoffs. Lastly, I have begun testing the addition of neoprene padding for comfort. 

The power bank also worked well, providing more than enough power to last the whole day (around 11 hours). With the components all nicely mounted to the chest mount, we were much more easily able to gather data. Previously, data collection needed to be done with two people, as we had to hold all the components together carefully. Now the device is basically hands-free, save for the laptop needed to SSH in and run the scripts. Hopefully, this can be replaced with a startup script once we finalize integrating all our software components together. 

Regarding the schedule, I am in sync with the Gantt Chart. I aim to continue working to get all the components mounted to the chest mount. I expect to work through some issues with screw hole dimensions, so I have blocked out a decent chunk of time for iterating through designs. 

As I designed, implemented, and debugged my project, I had to use some tools to fix issues with the camera. Specifically, I had to use nvarguscamerasrc, Argus daemon, GStreamer, and v4l2-ctl for debugging. Most of my knowledge for this was gained by using Google and the NVIDIA Developer Forums for people with similar issues. I also had to debug many issues by checking versioning issues with packages. This issue is still in progress, as we have some conflicts with what packages are needed.

Posted on

Andrew Wang’s Status Report 4/19/2025

This week, I worked on integrating the ML object detection with the hardware components that Will has set up. Most of the work that needed to be done was in the context of porting the model to the Jetson, then ensuring that the models themselves ran on the Orin GPU. This was a lot trickier than I anticipated, since Jetson Nano configurations with the underlying CUDA kernels aren’t as straightforward as using GPUs on the computing cluster that I had developed and initially evaluated the models on.
I was able to eventually get the object detection models running using a Docker container, which are custom modular builds that enable the usage of Jetson-specific GPU functionalities that are needed for the object detection models. After this started working, I observed a drop in inference time from 8 seconds/frame to 90 milliseconds/frame, so I am reasonably sure that this is working now.
I was also able to draft an implementation of a speech to text feedback loop in the navigation submodule, and I’ll be able to fully test it in the next few days to make sure the feedback is clear and timely.
With regards to progress, I believe that I am about on schedule. I am not particularly concerned about not being able to complete the necessary deliverables on my end, and in the coming weeks, I anticipate spending time working on the integration, possibly helping troubleshoot the other components when I can, which I started doing this week.

The next week, I also anticipate refining the navigation logic further, and getting a speech to text feedback loop tested on the Jetson, as mentioned before.

Posted on

Max Tang Status Report 4/19/2025

This week I worked on the final integration of the walk sign image classification model into the board and docker environment. We first further optimized the walk sign model. I trained it on two different datasets: one that included a lot of non-crosswalk images labeled as “don’t walk”, and one that only included crosswalk images. I also implemented the code logic for transitioning from walk sign image classification to cross walk object detection. Initially, the model worked fine on the board, but uploading the other object detection model, we realized there were Python dependency conflicts. This is still currently an issue, as there have been challenges in trying to convert the tensorflow model to a pytorch model. One attempt was to change the walk sign model from a tensorflow model to a pytorch model. This involved first saving the model in the .tf format, and then converting it to .onnx using (python -m tf2onnx.convert –saved-model walksignmodel –output walksignmodel.onnx), and then converting that to .pt. However, this has had many other python dependency issues with the onnx libraries too. My plan for this weekend is to resolve this issue as soon as possible.

Update on 4/20/2025: The pytorch issue has been resolved, see team status report for update.

Posted on

Team Status Report for 4/12/2025

We conducted a live test of the walk sign image classifier using the actual camera and with the model running on the Jetson. The performance did not meet our specifications, so we are now working on improving that model, as detailed in Max’s status report for this week. The good news is that the camera quality is decent, and despite some initial issues with linking the video stream to the model, everything seems to work smoothly.

The object detection models all seem to work reasonably well on the images sampled from the live video feed. Not only do the bounding boxes accurately center on the relevant objects, but the video feed is clearly high-resolution enough for the model to work correctly. As such, it has been straightforward to visually confirm that the object detection component is working as well as it did during our extensive testing phase, and we can focus our efforts on refining the navigation submodule logic.

Concerning hardware, finishing touches for the chest mount are arriving this weekend. This includes standoff components and the padding. The primary material of the chest mount has been changed from 6mm plywood to 3mm acrylic. This decision was made for weight, flexibility, compatibility (with mounting hardware), and aesthetics. We also have obtained our power supply, which reads an estimated 11-hour battery life. This should be more than enough to cover the day-to-day usage of most users.

Posted on

William Shaw’s Status Report for 04/12/2025

This week, I focused on the refinement of the chest mount, as well as some integration and finalization. For the chest mount, I worked on calibrating the designs for comfort. Now, the main part of the chest mount is made of 3mm acrylic (vs the 6mm plywood previously used). This allows for much smoother contact with straps, as the previous design had issues with fraying fabric.  It is also much lighter and more flexible, making the design more comfortable to wear. The standoff components are arriving tomorrow, which will let me mount the components to the chest piece. 

In other news, the power bank arrived this week. It took a while for this, as I had to ensure that the battery could consistently output the required wattage over the ports (an issue we ran into with existing power supplies). This allowed us to take the system out for testing, albeit awkwardly, as we held all the components, laptops, and wires by hand. The results from this testing were not great, as the classifiers could not correctly detect the correct signals most of the time. As such, we recorded a bunch of new data, which my teammates are using to retrain the models to make them more accurate. 

Another issue was that my default settings for the camera were lackluster. I believe that with some fiddling, I can get a better picture quality for the models. However, this higher resolution may not help or may even cause the models to take longer to run. Ultimately, more testing needs to be done for this. On a more positive note, the red tint issues from last week have been completely resolved using a .isp file I found online. 

Regarding the schedule, I am more or less in sync with the Gantt Chart (which was updated during demos). I aim to continue working tomorrow to get all the components mounted to the chest mount. I expect to work through some issues with screw hole dimensions, so I have blocked out a decent chunk of time for iterating through designs.

Posted on

Andrew Wang’s Status Report 4/12/2025

This week, I worked on evaluating our models on a real world video feed, sampling images from an actual crosswalk that reflects our use case, and implementing a basic form of our feedback logic in the navigation submodule.

 

Using our models, I manually implemented visualizations of the bounding boxes identified by the model in the images sampled, and printed out any feedback prompts that would be relayed to the user on the top left of each image.

 

Visually inspecting these images, we see that not only do the bounding boxes look accurate to their identified obstacles, but that we are able to use their positional information in the image to relay feedback. For sample, in the second image, there is a person coming up behind the user on the right side, so the user is instructed to move to the left. Additionally, the inference speed was very quick; the average inference time per image was roughly 8ms for the non-quantized models, which is an acceptably quick turnaround for a base model. I haven’t tried running inference on the quantized models, but previous experimentation has indicated that it would likely be faster. As such, since the current models are already acceptably quick, I plan on testing the base models on the Jetson Nano first before I attempt to use to quantized models to maximize performance accuracy.

 

Currently, I am about on schedule with regards to ML model integration and evaluation. For this week, I plan on working with the staff and the rest of our group to refine the feedback logic given by the object detection model, as currently the logic is pretty rudimentary, often giving multiple prompts to adjust the user’s path when only one or even none are actually necessary. I also plan on integrating the text to speech functionality of our navigation submodule with our hardware if time allows. Additionally, I will need to port the object detection models onto the Jetson Nano and more rigorously evaluate the speed on our hardware, so that we can determine if more intense model compression will be necessary to make sure inference and feedback are as quick as we need it to be.

Posted on

Max Tang’s Status Report for 4/12/2025

Last week, we conducted a live test of the walk sign image classifier using the initial model. This involved running the model on the Jetson and feeding it images captured by the camera once every 2 seconds. By also saving the image that was being fed into the model, I could then later view the image on my computer and see exactly what the model was seeing. The performance during the live test was not great and the model frequently predicted “GO” and only rarely predicted “RED”. To try and fix this, we collected 400 more images using the same camera and retrained the model. I re-evaluated the performance with a test dataset and it seems to be better now, but I still need to conduct more field tests and then likely repeat this process if it’s not performing well enough. I also need to work on implementing the rest of the control logic for switching from the walk sign classification to the object detection process.

Posted on

Andrew Wang’s Status Report: 3/29/2025

This week, I worked on quantizing the object detection models, such that we may have a smaller, more compressed model that will fit on the Jetson and also for faster inference times.

 

I am using TensorRT and Onnx packages to perform the quantization on the YOLO models, which I found online was the best method for quantizing ML models for Edge AI computing, due to the fact that this method results in the fastest speedups while shrinking the models appropriately. However, this process was pretty difficult as there are a lot of hardware specific issues I ran into in this process. Specifically, the process of quantizing using these packages involves at lot of under the hood implementation assumptions about how the layers are implemented and whether or not they are fuse-able, which is a key part of the process.

I did end up figuring it out, and ran some evaluations on the YOLOv12 model to see how performance changes with quantization:

 

Visually inspecting the prediction-labelled images and PR curve as well as the normalized confusion matrix, just like before, it generally appears that there isn’t much difference in performance with the base, un-quantized model, which is great news. This implies that we can compress our models in this way without sacrificing performance, which was the primary concern

Currently, I am about on schedule with regards to ML model development, but slightly behind on integration/testing the navigation components.

For this week, in tandem with the demos we are scheduled for, I plan on getting into the details of integration testing. I also would be interesting in benchmarking any inference speedups associated with the quantization, both in my development environment and on the Jetson itself in a formal testing setup.

Posted on

Team Status Report for 3/29/2025

This week we prepared for our interim demo by working on integrating each independent submodule onto the Jetson Orin Nano. This involved working on uploading the models and necessary Python libraries, ensuring that the camera interfaces with the board, and checking that the video streaming works. The walk sign model is in a near finalized state, and the final step for integrating it is being able to take in a video frame as input from the camera. This will likely require a new submodule that solely handles sending the camera feed to the two vision models. This would make our code more modular, but the challenge is making sure that it works and can interface with the two models correctly. For example, the walk sign model can take in image inputs either from the file system given a path to the image, or it might be possible to use some Python interface with the camera to get the data instead.

WIth regards to the object detection model, we spent this week working on quantizing the model to determine the smallest size possible for the models to fit onto the Jetson, while also maintaining performance. This tradeoff will likely need to be explored in a bit more detail, but this allows for small models and quicker inference, which may be a factor in deciding which versions of the object detectors will be in the final product.

Concerning the hardware, work is continuing on the chest mount. We are now on the second revision, which finalized some dimensions with accurate (caliper) measurements and fixed some glaring comfort issues with strap angles. The design is not yet finalized, but prototypes are being made and laser cut. We are also working on calibrating / fine tuning the camera, to resolve some issues with red tint on the picture.