This week I worked on compiling data for training the walk sign detection model. The model’s performance is only as good as the data that it is trained on, so I felt that it was important to get this step right. I spent a lot of time searching online for datasets of pedestrian traffic lights. However, I encountered significant challenges in finding datasets specific to American pedestrian traffic signals, which typically use a white pedestrian symbol for “Walk” and a red hand for “Don’t Walk.” The majority of publicly available datasets featured Chinese pedestrian signals that use a red pedestrian and green pedestrian symbol, which are not suitable for this model. I decided to instead compile my own dataset by scraping images from Google as well as Google maps. I will also augment this dataset with real world images, which I will begin next week. This progress so far is on schedule, perhaps a little behind. The lack of existing American datasets set my back a little, so I will need to expedite the data collection. Next week I hope to have a fully labeled dataset with multiple angles and lighting situations. This should be ready for model training, which will be the next step in the walk sign detection section.
William Shaw’s Status Report for 02/15/2025
This week, I primarily focused on ensuring that all our parts were in order. This was a much deeper dive into each hardware component than before, and I checked for things like compatibility, interfacing, and outputs. This caused me to revise a few of the prior hardware choices and realize that more needed to be added. In particular, the GPS module we planned to use did not give accurate heading data for stationary subjects. As such, I added a new IMU module to act as a compass, the Adafruit 9-DOF IMU Fusion Breakout BNO055. The module allows us to get accurate user heading without movement up to ±2°, while automatically compensating for head-tilt and sensor noise.
Another update regarding parts is the audio output and power supply. I had previously thought a Type C Power delivery power bank could power the Jetson board. However, for the Jetson Orin Nano Development Kit, the Type C port is data only and does not power the board. As such, I am looking into alternative power supplies/options for when we make the system portable. Additionally, the board does not come with a 3.5mm audio jack. While there is audio over the HDMI port, that is not a viable solution since we will not connect the board to a display. As such, I need to find a compatible USB sound card for the board.
So far, I have the Jetson Orin Nano on hand. The Arducam IMX219 (camera) and BNO055 IMU (Compass) are being shipped. These are necessary for us to begin testing the navigation system of our project, so we should be able to start testing actual inputs when they arrive (assuming that interfacing goes smoothly). There are a few remaining components to order (speakers, soundcard, portable power supply), but they are not system-critical for the work that needs to be done so far. I plan to order these components by the following weekly report. I am on schedule so far. By next week, I hope to have placed orders for all the components. I also aim to successfully interface the IMU and camera to the Jetson Orin Nano board.
Team Status Report for 2/8/2025
The most significant risks to the success of our project is the performance of the two image classification models and the integration of the hardware components. The accuracy of the image classification models need to be consistently high enough during real world testing in order for the helment to be able to transition between the two image classification and object detection states. The other issue is if the sensors we use will be compatible with our chosen microcontroller, the Jetson Nano. If, for example, the output of the camera is too high resolution and takes up too much memory, then this could be a problem for the limited memory on the microcontroller. These issues are still unclear since the ordered parts have not arrived yet, but the contingency plan is to simply try other parts such as lower resolution cameras that are still clear enough to be used for accurate image classification. No changes have been made to the existing design yet, as we have only just begun the implementation process and no issues have been discovered as of yet.
Andrew Wang’s Status Report for 2/8/2025
This week, I began looking into different object detection algorithms online that we can use as part of our first iteration of our implementation. Specifically, I installed a pre-trained YOLOv8 model from the YOLO package “ultralytics”, and was able to get it working on a CMU computing cluster. Since a rigorous evaluation and fine-tuning of the models will be necessary for integration, I’m planning on beginning to implement an fine-tuning and evaluation pipeline in the next few days to measure the model performance on unseen data, such as generic datasets containing images of streets such as BDD100K, EuroCity Persons, and Mapillary Vistas. Unfortunately, these datasets are way too big to store on the clusters I currently have access to, so I am working on obtaining access to alternative computing resources, which should be approved in the next few days.
With regards to progress, I believe that I am about on schedule. We have specifically set aside the upcoming week and next to evaluate and handle the ML side of our project based on our Gantt chart, and I am optimistic that we should be able to get this done in the next two weeks as the models themselves can simply be fine-tuned to any degree as we see fit with our constraints.
By the end of next week, I’d hope to have completed the download of the image datasets, as well as finished preliminary evaluation of the YOLOv8 model. We may also consider using different object detection models, although this is likely something we will consider more seriously as we get the first results from our YOLOv8 model.
William Shaw’s Status Report for 2/8/2025
This week, I spent most of my time looking into hardware components for our project. More specifically, I looked into the camera, GPS/Compass, and battery components. I also began drafting plans for mounting these components cleanly and comfortably. Currently, I plan to mount the camera to the helmet with a GoPro mount. This would let us adjust the camera angle based on testing results. Furthermore, I would like to create a case for the Jetson board and its components using laser cutting or 3D printing. If it is light enough, the board and battery can be mounted to the helmet, but otherwise, we may need to use a hip-mounted pack for comfort.
I also placed the order form for the Nvidia Jetson Orin Nano. I noticed a 4GB variant (non-Orin) was also available in the existing ready-to-use stock, but I am unsure if it is sufficient to run our models.
I am on track for our Gantt Chart Schedule. Although I have not placed all the orders for the other parts yet, testing of the code can begin once we receive our Jetson board. I would like to get this before placing orders for some of the parts (like the camera), so we can test the interfacing available. I also want to check how well it runs the model, as the top contender for the camera (Arducam IMX219) outputs a 3280 x 2464 pixel image. Although the FOV seems promising, the resolution might be too high to run our model at a suitable refresh rate.
For next week, I hope to finalize all of the parts and place the orders! However, I will emphasize that I want to ensure the components work first, which may require testing on the Jetson board beforehand.
Max Tang’s Status Report for 2/8/2025
This week I presented our group’s initial proposal presentation. The presentation went well, and I received many thought-provoking questions that have helped me realize that there were some aspects to our design that we have not considered, such as intersections that have multiple sidewalks. I began searching for suitable models that we can use to create our walk sign image classification model. One of these is an off-the-shelf YOLOv8 model that we can simply fine tune on walk sign images. Another potential solution I found is to gather as many images of walk signs as possible, as a combination of existing online datasets and self-taken images, and upload them to Edge Impulse. Then I can use Edge Impulse’s image classification model, which would be great for our project since Edge Impulse has a feature that lets you create quantized models, which use smaller data types for storing parameters and reduces the total memory required.
Progress is still on schedule. We allocated ourselves a large chunk of time for researching and making the model, and I believe that picking a suitable model at the beginning will help save time tuning and testing later. Next week I hope to be able to start the training and initial testing against validation datasets. This will give ample time for iteration if further improvements are required, which is very likely.