Author: mhtang

The only remaining work to be done is to add the audio component, which simply involves connecting the earbuds via bluetooth and then calling some Python library like “subprocess” to connect the earbuds, and then “pygame” to play an audio file. We have the logic for when each audio file should be played already. There is also the compass component, which should also be quick.
The final design for the chest mount has been completed. The final revision adds a front plate, neoprene padding, and uses black and smoke grey acrylic.

Unit tests:
YOLO model (90 ms inference)

The performance was good enough and the speed was fast enough to not have to make any design changes. We also observe that in practice, the model does well enough to identify obstacles in the field of view:

For the initial verson of the navigation, we provided feedback to the user based on how close a detected obstacle was to the camera. However, in common scenarios where a person is just overtaking the user, this logic is insufficient, such as in the example image below during our testing at a crosswalk on campus:

In order to address this, we needed to update our navigation submodule slightly, allowing for a slightly more nuanced logic with regards to whether or not to alert the user to specific obstacles. We will need to include in our system wide testing as well, to make sure that the navigation behaves as anticipated as part of a larger system.

ResNet model (30 ms inference)

Again, the performance was good enough and the speed was fast enough to not have to change the model or size.

For the overall system test, no numbers have been collected as of yet. However, we plan to run it as follows:
First, we wear the device (without being blindfolded). Next, the we wait at a crosswalk, ensuring that the “Walk Sign” cue is only played when there is a valid walk signal. Once the light displays “WALK”, we begin crossing the road. We will ensure that the device functions properly for both a clear crossing with no deviation, a crossing with objects in the path, and a crossing where we deviate from the path. At the end of the cross road, we also ensure that it resets back to the original Walk Sign Detection model (our idle state).

This week we prepared for our interim demo by working on integrating each independent submodule onto the Jetson Orin Nano. This involved working on uploading the models and necessary Python libraries, ensuring that the camera interfaces with the board, and checking that the video streaming works. The walk sign model is in a near finalized state, and the final step for integrating it is being able to take in a video frame as input from the camera. This will likely require a new submodule that solely handles sending the camera feed to the two vision models. This would make our code more modular, but the challenge is making sure that it works and can interface with the two models correctly. For example, the walk sign model can take in image inputs either from the file system given a path to the image, or it might be possible to use some Python interface with the camera to get the data instead.

WIth regards to the object detection model, we spent this week working on quantizing the model to determine the smallest size possible for the models to fit onto the Jetson, while also maintaining performance. This tradeoff will likely need to be explored in a bit more detail, but this allows for small models and quicker inference, which may be a factor in deciding which versions of the object detectors will be in the final product.

Concerning the hardware, work is continuing on the chest mount. We are now on the second revision, which finalized some dimensions with accurate (caliper) measurements and fixed some glaring comfort issues with strap angles. The design is not yet finalized, but prototypes are being made and laser cut. We are also working on calibrating / fine tuning the camera, to resolve some issues with red tint on the picture.

Enough progress has been made on each member’s respective subsystems for us to begin considering how to integrate all of the submodules onto the board together. The challenge is making sure that we can get all of the subsystems such as the walk sign classifier, obstacle detector, crosswalk navigator, speech module, etc. to work together. This will involve looking into how we can program the Jetson Orin Nano microcontroller and upload the machine learning models as well. The model performance is at a point where it is usable but could still use some finetuning, but it is more important at this point in time to make sure that we can actually run them on the board alongside the other modules.

With regards to the crosswalk navigation, we’ve begun implementing a basic feedback pipeline using text to speech libraries in python. No changes will be needed for right now, and we’ll likely need to wait until integration testing to determine if any further adjustments are needed in this submodule.

Concerning the hardware, work has begun on the design of the chest mount. Some images of the current design are included below. It will have straps at each of the four corners and will be worn like a chest harness. We plan to laser-cut the initial design out of wood as a proof of concept. The final version will be either ABS or wood, depending on which is more durable and easy to mount the devices onto. We will also likely add either a foam or TPU pad to the underside of the mount, as having a hard chestpiece would be uncomfortable for the user. With regards to the peripherals, the camera may have broken. This is system-critical, so will be the primary focus until it is resolved.

 

One issue that we have not really spent too much thought on is whether or not the machine learning models will fit on the microcontroller. The microcontroller itself already has a decent amount of memory, but it is possible that we can add external storage as well. The models themselves, from initial research, should definitely fit. Quantizing the models will make them smaller and possibly improve inference speed, but this can sacrifice performance and might not be necessary from a model size standpoint. No changes have been made to the design recently, and next week we should begin to explore how we can upload the models to the microcontroller and being developing the main program that is going to run the models and manage the control flow.

With regards to hardware, testing for integrating the peripherals is still ongoing. We forecast that it will be complete by the end of next week. One piece of good news is that we have confirmed that the USB C PD to DC cable can successfully power the Jetson Orin Nano, which will allow us to make it portable. For some reason, our existing power bank (which was not purchased but had on hand) cannot supply the correct wattage, which is probably due to it being old. We will now need to find a suitable power bank to buy. Initial designs have also started for mounting the Jetson Orin Nano. Based on our last meeting, we have decided to create both a helmet mount and a chest mount. Once complete, we can test and compare both designs for user comfort and camera shake. The testing will determine which mounting system is used in the final design.

With regards to the object detection models, we continue to have trouble with fine-tuning out of the box models. To give a greater range of models to evaluate, we have decided to pause the fine-tuning implementation, and work on evaluating different YOLO models available, such as YOLOv12. No changes otherwise have been made to the software implementation