Tag: status report

One issue that we have not really spent too much thought on is whether or not the machine learning models will fit on the microcontroller. The microcontroller itself already has a decent amount of memory, but it is possible that we can add external storage as well. The models themselves, from initial research, should definitely fit. Quantizing the models will make them smaller and possibly improve inference speed, but this can sacrifice performance and might not be necessary from a model size standpoint. No changes have been made to the design recently, and next week we should begin to explore how we can upload the models to the microcontroller and being developing the main program that is going to run the models and manage the control flow.

With regards to hardware, testing for integrating the peripherals is still ongoing. We forecast that it will be complete by the end of next week. One piece of good news is that we have confirmed that the USB C PD to DC cable can successfully power the Jetson Orin Nano, which will allow us to make it portable. For some reason, our existing power bank (which was not purchased but had on hand) cannot supply the correct wattage, which is probably due to it being old. We will now need to find a suitable power bank to buy. Initial designs have also started for mounting the Jetson Orin Nano. Based on our last meeting, we have decided to create both a helmet mount and a chest mount. Once complete, we can test and compare both designs for user comfort and camera shake. The testing will determine which mounting system is used in the final design.

With regards to the object detection models, we continue to have trouble with fine-tuning out of the box models. To give a greater range of models to evaluate, we have decided to pause the fine-tuning implementation, and work on evaluating different YOLO models available, such as YOLOv12. No changes otherwise have been made to the software implementation

A change was made to the existing design – specifically, the machine learning model used in the walk sign subsystem was changed from a YOLO object detection model to a ResNet image classification model. This is because the subsystem needs be able to actually classify images as either containing a WALK sign or DON’T WALK sign, so an object detection model would not suffice. No costs were incurred by this change other than the time spent adding bounding boxes to the collected dataset. One risk is the performance of the walk sign image classification model when evaluated in the real world. It is possible that images captured by the camera when mounted on the helmet are different (blurrier, taller angle, etc.) than the images the model is trained on. This can definitely affect its performance, but now that the camera has arrived, we can begin testing this and adjust our dataset accordingly.

Part A (written by Max): The target demographic of our product is the visually impaired pedestrian population, but the accessibility of pedestrian crosswalks around the world varies greatly across countries, cities, and even neighborhoods within a single city. It is common to see sidewalks with tactile bumps, pedestrian signals that announce the WALK sign and the name of the street, and other accessibility features in densely populated downtowns. However, sidewalks in rural neighborhoods or less developed countries often do not have any of these features. The benefit of the Self-Driving Human is that it would work at any crosswalk that has the signal indicator. As long as the camera can detect the walk sign, then the helmet is able to run the walk sign classification phase and navigation phases without any issues. Another global factor is the different symbols used to indicate WALK and DON’T WALK. For example, Asian countries often use an image of a green man to indicate WALK, while U.S. crosswalks use a white man. This can only be solved by training the model on country-specific datasets, which might not be as readily available in some parts of the world.

Part B (written by William): The Self-Driving Human has the potential to influence cultural factors by reshaping how society views assistive technology for the visually impaired. In particular, our project would increase mobility and reduce reliance on caregivers for its users. This can lead to cultural benefits like increased participation in certain social events as the user gains more autonomy. Ideally, this would lead to greater inclusivity in city design and social interactions. Additionally, our project could promote a standardized form of audio-based navigation, influencing positive expectations about accessible infrastructure and design. We hope this pushes for broader adoption of assistive technology-driven solutions, which could result in the development of even more inclusive and accessible technologies.

Part C (written by Andrew): The smart hat for visually impaired pedestrians addresses a critical need for independent and safe navigation while keeping key environmental factors in mind. By utilizing computer vision and GPS-based obstacle detection, the device minimizes reliance on physical infrastructure such as paving and audio signals, which may be unavailable or poorly maintained in certain areas. This reduces the dependency on city-wide accessibility upgrades, making the solution more scalable and effective across diverse environments. Additionally, by incorporating on-device processing, the system reduces the need for constant cloud connectivity, thereby lowering energy consumption and emissions associated with remote data processing. Finally, by enabling visually impaired individuals to navigate their surroundings independently, the device supports inclusive urban mobility while addressing environmental sustainability in its design and implementation.

The performance of the image classification and object detection models remain as the most significant risks, but these will only be revealed once we start actually testing them with data collected from our camera which has not arrived yet. For now, the contingency plan would be to switch models or perhaps make the scope of our input data or images that we want to classify smaller so that the models have an easier time with recognition. One change we made to the existing design was the camera we planned on using. We initially wanted a camera with a large field of view to try and capture as much of the environment as possible, but we realized that this would make the image size too large and make recognition harder.

With regards to the object detection model development, we plan to continue developing fine-tuned YOLO models. Initial testing of pre-trained models on out-of-distribution data (BDD100k validation dataset) yielded reasonable results, but we might want to consider leaning heavier on fine-tuned models for testing such that we have models trained on a wider variety of data. There is a significant risk that fine-tuning the existing models might not even be sufficient for accurate models when we integrate and test, however, and so our contingency plan is to continue collecting and processing more diverse datasets in an effort to boost performance.

In terms of hardware, we chose to delay ordering a sound card as we are considering using bone-conduction earphones for safety. They block less ambient noise and can be connected via Bluetooth. Testing for audio can be done through the DisplayPort connector, as the audio drivers should be identical regardless of which headphones we end up choosing. For power, we have ordered a USB-C PD to 15V 5A DC Barrel Jack converter. This fits into the power requirements while allowing us to use a PD Powerbank instead of a more esoteric Powerbank with a DC output.