Category: Team Status Reports

A change was made to the existing design – specifically, the machine learning model used in the walk sign subsystem was changed from a YOLO object detection model to a ResNet image classification model. This is because the subsystem needs be able to actually classify images as either containing a WALK sign or DON’T WALK sign, so an object detection model would not suffice. No costs were incurred by this change other than the time spent adding bounding boxes to the collected dataset. One risk is the performance of the walk sign image classification model when evaluated in the real world. It is possible that images captured by the camera when mounted on the helmet are different (blurrier, taller angle, etc.) than the images the model is trained on. This can definitely affect its performance, but now that the camera has arrived, we can begin testing this and adjust our dataset accordingly.

Part A (written by Max): The target demographic of our product is the visually impaired pedestrian population, but the accessibility of pedestrian crosswalks around the world varies greatly across countries, cities, and even neighborhoods within a single city. It is common to see sidewalks with tactile bumps, pedestrian signals that announce the WALK sign and the name of the street, and other accessibility features in densely populated downtowns. However, sidewalks in rural neighborhoods or less developed countries often do not have any of these features. The benefit of the Self-Driving Human is that it would work at any crosswalk that has the signal indicator. As long as the camera can detect the walk sign, then the helmet is able to run the walk sign classification phase and navigation phases without any issues. Another global factor is the different symbols used to indicate WALK and DON’T WALK. For example, Asian countries often use an image of a green man to indicate WALK, while U.S. crosswalks use a white man. This can only be solved by training the model on country-specific datasets, which might not be as readily available in some parts of the world.

Part B (written by William): The Self-Driving Human has the potential to influence cultural factors by reshaping how society views assistive technology for the visually impaired. In particular, our project would increase mobility and reduce reliance on caregivers for its users. This can lead to cultural benefits like increased participation in certain social events as the user gains more autonomy. Ideally, this would lead to greater inclusivity in city design and social interactions. Additionally, our project could promote a standardized form of audio-based navigation, influencing positive expectations about accessible infrastructure and design. We hope this pushes for broader adoption of assistive technology-driven solutions, which could result in the development of even more inclusive and accessible technologies.

 

Part C (written by Andrew): The smart hat for visually impaired pedestrians addresses a critical need for independent and safe navigation while keeping key environmental factors in mind. By utilizing computer vision and GPS-based obstacle detection, the device minimizes reliance on physical infrastructure such as paving and audio signals, which may be unavailable or poorly maintained in certain areas. This reduces the dependency on city-wide accessibility upgrades, making the solution more scalable and effective across diverse environments. Additionally, by incorporating on-device processing, the system reduces the need for constant cloud connectivity, thereby lowering energy consumption and emissions associated with remote data processing. Finally, by enabling visually impaired individuals to navigate their surroundings independently, the device supports inclusive urban mobility while addressing environmental sustainability in its design and implementation.

The performance of the image classification and object detection models remain as the most significant risks, but these will only be revealed once we start actually testing them with data collected from our camera which has not arrived yet. For now, the contingency plan would be to switch models or perhaps make the scope of our input data or images that we want to classify smaller so that the models have an easier time with recognition. One change we made to the existing design was the camera we planned on using. We initially wanted a camera with a large field of view to try and capture as much of the environment as possible, but we realized that this would make the image size too large and make recognition harder.

With regards to the object detection model development, we plan to continue developing fine-tuned YOLO models. Initial testing of pre-trained models on out-of-distribution data (BDD100k validation dataset) yielded reasonable results, but we might want to consider leaning heavier on fine-tuned models for testing such that we have models trained on a wider variety of data. There is a significant risk that fine-tuning the existing models might not even be sufficient for accurate models when we integrate and test, however, and so our contingency plan is to continue collecting and processing more diverse datasets in an effort to boost performance.

In terms of hardware, we chose to delay ordering a sound card as we are considering using bone-conduction earphones for safety. They block less ambient noise and can be connected via Bluetooth. Testing for audio can be done through the DisplayPort connector, as the audio drivers should be identical regardless of which headphones we end up choosing. For power, we have ordered a USB-C PD to 15V 5A DC Barrel Jack converter. This fits into the power requirements while allowing us to use a PD Powerbank instead of a more esoteric Powerbank with a DC output.

Currently, the most significant risk to the project is obtaining high-quality data to use for training our models. This is crucial, as no amount of hyperparameter optimization and tuning will overcome a lack of high-quality and well-labeled data. The images we require are rather specific, such as obstacles in a crosswalk from a pedestrian’s perspective and images of the pedestrian traffic light taken from the sidewalk. We are managing this risk by obtaining data from a variety of sources, such as online datasets, Google Images and Google Maps, and also real-world images. If this does not work, our contingency plan is to perhaps adjust the purpose of our model so that it does not require such specific data.

As outlined in William’s status report for this week, a few updates have been made to the hardware components. First, an additional IMU module is needed for accurate user heading. The FOV of the camera ordered was reduced from 175º (D)  to 105º (D), as we were concerned about image distortion and extraneous data from having such a wide FOV. We chose 105º after some comparisons made using an actual camera to better visualize each FOV’s effective viewport. Having the Jetson Orin Nano on hand also allowed us to realize that additional components were needed to have audio output (no 3.5mm jack was present) and to make the power supply portable (the type-c port does not supply power to the board). These changes did not require any additional cost incurred by incompatible parts, as we have been very careful to ensure compatibility before actually ordering.

Our schedule remains essentially the same as before. For the hardware side, all the system’s critical components will arrive on time to stay on schedule. For the software side, our object detection model development is slightly behind schedule as mentioned in Andrew’s status report for 2/15. We anticipate having several versions of models ready for testing by the end of next week, and will be able to hopefully implement code to integrate it into our broader setup.

We will now go over the week 2 specific status report questions. A was written by William, B was written by Max and C was written by Andrew.

Part A. The Self-Driving Human is a project that is designed to address the safety and well-being of visually impaired pedestrians, both in a physiological and psychological sense. Crossing the street as a visually impaired person is both scary and dangerous. Traditional aids can be absent or inconsistent. Our project provides real-time audio guidance that helps the user cross the road safely, detect walk signals, avoid obstacles, and stay on the crosswalk. Because it is an independent navigation aid, it provides the user with self-sufficiency, as they are not reliant on crosswalk aids being maintained to cross the road. This self-sufficiency is an aspect of welfare, as the ability to move freely and confidently is a basic need. Ideally, our project works to create a more accessible and inclusive environment.

Part B. From a social perspective, the helmet will improve accessibility and inclusivity for visually impaired people and allow them to participate more fully in public life. There are some cities where pedestrian infrastructure is less friendly and accommodating, so this helmet would enable users to still cross streets safely. Economically, this helmet could reduce the need for expensive public infrastructure changes. Politically, solutions like this for the visually impaired can help increase awareness of the need for accessible infrastructure.

Part C. The traditional method of assisted street crossing/pedestrian navigation for the visually impaired involves expensive solutions such as guide dogs. While there is a significant supply of assistance, these methods might not be broadly accessible to consumers in need of them with regard to economic concerns. As such, we envision our project to serve as a first step in presenting an economically viable solution, able to be engineered with a concrete budget. As all of the navigation and feedback capabilities will be built directly into our device and will have been appropriately developed before porting them to the hardware, we anticipate that our (relatively) lightweight technology can increase the accessibility of visually impaired navigation assistance on a budget, as the development and distribution our project can be scaled with the availability of hardware, helping resolve consumption patterns.

The most significant risks to the success of our project is the performance of the two image classification models and the integration of the hardware components. The accuracy of the image classification models need to be consistently high enough during real world testing in order for the helment to be able to transition between the two image classification and object detection states. The other issue is if the sensors we use will be compatible with our chosen microcontroller, the Jetson Nano. If, for example, the output of the camera is too high resolution and takes up too much memory, then this could be a problem for the limited memory on the microcontroller. These issues are still unclear since the ordered parts have not arrived yet, but the contingency plan is to simply try other parts such as lower resolution cameras that are still clear enough to be used for accurate image classification. No changes have been made to the existing design yet, as we have only just begun the implementation process and no issues have been discovered as of yet.