Category: Uncategorized

This week, I have mostly been doing some preparation work for finalizing our project in the final week. I have ordered some new parts to help rebuild the belt. Currently I am still waiting on the parts to arrive before I reroute the circuit. But I have tested out the individual parts and ruled out components that are broken (since vibration motor disks and sensors are quite cheap, they can be quite unrealiable). I have done some basic testing to make sure that the ultrasonic and vibration sensors work as expected, and only need to wait on a few components to finish up assembling the final form of the belt next week.

Apart from the belt, I am also looking into developing a visualization system for the belt’s functionality. The basic idea is to feed the processed information from the raspberry pi to another PC through USB. I would hope to include both the video feed from the depth camera, as well as the information from the sensors. The visuals would indicate how the belt is producing effective warnings and how it is recognizing risks, so that observers can understand the belt without having to put it on. This would help us demonstrate the belt much better.

Next week, I will look to finish up on the two points above, and transition to testing before the weekend.

For this week, I was focused on adjusting the physical model for the belt and fixing the arduino interface. Following last week’s progress, I have completed a working prototype that consists of most of the components we need other than the battery and depth camera. Furthermore, I have adjusted part of the arduino interface to allow for a faster response rate from the ultrasonic sensors.

Later on this week, I have not made too much progress in face of carnival, but in the following week, I plan to reroute most of my circuits, as well as replace some of the parts that I found to be malfunctioning. I will also add a attachment mechanism for the depth camera as well as the battery.

This week I mostly worked with Ning on modeling the depth image to rate threat levels of obstacles. My first attempt was based around the assumption that comparing with the same baseline surface used for calibration, the mean squared error generated from  depth image with obstacles and an extension of the baseline surface without obstacles will be different enough to enable thresholds. But this assumption proved to be incorrect.

Subsequently, I discussed with Ning and we agreed on using Oak-D’s feature tracker module to narrow the range of pixels used for computing the mean squared error. We observed that the features tracked are fairly stable across time but could vanish in the event of shaking the camera. To smooth out this outlier, we planned to take the average of several historical frames (taken within one tenth of a second assuming 60 FPS).

I also proposed adding a Stop mode where all six coin motors will vibrate in cases where the user could not simply walk around, e.g., when facing a staircase or wall.  To build this feature, I searched online for neural net model that can classify stairs and managed to feed depth image into an example TensorFlow Lite classification model.

This week we mostly worked on modeling the depth image. At first, the idea is to compute the mean squared error between baseline ground surface and the incoming image stream, and place a higher weight on closer data as illustrated below. However, it is discovered that the errors were relatively unchanged from a baseline surface to one with obstacle. 

Now, the updated plan is to use Oak-D’s feature tracker module to pinpoint significant pixels to compare with their counterparts in baseline.  Since these feature pixels represent qualitative difference from the baseline surface, we reasoned that the subsequent mean squared error difference will be separable from baseline. In terms of implementation, the RGB camera at the center will be used to align features detected from the left and right stereo cameras through Oak-D’s built-in alignment functionality.

Additionally, we are considering adding a Stop mode where all six coin motors will vibrate in cases where the user could use some help and not simply walk around an obstacle, e.g., when facing a staircase or wall.  To achieve this, we have looked into classic computer vision algorithm and neural net model that can classify objects. This feature would also make more sense with audio feedback, which we would most likely leave as future work.

This week I brainstormed with Ning on how exactly the depth camera will pass data to raspberry pi and how the initial calibration would determine a legitimate starting path. For now, we will use the center vector of the whole frame  as baseline “ground truth” to avoid noise from adjacent objects.

I also worked on belt put-up with Alex and added the feature of running the control script upon system boot of the raspberry pi, i.e., turning on the power switch.

This week, I am primarily focused on the physical prototype for our belt. We wanted to put a model of the belt together before we conduct more software testing. Using myself as a user, I have planned out the placing of our components on the belt. This mainly includes the ultrasonic sensors, vibration units, arduino board, breadboard, raspberrypi, depth camera and the battery. Overall our sketch seems to be valid. So far we have attached all of the sensors as well as the vibration units and arduino board and breadboard. A rough image of what the board would look like is included in the team report.

I will try to finish up the connection on Sunday so we can demo our current progress on Monday.

A minor arduino adjustment this week was I changed the rate in which the arduino interface sends signals to the raspberry pi. Instead of sending a data for each sensor we operate, we would be doing so for every cycle of  6 sensors.

Moving on, we would primarily focus on software testing. We are also considering improving the physica modeling to hide more of the circuit. The exact way to do this is still under discussion, but will likely be decided after our physical prototype is all complete.

This week I explored different ways of collecting and processing data. I found a way to calculate depth based on the disparity of the stereo cameras, and wrote a successful sample code generating a map of depth information. I have tried combining stereo depth with a YOLO neural network, but I find the classification of objects to be rather unstable. After a discussion with the team about how our system should interpret the depth map, we decided that we will split the depth map into horizontal layers which we will tune the warning thresholds differently.

Next week my work will be primarily focused on tuning the number of horizontal layers and warning thresholds through extensive testing.

This week I added the logic of alerting for only a duration instead of indefinitely for a stationary obstacle, and sending control messages from Raspberry Pi to Arduino based on Alex’s encoding of multiple vibrator control in one line. Another update following Alex’s suggestion is that messages are sent only when intended intensity for a vibrator changes for more efficient serial communication. Meanwhile, for processing depth camera data, I aligned with Ning on how to start feeding arrays of depth data associated with different vibrators to the Raspberry Pi alert model.

I also broke two of the vibrators as their wires are too thin. This experience led me to push for casing all components onto the belt so that the system can be tested consistently and safely. Consequently, I got a mini breadboard from techspark and ordered purchase of more ultrasonic sensors(now we only have 5 instead of 6), new vibration motors, and an acrylic Arduino case.

Next week, I plan to incorporate depth camera data into the model and hopefully start testing the system against a set environment and gauge the accuracy of our sensing.

This week, I mainly focused on testing the system together with kelton, and made sure that connections between the arduino and the raspberry pi were working as expected. So far, the interface seems fine and we would need to move on to more testing to make sure that our threat detection are working correctly. I also started making physical models and developed circuit connection schemes that we will carry on next week.  Moving on, we will start to assemble our physical belt together first, and conduct software testing from that point onward.

 

This week, we focused on testing the multiple sensor and vibration system, made progress on how the depth camera identifies ground level threats, and started to make physical parts for assembling the belt.

For the sensor and vibration system, we have finished developing both the arduino interface to communicate with the sensors and vibration units, as well as the python interface on the part of raspberry pi. After some basic testing, the serial communication works quite well, and we were able to operate the system with multiple sensors and vibrators active at the same time. We have  developed a basic threat determining model in our python code, and conducted some testing to ensure that the system operates as expected.

In the testing process, we also realized that our vibration unit cables were extremely thin and fragile, and may not satisfy the requirements for a wearable device. Therefore, we decided to order new models for the vibration units that were a little more durable. We have also started to develop protective cases for our arduino board and breadboard circuits. Since we now have a finalized circuit connection model, we can simply build our circuits and continue our software testing without ever having to modifying the circuits. We plan to purchase existing arduino protection cases from amazon and use laser cutting and acrylic boards to build a custom case for our breadboard.

For detailed updates on the depth camera development, please see Ning’s report.