Category: Weekly updates-Ning

This week my work primarily focused on testing. During testing, we found our system to be less robust than expected, so I worked with Kelton to increase the robustness. I also helped in developing the testing scenarios. Planned testing scenarios include:

• stationary obstacles: we place multiple objects of varying heights (chairs, boxes, etc.) and lead the blindfolded user wearing the belt (“user” in later occurrence) to the site. We will ask the user to tell us the direction of the obstacles.
• Moving obstacles: We ask the user to stay still while our team members move in front of the user, effectively acting as moving obstacles. We then ask the user to tell us the movement directions of the moving obstacles.
• Mixed scenarios: We will ask the user to walk down a hallway with obstacles we place in advance. We will time the movements and observe the occurrence of any collisions.

This week I mainly focused on working with Kelton on condensing the 720P depth map into 2 threat level numbers. We eventually settled on using hard-coded thresholds to classify threats.

Below is a showcase of our work. From a single frame, we use hard thresholds to generate masks that signal a threat of at least level 1 (bottom center), 2 (bottom left), or 3 (top right). In these masks, the yellow part is where the threat lies. When we add all three masks together and pass it through a validity mask (top center, yellow parts are valid), we get the frame simplified to 4 threat levels (bottom right; the brighter the color, the higher the threat level.)

We do notice that the ground is considered a threat level of 1/2; we have theorized a way that utilizes the baseline matrix that we managed to generate in previous weeks to filter it out. Hopefully, we can show our result in our presentation next Monday.

This week I mainly worked on incorporating different nodes of the python DepthAI package to work with our new algorithm for the depth camera.

Our depth camera now incorporates a feature tracker that finds and tracks image features from RGB camera frames, and I successfully incorporated the RGB frame with depth frames, so now for each feature generated by the feature tracker, we can know its position in the frame and the distance data in the corresponding depth frame (available for comparison with the baseline if necessary) across different frames.

I’ve also tested the performance of the feature tracker with Kelton, and we found that the feature tracker performs poorly when the camera is shaken, in which case we may lose track of the features. We are considering the necessity of adding an IMU node in our depth camera code, but more testing is required to decide.

This week I worked with Kelton on object detection logic for the depth camera, specifically calibration. We are adopting the strategy of calibrating the surroundings upon system boot and constructing a “ground truth” frame with which we compare to real-time depth information. We assume that the ground in front of the user is flat during calibration, and found a way to model the depth information with linear regression (see charts in the team report.) For simplicity and to avoid the interference of obstacles on the side, we only take one pixel column in the middle of the frame and assume that it contains the correct information about a flat ground. We find that the reciprocal of the samples can be modeled by simple Ridge regression. We observe that the samples at the top of the frame are sometimes unstable, so we assign a smaller weight to those samples. Please note that this is only based on one pixel columns; the construction of the entire “ground truth” frame should be completed by tomorrow.

Next week I will focus on:

• finishing assembly of the physical belt with Alex; and
• implementation of obstacle detection & threat classification based on the “ground truth” frame with Kelton.