Category: Team Status Report

The most significant risk that could jeopardize the success of our project is if one of the components break. Since our design requires a lot of components and we only have one of each, breaking one of them could jeopardize the functionality of our project. Each of these components are fully working right now, but some of them seem quite fragile with wires hanging loose. The way we design our final form factor for the belt will be crucial so that components are embedded securely onto it. We don’t want a belt that would have our components hanging loose or falling while the user is walking around. Since all the other components are fixed in place, the belt would be the most fragile part of our design and would need careful planning.

The only changes made to the existing design of the system is to do processing on a separate node instead of the Jetson. The Jetson does not have enough compute to segment the image, so we will be offloading that compute to the laptop instead.

The most significant risks that could jeopardize the success of our project is the error while overlaying the coordinate system given by the UWB sensors and the coordinate system that we have from the segmentation model. Since obstacles are in one coordinate system and user localization is in another, incorrect overlaying could cause our user to bump into obstacles or never be guided to the right location. This could definitely jeopardize our project working, and is the last main milestone we have to accomplish. To manage these risks, we came up with a calibration system that could recalibrate the UWB coordinate system and scale it to match the occupancy matrix in which we run path planning on. We would have to work on it further to ensure it is fully robust. Another slight risk is that the Jetson does not have enough processing power to run the segmentation model that identifies our environment as obstacles or free space. To manage this risk, we could offload the compute to the cloud or on a separate laptop. Since the SAM model is not exactly tiny or meant for edge devices, the cloud or a working laptop is definitely a better alternative.

Thus, the only possible change that we might make to our existing design is offloading the compute from the Jetson to the cloud or a laptop. This change is necessary because it is quite evident that the Jetson does not have enough processing power or RAM to run segmentation models. We could try troubleshooting this a bit more or consider alternatives, but this is a possible change we might make.

To verify our project starting from the beginning of the pipeline, we would first test the accuracy of the segmentation model. So far we have inputted around 5-6 example images from the bird’s eye view and analyzed how well it was able to segment free space and obstacles. It has done an accurate job so far, but once we are in the verification stage we would empirically test it on 10 different environments and make sure the accuracy is above 95%. For latency of the pipeline on segmentation, we already know it will be approximately 30 seconds. Thus, we would have to slightly adjust our use case requirements to a fixed environment but still monitor the user in real time. The rest of the pipeline has minimal latency (downsampling algorithm, A* path finding algorithm, socket communication between Raspberry Pi and laptop, etc), so the segmentation model is the only bottleneck. We would run it in different environments and note the latency to verify our use case requirements. For correct navigation of the user to the destination, the main concern is the overlay between the coordinate system given by the UWB sensors and the coordinate system given by the segmentation model. Getting the haptics to vibrate in the right direction should not be a huge issue, but the main issue is most likely having the person be at the correct location indicated by the UWB sensors. We can measure this empirically by noting the error between where the person is located and where our system thinks he / she is located. This metric could help guide us to the navigation accuracy metric.

The most significant risk that can jeopardize the success of the project is the overlay between the coordinate plane given by the camera and the coordinate plane given by the UWB sensors. Right now, we have an occupancy matrix of all the obstacles in the indoor environment. We got the basic functionality of the UWB sensors working but to localize the person with the UWB tag in the same frame we would need to overlay the two coordinate planes on top of each other. Since the occupancy matrix is downsampled by an arbitrary scale, we would need to calibrate the UWB sensors so the user’s movements are accurately reflected on the occupancy matrix. We believe this is the next main complexity of our project that we would need to tackle. With 4 UWB sensors (including one on the person), we would be able to localize the person in 3D space. Since our occupancy matrix is 2D, we would need some to use some geometry to match the movements of the person onto the grid. Since the camera used is a 120 degree FOV, we would also need to account for some warping effects on the sides. Once we have all 4 UWB sensors, we can definitely try to test that out.

Another risk that could jeopardize the success of the project is the compute power of the Jetson. Our current software pipeline is already taking a few minutes to run on the laptop, and we still have the UWB positioning left. Once we complete our pipeline, we were planning to move the processing to the Jetson. If the Jetson is unable to process this in a reasonable time, we may have to consider a more powerful Jetson or doing our processing on the laptop.

Some minor changes to the design of the system is that the destination selection will be fully configurable by the user now. When setting up the environment, the user will have the option to configure what locations they would like to request navigation to in the future. This will be set up with a UI that a caregiver can use.

The most significant risk that can jeopardize the success of this project is the UWB sensors not working as well as we expected. Our software pipeline is decently robust at this point, however, we are completely relying on the UWB sensors embedded on the user’s belt to determine the user’s location and update it on the D* Path Planning algorithm. The frame captured from the phone camera and the frame calculated from the UWB sensors might also have different dimensions, so the movements of the person may not completely align on the occupancy matrix we run path finding on. This misalignment could cause significant drift as the user moves and makes it difficult to guide him to the target.

These risks are being mitigated by having some of our team members look into libraries that could decrease the fish eye effect on the ends of the image. Since we needed wide lens camera to capture the entire frame, warping on the ends is something that we need to work with. Since our UWB sensors most likely would work uniformly throughout, it would probably be easiest to decrease warping from the CV pipeline.

Another risk is the compass orientation we receive and how we are going to integrate that with the D* path finding algorithm. Since these components take some time to arrive, we can currently work on the software stack right now. However, we are looking into drivers and libraries that could run these hardware components.

There are no changes to our schedule. This week, our main milestones were getting some sort of camera to work and send feed to the computer processing the data. We were also able to select a segmentation model that could classify free spaces and obstacles. Our D* Path Planning algorithm is mostly working, and we will focus on integration next week.

The most significant risks that can jeopardize the success of this project is the hardware stack. Since the Jetson is a budget computer, it has many compatibility issues with modules that were even created to be used with it. For example, the WaveShare IMX219-83 Stereo Camera was created to be used with the Jetson, but just for it to be detected on the Jetson I had to download multiple drivers (until I found the right spec) and configured port settings on the Jetson that were not explained on most tutorials. Since many of the hardware components are not widely used, there is a very small community of engineers who have used these products and could share their ideas. Thus, the tutorials online don’t usually capture the complexity of setting this up. Setting up the Jetson and the camera modules were supposed to be done within a few weeks, but we are already taking longer than that (due to Jetson being broken and now camera compatibility issues). We are managing these risks by having Charles focus on the software stack in parallel while Kevin and Talay make sure the hardware stack is ready. We are also using widely available tools for the software stack so that our programs are easy to debug and there is a programming community out there.

No changes were made to the existing design. One small change is that we are going to utilize more UWB sensors that require less setup as described in Kevin’s status report.

The most significant part risk that could jeopardize the success of the project is the Jetson being able to handle all the CV libraries and processing for our project. We designed our block diagram such that the Jetson is the center node. Thus, the camera inputs will come to the Jetson and the Jetson would have to run 3 CV libraries on it. From there, it would have to take in data from UWB sensors to create directional haptic feedback and send it to the haptic sensors. A lot of processing and data interconnect is on the Jetson, so it is crucial that the Jetson can handle all this seamlessly. Right now we are having trouble booting up the Jetson even with many different OS images found online, so we are falling slightly behind on this task. If the Jetson doesn’t end up working as well as we expected, we might have to shift our processing to be done on a laptop instead. A laptop will definitely be able to handle all the libraries and tasks we intend to use.

Since last week, no changes were currently made to our design. However, depending on how the Jetson experiments go, we might have to pivot our central processing unit to the laptop. This is a relatively minor system change though, since we simply have to send data from the camera to the laptop, but all other implementation details remain unchanged.

The most significant risks that could jeopardize the success of the project is pivoting the implementation details of our project without having talked to our professor and TA yet. We are deciding to mount the camera on the ceiling instead of the person, so that it captures both the user and the target in one frame and provides a better map for us to work from. We believe this is more feasible compared to our original idea of mounting the camera on the person. We will talk to both our professor and TA on Monday for their inputs on whether this seems like a feasible implementation.

Another potential risk is that by moving the camera from the person to the ceiling, we are solely relying on computer vision for navigation. Previously, when the camera was mounted on the person, we needed accelerometers and potentiometers to track the displacement of the person as they walked around the room. Now, since both objects are in the camera’s vision, we would need to find a library that can map the environment of the room in 2D space. From that output, we need to make sure that we can run path planning on it and send haptic navigational directions to the user wearing the belt. Since none of us have worked with computer vision extensively, this could become a risk later on in the project. We are managing these risks by considering alternatives if the camera does not work out, for example using LIDAR instead. Since LIDAR sends out a point cloud, the warping effect from a camera may not be as big of a problem.

The most significant risks that can jeopardize the success of this project largely involve the precision and accuracy of our sensors. In particular, we anticipate having to map out the distance of an object when navigating the user, in particular when the object is out of the camera’s field of vision. This requires us to estimate the location of the object relative to the user’s position, which can lead to many significant complications. We are concerned what happens if we arrive at the wrong destination, or if the sensors are able to locate the user even with erratic/unpredictable movement.

To address this risk, we plan to have an update mechanism, where the location of the object is dynamically updated as the user approaches the destination. If the device loses sight of the object, we plan on mapping the last seen location of the target destination. Moreover, more research on hardware components is necessary. We would like accelerometers/potentiometers that emphasize precision, since the speed/distance traveled by our user will be fairly small.

There were no major changes to the existing design of the system, but we plan to add the new navigation functionality. We anticipate this being fairly costly, and more complex than the original concept. In particular, it requires the use of multiple cameras capable of depth perception, as well as an accelerometer/linear potentiometer to track the movement/displacement of the user. Furthermore, this would require more performance from our processing unit, as well as more power consumption. Adding more hardware seems inevitable, but one thing we are trying to keep in mind is that this device is a wearable; selecting compact/low profile components will be important for maintaining the wearability for the user. Furthermore, power consumption can be reduced by keeping unused components idle, as well as using devices for multiple functionalities. For example, the same camera can be used for both obstacle detection as well as navigation.