Posted on

Talay’s Status Report for 3/8

This week I was able to successfully set up the NVIDIA Jetson Nano 2GB Dev Kit. From last week, we were having trouble setting up the Nano. We tried downloading many OS images from the website through various methods but were not able to get the Jetson booted up. We decided to reorder a new board (the same spec and version) and tested the new board with the OS image given on the Nvidia website. I was able to boot up the board and complete the first few steps to set up our account and sign the user agreements. I also looked into the Jetson connectivity with our computer to run different computer vision tasks. I consulted with my teammates and realized that the best option is to buy a separate wifi card to attach to the Jetson. We can download the CV frameworks and models (pre-trained) into the Jetson from our computer via ethernet or using the Nvidia SDK Manager, but our Jetson still needs to wirelessly connect to the Raspberry Pi 4 via bluetooth. I also tried to connect the stereo camera to the Jetson via CSI (Camera Serial Interface), but that isn’t working yet.

Our progress is slightly behind schedule as I should have the camera set up within this week. I will work to connect the stereo camera to the Jetson so that it can start reading in images from the processor. After the camera setup, we could start downloading pre-trained CV frameworks onto the Jetson. In parallel, we should also set up wifi connectivity from the Jetson to the Raspberry Pi 4.

In the next week, I will have the stereo camera connectivity complete. My teammates will be looking into the pre-trained model so that once the camera connectivity is done, we can start downloading models onto the board.

Posted on

Team Status Report for 2/22

The most significant part risk that could jeopardize the success of the project is the Jetson being able to handle all the CV libraries and processing for our project. We designed our block diagram such that the Jetson is the center node. Thus, the camera inputs will come to the Jetson and the Jetson would have to run 3 CV libraries on it. From there, it would have to take in data from UWB sensors to create directional haptic feedback and send it to the haptic sensors. A lot of processing and data interconnect is on the Jetson, so it is crucial that the Jetson can handle all this seamlessly. Right now we are having trouble booting up the Jetson even with many different OS images found online, so we are falling slightly behind on this task. If the Jetson doesn’t end up working as well as we expected, we might have to shift our processing to be done on a laptop instead. A laptop will definitely be able to handle all the libraries and tasks we intend to use.

Since last week, no changes were currently made to our design. However, depending on how the Jetson experiments go, we might have to pivot our central processing unit to the laptop. This is a relatively minor system change though, since we simply have to send data from the camera to the laptop, but all other implementation details remain unchanged.

Posted on

Talay’s Status Report for 2/22

This week, I tried to set up the Jetson Nano 2GB Dev Kit so we could start testing our camera and CV libraries on the board. We downloaded an OS image for this specific version onto an SD card and tried to boot up the Jetson. However, only the Nvidia logo showed up without the system booting up at all. We tried flashing other OS images onto the SD card with no luck, so we speculate that our Jetson board might either be broken or we have missed something while setting it up. We will have another team member try to look into this problem and troubleshoot it. If this doesn’t work, we would try to set up a different Jetson board. We also ordered some additional components this week, including the UWB sensors so that we could start testing them in parallel.

Our progress is slightly behind because setting up and booting the Jetson took longer than we expected, and we still don’t have it working. Despite using numerous OS installers (SDK Manager, balenaEtcher), we were still not able to get it to work. We would have to try to set up the Jetson as soon as possible in the next week and determine whether our camera is compatible with it. If not, we should try to find alternatives since the Jetson is the main processing unit of our entire project.

Next week we hope to boot up the Jetson and connect the camera to it. Hopefully we can start viewing images on the camera with the Jetson processing.

Posted on

Talay’s Status Report for 2/15

This week I drew out the block diagram and implementation details for our project. First, we decided to localize the person on the same frame of reference as the camera by using Ultra Wideband (UWB) instead of IMU to avoid drifting. I did some research on UWB models and came across DWM1000 which seems suitable for our use case. We put 2 in the order list so we can start testing them next week. With 2 anchors in the room, we figured out a way to locate the coordinates of the person. I also did some research on the stereo camera mounted on the wall. We decided on the stereo camera dual OV2311, since it provides depth perception and is also compatible with the Jetson. We will also set up the Jetson and start running certain libraries on the input images next week. We decided to use the OpenCV library to convert from stereo images to a 3D point cloud, and Open3D to convert from the 3D point cloud to a 2D occupancy matrix. We looked into these libraries and it seems appropriate for our use case. We also decided to use the D* path finding algorithm. This algorithm tracks updates continuously and gives future directions. With these high level implementation details ironed out, we also spent some time making the design presentation slides for next week. I believe our progress is on schedule.

Next week, we first plan on setting up the Jetson Nano. This in itself will take up some time. After we set it up, we would like to run the stereo camera on the Jetson and take some input images. We could try setting up the camera on the ceiling and see how it could classify images from the bird’s eye view. However, the priority is definitely setting up the Jetson and connecting it to the camera module.

Posted on

Team’s Status Report for 2/08

The most significant risks that could jeopardize the success of the project is pivoting the implementation details of our project without having talked to our professor and TA yet. We are deciding to mount the camera on the ceiling instead of the person, so that it captures both the user and the target in one frame and provides a better map for us to work from. We believe this is more feasible compared to our original idea of mounting the camera on the person. We will talk to both our professor and TA on Monday for their inputs on whether this seems like a feasible implementation.

Another potential risk is that by moving the camera from the person to the ceiling, we are solely relying on computer vision for navigation. Previously, when the camera was mounted on the person, we needed accelerometers and potentiometers to track the displacement of the person as they walked around the room. Now, since both objects are in the camera’s vision, we would need to find a library that can map the environment of the room in 2D space. From that output, we need to make sure that we can run path planning on it and send haptic navigational directions to the user wearing the belt. Since none of us have worked with computer vision extensively, this could become a risk later on in the project. We are managing these risks by considering alternatives if the camera does not work out, for example using LIDAR instead. Since LIDAR sends out a point cloud, the warping effect from a camera may not be as big of a problem.

Posted on

Talay’s Status Report for 2/08

This week, I did more research on the implementation details of our project. I proposed the idea to switch the camera from being mounted on the person to the ceiling instead. After digging into SLAM algorithms, we realized it may be difficult to mount a camera on a person and use that as a perception tool to map the person towards the target. Thus, we decided to potentially change the location of the camera to the ceiling instead, where both the user and the target is in view. From this single point of view, we can run a path planning algorithm to guide the person towards the target through haptic sensors. Our use case would be slightly shifted, where instead of having the blind person navigate unfamiliar indoor environments, they would be navigating inside an indoor setting that is blind-friendly. These environments could be their office or homes, where a one-time camera set-up including some pre-configurations would be necessary. After this, the blind person would wear a haptic vibrating belt that would guide them to their location with voice-assisted commands. We also tested an online image classification library with a top-down view image to see if there are existing libraries for our use case. The library we found was able to detect objects from a bird’s eye view quite well, so we believe this should be feasible.

Our progress is slightly behind schedule, since we have shifted the implementation details quite a bit. That was because we realized the original solution may not be feasible under the requirements given. To catch up to project schedule, we are planning to iron out specific implementation details of our project by next week. Once we know what toolchains, libraries, and technology stack we are using for each specific component, we can think about how they can all fit in together. This is our target for next week. We would also like to order all materials by early next week, so we can start working with each of them soon.

Posted on

Team Status Report for 2/01

The most significant risks that can jeopardize the success of this project largely involve the precision and accuracy of our sensors. In particular, we anticipate having to map out the distance of an object when navigating the user, in particular when the object is out of the camera’s field of vision. This requires us to estimate the location of the object relative to the user’s position, which can lead to many significant complications. We are concerned what happens if we arrive at the wrong destination, or if the sensors are able to locate the user even with erratic/unpredictable movement.

To address this risk, we plan to have an update mechanism, where the location of the object is dynamically updated as the user approaches the destination. If the device loses sight of the object, we plan on mapping the last seen location of the target destination. Moreover, more research on hardware components is necessary. We would like accelerometers/potentiometers that emphasize precision, since the speed/distance traveled by our user will be fairly small.

There were no major changes to the existing design of the system, but we plan to add the new navigation functionality. We anticipate this being fairly costly, and more complex than the original concept. In particular, it requires the use of multiple cameras capable of depth perception, as well as an accelerometer/linear potentiometer to track the movement/displacement of the user. Furthermore, this would require more performance from our processing unit, as well as more power consumption. Adding more hardware seems inevitable, but one thing we are trying to keep in mind is that this device is a wearable; selecting compact/low profile components will be important for maintaining the wearability for the user. Furthermore, power consumption can be reduced by keeping unused components idle, as well as using devices for multiple functionalities. For example, the same camera can be used for both obstacle detection as well as navigation.

Posted on

Talay’s Status Report for 2/01

This week I contributed to team efforts to iron out our project idea and come up with specific implementation details for it. We did some quick research to find out how feasible our idea was and if we could accomplish the MVP by the end of the semester. Our progress is on schedule as we have nailed down a lot of the specifics of our project scope and implementation. During the next week, we hope to come up with a list of materials needed and start budgeting our cost. We would also like to look into what existing technologies there are for some of the features in our project.