Posted on

Maya’s Status Report 3/29

Accomplishments this week:
This week, I integrated the haptics with the Jetson, cut the wood and materials for our cane, and helped Cynthia work out specific cases for when objects should be detected and to trigger the proper haptic signal.

Reflection on schedule:
We had a lot of progress as a group this week and we were able to get many big steps put together. We are in a great spot for the interim demo and are in a great spot in terms of our schedule.

Plans for next week:
Over the next week, I will be working on testing each of the elements for power and current to make sure they are all going to be safe before plugging them in to the portable charger. I will also work to set up the pressure pads if time allows this week, but it is less important than the power consumption.

Posted on

Team Status Report 3/29

Risks:
The only risk is that sometimes the depth stream from the LiDAR camera gives us the wrong data/doesn’t work in random holes of the frame, so it tells us objects are 0 meters away which will likely lower our accuracy but hopefully won’t end up interfering with too much.

Changes:
We changed our plan back to the original plan of using pyrealsense2.

Posted on

Cynthia’s Status Report 3/29

Accomplishments this week:
Finished debugging and integrated my fine-tuned YOLOv8 model that now classifies stairs along with objects (image included below — note that our Jetson has to be plugged in now, so we couldn’t bring it to actual stairs but I emulated stairs with the cart in the image).  Helped Kaya with fixing pyrealsense a little, then once it worked I used his code that gets distance from grid points to make the code that runs the model and creates bounding boxes now get the distance to the center points of the objects too.  I also worked with Maya to get the haptics working from the Jetson.  Lastly, I wrote the code to integrate/trigger the haptics and decide what action to suggest with the object detection model, based on objects detected, their location, and their distance — this currently works but the suggested actions are not all correct yet.

Reflection on schedule:
I believe I did a lot this week and caught up to what we planned.

Plans for next week:
Work with Kaya and get the wall detection working.  Fix the recommended action decision making code.

Posted on

Kaya’s Status Report 3/29

Accomplishments this week:
I worked on reconfiguring the jetson jetpack so that we can get the correct pylibrealsense module. This involved reflashing our SD card, researching the compatible modules with our new Ubuntu v20.04, and reinstalling all of the compatible modules. After that, I wrote code for distance detection and was able to get distance detection working at 9 different points. Lastly, I was able to integrate that distance detection code with Cynthia’s CV code (see photo below for distance detection on 9 points integrated with the CV code).

Fig 1: Distance Detection on 9 points integrated with CV algorithm.

Reflection on schedule:
We did a lot this week with integrating distance, CV, and haptics so we are on track.

Plans for next week:
My plan is to work on wall detection with the distance code. Additionally, we plan on writing code for the force sensitive resistors and plan to start building our cane.

Posted on

Cynthia’s Status Report 3/22

Accomplishments this week:
I worked with Kaya after the correct versions of the libraries we needed were installed (after a lot of trouble and many hours spent on this) to get the pre-trained ML model working on the live RGB stream (see photos below).  I additionally had to change the code I previously wrote to work around what we decided does not work (pyrealsense2) on the Jetson, which we were depending on for its depth stream.  Recently, I started working with Kaya to get depth data and figure out how we can use that data in my python scripts instead of just getting it as a terminal command output.

Reflection on schedule:
I think I am slightly behind what our schedule was for my portion, but that is because we switched to using the camera on the Jetson earlier than planned since I am not able to use the library I need on the desktops and cannot use the camera on my laptop, so I was never able to test my code until one day ago and have not included stairs in our model yet.  Additionally, I was sick (and still am) and was able to work less than planned.  Overall, because of the rearranging, we are still on schedule as a group, but I need to continue to make good progress with our model moving forward.

Plans for next week:
Write code to train the model to incorporate a dataset of stairs and work further on getting distance measurements without pyrealsense2.

Posted on

Team Status Report 03/22

Risks:

The only risk we have is that the only way we can access the distance data is through a terminal command. We need to come up with a way to run that in parallel with the CV algorithm.

 

Changes:

We are no longer using pylibrealsense2 for our ML model and instead using open cv with a yolo model.

Posted on

Kaya’s Status Report 03/22

Accomplishments this week:
This week, I configured the proper CV libraries to analyze our Lidar camera data. I did this by setting up a virtual environment with the proper libraries (Python3.7, OpenCV version 4.11.0).  Additionally, I configured our Jetson to analyze the distance data and wrote a script for this analysis.

Reflection on schedule:

We are slightly ahead of schedule now. We did a lot this week (object detection, distance detection, haptics).

We dedicated a lot to capstone this week, especially with the configuring of the libraries. We had to research and reconfigure the python and pip libraries to match the needs of the Lidar camera. After numerous tries of source installing and pip installing pylibrealsense, we came to the conclusion that we can’t use this library pylibrealsense. After trying other things out, we came to the conclusion that cv2 is the best python library for analyzing the camera data. Additionally, we came up with a separate command for getting distance data.

Plans for next week:

Work with Cynthia on writing a script to connect the distance detection with the computer vision code. Additionally, I plan on working with Maya on configuring the force sensitive resistors.

Posted on

Maya’s Status Report 3/22

Accomplishments this week:
This week, I setup the haptics and created a few sample patterns that we will be using with our Jetson.  The haptic patterns for each obstacle type is demonstrated here.

Reflection on schedule:
We were a bit behind schedule at the beginning of the week because we had some problems with the Jetson and L515 compatibility, but we put in a lot of hours this week to recover from this. Personally, that included me helping Kaya with the Jetson and L515 connections, and I also setup the haptics and created case statements for when each haptic pattern is set off.

Plans for next week:
Over the next week, I will be working to run the haptics over the Jetson and hopefully begin to connect the haptic responses through the CV code, and I will begin testing the overall power consumption to make sure it is under 30W and 5V.

Posted on

Cynthia’s Status Report

Accomplishments this week: I worked with my team to get our RealSense SDK visualizer working with getting the RGB and depth streams on the Jetson and I wrote the code to get the same streams through a Python script instead of the visualizer.

Reflection on schedule: We are a little behind schedule for the software, but also because we are doing things a different order than on our schedule.  We ended up doing the integration for the LiDAR camera with the Jetson this week, which we got working, but the correct versions that we need to run our python code needs to be worked on further, since our visualizer works for the data stream on the Jetson but not the Python code to view the stream because of the libraries.

Plans for next week: Over the next week I will be catching up on my object detection code by writing the code to obtain bounding boxes and getting it to work on the Jetson Nano.

Posted on

Maya’s Status Report 3/15

Accomplishments this week:
This week we linked the Jetson and the L515 using the SDK viewer, which is pictured below, and we are continuing to set up the L515 on the Jetson with our current code instead of the viewer code.

Reflection on schedule:
We are on schedule for the most part, but we have a heavy workday on Sunday to finish our goals for this week, which includes starting to integrate the haptics with the Jetson.

Plans for next week:
Over the next week, we will be working on the haptic logic and making sure it integrates with the Jetson.