Weekly Status Reports

Accomplishment:

• Implemented a python program that tests the OR model + DE feature against test images. It retrieves the closest object detected in the image and verifies accuracy by the respective image filename. As an example, if the filename is “person_test5.jpg”, the actual closest object is a person in the image. In the program, it filters out “person” from the filename and compares it with the detected closest object. 
• The program was run against chair (8), couch (6), person (5) images. The result came out as 100% accurate.
• Started working on deploying the OR module to Jetson. I transferred python files and reference images from my computer to Jetson.  

Progress:

I failed to meet the schedule due to the system setting of Nvidia Jetson. Importing the torch module on Jetson is taking more time than expected due to unexpected errors, so the schedule is postponed for a few days. Installing appropriate modules to Jetson is the critical component of the project, so I will make this as the highest priority and attempt to resolve the issue as fast as possible. 

Projected Deliverables:

By next week, I will finish deploying the OR model to Jetson, so that we can start testing the interaction between several subsystems. 

Now that you have some portions of your project built, and entering into the verification and validation phase of your project, provide a comprehensive update on what tests you have run or are planning to run. In particular, how will you analyze the anticipated measured results to verify your contribution to the project meets the engineering design requirements or the use case requirements?

I have implemented a python program that tests the OR model + DE feature against test images. It retrieves the closest object detected in the image and verifies accuracy by the respective image filename. As an example, if the filename is “person_test5.jpg”, the actual closest object is a person in the image. In the program, it filters out “person” from the filename and compares it with the detected closest object. The program was run against chair (8), couch (6), person (5) images. The result came out as 100% accurate, which is far greater than the use case requirement of 70% accuracy. If time permits, I am planning to include more indoor objects, so that the model can cover a wider range of objects while maintaining high accuracy. 

After the deployment of the OR model to Jetson, I am planning to use the same test file to run a testing on images taken from the Jetson camera and produce an accuracy report. In this case, since we are sending the images to the model in real time from the Jetson, we would not be able to rename the file in the format of the actual object. Therefore, I will instead use live outputs of detected closest objects from the Jetson and manually check whether the detection is accurate.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

• This week we connected the camera module to the Jetson and captured a few images. The camera lens causes a slight distortion to the image, and the images are lower resolution compared to the laptop camera we have been using to test the OR model. A risk associated with this is that we may experience lower accuracy of the model, and we may have to mitigate this by processing the images before sending them to the OR model.
• The accuracy of the DE of a detected object is a risk that we are currently facing. Although we can successfully determine which object is closer to the camera, the numerical value of the distance in meters is inaccurate. This is due to the difference in the width of the chair in the lab and from the reference image. This inaccuracy does not impact the output of the model as much, but it is an undeniable factor to the accuracy of the DE feature. We are planning to mitigate this risk by taking the reference images of the objects that we will be using for the test environment. In this way, we are able to make the width of the respective indoor objects the same (i.e. all chairs have the same width, all sofas have the same width, etc.). 

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There was one change from the design of the system. The OR model will use Yolov5 instead of Yolov4. This change is to increase the accuracy and improve the data latency of the model. Although it took more development time than necessary to program the DE feature, the result will have a better yield. 

Provide an updated schedule if changes have occurred.

Few more relevant objects will be added to the DE feature. The deployment of the OR model on Jetson will begin next week. 

Here is our update Gantt Chart as of Interim Demo (4/1/2024):

 

Gantt Chart – Timeline 1

 

 

Accomplishment: 

For this week, I have successfully implemented the Yolov5 OR model + DE feature. I used classes for the easier extraction of the reference images and filtered the detected objects so that it only outputs several indoor objects, such as a couch, person, mobile phone, and chair. I took several reference images from my laptop camera from a known distance and compared them with the images from online pairwise to determine whether the OR model successfully recognizes specific indoor objects and outputs relative distance from one another (which object is closer to the camera). After several instances of successful output, I used the image captured from the Jetson camera and ran it in my model. The following has the image taken and the output from the model. 

As shown in the image, it successfully outputs several chairs and their distances from the camera. However, although the order of distance from the camera makes sense, the numerical value of the estimation is too high. This is because the chair from the reference image has a different width from the chair from the test image. To resolve this problem, I am planning on taking the reference images of the objects that will be used in the test environment to increase the precision of the DE feature. 

Progress

I have successfully added relevant objects (coach, chair) to the DE feature and had some testing done. However, it is also important to test the OR model with more images from the Jetson camera to ensure the accuracy. I will need to do more testing with images and videos taken from a Jetson camera. 

Projected Deliverables

For next week, I will finish deploying the OR model to Jetson. At the same time, I will include more relevant objects, such as a table, to ensure sufficient range of indoor objects. The test results with Jetson camera will be documented for the final report.

Accomplishment: 

I have worked on trying to integrate a DE feature to Yolov9, but it was unsuccessful. I have noticed that the recent versions of the OR model restrict our abilities to manipulate with the detected output, which prevents us from using the data to detect the distance. 

Therefore, I decided to step back to Yolov4 with the DE feature considering the time constraint on the project. The current open source library only has the DE feature for a person and a mobile phone. To test my understanding of the source, I have added several reference images and measured the distance of the width of a sofa to detect the distance. As shown in the screenshot of the running model, the distance is shown along with the detected object. 

The accuracy of the distance will be improved by taking a reference image from a constant known value, which will be the distance between the object and the camera. 

Progress

Because we are stepping back to Yolov4 with the DE feature, I am catching up with the progress. I still need to add reference images of several more indoor objects to test out the DE feature. 

Projected Deliverables

By next week, I will add bench, cat, dog, backpack, handbag, suitcase, bottle, wine glass, cup, fork, knife, spoon, bowl, chair, sofa, potted plant, bed, dining table, tv monitor, laptop, mouse, keyboard, cell phone, microwave, oven, toaster, sink, refrigerator, book, scissors to the DE feature. I will also finish up the testing and begin to work on deploying the model to Nvidia Jetson. It will be a long week to prepare for the interim demo and ensure that the model runs in the Jetson well.