Category: Riley’s Status Reports

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I assisted Allie in debugging the color model when running tflite. I wrote a script that classified 80 subsequent images and easily collated the results to view to quickly determine if any improvements were made with one model over the other. Unfortunately, we have not yet found a color model that functions properly on the Jetson, but the script was invaluable for testing and provided much of the timing testing data seen in the group status report. Additionally, I attempted testing with saved models and keras models, but the machine ran out of memory or simply crashed when that was attempted. 

As an alternative, I attempted to write a color detection algorithm based off the dominant objects, and I managed to detect the outlines of the clothing, but the module I used didn’t consistently grab the whole piece of clothing, but often only grabbed parts of it and parts of the background, making the average inaccurate. I was unable to find time to properly debug this this week with the chaos of the final week of classes, but we are confident that there is an alternative method available for us if the model remains stubborn in its unwillingness to function with the Jetson’s version of tensorflow. 

I also implemented a pushbutton and learnt about GPIO for the Jetson using their in-built Python GPIO API Jetson.GPIO. This took longer than expected since I had to learn that the Jetson only accepts 3.3V input for its pins and does not consistently detect a rising/falling edge if from above that value. I also had to determine that since internal pull up resistors are not supported by our edge device, I would have to implement one and add it to the small circuit. Once I made those adjustments, Falling edges were detected consistently and I was able to integrate it to be the trigger for a picture. 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

My progress is on schedule.

What deliverables do you hope to complete in the next week?

I hope to help implement the color detection algorithm/test models to check for improvement. 

I also hope to make a stand/holder for the camera to allow for more intuitive use of the camera. 

Along with those items, I intend to make the pushbutton circuit smaller, explore options to miniaturize the circuit, and complete our final week of tasks. 

 What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

This week I fixed the camera bug that was preventing the openCV from displaying the images properly. Additionally I scaled the image to a level appropriate for the model and can continuously detect keyboard input to take pictures 4 seconds after the input was triggered. The issue with the openCV ended up being the fact that I needed to explicitly state what frame rate and resolution I was using, along with needing to add a delay for CV to display the image. 

Additionally, on Saturday I worked with Gabi to set up the Xavier NX with the database and API. We configured the hardware, finalized endpoints to send and store data and connected the server and the embedded computer. Currently I have gotten the device to send some dummy information to the backend server, which then can send a sql query to add the items to the database.

Below are images of my successful POST request via fastAPI.  The left image is showing the testing arguments I passed in successfully mirrored as a method to test the endpoint. The right image shows the fastAPI success, which is a little green OK at the bottom. If you see a couple of red failures above it, those issues were fixed 🙂

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

Progress is on schedule and nearly ready for the demo. Some more testing will be done between now and Wednesday but no new major features need to be added until the demo.

What deliverables do you hope to complete in the next week?

Next week I hope to set up a free trial with Amazon Educate and through that gain access to S3 file storage. With that we should have more capabilities in sending and storing images to display as a visual for the user. Additionally I hope to be able to detect the physical button input and use that as a trigger to take a picture. 

Verification Tests:

I tested the functionality of the camera by testing if the camera could display consistently between different boot ups and with the different frame rates and resolutions available. With the GUI the camera was able to consistently display and didn’t experience any issues in the testing. 

Additionally, I tested the cv code by running fifty different image capture requests and ensuring that each time, the event triggered as expected and displayed the image taken by openCV. This action worked 100% of the time. Further testing will be taken when the model is included in order to test if the results can be generated within 3 seconds for a sample dataset of fifty images taken by the camera. However since not all the models are ready and on the Jetson, this requirement will be tested in the coming weeks. 

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

This week I coded up an endpoint to send the data to the database and added the required modules on the Xavier NX to achieve this (fastapi, config, msql). Additionally I tested out the code to take pictures only after certain user input. The user input was tracked correctly, but something went wrong with opencv where I got an error message “cannot query video position: status=0, value=-1, duration=-1,”. 

I ensured that the camera worked as expected using the qv4l2 gui. And I did have to fix an issue with it only showing a green screen, but I got the camera to work again. 

Even with the working camera the cv library wasn’t able to take a picture using the opened camera. This is an issue that I will be heavily troubleshooting next week as it is essential and not a hardware issue. 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

Slightly behind schedule due to the camera not being able to take pictures but assuming that works as expected next week when I put more time into it, I will catch up.

What deliverables do you hope to complete in the next week?

Fix the camera issues, mock up a breadboard and connections to detect input on the pushbutton, and meet with Gabi to test sending data to the database from the XavierNX.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

This week I created the framework to take constant images and put them into the model. Unfortunately, due to other courses picking up in this time period for me, I was unable to complete more than that, specifically referring to my desire to link up the image and model results with the backend API. 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

On Schedule still, I will work more next week Friday and Saturday after my work clears up.

What deliverables do you hope to complete in the next week?

Next week I hope to, with the help of Gabi, be able to link up the Jetson with the backend through the use of an API. Additionally, I would like to test inputting an image into a default model to ensure the acceptance of the cv image type. 

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

In summary, my progress this week was installing VS Code and connecting Github on the Xavier NX, connecting the camera to the Xavier NX, and taking a picture using the camera using opencv. A lot more work went into this than the simple description allows for. Firstly, in the process of installing VS Code on the embedded device, I followed the instructions from VS Code after downloading the arm64 version. I originally downloaded the amd64 version, but that obviously didn’t work on the Xavier NX which uses arm. After I downloaded the deb file and followed the instructions I still ran into issues with the window not opening after clicking on the icon or typing code into the terminal. I tried debugging for a little while, and none of the most available resources (https://askubuntu.com/questions/1410992/vscode-not-opening-on-arm64-ubuntu-20-04 and https://askubuntu.com/questions/1022923/cannot-open-visual-studio-code) were unable to help. Eventually I stumbled upon a website that recommended running code —no-sandbox, which worked. To the best of my knowledge this occurs because in certain linux or other environments, launching the Chromium sandbox is impossible, so we needed to disable the feature and lose some safety features.  

After that, I connected the camera to the Xavier NX, and followed these Arducam instructions, and it worked very well. I was able to set up, view and change the camera settings with the gui installed in the instructions.

After connecting the camera, the next step was to use opencv to take a picture from the camera. This was the most frustrating part to debug since despite the fact that opencv was downloaded, it kept saying the module was not installed. I tried to debug on my own for a little while, trying the typical google searches and websites but eventually I stalled out and pivoted to other work for a little while. Eventually Allie and Gabi were able to help debug and Allie found that we needed to put the python path to the modules in bashrc as it hadn’t been done. Once we followed this it worked and we were able to run openCV. This process took 3-4 hours of debugging and should have worked far earlier than in practice but my guess is that some part of our environment might have made it so the installation didn’t “stick” correctly until we put the path in bashrc.

The actual openCV script that we used to take a still image can be found in many different places but I found it from a youtube video. I had to make a small modification to make it work with our camera, but the rest of the code is from the first demo of the video. Below is a picture of the code, with a couple extraneous print statements, and an image that was taken by our camera. The image looks slightly blurry due to the time it took to process and an unsteady hand. The time taken is something that we might have to reduce further, however I think this should well be possible and it’s just something from opencv that made the shutter window longer than it needed to be. After this I connected our github with the Xavier NX and pushed the code. 

OpenCV Starter Code

Resulting Image

If you want higher quality images please slack me! For some reason the process of moving it into the website really lowering the quality. I have higher quality versions that can I can show if desired!

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

On Schedule

What deliverables do you hope to complete in the next week?

Next week I want to make the openCV continuously send images using a script. Once this is done with a low enough time interval to meet our timing constraints, I will then try to move the models to the Xavier NX and run some of these images through the model or a sample model to ensure that the formatting of the images is as expected on both sides.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

The past two weeks, since the last status report I have primarily accomplished two things. The first, and more time consuming of which was the writing, editing and integration of the design report. This took a good amount of time to write, even though we had gone over all the information in various forms. Expanding on said information and collating it in a report format was not a trivial task for our team. Past that, this week over break I have received news that the camera has arrived, and in response I have begun the preparation to understand what format the camera will give and how to connect it while getting information to and from the Xavier NX. (might need to do research here). Specifically, I have looked at several OpenCV tutorials, and determined which image format the video will arrive in. With that information, it will be possible to lower the learning curve that comes with implementing and integrating the camera. Additionally, I have downloaded the frontend repo and have gotten acquainted with the structure.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

My progress is on schedule, the hardware components took some time to arrive but not so much that there is any concern yet. Additionally. I have been able to do some work to speed up this integration as well in preparation. 

What deliverables do you hope to complete in the next week?

Next week I intend to receive the camera and detect images from it and store them in an openCV format, all on the Xavier NX. Additionally, if time allows I intend to download our model or a test model onto the Xavier NX, attempt to run it and send information to our API or simply collate it into a format to do just that without actually sending it depending on my and my teammates progress.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours.

This week I received a couple of items, the LED light to get better consistency when scanning clothing, and more importantly, the Nvidia Jetson Xavier NX. One thing that I found was that to properly boot up the Xavier NX, a couple of additional materials were needed. A USB-Keyboard, HDMI Cable, a spare Monitor, and a 16GB SD card. The first 3 additional materials are easily acquired as we can make use of the computer labs along with our own supplies. However, we did not have an SD card capable of flashing the Xavier NX, so I promptly ordered one. We are still waiting for this SD card and the camera, so I was unable to start the physical boot up process. In the meantime, I worked on the design report and presentation, as it is upcoming. Additionally, I did a fair bit of research on the Xavier NX to try and determine what else I would need so I would be caught off guard again. Using resources such as the starting guide, the Jetpack SDK, the User Guide, the Developer Guide, and many more, I determined that an Ubuntu 22.04 Linux environment would lead to the least complications when using OpenCV and other modules on the Xavier NX. To do this, I downloaded VMWare and created a workstation with the requirements above. I chose not to use WSL because of the complications that can and have arisen when dealing with peripherals on my computer with WSL. Below is a small picture of the workstation. 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the proiect schedule?

My progress is slightly behind, I hoped to be able to flash and boot up the Xavier NX this week, but I misjudged which materials were needed and am now waiting on the SD card to arrive. It was ordered with Amazon Prime so the delay should not be overwhelming, and as soon as I get that item, I will start the process next week. To catch up to the project schedule, next week I will also work on getting the required modules and development environment of the Xavier NX set up, beyond simply booting it up.

What deliverables do you hope to complete in the next week?

I hope to turn on the Xavier NX, download the required software and modules onto it, and run basic programs using the Xavier NX. I believe this should be possible but time consuming given the intricacies and unexpected issues that often crop up in installation and set-up.

Status Report: Slightly Behind but manageable