This was a very eventful week for me. I spent over 24 hours this week working on various aspects of the project.
On Monday, I gave our final presentation. Later in the week, I tested hosting both the web server and camera on the Jetson which worked as expected. Once I had completed that I tested our pipeline with live video footage. I found that our object detection algorithm was quite good at detecting chairs but was too sensitive and mistakenly marked a backpack as a chair. Keeping this in mind, I wrote a script to capture images with extra chairs to serve for test images for later. I was able to do this as I knew how many chairs I expected it to find. After this, I brainstormed how we might fix this issue. I looked into some inference parameters first and tried playing with the confidence threshold, NMS IOU threshold (which dictates how closes 2 bounding boxes can be before being considered as the same object) and lastly the model classes. Playing with the confidence threshold and NMS IOU did not make a significant impact but using model classes increased the confidence in chairs and people which was a helpful asset. That being said, the output classes for our custom trained model only looked for people, chairs and tables. I had a feeling that the algorithm was still identifying objects as we had frozen most of the earlier layers but since it only had 3 classification options it was trying to fit the object to one of the classes. With this in mind, I re-annotated our custom dataset to match all the classified objects in COCO including backpacks and laptops. I additionally tried to see if there were other dataset with less classes but had the classes we were looking for and found that possibly using PASCAL’s VOC dataset may be a better fit. I tried training with an increased number of classes with the relabelled data but my initial results gave undesirable results. I also wrote our sample thresholding algorithm.
Later on in the week, I tried integrating the 2 cameras. I got both cameras working with the pipeline separately but when I ran them at the same time the Jetson was taking a very long time to load the model. I used a resource called jetson stats to monitor the activity on the Jetson and found that the CPU’s and GPU’s were not being overloaded but the memory was maxing out. After reading a little bit more I figured I might have to switch from PyTorch inference to TensorRT or DeepStream. I spent 5+ hours on Saturday working on this but managed to get YOLOv5 working with TensorRT. I will need to port our code to work with TensorRT but I am quite proud of this accomplishment along with everything else I managed to get done this week.