Author: mgoli

Most of my work this week was in trying out further model training configurations to account for some false positives we discovered in our design.

On Sunday,  I worked on adding details of our old approach and restructured approach, our computer vision training accuracy – the complete solution and the final gantt chart for up until the last week. From Monday to Wednesday I was busy with exams and deadlines for some other classes, so I instead put some time into figuring out what tasks we had left as a group and assigning overall priorities to tasks during this time. Work between these few days took about 4-5 hours to compile everything together.

From Thursday onward, we noted that the model was giving some high confidence false positives that the sample thresholding couldn’t account for. Specifically, some occurrences of backpacks were detected as people with >60% confidence or the TV in the room came up as a dining table with >60% confidence as well.

This was when I realized our custom dataset didn’t have enough background information, or samples of objects not to look for in the images – such as TV, or phones etc. Aditi suggested using the Pascal VOC dataset to add to our training, adding in some needed background information. So from Thursday onward, I worked on retraining the model with various combinations of the VOC dataset and our custom dataset. To prep, I wrote a script to eliminate images without tables/chairs/people from the VOC dataset, drop instances of classes we weren’t looking for and to switch the class mapping to our custom class mapping. With the larger dataset I decided to switch over to a larger AWS training instance (P2.xlarge -> P3.2xlarge). I encountered some problems with training as my Allocated CPU limit on AWS didn’t account for the larger instance. Unfortunately, I found this out in the middle of training with my connection closing out and I was unable to restart any instances to save any data. i immediately sent in a new CPU limit increase request to AWS but most of my time during Thursday/Friday was thus spent trying to save all the training data that was stored across the two instances I had (solved using an S3 bucket). In total, I spent about 8-10 hours from cleaning the Pascal VOC dataset to running training on Pascal, to dealing with the AWS server issue.

Another thing we noted with regards to training was how switching class mappings and the overall output layer requires some retraining from the model to remap to the new mapping scheme while also training for higher accuracy in the target classes. Because the classes we are targeting are already part of YOLO’s pretained class mappings (COCO labelling), training using those existing mappings and our dataset will help with the overall accuracy of the model (specifically lowering the rate of false positives as those items now have a specified class). I spent up to 7-10 hours on Saturday reprepping our data to work with the COCO labelling scheme. This involved redownloading our re-labelled custom dataset, running my random data augmentation script to increase the dataset size, writing a new script to switch  to the COCO mappings and running the dataset through the remapping script. On the VOC side as well, I rewrote the VOC remapping script to remap to COCO labellings. From here, I backed up the datasets to an S3 bucket as well.

On Saturday as well, there was an issue trying to use my spot instances to train/move data. Spot capacity works with leftover available CPUs, and no spot capacity was available in the us area regions where my instances were.  I thus also spent time Saturday working with Google Colab to set up a notebook to move our training to. By the night however,  there was spot capacity available so I was able to start training.

I tested 4 configuration of training to start off with,  to test how the model would train in terms of accuracy improvement:

• Custom for 50 Epochs w/Backbone (Layers 1-10) Frozen
• Custom for 50 Epochs w/Backbone (Layers 1-10) Frozen + Custom for 10 Epochs w/Layer 1-22 Frozen
• Pascal VOC for 50 Epochs w/Backbone (Layers 1-10) Frozen
• Pascal VOC for 50 Epochs w/Backbone (Layers 1-10) Frozen + Custom for 10 Epochs w/Layer 1-22 Frozen

Training is still ongoing, but so far the Custom Training using COCO labelling for 50 epochs with just the backbone frozen seems to have good performance.

Moving into the final demo and onward, I will be training the rest of these configurations and working on compiling my training data for the final report. In addition, working with my team to test remaining metrics and work on the final report/video.

This past week the majority of my time was spent cleaning uo our custom dataset, further training the model and tying together the computer vision pipeline as an independently running module.

The past few weeks before I had tried to retrain the YOLO model using transfer learning with various existing image datasets. Looking between ImageNet and OpenImages and running a few test runs on the existing dataset, i found that existing datasets actually didn’t prove to be extremely helpful in training.

Both datasets have many more classes than those we need to train for. While it is possible to run a simple script to relabel the training data only for our target classes,  the image sets themselves are also too varied to help with training for our specific use cases. As the models are already pertained, training on a custom dataset for our use case is what will help with further increasing accuracy and the confidence level of the system’s detections.

One other thing that came of note was that YOLO was initially pertained with 80 classes instead of our target 3, the change in the output dimensions is also something extra training will need to count for. One consideration was to maybe continue training with the 80 class scheme and to add an extra output layer to only consider the results of the target three classes. However, I also noted that this introduced more overhead in creating the custom dataset – as instances of all 80 classes will need to be labeled for training. So I determined it was best to use transfer learning with only our custom dataset with just the 3 classes.

After collecting the data as a group, I went through and labeled the data for training purposes using roboflow – a cv platform with functionality to label data. Our initial dataset was only around ~40 images, so as per TA suggestion – I looked into Data Augmentation to introduce noise, contrast change (etc) to artificially produce more data. One issue that came up was looking into how the bounding box detection txt files in the training could be augmented as well. I found a Github codebase https://github.com/Paperspace/DataAugmentationForObjectDetection

with functionality to augment the images and the txt detections. Writing a script to augment our data, I used the codebases augmentation features to artificially increase the dataset to 264 images.

For the training itself, I noted that he YOLO architecture has a 10-layer base a s backbone for feature detection and an additional 13-layer head for object detection and classification (in the same step).  So for training, I tested freezing just the backbone  and also leaving the last 1/2/3 layers unfrozen in training. Finding that  training largely stalled at a precision (portion of detections that were true positives) of about 0.5 on most rounds.

This past week specifically, I committed this by removing our extraneous backpack class to try to we had initially put in to account for extra cases of people temporarily leaving a room (a backpack would then be used to indicate that the seat was occupied). The backpacks were easily conflated with some of the chairs and we were choosing to let go of this extra case in rescoping – so I removed the class. One other change in the dataset was limiting the data augmentation to in place changes (removing any changes that messed with scale/shear/tranlating the image data). With that I removed the backpack class and ran the dat augmentation script again to have the 264 training samples.

Training from there, I was able to achieve >0.90 precision with training just the backbone.  This was all that I had worked on until Wednesday specifically, From there, I was mainly writing code to tie the CV module components into standalone module that could run on its own. This took about another 3-4 hours to write out and debug fully.

This past week most of my work has been in integration and environment setup for training. I caught the flu earlier in the week and wasn’t able to work until about Thursday, so training has been pushed back by about a week now. But I am working on finishing environment setup and training will be done this week.  In terms of integration – I spent time writing code to connect the camera, preprocessing and model together. I spent another 5-7 hours on my own rewriting and tweaking the code to integrate seamlessly with the web app. In this next week, I will need to work more this week to get us back on track for model training.

Overall, as a team this week – we mainly each worked individually on our assigned sections.  As team, we worked on resplitting the computer vision workload to make sure everyone had an equal contribution. Namely,  all members will now work on data collection and labeling, Aditi will additionally work on post processing (sample combination/averaging) and will be in charge of the overall code structure to integrate all portions, Mehar will work on preprocessing and model training. This week, we also went through  the design review feedback and made notes to work on testing the post processing plan we had noted in the paper, and that we will need to expand our testing for any unintended factors such as lighting changes.

This week, I continued to implement the preprocessing section of the computer vision pipeline. The three main goals for the preprocessing are to denoise and increase the contrast and brightness of the the input images. Initially, I used the builtin denoising methods included in the OpenCV library and a simple scaling function to work on contrast and brightness. I found that simply scaling up the image RGB values for contrast and brightness was very rudimentary. Many areas in the image were at risk of overexposure, and the method leaves little room for the preprocessing to adapt for each image. Researching further,  another contrast/brightness adjustment method I found was the clahe method which involves normalization of the image values. This sort of system is adaptive for each image and I found it was much better for increasing contrast without overexposing portions of the image.

This week, overall – I wasn’t able to work as much due to assignments in other classes. So the goals for next week include finishing up preprocessing and working model training.

 

The week before break, most of my time was spent working on the design report – I researched further into feature descriptor-based object detection and did compiled my benchmark results for the report. In the report, I was responsible for the computer vision trade studies and design implementation sections, the introduction, schedule and team member responsibility sections. This overall was about 9+ hours of work.

Beyond this, during the break I started implementing the preprocessing and overall code structure of our project to integrate the preprocessing/model/postprocessing sections.  This took about 3-4 hours to research and formulate overall.

My work was focused around pulling together the final presentation components and preparing for the proposal presentation, along with researching and planning the computer vision component of our project

My biggest accomplishment this week was the presentation, I did some reformatting through Sunday night along with rehearsing through Monday and Tuesday. I specifically added in some logos (Scotty dog, and ‘scottyseat’) to our slides and worked some more on our implementation slides

In terms of the overall project itself, my tasks included finding a study room to test in, deciding the final model for the computer vision component and studying OpenCV. As far as the study room, there was no need to look for one as we decided on the ECE lounge study rooms together.

I spent a bit of time going over old OpenCV notes I had and looking through the GitHub depositary for the current object detection model we are looking at, Faster R-CNN. The bulk of my time however was spent trying to rescope our computer vision segment and my portion of the Gantt chart.

With the feedback from the proposal, there were many new specifications to be met so I spent quite a bit of time to figure out what steps I need to take in the next few months to meet MVP.

With the rescoping process, I am a slightly behind on what I planned to study of OpenCV this week but I also now have a better idea of what I can focus on learning for the next week.

In the next week I hope to continue researching OpenCV to have the initial design of the computer vision backend, along with an initial version of the computer vision model up and running. This initial version would be implementing the existing pretrained model.