Summary
I spent this week working on general areas as our project is coming to a close:
- Testing the CV subsystem
- Refining the accuracy and robustness of the CV subsystem
- Helping to prepare our team’s final presentation
Testing the CV subsystem
The most critical task of this week regarding the CV subsystem was testing its the accuracy and robustness. The subsystem fundamentally performs two tasks: object detection and potential contact detection.
For testing object detection, I wanted to determine how accurate the system is at detecting the correct objects in each frame. To test this, I performed a series of trials of 60 seconds. In each trial, I what proportion of the total frames had the correct number and types of objects detected. Across the trials, I varied the types of objects, the movement of the camera, and whether the object(s) were on the table or held in the hand. I will present these findings in our final presentation, and they are summarized below in Table 1. The data shows that we were able to reliably track our chosen objects (mayo jar, coffee cup, scissors) to our ideal and minimum requirements when the object was not held. In the case that the object was held, we fell somewhat short of the goal.
Table 1. Object Detection Testing
| Object | Stationary Camera | Moving Camera | Holding Object |
| Mayo Jar | 99% | 93% | 65% |
| Coffee Cup | 100% | 72% | 71% |
| Scissors | 99% | 97% | 67% |
| Book | 78% | – | – |
I was not able to get to testing potential contact detection this week, which I plan on finishing tomorrow or after the presentation next week. Even though our object detection testing fell short of our 90% minimum accuracy metric in some of the testing conditions, I am hopeful that potential contact detection will not suffer as much given the refinements described in the next section.
CV subsystem refinements
I also spent this week refining a few areas of the CV subsystem: I improved object detection through what I am calling “object aliasing”; and I improved potential contact detection by generalizing and expanding the object “memory” used for locking onto an object that I implemented last week.
Object Aliasing
I recognized that after choosing our set of 3 objects and lowering the prediction confidence threshold (in order to report more detections from the network), some of the objects were consistently being misidentified by other objects in the network’s model and outside of our chosen set of 3. For example, at a 30% confidence threshold, and when holding the coffee cup at a 30 degree angle, it would sometimes be identified as a remote. I continued testing and identified a set of auxiliary objects that would uniquely be mistaken by the detection network for each of the objects (figure 1). In order to keep the stability benefits of a low threshold but not sacrifice accuracy, I experimented with manually remapping such incorrect object classifications to the correct object class (e.g. every time a remote was detected, it would remap that detection to a coffee cup), and in general the process has been successful.
Figure 1. Unique object aliases identified during testing
Generalizing memory length
I also improved the robustness of potential contact detection by parametrizing the number of previous frames to consider when determining if to lock onto a new object. Previously, I programmed the system to lock onto an object only if it the glove hitbox overlapped with that object for a set number of frames continuously. However, this was not very robust, since it is common for either the object detection or fiducial marker detection to intermittently cut out for a frame or two.
To address this problem, I loosened the constraint of the conditional, and allowed for tracking if the same overlap had occurred at least once over a parametrized period of previous frames. Hopefully this will get around some of the weaker object detection testing results and boost the potential contact detection performance.
Presentation
Finally, I helped my team with putting together our group presentation, filling my relevant slides, and preparing to present it next week.