Month: April 2022

As stated in the team status report, nothing much was done this week outside of the poster and the video. For me specifically, I just haven’t had the time to do any additional optimizations on the CV code. All of my work/finals except for this capstone should be complete as of May 3rd, so I expect that I will resume working on the capstone then, so we can have some better test results for the final paper.

So, this week, I spent the majority of my time finalizing the last for secondary checks for the CV code, and actually did the testing. I started the actual testing itself far too late. I had hoped to finish by today, but even with Jay’s assistance, we were unable to get all of the taken/identified before the end of today.

 

There was a lot of just manually iterating, testing and retesting suitable HSV color bounds, which took more time than I expected. Also, I found that the dimensions of our image post localization generally were not very reliable. The SSIM method I was using for localization tends to pretty regularly overestimate the bounds, so it’s difficult to . The overinclusion of the background also had a cascading effect on the color tests, since it was harder to tell how much of an item was a particular color when we don’t know how much of the image is the background. As such, a lot of the secondary checks ended up being a lot less effective than I had hoped. I think we might possibly be able to mitigate this with a second localization pass using SIFT, but I’m not sure if it’s feasible at this point.

 

There’s not much to say about the testing, I just vastly underestimated the amount of time it would take. It’s not difficult to do, it’s just that our testing plan expects us to take >200 images manually, which takes a large amount of time. Preliminary results based on what we have done are mixed. We are well within the amount of time needed, and we get enough correct positives. However, we also misidentify objects (ID object A as object B) far more commonly then we fail to identify objects (ID object A as an unknown object). This is unfortunately exactly the opposite of what we wanted to occur.

 

Next week, and probably for all of the following weeks until the final live demo/paper, I will be working on trying to get the results such that we can meet our use case. I expect this to be fully possible given where we’re at, it’s literally the only thing left, and I can think of several possible secondary checks that I didn’t get to implement on the first p

This week, I mostly worked on doing integration, specifically handling a fair number of edge cases with regards to unregistered items. This consisted mostly of work in the Django code, and a few minor updates to the CV code. Both Harry and Jay helped me quite a bit as I didn’t have much experience with Django prior.

The CV code previously just gave it’s best guess out of the approved iconic image classes, or none if it wasn’t sure. The CV code was changed to accept a set of items in the current inventory. This is used in the event that an item was removed, so we don’t have to check every possibility, which should help with accuracy. It also deals with the edge case where the cv code says that an item was removed, that it didn’t say was in the cabinet in the first place. The CV code was also modified to accept arbitrary sets of additional Iconic images, to support user item registration. Both of these changes were relatively minor, and mostly just involved filtering/adding to the set of iconic image checked against.

The majority of the code work was done in Django, dealing with user registered items. Essentially, we needed to be able to be able to add unidentified items to the cabinets, and later change their category when the user identified them. We also needed to be able to handle them, in the case that the user decided to place the items and removed them, all without manually identifying them. Long story short this was resolved by creating a Iconic Image model item, with an optional associated item field, that was assigned a UNKNOWN category be default. Essentially, we then treat it as an a regular iconic image, until the user manually identifies it, at which point, we can propagate the identified category to the associated item object. We can also easily generate the prompts the user by filtering iconic models by category/created user. This handles most of the edge cases, and doesn’t have a huge drawback unless the user has huge amounts of unregistered items in their cabinets concurrently, which is unlikely. We still need to do some UI/CSS work here (the alert/request to identify items needs to be placed in a few different places and made much more apparent), but we have the needed functionality. There was also some minor technical debt that I had to fix regards to categories (IE we were making every item have a unique category. instead of re-using them).

Overall, for not knowing Django very well before this week, I’d say I did a pretty good job. Again, I obviously had a large amount of help from Harry and Jay, but I think I learned fairly quickly and got a good amount done. For the next week, I plan to almost exclusively work on the CV component, and try and get it within our needed criteria, and get the testing done so we have it for the slides.