Month: April 2021

This week and the past weeks I caught up on a lot of the stuff I needed to get done. Last week I ported the code to C++ and got everything done on the schedule. This week the team has been trying to pivot to using media pipe to get out of using a glove. For the first half of next week likely I will try to get media pipe working on my system as well. That way we should have a better time creating build packages that work in general for Linux systems.

For this past week, our group decided to do a pivot to trying to integrate mediapipe’s handtracking into Whiteboard Pal. Because the c++ library is rather convoluted and lack any documentation, I first wrote all the gesture detection and finger tracking code in Python in a separate file. In the case that later we are unable to integrate the c++ mediapipe library, this Python file can also be used as a fallback to be integrated using ros icp or another method. The Python gesture detector I wrote has a near 100% accuracy when there is a hand on the webcam. The only issue is if there is no hand at all and you have a complicated background, it might detect some part of the background as a hand (but it’ll still detect it likely as no gesture).

In addition to the Python code I wrote, I also managed to get the c++ mediapipe hand tracking demo that’s provided by the library to build and run successfully. Currently, I’m in the middle of trying to use the same algorithm as my Python code, except turn it into a mediapipe calculator in c++ to be integrated with our Whiteboard Pal.

For this past week, I successfully fully integrated my ML model that I made with Python into c++. So now it is a part of the pipeline and returns the classification as a boolean. Additionally, I also print out the classification to stdout for demo purposes.

To integrate, I had to create a different model, since it appears the Mat format in c++ had only 1 channel in grayscale. As a result, I changed around my python model a bit to have the input dimensions match with the c++ image dimensions, and the resulting one has 67% validation accuracy.

Right now my part is demo-ready, and I will further work on improving the accuracy of the model in the upcoming weeks.

Earlier this week we got together and finally integrated everything together in C++, which entailed working out some dependency management troubles with my teammates which use different linux distributions. Though probably better to figure out that stuff now than latter. Our meeting with Professor Kim earlier this week was elucidating to the fact that “draw on screen” is far more useful than “draw on camera input”. So I spent some time reading the X11 manual to see how best to get a feed of what’s being drawn onto the screen. At first glance the `getImage` function seemed to provide that functionality but unfortunately it’s quite slow, as it blocks the rendering process while copying the entire frame, including transparency. Instead it turns out it’s faster to just create a secondary consumer of X11 input from the UI and read that directly into a CV matrix. I have something that *mostly* works – the images are still a bit messed up because the field ordering in X11’s RGB image format is different than OpenCV’s. Additionally I abstracted away our input so we should be able to swap between different input sources while the application is running.

This week The team got together and got the entire pipeline put together. We can now read camera input modify it in software and allow other applications to use that software camera as an input. We have it working on Arch Linux and Linux Mint (Which as a Fork of Ubuntu). Jenny is using Ubuntu 20.04 and just got it up and running this week and will be setting her stuff up this week as well. On my own front I began to find a more sophisticated way of doing background subtraction for the purpose of hand detection. So far in my research I found a good paper which also pointed to other papers which give a survey of many other methods that are used. Along with that I started looking at media pipe which seems to be even more ideal for what we want to do. Unfortunately this week I have not been able to get these new methods tested at yet due to assignments being due.

Paper that gives a survey of background subtraction methods. https://ipsjcva.springeropen.com/articles/10.1186/s41074-017-0036-1

For this past week, I have been caught up with the housekeeping aspects of the project and integration. Since Zacchaeus and Sebastien both currently do not have Ubuntu as their version of Linux, and since Ubuntu is one of the most popular distributions, we wanted to ensure that our project will run on that. Thus, I installed Ubuntu on my Windows machine, enabled dual boot, and set up Whiteboard Pal on my Linux, which involved needing to build all the libraries again and ensuring Whiteboard Pal could find them, namely opencv and Tensorflow being the big ones.

I also have abandoned the path of using a third-party written library to import a keras model into c++, and instead, looked into importing protobuf files (.pb, which is the file type used by tensorflow when saving a model) into c++ through direct tensorflow library functions. I will be testing the different ways to import in the next three days, and will hopefully have found one that works by Wednesday meeting with Professor Kim.