Team C8: Whiteboard Pal

This week I implemented a significant portion of the our system’s pipeline, which is more or less a wrapper of threads and synchronization around the CV and drawing code (the part that maintains a mask of pixels to be applied to every frame). I quickly discovered that build, dependency management, and a lot of other things are quite clunky and tedious when using C++ compared to Rust, a language I am more comfortable with, so I used Rust for this instead. But since OpenCV itself is written in C++, I wrote some foreign-function-interface (FFI) bindings and a well-defined (and typed 😀) function signature for our CV models, which will be written in C++, to implement. In other words, the pipeline code can simply call functions written in C++ that perform any CV / ML tasks using OpenCV and return their respective outputs. And we can use Rust’s wonderful build, dependency management, installation, and testing tool, cargo, to compile and link the C++ code as well without any makefiles or mess of headers.

During the process I gathered more specific details about the each of the system’s functional blocks, how exactly to do the thread synchronization, what parameters they take, and what data structures they use – which we can include in our design report, though all of these things may change in the future.

Right now the C++ model functions are just dummies that always return the same result, since right now my focus is making sure that we can get from a camera feed of frames to ((x,y), gesture_state) pairs to a virtual loopback camera, and to have it be fast and free of concurrency bugs. At this point I’ve got the FFI (mostly) working and a rough first pass at thread synchronization working as well. Next week I’ll be fixing that minor FFI bug and working on the loopback camera.

For this week, our team presented our design review slides and looked over the feedback given by our faculty and peers. We also set out a work plan for completing the design review paper, which will include some changes that has been made as a result of feedback we have received throughout the last week.

In addition, our team has each individually made progress in the areas that we are responsible for completing, which is detailed more in our individual status reports.

This week I mostly spent time reading about how Linux and Mac handle virtual devices and mentally cataloguing OpenCV’s very large set of rather useful API’s. A lot of this was because we needed to make a precise decision about what we were doing and to some extent there wasn’t much of a consensus at first, so we all spent some time doing some more reading and learning about each of the possibilities and how much time they would take. In particular, I was looking into Professor Kim’s suggestion of being able to draw straight onto the screen and then pipe the result into zoom (or wherever in theory), which means a virtual camera interface. On linux, there’s actually a kernel module, a part of the Video 4 Linux project, that creates loopback devices, which are more or less a file that a program can write frame data to that has a corresponding file from which said data can be read by other programs. Mac has a dedicated library that has far less documentation and it seems far less simple to implement, so we decided to stick to Linux as our targeted platform and write the entire thing in C++. I also created an updated software architecture diagram for the system, which as of now will be a pipeline consisting of 4 threads that use message passing channels from the boost library for synchronization.

This week I made progress on my portion of the project. I needed to figure out how to do real time finger tracking. I found a method that was used and this will be my starting point. I worked this week with the  team on the design presentation and began to think about how I will be presenting on Monday.

This week, after I made the some notes, spent some times thinking about what I was going to say, and gave the presentation, on Friday, Zacchaeus, Jenny, and I went over feedback from the presentation and agreed that though the course wants quantifiable metrics, ultimately this project’s use case is rather hard to quantify. Aside from latency and framerate numbers, which are more concerned with the associated utility that can render the product unusable, there aren’t really any good and sane ways to measure anything that falls under the umbrella of “accuracy”. The former can be improved by optimizing a solution, but that means having a solution that’s “good” in the first place, and “good” in this case is hard to measure. So we more or less decided that in the end it’s best that we just iterate as fast as possible. So to those ends I opened up a team github repo and implemented a suite of abstractions that should make it as easy as possible to iterate on models and their associated pre-processing steps. I also started implementing a tool for collecting and labeling raw frame data for gesture detection, which is most likely the harder part of this.

This week I worked with my group to fixup our presentation. Along with that I met with my group to work on and expand on our Gantt Chart. Following this I also did research into how finger tracking works. Along with this I also did some research into Hand Pose Estimation and found several research papers on this along with the code and the data sets for hand pose estimation that the researchers used. This will help me set up for next week.