This week I implemented a significant portion of the our system’s pipeline, which is more or less a wrapper of threads and synchronization around the CV and drawing code (the part that maintains a mask of pixels to be applied to every frame). I quickly discovered that build, dependency management, and a lot of other things are quite clunky and tedious when using C++ compared to Rust, a language I am more comfortable with, so I used Rust for this instead. But since OpenCV itself is written in C++, I wrote some foreign-function-interface (FFI) bindings and a well-defined (and typed 😀) function signature for our CV models, which will be written in C++, to implement. In other words, the pipeline code can simply call functions written in C++ that perform any CV / ML tasks using OpenCV and return their respective outputs. And we can use Rust’s wonderful build, dependency management, installation, and testing tool, cargo, to compile and link the C++ code as well without any makefiles or mess of headers.
During the process I gathered more specific details about the each of the system’s functional blocks, how exactly to do the thread synchronization, what parameters they take, and what data structures they use – which we can include in our design report, though all of these things may change in the future.
Right now the C++ model functions are just dummies that always return the same result, since right now my focus is making sure that we can get from a camera feed of frames to ((x,y), gesture_state) pairs to a virtual loopback camera, and to have it be fast and free of concurrency bugs. At this point I’ve got the FFI (mostly) working and a rough first pass at thread synchronization working as well. Next week I’ll be fixing that minor FFI bug and working on the loopback camera.