Author: agao2

This week consisted of: refining the cue stick system, creating the local application, and testing the projector output system. When I began building the cue stick system, I realized that the Arduino Nano was redundant since the ESP32 acts as a microcontroller just augmented with WiFi/Bluetooth. The first version of this module utilized only the Arduino Nano and was wired. Because this was a lot easier to set up, I built the V1 of our cue stick system with just the Arduino Nano to test functionality. Once I verified it was working, I replaced the Arduino Nano with the ESP32 module. It works by acting as a server accessible through WiFi on a specified localhost port. When endpoints “/accel”, “/gyro” are queried, I construct an HTTP request containing accelerometer and gyroscope data respectively.

This leads into the local application. I built a first version of the webapp using React/Flask. The Flask backend acts as a sort of lightweight storage for the webapp. It routinely polls the ESP32 module fetching recent gyroscope and accelerometer data. It stores it in a local SQLite database and returns paginated data to the React frontend when requested. I initially created this Flask backend because I didn’t think WiFi speeds were fast enough to accommodate how much data we were going to receive per few milliseconds/seconds. However, just by running back-of-the-envelope calculations on typical WiFi router speeds, it might be possible to do away with the entire backend and poll the ESP32 module directly actually. Though an argument to be made for keeping the backend is to eventually integrate it with the physics engine, in which we would need storage or more complex behavior. I also tested the projection system and had to modify the coloring scheme. It turns out the projector is too weak in well-lit environments, so you have to use only black and white in the projection to have enough contrast to be clear and visible.

My progress this week is on-schedule. For next week, I hope to flesh out the webapp more by building an intuitive, user-friendly display for the accelerometer, gyroscope data for the web application. I’m also integrating the camera feed to the web application.

This week I finished up the AprilTag module. I modified an existing AprilTag library, extended it, and provided an API that detects AprilTags within an image and creates a coordinate plane with it. This allows the computer vision The AprilTag module is written in C/C++ and Python. I also wrote some unit tests, documented the usage of this AprilTag API, and handed it off to the rest of my team.

After this, I looked into the hardware components we need to assemble this system together. For detecting pool stick motion and player behavior, I decided on an Arduino Nano + 9DOF IMU + ESP32 module. This encapsulates the motion processing system. The Nano gets data from the IMU, then streams it to our laptop via the ESP32 module over WiFi. I also wrote some of the motion detection software for Arduino, though I don’t have a chance to test it on hardware until the parts arrive.

I also started writing code for our projection system (projecting our predictions onto the actual pool table). Currently, the predictions are outputted as lines drawn onto an image/frame of the pool table. I wrote out v1 of the projection system which currently just displays each frame live as the projection system outputs them. I’m going to improve on this by just displaying the trajectories and being able to automatically calibrate the projector by scaling the image size up and down depending on the AprilTag coordinates.