This week I worked on focusing more on the orientation of the pen to better estimate the position of the cursor. I used the orientation to indicate which direction the “tip” of the pen would be facing relative to the user’s hand, which presumably would be holding the pen. I also integrated this functionality into the CV functionality that detects the center of the user’s hand. Combining these two points of information gives us a relatively high accuracy way of detecting the tip of the pen. After this, I added three buttons to the arduino and fed those clicks to the python program. Currently, one button is used as a left mouse button; as long as the button connected to the arduino is held down, so is the left click mouse button. Currently, one of the other two buttons is used to calibrate the orientation of the pen in case drift happens.

This week, I worked more on the pen tracking algorithm. I implemented a multithreaded producer/consumer queue model for the software (producer is arduino sending accelerometer/gyroscope readings and consumer is taking those readings to decide where to move the cursor). We ran into the expected problem of having sensor drift and not fully accurate readings. I researched solutions to this problem. The most promising solutions are using a kalman filter (implemented either in Python or directly on the arduino), especially since the sensor that we are currently using is very popular and many users online have run into similar issues, and using a more accurate IMU that has a magnetometer built in. I have begun to start implementing the kalman filter solution but have not yet completed it.