Category: Saumya’s Status Updates

This week I worked on finalizing code for our demo as well as the final presentation slides and poster. I created a final flow chart that contained a full description of the software architecture we implemented. I also worked on expanding the IMU code to the Z-axis and translating this to the movements onscreen. I also worked on finishing up the GUI.

In terms of the Gantt chart, we are right on track to finish by the demo. We just have to code the remaining parts of our project (GUI, touching up IMU movement, etc) and practice for our demo on Monday.

This week, I worked on code for reading in the IMU data and mapping this to mouse movements. Working with Rosina, I experimented with the calibrated linear acceleration data coming from the IMU. We noticed that our data was decently accurate thanks to the IMU’s software for bias-reduction and combating drift. We used this data accuracy to our advantage when designing our position calculator. We utilized a 3-state Kalman filter with position, velocity, and acceleration as our states. On each iteration of the filter, we used a double integration method to calculate the position.

When experimenting with this, we noticed that the Kalman filter resulted in a good amount of smoothing of our position curve, but the position would shift back a little bit due to our velocity values. To correct for this, we implemented code that would snap the velocity to zero if several acceleration measurements of 0 were detected. That way the drift would be minimal. When we implemented our mouse movement code, this resulted in pretty good movement. In order to make this as accurate as possible, we plan on using our failsafe button as an interrupt to snap the mouse position in place and avoid the recoil. Now that this works for a single dimension, we plan to implement a second kalman filter to handle vertical movement at the same time.

In terms of the Gantt chart, I still have a lot of progress to make towards polishing our product, but a lot of the main challenges are completed.

This week, I worked on completing the networking code for keystroke detection as well as completing a UI for users to attribute gestures to specific keystrokes. I came up with the idea of using spotlight search on the Mac to quickly open common websites for viewing movies (ex. Netflix, Hulu, YouTube), and I played around with polling and delays to get a really quick connection between the sensor movement and the keystrokes. By Rosina’s suggestion, I also added a longer delay when a gesture was registered compared to when no gesture was registered to avoid the system registering the same opcode multiple times on a single press.

One bug I had to fix dealt with the pyautogui hotkey function, which is supposed to handle keyboard shortcuts (like pressing ctrl+z). I noticed that it sometimes didn’t press the keys at the same exact time and the shortcuts were not always carried out. To fix this, I wrote my own version using other pyautogui functions that ensured that all keys stayed pressed at the same time. This likely is less optimized than the hotkey function, but the latency difference was undetectable, and it worked with good reliability. I also worked on a protocol for sending IMU data. If the fail-safe is triggered, while it stays triggered, I’ll continuously transmit IMU data and have the mouse move in the IMU data callback function. This failsafe condition will be checked on the ESP before any of the other sensors are checked to ensure that other sensors are ignored when the failsafe is pressed.

In terms of the Gantt chart, I am on track based on my tasks. I should be ready to integrate with Rosina’s part and hopefully we can have a finished product ready to user test by the end of Thanksgiving break.

This week, I was able to establish a BLE connection between the ESP32 and our Python receiver, and I worked on a simple script that created a connection between the Python client and the ESP32 server (BLE uses server/client terminology in a different way from the traditional definition). I spent some time exploring creating the connection via an MQTT server since BLE hadn’t been working, but ultimately, I got the code to work and found the BLE code to be simpler for demo purposes. 

I’m also currently working on the interim demo. I’m making a more specific flow chart of what will be processed locally and what specifically will be sent to the Python receiver to get our glove to transmit data and actually get our sensors to register keystrokes and movement onscreen.

With regards to the Gantt chart, I think I’ve caught up a little bit, but not having our IMU for a while and the BLE issues set back our tasks slightly. Having this BLE communication protocol done will allow me to finish keystroke detection likely within the week so Rosina and I can really focus on the IMU portion.

This week, I worked on trying to get communication between the ESP32 and our Python receiver to work via BLE. I also helped Sarah talk through some ideas while she worked on our PCB. I wasn’t able to get Bluetooth to work with my M1 even using a different Python package, so I may fall back on our backup plan of either using a VM/different OS or trying WiFi with the ESP32. I also worked on the ethics assignment individually and with my group for the group discussion portion. I was unfortunately sick for most of the week as well, so I am a bit behind schedule with regards to our Gantt chart. To get back on track, I will spend the first half of the week catching up with my other classes including a makeup exam until Tuesday, and then dedicate the rest of the week to working on capstone. During my breaks in the first half of the week, I’ll also read up on WiFi projects and set up my VM so I can get started with our backup plans as soon as Wednesday hits.

This week was our budgeted slack week. We largely used this time to enjoy our break since we’re mostly on track, but we did a small amount of work to keep progressing towards our Gantt chart goals. Specifically, I looked into sample projects using the ESP32 and Bluetooth and downloaded the required Bluetooth modules. To work with my M1, I determined that I wouldn’t be able to use the bleak package, but pybluez might be compatible. As per the Canvas question, I’m looking into learning the Bluetooth protocol (at the abstracted level of the ESP32 and Python modules) as well as programming for the ESP32 in particular.

This week, I helped Sarah with sensor setup and debugging an issue with the board. Specifically, after some experimentation, we noticed that two of the pins seemed to be shorted on the ESP board we received. In addition, I did some research on how to implement BLE with our specific board as well as reading sample project code.As the presenter for the design presentation, I also went over my presentation a little bit. Based on my Gantt chart tasks, we’re on track for sensor and board setup, but a little behind with Bluetooth since it took a bit to get our board. I think I now have enough information to proceed, so I should have the Bluetooth transmitter set up by next week’s update. I also will be working on the design review report this week.

This week, I experimented with pyautogui to explore its capabilities and how we can potentially use it in our project. I also worked on the design presentation. As this week’s presenter, I also practiced presenting and worked on my script. In addition, I helped Sarah with hardware selection based on our software architecture requirements and power requirements. Based on my tasks on the Gantt chart, I’m on track with regards to key mapping research, but for Bluetooth, I was waiting on our finalization of the board to look into specific networking information. Now that we’ve selected our board, I can look into sample projects using Bluetooth connections to the microcontroller and figure out how I can get a similar connection established between our computer and mouse board in the first half of the week so the Bluetooth transmitter task can be completed by the end of the week. By next week’s update, I plan to make progress towards getting Bluetooth set up.

The classes that helped me the most with the design of our project were 18349 (Embedded Systems) and 18441 (Computer Networks). I hope to apply some knowledge from my internship, where I learned more about Bluetooth. I also took Physics C in high school, which will help me with the IMU math.

This past week, I spent time creating our software architecture plan. I also researched the Bluetooth portion of our project, specifically looking into Python libraries that would be compatible with the various microcontrollers we were deciding between (Arduinos, STM boards, etc). I identified the pyautogui library as one that we could use to take control of both the mouse and keyboard actions. In addition, I collaborated with my group members to refine our final proposal presentation slides. We worked to finalize our design plan, making sure that the scope of our MVP was reasonable given the time constraints. When creating our Gantt chart, I delegated tasks related to networking and physics aspects of our project to myself, since I have a little bit of experience and a lot of interest in those areas. Finally, I started looking into the inertial measurement unit sensors and what physics calculations we might need to translate hand movements into mouse actions.