This week I did some more work on the IMU. I have successfully added the calibration process, however the movements are not as accurate as we would like. I am still on schedule, and I’m hoping that this part can be finished by next week. My goal for the next week is to work on the accuracy and get the IMU working as we planned to. As a team, I think we all learned how to work together and delegate tasks. Since we all need to work on the same hardware, we made sure to set weekly deadlines to get our individual work done so that everyone gets the parts they need. We would also try to work together in the same place so that everyone on the team knows what’s going on in other other sectors.
Sarah’s Status Update 11/18
This week, I worked more on interfacing the sensors with the board. I was able to solder some wires to the sensors to wires for them to connect to the board. Additionally, I did more research on the sensors and their resistance limit. I’m fairly sure the resistors I put in place (10K) will suffice for any amount of combination for sensor resistances, but I will also look into increasing the resistance limit for more calibration purposes. I also explored the layout that we would like to use for the sensor and board placement on the glove. Next week, I will focus on soldering the PCB since it will likely arrive in the next few days, and attaching everything to the glove for a working model. We will also decide on which battery to use, as we would like it to be as light as possible. We are looking into 5V batteries with around 2000mAh since our tests indicated this would be sufficient power for more than 2-3 hours, which was our requirement. We will most likely order a backup battery and connect via usb. We are largely on track for this project and will look into getting more data and running more tests when the board arrives.
Saumya’s Status Update 11/18
This week, I worked on completing the networking code for keystroke detection as well as completing a UI for users to attribute gestures to specific keystrokes. I came up with the idea of using spotlight search on the Mac to quickly open common websites for viewing movies (ex. Netflix, Hulu, YouTube), and I played around with polling and delays to get a really quick connection between the sensor movement and the keystrokes. By Rosina’s suggestion, I also added a longer delay when a gesture was registered compared to when no gesture was registered to avoid the system registering the same opcode multiple times on a single press.
One bug I had to fix dealt with the pyautogui hotkey function, which is supposed to handle keyboard shortcuts (like pressing ctrl+z). I noticed that it sometimes didn’t press the keys at the same exact time and the shortcuts were not always carried out. To fix this, I wrote my own version using other pyautogui functions that ensured that all keys stayed pressed at the same time. This likely is less optimized than the hotkey function, but the latency difference was undetectable, and it worked with good reliability. I also worked on a protocol for sending IMU data. If the fail-safe is triggered, while it stays triggered, I’ll continuously transmit IMU data and have the mouse move in the IMU data callback function. This failsafe condition will be checked on the ESP before any of the other sensors are checked to ensure that other sensors are ignored when the failsafe is pressed.
In terms of the Gantt chart, I am on track based on my tasks. I should be ready to integrate with Rosina’s part and hopefully we can have a finished product ready to user test by the end of Thanksgiving break.
Team Status Update 11/11
This week we all focused on getting our parts done for the interim demo. Saumya had the bluetooth portion, Sarah had the sensors and hardware, and Rosina had the IMU. We are currently working on improving our respective parts and integrating them together. As for changes to the design, we went back to the bluetooth protocol instead of Wifi. We are also making new changes to our PCB board to add in resistors and make it smaller in size. Our costs are still mitigated and was necessary in terms of creating a more accessible device. Some risks we are currently facing is making sure we don’t use up too much memory since only limited RAM is available on the ESP32. After we get out parts working we will focus on minimizing the memory.
Rosina’s Status Update 11/11
This week was the interim demo so I focused on getting something visually working for the IMU. I had the algorithm done that outputted x and y coordinates, but I then added code so that it would translate to mouse movements on the screen. I am currently on schedule with this. I also hope to continue working on the IMU calibration and the speed at which is moves and get it closer to working as desired. For testing the product, we heavily rely on user input. We plan to get a variety of people to test the product and use their feedback to improve the project. I will also be running latency tests and battery tests.
Sarah’s Status Update 11/11
This week, we had our interim demonstrations, so I worked on getting ready for that. I set up a breadboard with 4 touch sensors and 3 flex sensors and connected them all to the ESP32. However, the flex sensors’ resistance was too large to be recognized by the ESP. I then added 10K Ohm resistors in parallel for each flex sensor to lower the effective resistance of each flex sensor circuit. Additionally, when combining all the sensors on the same power line, there seemed to be the same problem with too much resistance on each line. I then added 10K Ohm resistors in parallel for each sensor, which seemed to solve our problem. We also received our IMU earlier this week. Since the communication between the ESP and IMU is I2C, I created an I2C scanner to find the peripheral address of the IMU. I then read through the IMU manuals to figure out how to enable and display the readings from each sensor on the IMU. Since the IMU is the only peripheral on the I2C bus, I treated it as a sensor and constantly polled for data. This eliminated the need for interrupts, so the only wires connecting the ESP and the IMU are power, mode selection for the communication protocol, and the data and clock lines. We ended up implementing a library especially designed for the FSM300, which allowed us to print gyroscope and accelerometer readings to the serial display. After adding in the resistors to the demo breadboard, I updated the PCB files to include these added resistors, using SMD 0805 10K resistors. For ease of use and to make our entire footprint smaller, I sent out the files for the PCB to be fabricated, and I expect this order to arrive early next week. I also placed an order for the correct resistors needed as I will do the PCB assembly by hand. Finally, I tested out the battery life needed for the entire system, and found that the battery life lasted for over 15 hours with about 7500 mAh of charge. Next week, I will start assembling the PCB, attaching the sensors to the glove, and do some more stress testing on the battery so we can hopefully order either a small power bank or lithium ion battery for the final project. We are on schedule to finish on time.
Saumya’s Status Update 11/11
This week, I completed a networking proof of concept for the interim demos and iterated on my design to solve some of the challenges brought up by the TAs during our demos. I created a protocol for exchanging information between the glove (server) and our Python receiver client, and finished coding up the communication protocol for the various gestures. The protocol involved transmitting GATT profile characteristics containing an opcode (referring to a gesture) and IMU data (referring to acceleration and gyroscope velocity values). I also made a software flowchart to make it easier to explain our flow. I also integrated Sarah’s code with mine to make sure memory wasn’t an issue with our ESP32 and that opcodes were being detected correctly.
One challenge brought up by the TAs was making sure to register gestures only a single time. To fix this, I worked on tweaking the protocol to be based on callback notifications to the Python client when a characteristic was updated.
With regards to the Gantt chart, our team is right on track to begin some testing while finishing up the IMU portion of the code, and I am on schedule with my tasks. The main tests I’ve run based on my own part have been latency and range. I tested the ESP’s range and my laptop is able to pick up its signal from the amount we indicated (2.28 meters) in the use case requirements without issues. For latency, however, the opcodes took a little bit of time (a few seconds) to register movement due to the polling-style nature of how I was receiving Bluetooth notifications. The actual transmission latency between the two devices isn’t a lot, so I think I can fix this by not polling for notifications. I plan to continue timing the latency with my new protocol design and trying to process keystrokes faster to keep us within the delay we indicated in our use case requirements.
Rosina’s Status Update 11/4
This week we received the IMU so I started to test my code and get the IMU interface it with the computer. Since we have demos due, I’ve been working on getting the IMU to output x and y coordinates as accurate as possible. Next week, I hope to use PyAutoGUI and translate the coordinates to position the mouse on the laptop screen.
Saumya’s Status Update 11/4
This week, I was able to establish a BLE connection between the ESP32 and our Python receiver, and I worked on a simple script that created a connection between the Python client and the ESP32 server (BLE uses server/client terminology in a different way from the traditional definition). I spent some time exploring creating the connection via an MQTT server since BLE hadn’t been working, but ultimately, I got the code to work and found the BLE code to be simpler for demo purposes.
I’m also currently working on the interim demo. I’m making a more specific flow chart of what will be processed locally and what specifically will be sent to the Python receiver to get our glove to transmit data and actually get our sensors to register keystrokes and movement onscreen.
With regards to the Gantt chart, I think I’ve caught up a little bit, but not having our IMU for a while and the BLE issues set back our tasks slightly. Having this BLE communication protocol done will allow me to finish keystroke detection likely within the week so Rosina and I can really focus on the IMU portion.
Team Status Update
This week, we worked on our ethics assignment and thought about ways in which our product would affect users in an ethical manner. One of the cases we thought about specifically was the situation in which someone could control someone else’s device without their consent. We thought about ways in which we could combat this, including limiting information collected from the user as well as security measures to make sure the user knows exactly which keystrokes do what and the risks that come with using the device. Additionally, we received our IMU this week and soldered jumper cables onto it. We will work next week on creating a demonstration for the class as well as working with the IMU to interface it with the ESP32, specifically implementing communication protocols as well as figuring out how to access information from the device. Some risks that could jeopardize the project are, because we received the IMU somewhat later in the semester due to shipping complications, we need to start interfacing and familiarizing ourselves with the software and calibration techniques that come with the device. No considerable changes have occurred to our design, and we are seemingly on track with the rest of the project.