Category: Gram’s Status Reports

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I fixed the flex sensors to the glove at their base using hot glue and used tape to fix them through their midsections and tip. This took some time to accomplish since I had to readjust the sensors so they would be straight to minimize strain on the sensors.

I also worked on integrating the end-to-end systems and setup demos of our independent subsystems for our presentation. Here, I added a debounce to our classifier to create a more seamless experience when performing gestures. Essentially, when a gesture is classified, a timeout is applied before the next gesture is applied to prevent the host cursor from jittering across the screen.

I also worked on the slides for the final demo and practiced speaking since I presented.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is a bit behind because after integration, some parts malfunctioned (some wires detached, driver malfunctioned). However, we will resolve them this week and get the entire system working.

What deliverables do you hope to complete in the next week?

I plan to work on testing the last untested parts and work on the final report.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I configured the compute module to automatically start the system on startup, since a bug in the earlier configuration caused the program to keep crashing on startup.

I improved the gesture classification model to become more robust against random noise that was being collected by the sensors. I also addressed an edge case in the gesture classification model where pan/rotate gestures weren’t being detected if they rotated slightly while panning or panned slightly while rotating.

I also sewed 10 cloth straps onto the gloves to hold the flex sensors in place. However, after putting this together and trying it with the glove module, we discovered that it wasn’t providing enough tension for the flex sensors to pick up. I experimented with using rubber bands and tape as alternative attachment schemes and will likely stick with tape for now.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is a bit behind since there were a lot of hiccups with integration, but since the integration is pretty much done, we can get started with testing this week after our presentation.

What deliverables do you hope to complete in the next week?

I plan to work on the demo presentation slides and finish testing our system.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I worked on integrating the binary packed input from the glove module with the MQTT receiver on the compute module. I also worked on integrating the compute module with the HID receiver on the computer. I also worked on refining the gesture classification model based on real values from the glove module.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is a bit behind since we should be starting to test this week. I’ll ensure the end-to-end integration is completed by the start of next week so that we can spend most of next week working on testing and evaluation.

What deliverables do you hope to complete in the next week?

I plan to refine the gesture classification model more so that it can work seamlessly with the input from the glove and dispatch more seamlessly with the HID receiver. Afterwards, I plan to work on some benchmarking and evaluation.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I worked on the gesture dispatch mapping on the HID controller to work in conjuction with the gesture classification model. Specifically, I worked on code that used pyautogui to dispatch HID controls that mapped to specific gestures on Geogebra. I also built an API to make this work better with the input that comes in from the gesture classification model.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is on schedule.

What deliverables do you hope to complete in the next week?

Aside from the interim demo, I plan to flesh out integration in time for the interim demo as well as more rigorously test the gesture classification model.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I worked on smoothening the output of the decision tree over adjacent windows of data. I also implemented a system to map the magnitude of the gesture to the intensity of the action being performed on the computer.

Since I finished this part relatively early, I worked with David to start implementing the functionality for the gesture on the HID side. This will work well for both of us since this is closely related to the output of the gesture classification model I have been working on.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is on schedule.

What deliverables do you hope to complete in the next week?

This week, I plan to finish the gesture dispatch on the HID and integrate that with the gesture classification model.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week, I wrote code for the controller to collect and aggregate data published by the glove on MQTT. I also worked on the gesture classification model, which took some more time. I built a decision tree which analyzed the trends of the incoming data in each window to classify what gesture was being performed. Although I was not able to implement smoothing of the data and more seamless transitions between windows of sensor data, I was able to write tests for the various gestures to ensure that as the models change, we can quickly verify the correctness of the underlying logic.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is slightly behind schedule on my part because of the decision tree, so I will try to finish this before the middle of next week so I can finish all of next week’s tasks on time.

What deliverables do you hope to complete in the next week?

Next week, I plan to fully complete the code for the decision tree in terms of smoothening out recognition over adjacent windows of data. Moreover, I will establish bounds (min/max) for gesture intensity output so I can map them to the intensity of the input data.

I will also work on establishing a rapid data collection script so we can create a dataset that we can use to analyze future models without doing live testing.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

The past week, I worked on writing the design report. Specifically, I worked on the Introduction, Use Case Requirements, parts of the Design Trade Studies, System Implementation, and Project Management sections, Related Work, and Summary.

I also began writing the controller code for the compute module. I setup the Mosquitto (MQTT) broker on the Raspberry Pi so that it can listen for incoming messages from a connected glove. I also wrote a Python program that subscribed to the data packets that would be broadcasted by the glove. Since the glove MQTT client is still in progress, I tested this functionality by writing a dummy MQTT client in Python to publish messages.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is still on schedule.

What deliverables do you hope to complete in the next week?

I plan to finish writing the controller code this week. I will need to write the logic for aggregating received data packets into windows that will be fed into the decision tree. Depending on how quickly the system can process incoming packets, I may also need to implement some sort of buffer queue to handle the case of data packets arriving faster than the system can process them.

Furthermore, I will also finish implementing the decision tree. I will try to implement the logic for classifying what gesture a user is doing without necessarily correctly identifying the degree/intensity of the action yet. If the decision tree logic is finished quickly, I hope to implement the degree/intensity determination logic as well.

As you’ve now established a set of sub-systems necessary to implement your project, what new tools have your team determined will be necessary for you to learn to be able to accomplish these tasks?

I will need to learn more about windowed analysis algorithms over time series data because I need to identify what optimal window size I should use. Moreover, I will likely need to learn more about data smoothing algorithms to prevent jittering. Furthermore, I will need to learn how to effectively remove the gravity vector from the accelerometer data. This will be challenging since the gravity vector may be split up across the three axes, depending on the orientation of the hand.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I helped prepare the slides for the design presentation and started working on the design report, incorporating some of the feedback we received from the design presentation.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

We’re slightly behind based on our original schedule, but also a lot of the tasks we had originally listed were allotted more time than they actually needed. We updated our Gantt chart to reflect this which puts us back on schedule.

What deliverables do you hope to complete in the next week?

I will continue working on the design report with the rest of my team and finish it by then. Additionally, I’ll work on assembling the ML pipeline so we can start testing and collecting training data.

Updates

Block diagram updates

This week, I identified more specifically what sensors and communication protocols we will be using for our device. I also worked on the wiring between the different components and microcontrollers — illustrating how many wires we would actually need between components. I updated our block diagram to reflect this:

Communication Protocols

For the accelerometer, we found from the MPU-6050 IMU datasheet that it communicates using the I2C protocol. For communication between the ESP32 (gloves) and the Raspberry Pi (compute module), we originally only said that we were communicating over Wi-Fi but we had not yet thought about what protocol we would be using to communicate. I researched into what protocols were common for high-frequency updates from IoT devices and found the MQTT protocol to be the best fit for our setup. MQTT works similar to various other pub-sub systems while minimizing communication overhead. Moreover, there are existing MQTT libraries we can use for the ESP which can connect to an MQTT (Mosquitto) broker on the Raspberry Pi server.

Gestures

This week, I also worked on refining the exact gestures we would recognize. I worked to help break down our conceptual actions (zoom, pan, and rotation) into concrete gestures that we could recognize. We managed to break this down into 5 specific gestures:

• Enable transmission
• Disable transmission
• Zoom in
• Zoom out
• Rotate
• Pan

The gestures are explained in more detail here: Gestures

Schedule

Currently, we are still on schedule to finish the project in a timely manner.

Upcoming Deliverables

This coming week, I will be working on formalizing these design decisions in the design report. Specifically, I will work on finding resources to back up our design choices and, if necessary, modify them if resources state otherwise.

Relevant ECE Courses

• 18-100, 18-220, 18-240, 18-794,