Category: David’s Status Reports

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I assisted in making sure all of the components were attached properly on the glove. I also tested the battery life of the glove. Currently, the system is not working smoothly end-to-end, and I am still investigating why the HID controller is not working as expected.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is slightly behind due to the unforeseen problems with the HID controller. Since I will have more free time next week, I will work on getting everything working end to end.

What deliverables do you hope to complete in the next week?

Fix the HID controller and also run more tests on end to end accuracy and latency.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I was able to get the output reports working on the HID controller. This means that we will be able to get the left and right handed mode working, as well as the calibration protocol. This was not working at first because I forgot to include the output declaration in the USB HID report descriptor. In addition, I realized that using output reports rather than feature reports were easier to implement since there are many examples of them online (e.g., keyboard HIDs).

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Progress is on schedule.

What deliverables do you hope to complete in the next week?

This week, I plan to complete integration with left/right handed mode and calibration, as well as start some testing.

Now that you are entering into the verification and validation phase of your project, provide a comprehensive update on what tests you have you run or are planning to run. In particular, how will you analyze the anticipated measured results to verify your contribution to the project meets the engineering design requirements or the use case requirements?

I am planning to run tests on end to end latency as well as the usability and responsiveness of the HID commands to macro conversions on the host computer. I will analyze the responsiveness of the HID commands by running them end to end from the glove to the host computer. In addition, I will also test whether the largest possible magnitude of each command is a reasonable amount. These two will match directly to usability and latency requirements on the original report.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I worked on getting the integration between the host PC controller and the peripheral controller working across the HID interface. Now, I am able to receive specific HID commands from the host PC’s program and parse it to the appropriate command. I implemented this in a multithreaded manner, so that I can read, write, and perform actions simultaneously. Here is a short video demonstrating this:  Video on Google Drive

I was only able to get this working on Windows. I tried to make it run on Mac, but the HIDAPI was not reading the device properly, so it produced an error when attempting to open the device.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I on track. Gram and I were also able to finish the unfinished tasks from last week. The next things to do is to further test the integration for each subsystem.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I worked on getting the host HID controller working. This controller consists of two parts: (1) reading data from the HID interface, (2) performing the macros on the host PC.

Most of the work I have completed was figuring out how to perform the macros on the host PC. I have tested the pan gesture already, but it still needs to be calibrated since the sensitivity of drag actions on Geogebra is high. Nevertheless, I was able to control the movement of the mouse and the graph with the program. Since this requires significant work, I tapped Gram to assist in this part. He mentioned that he is currently waiting on the physical module to be finished and is available to provide support. In the next week, we will work on finishing up all the gesture macros on Geogebra.

I am also working on getting the HID interface working on the host PC side. Currently, I have some written some code using HIDAPI to read the data, but I am still designing the program to be multithreaded so that we can read data and perform actions simultaneously. I also discovered that the HID device class spec allows for what is called a Feature Report. This allows me to send calibration settings to the peripheral HID interface. For instance, we can specify handedness with this feature report.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am slightly behind schedule. About half a week or so. To compensate for this, I tapped Gram to assist in writing some of the macros for Geogebra.

 

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I spent my time writing the peripheral HID controller code and getting it to work on the Raspberry Pi.  I also wrote a script in Python that tests every data type in the HID report. After writing the code, I tested this system on the Pi with the install script. I had to test two things: (1) if the Pi connection through the USB-C port and the power splitter was working as intended, and (2) if the Pi is recognized as a peripheral device.

When I installed the drivers and the kernel configuration with configfs with the install script, I found that the connection through the USB-C port was working as intended. Plugging the Pi to my laptop caused it to show up in Windows (in Parallels) as a USB HID device. However, it initially did not work and failed to start due to a setup error (error 10) as seen in Windows Device Manager. I was able to trace the problem to the install script. In order to easily debug the HID report descriptor, I wrote it in a Python file that is relatively easy to read. Unfortunately, I did not notice that the Python file was being called in the setup script without it being present in the same folder. To fix this, I hard-coded the report descriptor value result from the Python script into the setup script. After I made this change, the Pi was successfully recognized as an HID gamepad.

Here is a demo of Windows receiving the test inputs from my test script. The test script exercises some positive and negative values for each data axis.

https://drive.google.com/file/d/1EWyLNByB8uvyoV85uuTF6MlHb0v3GAD7/view?usp=sharing

Next week, I will work on getting the macros working on Windows from a Python program. Additionally, I should also integrate the setup module with the HID controls.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am on schedule. See Gantt Chart below

As you’ve now established a set of sub-systems necessary to implement your project, what new tools have your team determined will be necessary for you to learn to be able to accomplish these tasks?

Since the next part is figuring out how to control the cursor and inputs on Windows, I have to learn how to do that from Python. In addition, I have to learn how to use HIDAPI on Windows to parse the HID input.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

In the past 2 weeks, I worked on solidifying the plan for the entire project integration and also the HID section specifically. I thought about alternative ways to communicate to the host PC while making set up easy.

For instance, I was considering turning the controller into a mouse/keyboard hybrid device that will automatically control the mouse/keyboard inputs on the host PC, but the problem now is that the cursor has to always be in the center of the screen. One way to solve this issue is to send the cursor location as well as the screen area to the device so it can perform the necessary calculations. However, I thought this method would be a bit convoluted and makes many assumptions.

The solution I ended up choosing is to send only the motion data to the computer as if it were a video game controller. Then a Python application running on the PC would translate these into the appropriate commands. This way, we can build in the application-specific controls as well as mouse cursor and display tracking.

I also did more research into the process to implement the HID controller. It’s quite poorly documented, but thanks to some engineers and their blogs, I was able to figure out how it works.

First, I would have to configure the RPi to act as a USB peripheral device through the HID device configuration with configfs. Then, I would write the software that sends the appropriate data to the HID file descriptor. On the other hand, I would also have to test whether the RPi is correctly being detected as a gamepad on a Windows PC as well as the data it’s sending to the PC. Once this is done, I can start on the host PC software to parse and translate the data into commands.

blog post about HID and how it should be set up

configfs documentation

reference implementation for RPi HID (GitHub)

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

The project is currently on schedule. See Gantt Chart below

As you’ve now established a set of sub-systems necessary to implement your project, what new tools have your team determined will be necessary for you to learn to be able to accomplish these tasks?

We determined that I have to learn how to use the HID device specification and driver provided by the Linux kernel as well as the HIDAPI on Windows to accomplish the tasks.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I presented our design proposal in class with our updated key design decisions. We also started writing the final report on the detailed design of our project. However, we did not make significant progress yet since the design proposal feedback was only sent yesterday.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Our original schedule did not take into account the entire design review process, so by our original Gantt chart, we are behind.

As a result, I updated our Gantt chart to take the adjustments in schedule into consideration. I realized that some tasks I indicated on the chart will not require as much time as I expected like assembling the circuit, so I adjusted the chart accordingly.

Planning out the entire schedule was quite difficult since we did not properly look into the exact timeline of the design report. I also failed to take into account Spring Break, so I had to make some adjustments to the schedule. I expect that we’ll have to adjust the schedule again since some tasks may end up taking longer than expected in the future. The key is to perform as many parallel tasks as possible if the schedule needs to be adjusted.

What deliverables do you hope to complete in the next week?

In the next week, I hope to finalize and submit the design report.

This week, I spent my time working with Gram and Xuan on finalizing some key design decisions. We discussed the the specifications for the gestures that we intend to implement in our project as well as determine if these gestures are intuitive. This part took a bit of thinking because we wanted to make the gestures distinct that users would not accidentally perform a gesture they did not intend. In addition, we are also considering an alternative design for a non-ML-based system which would end up having smaller latency.

Aside from discussing these design decisions, I also assisted in creating a more detailed block diagram for the entire system which elaborates on the protocols between each of the modules as well as enumerates all hardware components in each module. We can then tie all the things we purchased back to this diagram.

So far, I think we are on time in terms of our schedule and the next part of the work would be to finalize all of the design decisions formally in the report.

Gestures

Block Diagram

Relevant ECE Courses: 18-100, 18-220, 18-349