Category: Lilly’s Status Reports

Items done this week:

1. Sewed all the pockets for the parts on the hat, finalized the layout of all the components, and soldered ESP32 to IMU so they can be inserted into the hat without terrible wires sticking out.
2. Wrote/tested recalibration code using server<->Pi communication, and synchronized this with requests from the browser extension. I decided to just handle all the recalibration on the Pi since I realized the process doesn’t have to be that complicated to work properly.
3. Implemented simple low pass filtering (just a moving average) to kill some of the bumpiness in the angle data. Still using the Kalman filtering/sensor fusion algorithm from the interim demo. Seems to work well enough (main thing is it doesn’t drift up/down at rest) but I need to test with a camera instead of just eyeballing to see how accurate the “real-time” angles are.
4. Tweaked code for sending alerts to avoid over-alerting the user.
5. Tested and debugged wireless setup for hat system.
6. Roughly tested with a person wearing the hat.

Progress

Ready to start verifying the angle calculations at this point now that everything is attached to the hat and the wireless mode is fixed, so I’m not too behind schedule. One unfortunate thing is that the JST connectors I ordered ended up not being the right size so I will have to do some more soldering on Monday to extend the connector on the battery. This is not too much of a roadblock for testing, since there’s a convenient little flap on the hat that the battery can be tucked into, close to the ESP32. And, the soldering job will be quick anyway. Overall I don’t think there will be an issue with finishing the final tweaks + testing of this subsystem in time for the final demo.

Deliverables for next week

1. Make components more secure on the hat + solder extension wires to battery connectors
2. Test calculated angles with a camera
3. Configure a demo and “real” mode.

Verification

1. User testing with the hat – have someone wear the hat and do some work at a computer, record from side view (~10 min), calibrate ~90 degrees (upright) as starting position, have a screen with the current angles being printed in the frame, take a sample of ~20 frames from the video, use an image analysis tool (e.g. Kinovea) to get a ground truth angle to compare the “real-time” data with. Test “passes” if the ground truth angle and calculated angles are within 5 degrees in all frames. I also want to repeat this test on a user working for an hour, and take an even sampling of 20 frames across this hour to see if the accuracy gets worse over time.

Video of initial testing (this still with the jumper wires, forgot to re-record with soldered version):  https://drive.google.com/file/d/1_j-dwfMfuiTcbsXzWj6ul1kb8jZTK9rN/view?usp=sharing

Here’s the layout of parts on the hat, from the middle of my adventures in sewing earlier this week:

Items done this week:

1. Fully fixed the BLE communication from ESP32->RPi (ability to send the full float at once instead of separate bytes, which was a problem for some reason last week). Pi->server communication for sending over the angle data as received from the ESP32 is set up as well.
2. Tested sensor fusion algorithms to improve the angle calculations using the acceleration, gyro, and magnetic field data). These readings are still a bit buggy so I still need to implement some filtering on the sensors readings + calculated outputs so the values being sent to the extension do not jump up and down so much. However, they’re still a lot better than last week and get a lot closer to the correct angle, especially when moving the sensor slowly.

Progress?

I had hoped to have much cleaner angle data by now so I would say I’m still behind on that front. I also did not have time to fix the Pi->ESP32 sending issues as I focused on the angle calculation algorithms more this week. However, this is not that much of a problem as the main thing that the Pi/server needs to communicate with the ESP32 is when to start calibration, but this can just be synced with the timing of the Pi’s connection to the ESP32 via Bluetooth (instead of sending some value to indicate the calibration start). Still, I find this messy and I would like to implement the Pi->ESP32 control in the coming weeks so the “user” (and us developers…) can recalibrate/restart sampling via a command in the browser extension. I also decided to not start assembling the hat since our real visor order hasn’t come in yet, and I didn’t want to waste time making something that wouldn’t be in our final product, even though the interim demo is on Monday.

For the interim demo, I would like to have a cleaner (and more accurate) stream of angle data available to send to the Pi/extension, and implement some way of zeroing out the angle/recalibrating mid-run (ideally controlled from the extension).

Deliverables for next week:

1. Further improve angle processing code (probably need more low-pass filtering on output and input data)
2. Start assembling the components on the hat when the visor comes in.
3. Figure out the BLE (Pi->ESP32) issues so calibration can be more streamlined.

https://drive.google.com/file/d/1wddo5KdtA5FNAKvYoB1es1snJE4sNH3T/view?usp=sharing 

Video of the calculated angles when rotating the board from 0 -> 90 degrees in one direction (board offscreen). You can see that it’s still a bit off (only reaches -80 to -85 degrees at the “90 degree” point) and oscillates quite a bit before stabilizing when I stop moving. One good thing is that I’ve calibrated the gyroscopes more extensively so there’s a lot less drift (changing angle when the sensor isn’t even moving).

Items done:

1. Wrote/tested initial angle calculation + user calibration code for the ESP32. Changes in angles reported are broadly correct but a lot of refinement is needed to make this usable.
2. Finished the design report, which ate up most of the week before break.

 

Unfortunately my progress is a bit behind schedule, as I had aimed to have written and tested a basic BLE script for sending data between the ESP32 and RPi by now, and have a more stable version of the angle calculation code as well. I think the angle calculation code shouldn’t take too long since I know what I want to implement to fix the drifting values, so I will likely have enough time this week to figure out the RPi communications after coming back from break and getting a midterm over with. I had also wanted to put in an order for the non-electronic hat parts before the break, but this will be done on Monday now. We may also use a random hat we have, and fabric from Techspark to prototype for now if these parts do not arrive soon.

 

Deliverables for next week!

1. Add complementary filtering to the angle calculation algorithm, and try offsetting drift values in the code. Also see how much the data improves from averaging over two sensors to decide if this is worth doing.
2. Write BLE script for the communication to the RPi and set up calibration flow in the code (need to work with Cora for UI parts).
3. Order a hat and mesh fabric for the wearable parts.

Items completed this week:

1. Tested the ESP32 microcontroller to make sure it worked alright (could be flashed with an example program, didn’t overheat). Researched different development environments (PlatformIO, Arduino) but ultimately decided to stick with the Arduino IDE for simplicity and ease of connecting to our specific microcontroller. I initially had concerns about the compiled code size using the Arduino IDE vs. other platforms but there should be more than enough memory on the board to handle this!
2. Presented our design review to the class.
3. Soldered pin headers to the ESP32 and the LSM9DS1 sensors so I can start breadboard prototyping and setting up I2C communications between them.
4. Researched how to set up the BLE connection from the ESP32.

The I2C setup will hopefully be done tomorrow (on Sunday)! Along with connectivity checks for my soldering… Otherwise I’m pretty on-schedule, especially since I have all the necessary parts now.

 

Deliverables for next week:

1. Finish up the design report.
2. Test sending data over BLE between RPi and ESP32.
3. Code up initial filtering/data processing algorithms for the gyro sensor data.
4. Design mounting system for hat.

This week, I completed the following items:

1. Researched, finalized, and ordered materials for the neck angle-calculation module. These included the gyroscope/accelerometer module, an ESP32-C6, and a battery to power these components. I also researched the communication protocols that I will need to configure and use to send data from the sensor to the ESP32 (I2C), and from the ESP32 to the main microcontroller (BLE).

2. Researched objective measures for determining poor neck posture. I hope to implement an approximation of craniovertebral angle, which is actually used for diagnosing forward head posture, but this will require good positional calculations from the accelerometer, which may not be feasible. To achieve this, I researched methods of improving calculation accuracy for angle/position, including Kalman filtering. An alternative threshold for neck strain is simply a 15 degree downward rotation of the neck.

3. Updated the design of the neck module to use a cap/visor instead of a neckband to improve wearability, comfort, and ease of component layout.

 

4. Tested and did initial calibrations on the OpenCV Python script that we will be adopting for the blink detection part of this project. From this testing, I confirmed that rudimentary blink detection worked, and identified areas that we will need to improve upon and modify to meet our use-case requirements. These included support for single-eye blink detection, ensuring that our rate-calculations do not factor in time where neither eye is in view of the camera, improving functionality in dimmer lighting, and improving landmark detection for glasses-wearers. However, as difficulties with running a Python script from the browser extension arose, Cora will be working on converting the script to JavaScript using OpenCV.js.

5. Prepared slides and presentation for the upcoming design review. 

A couple of my deliverables from last week were not completed unfortunately (figuring out the Bluetooth implementation between ESP32/RPI, and drafting code for gyroscope calculations), as I spent much more time than expected on researching and selecting a microcontroller, and working on the design review presentation. However, these two items would be more effectively completed with the parts in hand, so this isn’t super detrimental to our schedule.

 

Deliverables for next week:

• Present design review and write design report.
• Set up the development environment for flashing the ESP32.
• Research how to implement bluetooth communication between ESP32 and RPi. If ESP32 arrives on Mon/Wed, work on actually setting up this connection too.
• If sensors arrive as well, start setting up the I2C interfaces and test sensor reliability.

This week I focused on researching components for the neck angle measuring system, mainly the gyroscopes.

While we were initially talking about using a gyroscope for tracking neck angle, I also researched inclinometers as another option since they can be used for angle measurement too, especially in steadier conditions. It seems that inclinometers on the market are mainly intended for industrial applications and are too clunky (I’m looking for something relatively flat to avoid disturbing the user) and expensive relative to our budget. So, for something wearable I continued to focus on gyroscope sensors.

As I researched, I looked into the potential of biaxial angle tracking, as forward neck posture is not the only way a user can experience neck strain after extended work periods. Most gyros have this capability anyway. I focused on options that allowed for angle tracking along at least 2 axes (sagittal and coronal) and researched the circuitry each sensor would need to work, with the aim of selecting a sensor with minimal circuitry and accuracy within our requirements, ideally much tighter as our initial goal of +/- 5 degree measurement accuracy is quite wide. Drift in the gyros also came up as a problem, which could be mitigated by doubling up the sensors – and would allow the band to be more symmetric in terms of weight on the user. I also researched/tested the range of angular velocities that our sensor would need to handle – a maximum speed of 200 deg/sec is what I looked for. Further, while an analog gyroscope would be nice so we can control the sampling rate/resolution better, I found that digital gyroscopes will be easier to find and use. A potential option is the L3GD20 gyro, which is small, has low input voltage requirements and a good-enough range.

I am on schedule for now. Further research on wearable batteries and gyroscope part selections will be completed tomorrow before our team meeting on Monday.

Deliverables for the next week:

• Make purchases for neckband components after cross-checking with the team (Monday).
• Research wireless connection between ESP32 + RPI.
• Finalize design for how the neckband will be worn by the user.
• Start drafting code for angle calculation with gyroscope data for the ESP32.