Posted on

Elliot’s Status Report for 12/7

This week was spent improving and testing the Bluetooth code to verify its performance. Following last week’s changes, there appeared to be some lingering interference between the two sticks (~100ms delay) when playing together. Paired with the CV processing, I concluded that this was not acceptable latency, so I changed the firmware and host code to run BT Classic rather than BLE. The main changes involved identifying serial ports for each of our ESPs as well as advertising and sending data as a byte array with the BluetoothSerial library rather than notifying with BLEdevice. The laptop code was relatively simple to implement, considering I only needed to work with serial input similar to UART. Following these changes, I set up a test to verify that the interference between the two sticks was lowered; I used matplotlib to take timestamps between when the accelerometer data was first received and when the sound output played, and I overlayed the delays of the two sticks to compare their relative performances.

Test Results (<6ms delay for both drumsticks)

I’m satisfied with these results, since this was our most persistent issue and both sticks are now operating at similar performance. It also gives us an opportunity to highlight the design tradeoffs between lower power consumption and reduced interference on the 2.4GHz band for BLE and BT Classic, respectively. I believe our team is on track to have a fully working project before the demo date, and I plan to spend this week extensively testing the system in various environments and conditions to locate any edge-case bugs.

Posted on

Elliot’s Status Report for 11/30

This week I worked more on the system’s Bluetooth component and gathered verification metrics in preparation for our final presentation. One of the team’s design requirements is to ensure a packet loss under 2% for the BLE, so I performed a packet counting test between the MCUs and the laptop. The strategy was to have a counter incremented before notifying the central device of incoming data, and conversely have a separate counter to be incremented upon entry to that ESP’s corresponding notification handler. I ran this test for five minutes with the two sticks communicating simultaneously, and by taking the difference of the two counters I came to a packet loss of 2.01 percent (26,815 packets received vs. 27,364 packets sent). This was a surprisingly high data loss for our use case, leading me to confirm that our issues with latency were most likely stemming from the wireless transmission. Looking back at my firmware implementation, the latent drumstick would delay for a few seconds, then output a stream of data like this:

This was ultimately a hint that the ESP was forced to retransmit as well as queue packets across multiple connection intervals. After reading more about the Bluetooth stack, I realized that a fixed connection interval of 7.5ms was too short to allow the central device to schedule events, therefore resulting in packet collisions between the two boards. I also found that sending multiple notifications to the laptop as quickly as possible would overwhelm the event processing queue and cause it to fall behind in timing (similar to our struggles with the audio output). The solution was to raise the connection intervals to 20ms to allow for more schedulability between the devices, and to also raise the notification rates from 1ms up to 21 and 23ms, staggering them to further prevent queue congestion. This led to a much smoother response between the two drumsticks, and the safer approach did not seem to have a noticeable impact on performance.

One skill I’ve picked up while working on this capstone project is quickly reading through online documentation for relevant information. In order to make our Bluetooth, OpenCV and multithreaded audio modules cooperate, I’ve read everything from online tutorials by Nordic Semiconductor, web articles on Geeksforgeeks, and pure datasheets for our microcontrollers while problem solving. I’ve also learned to take as much input as possible from people with experience, such as the teaching staff and faculty, which has made the setbacks we’ve encountered much more manageable.

This week, I plan to help further optimize the system’s CV and Bluetooth processing. The problems we currently face are the HSV lighting inconsistencies along with a dip in performance when drumsticks are in view of the camera. I believe we’re still on track with our schedule, although we may be approaching significant design tradeoff decisions to be able to bring down the response time.

Posted on

Elliot’s Status Report for 11/9

This week was spent working with the rest of the team to bring up a testable prototype for the interim demo. I integrated my bluetooth client code to the system controller in our repository, and together we sorted out file dependencies to get a running program that plays audio upon valid accelerometer spikes. I also worked on planning ahead with the multithreaded code in mind, in which we will need to spawn separate threads for each drumstick–a foreseeable issue in our development is undoubtedly the timing synchronization between the accelerometer readings, computer vision detection, and audio playback, and I plan to meet with Ben and Belle to continue to test their interaction with the shared buffer thoroughly.  Once the speed of the system, especially the CV, is confidently established, I may also update the rate at which the ESP boards notify the laptop with new readings, or even switch to a polling-based implementation.

The other potential concern is the interference on the 2.4GHz band once the second microcontroller is incorporated. In our weekly meetings with Tjun Jet and Professor Bain, we considered utilizing the Wi-Fi capabilities of the ESP32 rather than BLE to ensure adequate throughput and connectivity. With our testing this week, however, it seems that Bluetooth offers an appropriate latency for the needs of the project, and so our reason for choosing Wi-Fi would depend solely on the packet loss behavior of the two sticks running together. There could also be tradeoffs if we choose to redirect the system away from BLE in the form of setup time and ease of pairing, which would potentially neglect our use case requirement for versatility. As such, my plan for this upcoming week is to integrate the second ESP32 board to begin testing with two BLE devices, and to conduct a trade study between Bluetooth and Wi-Fi for our specific use case. I believe the team is on schedule and providing ample time for testing, therefore allowing us to identify important considerations–such as lighting–earlier in the process.

Posted on

Elliot’s Status Report for 11/2

I spent this week cleaning up the system’s Bluetooth module, determining the one-way latency of our wireless data transmission, and establishing a consistent threshold for the incoming accelerometer values on the host device.

To obtain latency metrics, I chose to implement a Round Trip Time (RTT) test. The strategy was to take an initial timestamp on the ESP with the system clock, update the server characteristic and notify the client, wait for a response by observing a change in the server entry, and take the time difference. This came with a few minor issues to be resolved: first, I observed that the characteristic updates were inconsistent and the test resulted in significantly different output values across runs. This was due to the client updating the same buffer as the ESP32 during its response, thus introducing concurrency issues when the devices attempted to update the characteristic simultaneously. I fixed this by separating the transmission and reception to two distinct characteristics, allowing for continuous processing on both sides. Once this was resolved, I noticed that the resulting delay was still too high–around 100ms. After searching online, I came across this article, stating that the default connection interval for the ESP32 ranges from 7.5ms up to as much as 4s: https://docs.espressif.com/projects/esp-idf/en/release-v5.2/esp32c6/api-guides/ble/get-started/ble-connection.html. Having this variance was unacceptable for our purposes, and so I made use of the esp_gap_ble_api library to manually set the maximum connection interval to 20ms. This change greatly reduced the final delay of the test, but having the shorter connection interval means I’ll have to be aware of interference as we integrate a second microcontroller on the 2.4GHz band. The final value of my testing procedure landed our one-way latency at around 40ms, but my belief is that the actual value is even less; this is because of the inherent overhead introduced across the testing code–the operations of looping in the arduino firmware, polling for the client response, and unpacking data all contribute a nonzero latency to the result. Hence, I tested the implementation qualitatively by manually setting a fixed accelerometer threshold and printing over USB on valid spikes. This test produced favorable results, suggesting that the latency could certainly be under 40ms. I was also able to determine an appropriate threshold value for data processing while doing this, which I concluded to be 10 m/s2. This value achieved a reasonable hit detection rate, but we may choose to store multiple thresholds corresponding to different surfaces if the user wishes to play with a uniform actuation force across all surface types. Ultimately, these tests were helpful in our planning towards a low-latency solution, and I believe I’m still on track with the team’s schedule.

In this upcoming week, I plan to move my Bluetooth code into the system controller and assist Ben with audio buffer delay. Specifically, I will:

  1. Create a functional controller to detect accelerometer hits and play specified audio files before introducing CV.
  2. Explore ways to minimize audio output latency as much as possible, such as diving into the PyAudio stack, finding a different library, or considering the MIDI controller route suggested to us by Professor Bain.
Posted on

Elliot’s Status Report for 10/26

My past week was spent working with the MPU6050 chips and cleaning up the Bluetooth module in preparation for integration with the system controller. My goals were to collect data from the 3-axis accelerometers and set up the client notifications to minimize latency. I first soldered the accelerometers and connected the serial clock, data, power, and GND wires, then used the Adafruit MPU6050 libraries to update the firmware. I used the getEvent function to return sensors_event_t data types and also utilized built-in macros to define configuration parameters such as the accelerometer output range, all of which I found from this resource. I packed the three axes of data to one characteristic, and unpacked the content from the server on the client side accordingly. I attached the accelerometer to the drumstick with the Y-axis parallel, and so I averaged the absolute measurements between the X and Z sensors to achieve a desirable output magnitude.

One of the issues I ran into last week was the persistence of the connection, in which the ESP disconnected from the client and was not able to reestablish. I fixed this by adjusting the callback functions to restart advertising automatically following any disconnection. Another potential concern was that I accessed the GATT characteristic by polling manually from the client side, which could add time to our final latency and block relevant processing. If we plan to play our final product at a moderate speed, asynchronous notifications will be required while we evaluate frames for previous hits. Developing the notify behavior brought up a problem, however, namely in the permissions of the BLE service. When I ran start_notify on my laptop, I observed a runtime error saying the attribute could not be written–I eventually realized it was because I had chosen standardized service and characteristic UUIDs with predetermined permission flags. By creating custom UUIDs, I was able to enable notify behavior manually as well as write directly to the characteristic from my laptop.

The write permission I described above is also relevant for the RTT testing I’m currently working on. My strategy is to notify the laptop using an update from the ESP, use a handler in the host code to send an update back, and derive the timestamp offset from start to finish. This, however, is taking longer than expected to achieve an accurate estimate, because having the client access the same buffer as the server introduces concurrency and extraneous latency factors.

I believe I’ve caught up in terms of progress, but I’m aware that the bulk of our team’s difficulty is still ahead in bringing down output delay once we have a functional system. My plan for this upcoming week is to:

  1. Establish a reliable measurement for the one-way latency of our Bluetooth
  2. Begin integrating BLE code with the other modules
  3. Work on the system controller to make more progress towards a testable solution

Posted on

Elliot’s Status Report for 10/19

For this week’s tasks, I put my efforts towards developing the client and server code to transmit accelerometer data over BLE. The firmware for this project’s microcontrollers will be oriented around an Arduino-based framework, providing us access to abstracted libraries for I2C and serial debugging as well as a straightforward IDE to compile, flash, and monitor our code. Because I prefer to work with VS code over the native Arduino platform, I used a third-party extension, PlatformIO, for embedded development on Arduino boards.

I first set up a C++ file to represent the server initialization onboard the ESP32. The code is structured with the standard Arduino setup and loop, with added callback functions declared using the BLEServer library to handle connection status. In initialization, I set the serial baud rate to the UART standard of 115200 in order to allow USB communication to an output monitor. Using this output, I was able to find the MAC address of the microcontroller by printing it with a method from the BLEDevice library. I found that typecasting between the Arduino String type, the C++ std::string, and the C char array was a bit convoluted, which is something I will keep in mind in case we decide to append timestamps with the ESPs rather than the host controller. I then created the generic service for accelerometer data and added a characteristic to store the intended floating point value–the UUIDs used in these two operations were defined globally and found from sections 3.4 and 3.8 of the SIG group’s official identifiers found here:  https://www.bluetooth.com/wp-content/uploads/Files/Specification/HTML/Assigned_Numbers/out/en/Assigned_Numbers.pdf?v=1729378675069. The board then starts advertising and loops on the condition of a valid connection.

Output of MAC address and successful init

I also created the client side code which connects to the device address using bleak. For this stage of development, my goal was to simply get some form of communication between the two devices, and so I opted for a simple polling loop with the asyncio library. I did this by reading straight from the GATT characteristic and unpacking these bytes to a comprehensible float. For future improvements to latency, I plan to use the server to notify the host controller as opposed to the current blocking behavior.  For testing my current program, the loop on the flashed code sets the characteristic to an arbitrary value and increments at a one second interval, which the client then reads directly.

Output of fake data over BLE

This code is a good step forward, but I am a bit behind currently, considering I have not yet soldered the accelerometers to the ESP boards. Moving into the next week, my goal is to lower the latency as much as possible and start incorporating the MPU6050s to get a better idea of what the overall output lag will be. Specifically, this week I will:

  1. Clean up the Bluetooth program to make it modular for integration with Ben and Belle’s work while also ensuring versatility across different devices.
  2. Make an RTT test to get a baseline delay metric for the BLE module.
  3. Connect accelerometers to the ESP32s and start collecting data.
  4. Work on the system controller and the multithreaded code. There will likely be concurrency conflicts even with the system broken into separate threads, meaning that getting an estimate of the delay is our most important objective.
Posted on

Elliot’s Status Report for 10/5

Following the design presentation this past week, I worked on the implementation details for our final design report writeup, where I outline our libraries, equations, and general strategies internally for how we’ll communicate between modules. I spoke with Ben and Belle about how we’d carry out the 30mm buffer zone in our use case requirement, how many frames we would have to process from our sliding window on the event of any given hit, and how the resolution and field of view of our chosen camera would impact the detection of rings on the table. Hence, given that we were not able to successfully place an order for a web camera off Amazon, I had the opportunity to search for a suitable camera with our new priorities being the ability to process 20 relevant frames from our frame window while also avoiding the optical distortion resulting from a high field of view. I found a 1080p, 60FPS camera with an 84 degree field of view in the Elgato MK.2, which we’ll be considering alongside other options within the next few days; the most crucial requirements were the framerate, where I decided that a web camera running at 60 frames per second should allow us to gather 20 frames of relevant imaging (up to 0.33 seconds pre-timestamp), and the field of view, where the team concluded that anything higher than 90 degrees could distort our pixel-based sizing calculations. Apart from exploring our hardware needs, I finalized the connectivity of our BLE, since the online simulation I used only operated up to the advertising stage and wasn’t able to emulate over-the-air pairing. The host device code should be simple, where we’ll run two threads pairing with separate MAC addresses for the microcontrollers and subscribing to the UUIDs of the accelerometer characteristics, although I’m waiting for parts to arrive for testing. Overall, myself and the team are on schedule, and we should be well prepared for bug-fixing and unit testing post-break. This week, I personally plan to:

  1. Work hands-on with the ESP32 and MPU-6050. I plan to solder the jumper cables between the two for the I2C serial communication and flash firmware code to start advertising over bluetooth.
  2. Finalize our design report. I’m looking to complete my bluetooth testing by Thursday, from which point I’ll incorporate it into the repository and describe the technical details within the implementation section of our writeup.
Posted on

Elliot’s Status Report for 9/28

This week, I was tasked with establishing a basis for communicating with the ESP32 boards and their corresponding MPU-6050 accelerometers. I spent some time looking into the BLE stack to determine the complexity needed for our GATT services and characteristics–the available options were the native ESP IoT Dev Framework, the Arduino IDE, and a MicroPython-based firmware. I concluded that while the ESP-IDF would give us the most control over the pipeline we implement, since our main purpose is to simply transmit the accelerometer data and its timestamp, the service complexity does not call for any fine tuning. Between the Arduino framework and MicroPython, it would be be best to use a compiled language rather than an interpreted one for the purpose of lower latency. Therefore, I started developing some of the C++ code we’ll eventually be flashing to our microcontrollers; to test functionality, I worked on an ESP simulator on Wokwi to set up a bluetooth connection and send accelerometer data to notified clients. Some libraries necessary for the arduino framework include Wire.h for I2C, BLEUtils for initializing the advertising and notifications, and the MPU6050 device driver.

I also practiced my presentation approach for Monday, where I’ll be talking about how our bluetooth, computer vision, and web server modules interact. I emphasized time spent on identifying why we chose the components we chose, as well as developing concrete requirements to link back to our product use case requirements.

For next week, my plan is to:

  1. Hopefully receive our hardware and begin testing the accelerometer thresholds. I’ll set up communication over bluetooth to relay data, and based on our drum hits, we’ll then look at what signal spikes would indicate an adequate hit.
  2. Test bluetooth latency and packet loss. Once we have the ESP32’s wired up, I can insert packet misses to determine adequate rates, as well as measure the latency of our transmissions with timestamps. This is especially important to us, given the fact that we’ll be using two transmission devices in a low latency environment.
Posted on

Elliot’s Status Report for 9/21

This past week, I helped in finalizing the parts list we would request for purchase and for reservation from existing inventory. I spoke with Professor Mukherjee and our team’s TA, Tjun Jet, about some of the components and their purposes. I also worked on the slide deck for the upcoming design presentation and covered a few solutions regarding the interconnectivity of our components–I began researching how to interface with the accelerometers using the ESP32 MCUs as well as the bluetooth stack we’ll be using to relay data back to the host device. I also helped establish our fallback plans in case any given module from our block diagram does not work as expected. Overall, the team is currently aligned with the schedule we laid out in our Gantt chart, but there is still a considerable amount of research needed before I can confidently outline our technical design to the class. This upcoming week, I plan to do the following in preparation for the design submission:

  1. Look into our options for BLE abstraction libraries to easily communicate with the microcontroller. My hope is that Python will have an existing API available for the ESP32, but if not, I am prepared to read the documentation for our device’s Bluetooth module and initialize the advertising, connection, and packet transactions manually.
  2. Similarly, I need to find a way to accept accelerometer data through the MCU. I looked at a few datasheets for the ESP, and it doesn’t use the Cortex M-series microprocessors I’m most familiar with, so I’m not sure if I’ll have to manually write a device driver for the I2C peripheral to communicate with the MPU-6050. Again, hopefully Python has some level of abstraction available for us to use.
  3. Belle will be mostly working on the repository code for the OpenCV, but I plan to also help with the CV code so that I can get a full understanding of our algorithm’s capabilities before we actually enter testing. It will also help me familiarize myself with the repo we’ll be working in, where we plan to put together Ben’s front and backend implementations, the CV code, and the Bluetooth interface for collecting accelerometer data. 
  4. Prepare slides and practice verbal delivery. Once all of the above details are established, I can describe our technical strategies in greater detail and identify appropriate visuals to include in order to communicate our project vision clearly.