Posted on

Elliot’s Status Report for 12/7

This week was spent improving and testing the Bluetooth code to verify its performance. Following last week’s changes, there appeared to be some lingering interference between the two sticks (~100ms delay) when playing together. Paired with the CV processing, I concluded that this was not acceptable latency, so I changed the firmware and host code to run BT Classic rather than BLE. The main changes involved identifying serial ports for each of our ESPs as well as advertising and sending data as a byte array with the BluetoothSerial library rather than notifying with BLEdevice. The laptop code was relatively simple to implement, considering I only needed to work with serial input similar to UART. Following these changes, I set up a test to verify that the interference between the two sticks was lowered; I used matplotlib to take timestamps between when the accelerometer data was first received and when the sound output played, and I overlayed the delays of the two sticks to compare their relative performances.

Test Results (<6ms delay for both drumsticks)

I’m satisfied with these results, since this was our most persistent issue and both sticks are now operating at similar performance. It also gives us an opportunity to highlight the design tradeoffs between lower power consumption and reduced interference on the 2.4GHz band for BLE and BT Classic, respectively. I believe our team is on track to have a fully working project before the demo date, and I plan to spend this week extensively testing the system in various environments and conditions to locate any edge-case bugs.

Posted on

Elliot’s Status Report for 11/30

This week I worked more on the system’s Bluetooth component and gathered verification metrics in preparation for our final presentation. One of the team’s design requirements is to ensure a packet loss under 2% for the BLE, so I performed a packet counting test between the MCUs and the laptop. The strategy was to have a counter incremented before notifying the central device of incoming data, and conversely have a separate counter to be incremented upon entry to that ESP’s corresponding notification handler. I ran this test for five minutes with the two sticks communicating simultaneously, and by taking the difference of the two counters I came to a packet loss of 2.01 percent (26,815 packets received vs. 27,364 packets sent). This was a surprisingly high data loss for our use case, leading me to confirm that our issues with latency were most likely stemming from the wireless transmission. Looking back at my firmware implementation, the latent drumstick would delay for a few seconds, then output a stream of data like this:

This was ultimately a hint that the ESP was forced to retransmit as well as queue packets across multiple connection intervals. After reading more about the Bluetooth stack, I realized that a fixed connection interval of 7.5ms was too short to allow the central device to schedule events, therefore resulting in packet collisions between the two boards. I also found that sending multiple notifications to the laptop as quickly as possible would overwhelm the event processing queue and cause it to fall behind in timing (similar to our struggles with the audio output). The solution was to raise the connection intervals to 20ms to allow for more schedulability between the devices, and to also raise the notification rates from 1ms up to 21 and 23ms, staggering them to further prevent queue congestion. This led to a much smoother response between the two drumsticks, and the safer approach did not seem to have a noticeable impact on performance.

One skill I’ve picked up while working on this capstone project is quickly reading through online documentation for relevant information. In order to make our Bluetooth, OpenCV and multithreaded audio modules cooperate, I’ve read everything from online tutorials by Nordic Semiconductor, web articles on Geeksforgeeks, and pure datasheets for our microcontrollers while problem solving. I’ve also learned to take as much input as possible from people with experience, such as the teaching staff and faculty, which has made the setbacks we’ve encountered much more manageable.

This week, I plan to help further optimize the system’s CV and Bluetooth processing. The problems we currently face are the HSV lighting inconsistencies along with a dip in performance when drumsticks are in view of the camera. I believe we’re still on track with our schedule, although we may be approaching significant design tradeoff decisions to be able to bring down the response time.

Posted on

Elliot’s Status Report for 11/16

This week I mainly worked towards incorporating the second BLE drumstick to the system controller and fine-tuning our accelerometer thresholds upon integration with the rest of the code. After wiring the second ESP and accelerometer, I used serial output to find the new MAC address, and I dedicated the laptop’s connection functionality across separate threads for each of the boards. Once I was able to reliably connect to both of the sticks, I met with my group to test the new configuration. At that point, Ben had already spent some time deriving the required behavior for our MPU6050 output waveforms, but we hadn’t yet achieved an appropriate method for distinguishing hits on the table from swinging drumsticks in the air. After testing across various surfaces and static thresholds, I noticed that taking the magnitude of our data readings was not sufficient for identifying hit patterns–the sequence of high-magnitude accelerometer output to low-magnitude was too general to characterize a rebound off the playing surface, and instead triggered a high frequency of false readings while idle. I modified the notification handlers in my bluetooth code to attach the sign of the z-axis to our incoming data, thereby allowing us to identify upward and downward swings independently, and using the graph from our prior testing I was able to set up a sequence for detecting valid spikes. By polling for a sufficient negative reading, blocking with a delay, and then taking a subsequent reading, we were able to correctly identify downward hits given the two-point difference. One issue we found, however, was that while the two sticks operated well separately, one of them would suffer from a noticeable output delay when playing simultaneously. An important point to consider was that for any run, whichever ESP made the connection to the laptop second was the one that experienced the performance drop. This problem could be a result of the bluetooth connection or it could be a flaw in our threading architecture; I am hesitant to blame our BLE connection interval simply due to the fact that both microcontrollers are running the same firmware code, but I plan to resolve this issue within the next few days. Overall, I believe my progress and our team’s progress is on track, and this upcoming week I plan to meet with the team to flesh out the system’s HSV/lighting concerns as well as continue to test the CV module extensively.

Posted on

Team Status Report for 11/16

This week our team focused on implementing the second Bluetooth drumstick as well as a reliable hit detection mechanism. This led to a few changes in our system controller–first, we refactored the callback functions in the BLE module to more accurately characterize the accelerometer readings, and we also modified the ESP threads to measure hits based off changes in acceleration rather than a static threshold. One risk to the system we are currently facing is the increased latency of the drumsticks when operating simultaneously; our mitigation plan is to explore other threading libraries such as multiprocessing or to research additional functionality in the asyncio module to better handle concurrent execution. In regards to the lighting concerns brought up in last week’s status report, we are in the process of testing with additional overhead lights, which appears to be an effective mitigation strategy for ensuring consistent output.

Our team has run a number of tests thus far in the development process: while much of our system testing is qualitative (measuring audio buffer latency being impractical with timestamps), some of our formal testing includes round-trip time for the BLE, accelerometer waveform generation with the MPU6050’s, and CV frame-by-frame  delay measurements. Additionally, as we move into a fully functional deliverable this week, we plan to conduct an end-to-end latency measurement between the accelerometer spike and the sound playback, and we will also validate our 30mm use case requirement by ensuring that the perimeters for our each of our drum pads remain sensitive to input, even across varying environments, lighting, and drum configurations.

Figure 1. MPU 6050 hit sequence (prior to z-axis signed magnitude changes)

The results of our tests will be compared directly to the use case and design requirements we set out to fulfill, and our goal of diversity and inclusion will be achieved through rigorous testing to reach a comfortable user interface. We do not have new changes in our schedule to report, and we intend to approach a viable product in the coming weeks.

Posted on

Elliot’s Status Report for 11/9

This week was spent working with the rest of the team to bring up a testable prototype for the interim demo. I integrated my bluetooth client code to the system controller in our repository, and together we sorted out file dependencies to get a running program that plays audio upon valid accelerometer spikes. I also worked on planning ahead with the multithreaded code in mind, in which we will need to spawn separate threads for each drumstick–a foreseeable issue in our development is undoubtedly the timing synchronization between the accelerometer readings, computer vision detection, and audio playback, and I plan to meet with Ben and Belle to continue to test their interaction with the shared buffer thoroughly.  Once the speed of the system, especially the CV, is confidently established, I may also update the rate at which the ESP boards notify the laptop with new readings, or even switch to a polling-based implementation.

The other potential concern is the interference on the 2.4GHz band once the second microcontroller is incorporated. In our weekly meetings with Tjun Jet and Professor Bain, we considered utilizing the Wi-Fi capabilities of the ESP32 rather than BLE to ensure adequate throughput and connectivity. With our testing this week, however, it seems that Bluetooth offers an appropriate latency for the needs of the project, and so our reason for choosing Wi-Fi would depend solely on the packet loss behavior of the two sticks running together. There could also be tradeoffs if we choose to redirect the system away from BLE in the form of setup time and ease of pairing, which would potentially neglect our use case requirement for versatility. As such, my plan for this upcoming week is to integrate the second ESP32 board to begin testing with two BLE devices, and to conduct a trade study between Bluetooth and Wi-Fi for our specific use case. I believe the team is on schedule and providing ample time for testing, therefore allowing us to identify important considerations–such as lighting–earlier in the process.

Posted on

Elliot’s Status Report for 11/2

I spent this week cleaning up the system’s Bluetooth module, determining the one-way latency of our wireless data transmission, and establishing a consistent threshold for the incoming accelerometer values on the host device.

To obtain latency metrics, I chose to implement a Round Trip Time (RTT) test. The strategy was to take an initial timestamp on the ESP with the system clock, update the server characteristic and notify the client, wait for a response by observing a change in the server entry, and take the time difference. This came with a few minor issues to be resolved: first, I observed that the characteristic updates were inconsistent and the test resulted in significantly different output values across runs. This was due to the client updating the same buffer as the ESP32 during its response, thus introducing concurrency issues when the devices attempted to update the characteristic simultaneously. I fixed this by separating the transmission and reception to two distinct characteristics, allowing for continuous processing on both sides. Once this was resolved, I noticed that the resulting delay was still too high–around 100ms. After searching online, I came across this article, stating that the default connection interval for the ESP32 ranges from 7.5ms up to as much as 4s: https://docs.espressif.com/projects/esp-idf/en/release-v5.2/esp32c6/api-guides/ble/get-started/ble-connection.html. Having this variance was unacceptable for our purposes, and so I made use of the esp_gap_ble_api library to manually set the maximum connection interval to 20ms. This change greatly reduced the final delay of the test, but having the shorter connection interval means I’ll have to be aware of interference as we integrate a second microcontroller on the 2.4GHz band. The final value of my testing procedure landed our one-way latency at around 40ms, but my belief is that the actual value is even less; this is because of the inherent overhead introduced across the testing code–the operations of looping in the arduino firmware, polling for the client response, and unpacking data all contribute a nonzero latency to the result. Hence, I tested the implementation qualitatively by manually setting a fixed accelerometer threshold and printing over USB on valid spikes. This test produced favorable results, suggesting that the latency could certainly be under 40ms. I was also able to determine an appropriate threshold value for data processing while doing this, which I concluded to be 10 m/s2. This value achieved a reasonable hit detection rate, but we may choose to store multiple thresholds corresponding to different surfaces if the user wishes to play with a uniform actuation force across all surface types. Ultimately, these tests were helpful in our planning towards a low-latency solution, and I believe I’m still on track with the team’s schedule.

In this upcoming week, I plan to move my Bluetooth code into the system controller and assist Ben with audio buffer delay. Specifically, I will:

  1. Create a functional controller to detect accelerometer hits and play specified audio files before introducing CV.
  2. Explore ways to minimize audio output latency as much as possible, such as diving into the PyAudio stack, finding a different library, or considering the MIDI controller route suggested to us by Professor Bain.
Posted on

Elliot’s Status Report for 10/26

My past week was spent working with the MPU6050 chips and cleaning up the Bluetooth module in preparation for integration with the system controller. My goals were to collect data from the 3-axis accelerometers and set up the client notifications to minimize latency. I first soldered the accelerometers and connected the serial clock, data, power, and GND wires, then used the Adafruit MPU6050 libraries to update the firmware. I used the getEvent function to return sensors_event_t data types and also utilized built-in macros to define configuration parameters such as the accelerometer output range, all of which I found from this resource. I packed the three axes of data to one characteristic, and unpacked the content from the server on the client side accordingly. I attached the accelerometer to the drumstick with the Y-axis parallel, and so I averaged the absolute measurements between the X and Z sensors to achieve a desirable output magnitude.

One of the issues I ran into last week was the persistence of the connection, in which the ESP disconnected from the client and was not able to reestablish. I fixed this by adjusting the callback functions to restart advertising automatically following any disconnection. Another potential concern was that I accessed the GATT characteristic by polling manually from the client side, which could add time to our final latency and block relevant processing. If we plan to play our final product at a moderate speed, asynchronous notifications will be required while we evaluate frames for previous hits. Developing the notify behavior brought up a problem, however, namely in the permissions of the BLE service. When I ran start_notify on my laptop, I observed a runtime error saying the attribute could not be written–I eventually realized it was because I had chosen standardized service and characteristic UUIDs with predetermined permission flags. By creating custom UUIDs, I was able to enable notify behavior manually as well as write directly to the characteristic from my laptop.

The write permission I described above is also relevant for the RTT testing I’m currently working on. My strategy is to notify the laptop using an update from the ESP, use a handler in the host code to send an update back, and derive the timestamp offset from start to finish. This, however, is taking longer than expected to achieve an accurate estimate, because having the client access the same buffer as the server introduces concurrency and extraneous latency factors.

I believe I’ve caught up in terms of progress, but I’m aware that the bulk of our team’s difficulty is still ahead in bringing down output delay once we have a functional system. My plan for this upcoming week is to:

  1. Establish a reliable measurement for the one-way latency of our Bluetooth
  2. Begin integrating BLE code with the other modules
  3. Work on the system controller to make more progress towards a testable solution

Posted on

Elliot’s Status Report for 10/19

For this week’s tasks, I put my efforts towards developing the client and server code to transmit accelerometer data over BLE. The firmware for this project’s microcontrollers will be oriented around an Arduino-based framework, providing us access to abstracted libraries for I2C and serial debugging as well as a straightforward IDE to compile, flash, and monitor our code. Because I prefer to work with VS code over the native Arduino platform, I used a third-party extension, PlatformIO, for embedded development on Arduino boards.

I first set up a C++ file to represent the server initialization onboard the ESP32. The code is structured with the standard Arduino setup and loop, with added callback functions declared using the BLEServer library to handle connection status. In initialization, I set the serial baud rate to the UART standard of 115200 in order to allow USB communication to an output monitor. Using this output, I was able to find the MAC address of the microcontroller by printing it with a method from the BLEDevice library. I found that typecasting between the Arduino String type, the C++ std::string, and the C char array was a bit convoluted, which is something I will keep in mind in case we decide to append timestamps with the ESPs rather than the host controller. I then created the generic service for accelerometer data and added a characteristic to store the intended floating point value–the UUIDs used in these two operations were defined globally and found from sections 3.4 and 3.8 of the SIG group’s official identifiers found here:  https://www.bluetooth.com/wp-content/uploads/Files/Specification/HTML/Assigned_Numbers/out/en/Assigned_Numbers.pdf?v=1729378675069. The board then starts advertising and loops on the condition of a valid connection.

Output of MAC address and successful init

I also created the client side code which connects to the device address using bleak. For this stage of development, my goal was to simply get some form of communication between the two devices, and so I opted for a simple polling loop with the asyncio library. I did this by reading straight from the GATT characteristic and unpacking these bytes to a comprehensible float. For future improvements to latency, I plan to use the server to notify the host controller as opposed to the current blocking behavior.  For testing my current program, the loop on the flashed code sets the characteristic to an arbitrary value and increments at a one second interval, which the client then reads directly.

Output of fake data over BLE

This code is a good step forward, but I am a bit behind currently, considering I have not yet soldered the accelerometers to the ESP boards. Moving into the next week, my goal is to lower the latency as much as possible and start incorporating the MPU6050s to get a better idea of what the overall output lag will be. Specifically, this week I will:

  1. Clean up the Bluetooth program to make it modular for integration with Ben and Belle’s work while also ensuring versatility across different devices.
  2. Make an RTT test to get a baseline delay metric for the BLE module.
  3. Connect accelerometers to the ESP32s and start collecting data.
  4. Work on the system controller and the multithreaded code. There will likely be concurrency conflicts even with the system broken into separate threads, meaning that getting an estimate of the delay is our most important objective.
Posted on

Elliot’s Status Report for 10/5

Following the design presentation this past week, I worked on the implementation details for our final design report writeup, where I outline our libraries, equations, and general strategies internally for how we’ll communicate between modules. I spoke with Ben and Belle about how we’d carry out the 30mm buffer zone in our use case requirement, how many frames we would have to process from our sliding window on the event of any given hit, and how the resolution and field of view of our chosen camera would impact the detection of rings on the table. Hence, given that we were not able to successfully place an order for a web camera off Amazon, I had the opportunity to search for a suitable camera with our new priorities being the ability to process 20 relevant frames from our frame window while also avoiding the optical distortion resulting from a high field of view. I found a 1080p, 60FPS camera with an 84 degree field of view in the Elgato MK.2, which we’ll be considering alongside other options within the next few days; the most crucial requirements were the framerate, where I decided that a web camera running at 60 frames per second should allow us to gather 20 frames of relevant imaging (up to 0.33 seconds pre-timestamp), and the field of view, where the team concluded that anything higher than 90 degrees could distort our pixel-based sizing calculations. Apart from exploring our hardware needs, I finalized the connectivity of our BLE, since the online simulation I used only operated up to the advertising stage and wasn’t able to emulate over-the-air pairing. The host device code should be simple, where we’ll run two threads pairing with separate MAC addresses for the microcontrollers and subscribing to the UUIDs of the accelerometer characteristics, although I’m waiting for parts to arrive for testing. Overall, myself and the team are on schedule, and we should be well prepared for bug-fixing and unit testing post-break. This week, I personally plan to:

  1. Work hands-on with the ESP32 and MPU-6050. I plan to solder the jumper cables between the two for the I2C serial communication and flash firmware code to start advertising over bluetooth.
  2. Finalize our design report. I’m looking to complete my bluetooth testing by Thursday, from which point I’ll incorporate it into the repository and describe the technical details within the implementation section of our writeup.
Posted on

Team Status Report for 10/5

For this week, our team worked mainly on the writeup for our design report to fully plan out our final product. We took the time to tackle a few edge cases from our initial blueprint, specifically focusing on the more nuanced details of our design requirements and implementation strategies so that we can better explain our architecture to any reader of the design report. Our schedule remains the same, with Ben developing the web application, Elliot handling the Bluetooth data processing, and Belle covering the computer vision computation onboard the host; we chose, however, to split this week’s stage of our design process differently, with each member focusing on a specific section of the report. We delegated the introduction and requirements to Ben, the architecture and implementation to Elliot, and the testing and tradeoffs to Belle. We decided that this would result in a more well-rounded final product by giving each team member an opportunity to view the project from a holistic perspective before we begin to integrate our modules together. Having each team member dive into other components of the block diagram brought up a few potential concerns we hadn’t considered prior, each of which we then created a mitigation plan for. Some details we worked out this week included the following:

1.) 30mm scalability requirement: As outlined in our proposal and design presentations, one of our use case requirements is to provide the user a 30mm error zone to account for the rubber drumheads deviating from their original position upon impact from the drumsticks. The design requirement we mapped to it for traceability involved deriving a fixed scaling factor to apply to the gathered radii upon detection with the HoughCircles library.  We realized, however, that a single scaling factor across all four drums would not achieve a constant 30mm margin for each drum (as they differ in size), and that the relative diameters in pixels between the drumheads would not be sufficient to determine a scaling factor (an absolute metric is required if our solution is to be applicable for varying camera heights). Hence, our new implementation is to store the absolute sizes of each ring within an internal array and scale based on these known sizes. We can then detect the rings based on their relative sizes, map them to their stored dimensions, and apply a simple separate scaling factor to the radii accordingly. This will prove to be a less error-prone approach as opposed to a purely relative solution where we may have encountered issues if the user did not place all rings in view of the camera, or if the camera was too far from the table to detect small variances in the diameters.

2.) Reliability of BLE packet transmission: Another one of our use case requirements was to ensure a reliable connection within 3m of the laptop, for which we decided to aim for a packet loss of under 2%. Given our original research on the Bluetooth stack and the specifications for the ESP32’s performance, we figured that 2% would be a very reasonable goal. With the second microcontroller also transmitting accelerometer data, however, we run the risk of interference and packet loss, for which we had not developed a mitigation plan. This week, Ben searched for options to lower the packet loss in the event that we do not meet this requirement, eventually landing on the solution of raising the connection interval. Elliot then explored the firmware libraries available in Arduino and confirmed our ability to increase the connection interval with the host device at the cost of over-the-air latency.

3.) Audio output delay: One element we completely overlooked was the main thread’s method of playing audio files, for which we chose to use the pygame mixer. This week, however, our team discovered that this library introduces an unacceptable amount of output latency–we decided to pivot to the use of PyAudio, which is optimized with smaller audio buffers to achieve a much lower processing delay.

4.) Camera specifications: This week, while exploring strategies to most efficiently deploy our computer vision model, we evaluated the effect that a 120 degree field of view camera would have on our CV calculations. We found that wide angle cameras could potentially introduce a form of optical distortion, resulting in stretched pixels and slightly elliptical drumheads, and therefore less precise detection altogether under our framework. We also came to a decision regarding our sliding window, where we chose to now take 0.33 seconds worth of frames before the relevant accelerometer timestamp, since anything higher could lead to potentially false readings. Given these new requirements, we set out to find a high framerate, approximately 90 degree FOV camera, for which we plan to make an order early next week. Below is a diagram we created to help us map out how we’ll use this new field of view:

Next week we plan to stay on schedule and begin working with the physical components we ordered. By Friday, we intend to have a complete report for describing our requirements, strategies, and conscious design decisions in creating our CV-based drumset.