Posted on

Elliot’s Status Report for 11/30

This week I worked more on the system’s Bluetooth component and gathered verification metrics in preparation for our final presentation. One of the team’s design requirements is to ensure a packet loss under 2% for the BLE, so I performed a packet counting test between the MCUs and the laptop. The strategy was to have a counter incremented before notifying the central device of incoming data, and conversely have a separate counter to be incremented upon entry to that ESP’s corresponding notification handler. I ran this test for five minutes with the two sticks communicating simultaneously, and by taking the difference of the two counters I came to a packet loss of 2.01 percent (26,815 packets received vs. 27,364 packets sent). This was a surprisingly high data loss for our use case, leading me to confirm that our issues with latency were most likely stemming from the wireless transmission. Looking back at my firmware implementation, the latent drumstick would delay for a few seconds, then output a stream of data like this:

This was ultimately a hint that the ESP was forced to retransmit as well as queue packets across multiple connection intervals. After reading more about the Bluetooth stack, I realized that a fixed connection interval of 7.5ms was too short to allow the central device to schedule events, therefore resulting in packet collisions between the two boards. I also found that sending multiple notifications to the laptop as quickly as possible would overwhelm the event processing queue and cause it to fall behind in timing (similar to our struggles with the audio output). The solution was to raise the connection intervals to 20ms to allow for more schedulability between the devices, and to also raise the notification rates from 1ms up to 21 and 23ms, staggering them to further prevent queue congestion. This led to a much smoother response between the two drumsticks, and the safer approach did not seem to have a noticeable impact on performance.

One skill I’ve picked up while working on this capstone project is quickly reading through online documentation for relevant information. In order to make our Bluetooth, OpenCV and multithreaded audio modules cooperate, I’ve read everything from online tutorials by Nordic Semiconductor, web articles on Geeksforgeeks, and pure datasheets for our microcontrollers while problem solving. I’ve also learned to take as much input as possible from people with experience, such as the teaching staff and faculty, which has made the setbacks we’ve encountered much more manageable.

This week, I plan to help further optimize the system’s CV and Bluetooth processing. The problems we currently face are the HSV lighting inconsistencies along with a dip in performance when drumsticks are in view of the camera. I believe we’re still on track with our schedule, although we may be approaching significant design tradeoff decisions to be able to bring down the response time.

Posted on

Belle’s Status Report for 11/30

This week, I focused on preparing the slides for the final presentation – incorporating the results from our verification and validation tests – and contributed to the drumstick detection portion of our project.

The former involved organizing and presenting the data in a way that highlights how our project meets its use case and design requirements, as well as practicing the general flow of how I would present the relevant tests (since there are many of them but there is not much time allotted for each presentation, so I have to be concise).

As for the drumstick detection, one key aspect of our design was the use of exponential weighting to account for latency when the video frame taken at the moment of an accelerometer impact did not reflect the correct position of the drumstick tip (i.e., it would show the drumstick tip as being in the previous drum’s boundary, rather than the drum that was actually hit). This was particularly a concern because of the potential delay between the moment of impact and the processing of the frame, as we were not sure what said latency would look like.

However, during further testing, we found that this issue was quite rare. The camera’s FPS was sufficiently high, and the CV processing latency was small enough that frames typically matched up with the correct impact timing. As a result, we found that exponential weighting was unnecessary for most scenarios. Additionally, the mutexes required to protect the buffer used for the calculation were introducing unnecessary and unwanted latency. In order to simplify the system and improve overall responsiveness, we scrapped the buffer and exponential weighting completely, which led to a noticeable reduction in latency and slightly smoother performance in general.

Previously, we also found a way to have the user tweak the hsv values themselves using several sliders and a visualizer and changed one of the drumstick tips from blue to red, so the relevant issues were solved. As a result, I feel as though the drumstick detection portion of the project is mostly done.

According to our gantt chart, I should still be working with Elliot and Ben to to integrate all of our individual components of our project, so I believe I am on track. Therefore, next steps include finalizing preparations for the presentation and continuing to troubleshoot the Bluetooth latency discrepancy between the drumsticks.

Posted on

Team Status Report for 11/30

This week, our team mainly focused on implementing several validation and verification tests for our project, as well as finalizing our slides for next week’s Final Presentation.

Validation Tests

  • Drumstick Weight: Both drumsticks weigh 147g, well under the 190.4g limit.
  • Minimum Layout: The drum setup achieved a layout area of 1638.3cm², close to (but still under) the required 1644cm².
  • Drum Ring Detection: A 15mm margin was implemented to reduce overlapping issues, successfully scaling drum pad radii.
  • Reliable BLE Connection: At a 3m distance, all impacts were detected with no packet loss.
  • Correct Sound Playback: The system achieved an 89% accuracy for correct drum sound playback. This slightly missed the 90% target and will thus require refinement.
  • Audio Response: Average latency between drum impact and sound playback was 94.61ms, meeting the 100ms limit (despite a notable outlier).

Verification Tests

  • Connective Components: The drumsticks’ wires and tape weighed only 5g per drumstick, far below the 14.45g limit.
  • Latency: Measured latencies include:
    • BLE transmission: 0.088ms (RTT/2) – under 30ms requirement
    • CV processing: 33.2ms per frame – under 60ms requirement
    • Accelerometer processing: 5ms – under 5ms requirement

We were happy with the majority of these results, as they prove that we were indeed able to meet the constraints that we initially placed on ourselves for this project.

We still are facing a few challenges, however:  we have found that when both drumsticks are connected via Bluetooth, one often experiences noticeably higher latency than the other. The root cause is unclear and under investigation, so resolving this issue is our current/next major priority.

Nonetheless, we have incorporated the results of these tests into our presentation slides and will continue working to resolve the Bluetooth latency issue.

Posted on

Belle’s Status Report for 11/16

This week, I mostly worked with Ben and Elliot to continue integrating & fine-tuning various components of DrumLite to prepare for the Interim Demo happening this upcoming week.

In particular, my main contribution focused on fine-tuning the accelerometer readings. To refine our accelerometer threshold values, we utilized Matplotlib to continuously plot accelerometer data in real-time during testing. In these plots, the x-value represented time, and the y-value represented the average of the x and z components of the accelerometer output. This visualization helped us identify a distinct pattern: each drumstick hit produced a noticeable upward spike, followed by a downward spike in the accelerometer readings (as per the sample output screenshot below, which was created after hitting a machined drumstick on a drum pad four times):

Initially, we attempted to detect these hits by capturing the “high” value, followed by the “low” value. However, upon further analysis, we determined that simply calculating the difference between the two values would be sufficient for reliable detection. To implement this, we introduced a short delay of 1ms between sampling, which allowed us to consistently measure the low-high difference. Additionally, we decided to incorporated the sign of the z-component of the accelerometer output rather than taking its absolute value. This helped us better account for behaviors such as upward flicks of the wrist, which were sometimes mistakenly identified as downward drumstick hits (and were therefore incorrectly triggering a drum sound to be played). Thus, we were able to filter out other similar movements that weren’t downward drumstick swipes onto the drum pad/a solid surface, further refining the precision and reliability of our hit detection logic.

To address lighting inconsistencies from previous tests, we acquired another lamp, ensuring the testing desk is now fully illuminated. This adjustment will significantly improved the consistency of our drumstick tip detection, reducing the impact of shadows and uneven lighting. While we are still in the process of testing this 2-lamp setup, I currently believe using a YOLO/SSD model for object detection is unnecessary. These models are great for complex environments with many objects, but the simplicity of our current setup — with (mostly) controlled lighting and focused object tracking — is key. Also, implementing YOLO/SSD models would introduce significant computational overhead, which we aim to avoid given our desired sub-100ms-latency use case requirement. Therefore, I would prefer for this to remain as a last-resort solution to the lighting issue.

As per our timeline, since we should be fine-tuning/integrating different project components and are essentially done setting the accelerometer threshold values, we are indeed on track. Currently, specifically picking an HSV value for each drumstick is a bit cumbersome and unpredictable, especially in areas with a large amount of ambient lighting. Therefore, next week,  I aim to further test drumstick tip detection under varying lighting conditions and try to simplify the aforementioned process, as I believe it is the least-solid aspect of our implementation at the moment. 

Posted on

Elliot’s Status Report for 11/16

This week I mainly worked towards incorporating the second BLE drumstick to the system controller and fine-tuning our accelerometer thresholds upon integration with the rest of the code. After wiring the second ESP and accelerometer, I used serial output to find the new MAC address, and I dedicated the laptop’s connection functionality across separate threads for each of the boards. Once I was able to reliably connect to both of the sticks, I met with my group to test the new configuration. At that point, Ben had already spent some time deriving the required behavior for our MPU6050 output waveforms, but we hadn’t yet achieved an appropriate method for distinguishing hits on the table from swinging drumsticks in the air. After testing across various surfaces and static thresholds, I noticed that taking the magnitude of our data readings was not sufficient for identifying hit patterns–the sequence of high-magnitude accelerometer output to low-magnitude was too general to characterize a rebound off the playing surface, and instead triggered a high frequency of false readings while idle. I modified the notification handlers in my bluetooth code to attach the sign of the z-axis to our incoming data, thereby allowing us to identify upward and downward swings independently, and using the graph from our prior testing I was able to set up a sequence for detecting valid spikes. By polling for a sufficient negative reading, blocking with a delay, and then taking a subsequent reading, we were able to correctly identify downward hits given the two-point difference. One issue we found, however, was that while the two sticks operated well separately, one of them would suffer from a noticeable output delay when playing simultaneously. An important point to consider was that for any run, whichever ESP made the connection to the laptop second was the one that experienced the performance drop. This problem could be a result of the bluetooth connection or it could be a flaw in our threading architecture; I am hesitant to blame our BLE connection interval simply due to the fact that both microcontrollers are running the same firmware code, but I plan to resolve this issue within the next few days. Overall, I believe my progress and our team’s progress is on track, and this upcoming week I plan to meet with the team to flesh out the system’s HSV/lighting concerns as well as continue to test the CV module extensively.

Posted on

Team Status Report for 11/16

This week our team focused on implementing the second Bluetooth drumstick as well as a reliable hit detection mechanism. This led to a few changes in our system controller–first, we refactored the callback functions in the BLE module to more accurately characterize the accelerometer readings, and we also modified the ESP threads to measure hits based off changes in acceleration rather than a static threshold. One risk to the system we are currently facing is the increased latency of the drumsticks when operating simultaneously; our mitigation plan is to explore other threading libraries such as multiprocessing or to research additional functionality in the asyncio module to better handle concurrent execution. In regards to the lighting concerns brought up in last week’s status report, we are in the process of testing with additional overhead lights, which appears to be an effective mitigation strategy for ensuring consistent output.

Our team has run a number of tests thus far in the development process: while much of our system testing is qualitative (measuring audio buffer latency being impractical with timestamps), some of our formal testing includes round-trip time for the BLE, accelerometer waveform generation with the MPU6050’s, and CV frame-by-frame  delay measurements. Additionally, as we move into a fully functional deliverable this week, we plan to conduct an end-to-end latency measurement between the accelerometer spike and the sound playback, and we will also validate our 30mm use case requirement by ensuring that the perimeters for our each of our drum pads remain sensitive to input, even across varying environments, lighting, and drum configurations.

Figure 1. MPU 6050 hit sequence (prior to z-axis signed magnitude changes)

The results of our tests will be compared directly to the use case and design requirements we set out to fulfill, and our goal of diversity and inclusion will be achieved through rigorous testing to reach a comfortable user interface. We do not have new changes in our schedule to report, and we intend to approach a viable product in the coming weeks.

Posted on

Ben Solo’s Status Report for 11/16

This week I worked on determining better threshold values for what qualifies as an impact. Prior to this week we had a functional system where the accelerometer ESP32 system would successfully relay x, y, z acceleration data to the user’s laptop, and trigger a sound to play. However, we would essentially treat any acceleration as an impact and thus trigger a sound. We configured our accelerometer to align its Y-axis parallel to the drumstick, the X-axis to be parallel to the floor and perpendicular to the drumstick (left right motion), and the Z-axis to be perpendicular to the floor and perpendicular to the drumstick (up-down motion). This is shown in the image below:

Thus, we have two possible ways to retrieve relevant accelerometer data: 1.) reading just the Z-axis acceleration, and 2.) reading the average of the X and Z axis acceleration, since these are the two axis with relevant motion. So on top of finding better threshold values for what constitutes an impact, I needed to determine what axis to use when reading and storing the acceleration.  To determine both of these factors I ran a series of tests where I mounted the accelerometer/ESP32 system to the drumstick as shown in the image above and ran two test sequences. In the first test I used just the Z-axis acceleration values and in the second I used the average of the X and Z-axis acceleration. For each test sequence, I recorded 25 clear impacts on a rubber pad on my table. Before starting the tests, I did preformed a few sample impacts so I could see what the readings for an impact resembled. I noted that when an impact occurs (the stick hits the pad) the acceleration reading for that moment is relatively low. So for the tests purposes, an impact was identifiable by seeing a long chain of constant accelerometer data (just holding the stick in the air), followed by an increase in acceleration, followed by a low acceleration reading.
Once I established this, I started collecting samples, first for the test using just the Z-Axis. I performed a series of hits, recording the 25 readings where I could clearly discern an impact had occurred from the output data stream. Cases where an impact was not clearly identifiable from the data were discarded and that sample was repeated. For each sample, I stored the acceleration at the impact, the acceleration just prior to the impact, and the difference between the two (i.e. A(t-1) – A(t)). I then determined the mean and standard deviation for the acceleration, prior acceleration, and the difference between the two acceleration readings. Additionally, I calculated what I termed the upper bound (mean + 1 StdDev) and the lower bound (mean – 1 StdDev). The values are displayed below:

I repeated the same process for the second test, but this time using the average of the X and Z-axis acceleration. The results for this test sequence are shown below:

As you can see, almost unanimously, the values calculated from just the Z-axis acceleration are higher than when using the X,Z average. To then determine what the best threshold values to use would be I proceeded with 4 more tests:
1.) I set the system to use just the Z-axis acceleration and detected a impact with the following condition:

if accel < 3.151 and accel > 1.072: play sound

Here I was just casing on the acceleration to detect an impact, using the upper and lower bound for Acceleration(t).

2.) I set the system to use just the Z-axis acceleration and detected an impact with the following condition:

if (prior_Accel - Accel) < 5.249 and (prior_Accel - Accel) > 2.105: play sound

Here, I was casing on the difference of the previous acceleration reading and the current acceleration reading, using the upper and lower bounds for Ax(t-1) – Ax(t)

3.)  I set the system to use the  average of the X and Z-axis accelerations and detected an impact with the following condition:

if accel < 2.401 and accel > 1.031: play sound

Here I was casing on just the current acceleration reading, using the upper and lower bounds for Acceleration(t).

4.)  I set the system to use the average of the X and Z-axis accelerations and detected an impact with the following condition:

if (prior_Accel - Accel) < 5.249 and (prior_Accel - Accel) > 2.105: play sound

Here I cased on the difference of the prior acceleration minus the current acceleration and used the upper and lower bounds for Ax(t-1) – Ax(t).

After testing each configuration out, I determined two things:
1.) The thresholds taken using the average of the X and Z-axis accelerations resulted in higher correct impact detection than just using the Z-axis acceleration, regardless of whether casing on the (prior_Accel – Accel) or just Accel.

2.) Using the difference between the previous acceleration reading and the current one resulted in better detection of impacts.

Thus, the thresholds we are now using are defined by the upper and lower bound of the difference between the prior acceleration and the current acceleration (Ax(t-1) – Ax(t)), so the condition listed in test 4.) above.

While this system is now far better at not playing sounds when the user is just moving the drumstick about in the air, it still needs to be further tuned to detect impacts. From my experience testing the system out, it seems that about 75% of the instances when I hit the drum pad correctly register as impacts, while ~25% do not. We will need to conduct further tests as well as some trial and error in order to find better thresholds that more accurately detect impacts.

My progress is on schedule this week. In the coming week, I want to focus on getting better threshold values for the accelerometer so impacts are detected more accurately. Additionally, I want to work on the refinement of the integration of all of our systems. As I said last week, we now have a functional system for one drumstick. However, we recently constructed the second drumstick and need to make sure that the additional three threads that need to run concurrently (One controller thread, one BLE thread, and one CV thread) work correctly and do not interfere with one another. Make sure this process goes smoothly will be a top priority in the coming weeks.

Posted on

Belle’s Status Report for 11/9

This week, we mainly focused on integrating the different components of our project to prepare for the Interim Demo, which is coming up soon.

We first successfully integrated Elliot’s bluetooth/accelerometer code into the main code. The evidence of said success was an audio response (a drum beat/sound) being triggered by making a hit motion with the accelerometer and ESP32 in-hand.

We then aimed to integrate my drumstick tip detection code, which was a bit more of a challenge. The main issue concerned picking the correct HSV/RGB color values with respect to lighting and the shape of the drumstick tip. We positioned the drumstick tip on the desk (which we colored bright red), in-view of the webcam, and took a screenshot of the output. I then took this image and used a HSV color picker website in order to get HSV values for specific pixels in the screenshot. However, because of its rounded, oval-like shape, we have to consider multiple shadow, highlight, and mid-tone values. Picking a pixel that was too light or too dark would cause the drumstick tip to only be “seen” sometimes, or cause too many things to be “seen”. For example: sometimes the red undertones in my skin would be tracked along with the drumstick tip, or the tip would only be visible when in the more brightly-lit areas of the table.

In order to remedy this issue, we are experimenting with lighting to find an ideal setup. Currently we are using a flexible lamp that clamps onto the desk that the drum pads are laid on, but it only properly illuminates half of the desk. Thus, we put in an order for another lamp so that both halves of the desk can be properly lit, which should make the lighting more consistent.

As per our gantt chart, we are supposed to be configuring accelerometer thresholds and integrating all of our code at the moment, so we are surely on track. Next week, I plan to look into other object/color tracking such as CamShift, Background Subtraction, or even YOLO/SSD systems in case the lighting situation becomes overly complicated. I also would like to work on fine-tuning the accelerometer threshold values, as we are currently just holding the accelerometer and making a hit-like motion rather than strapping it to a drumstick and hitting the table.

 

Posted on

Elliot’s Status Report for 11/9

This week was spent working with the rest of the team to bring up a testable prototype for the interim demo. I integrated my bluetooth client code to the system controller in our repository, and together we sorted out file dependencies to get a running program that plays audio upon valid accelerometer spikes. I also worked on planning ahead with the multithreaded code in mind, in which we will need to spawn separate threads for each drumstick–a foreseeable issue in our development is undoubtedly the timing synchronization between the accelerometer readings, computer vision detection, and audio playback, and I plan to meet with Ben and Belle to continue to test their interaction with the shared buffer thoroughly.  Once the speed of the system, especially the CV, is confidently established, I may also update the rate at which the ESP boards notify the laptop with new readings, or even switch to a polling-based implementation.

The other potential concern is the interference on the 2.4GHz band once the second microcontroller is incorporated. In our weekly meetings with Tjun Jet and Professor Bain, we considered utilizing the Wi-Fi capabilities of the ESP32 rather than BLE to ensure adequate throughput and connectivity. With our testing this week, however, it seems that Bluetooth offers an appropriate latency for the needs of the project, and so our reason for choosing Wi-Fi would depend solely on the packet loss behavior of the two sticks running together. There could also be tradeoffs if we choose to redirect the system away from BLE in the form of setup time and ease of pairing, which would potentially neglect our use case requirement for versatility. As such, my plan for this upcoming week is to integrate the second ESP32 board to begin testing with two BLE devices, and to conduct a trade study between Bluetooth and Wi-Fi for our specific use case. I believe the team is on schedule and providing ample time for testing, therefore allowing us to identify important considerations–such as lighting–earlier in the process.

Posted on

Team Status Report for 11/9

This week we made big strides towards the completion of our project. We incrementally combined the various components each of us had built into one unified system that operates as defined in our use case for 1 drum stick. Essentially, we have a system where when the drumstick hits a given pad, it triggers an impact event and plays the sound corresponding to that drum pad with very low latency. Note however that this is currently only implemented for 1 drum stick, and not both. That will be our coming week’s goal. The biggest risk we identified which we had not anticipated was how much variation in lighting affect the ability of the object tracking module to identify the red colored drum stick tip. By trying out different light intensities (no light, overhead beam light, phone lights, etc.) we determined that without consistent lighting the system would not operate. During our testing, every time the light changed, we would have to capture an image of the drum stick tip, find its corresponding HSV value, and update the filter in our code before actually trying the system out. If we are unable to find a way to provide consistent lighting given any amount of ambient lighting, this will severely impact how usable this project is. The current plan is to purchase two very bright clip on lamps that can be oriented and positioned to equally distribute light over all 4 drum rings. If this doesn’t work, our backup plan is to line each drum pad with LED strips so each has consistent light regardless of its position relative to the camera. The backup is less favorable because it would require that either batteries be attached to each drum pad, or that each drum pad must be close enough to an outlet to be plugged in, which deteriorates our portability and versatility goal defined in the use case.

The second risk we identified was the potential for packet interference when transmitting from two ESP32’s simultaneously. There is a chance that when we try and use two drumsticks, both transmitting accelerometer data simultaneously, the transmissions will interfere with one another resulting in packet loss. The backup plan for this is to switch to WIFI, but this would require serious overhead work to implement. Our hope is that since most of the time impacts from two drum sticks occur sequentially, the two shouldn’t interfere, but we’ll have to see what the actual operation is like this week to be sure.

The following are some basic changes to the design of DrumLite we made this week:
1.) We are no longer using rubber rings and instead using circular rubber pads. The reason for this is as follows. When we detect the drum pad’s locations and radii and use rings, there are two circles that could potentially bet detected: 1 being the outer circle and one being the inner circle. Since the ins no good way to tell the system which one to choose, we decided to switch to a drum pad instead where only 1 circle can ever be detected. Additionally, this also makes getting a threshold acceleration much easier since the surface being hit will now be uniform. This system works very well and the detection output is shown below:

2.) We decided to continually record the predicted drum ring which the drum stick is in throughout the playing session. This way, when an impact occurs, we don’t actually have to do any CV and can instead just perform the exponential weighting on the predicted drum rings to determine which pad was hit.

We are on schedule and hope to continue at this healthy pace throughout the rest of the semester. Below is an image of the whole setup so far: