Our final report is attached here: Team_B5

 

 

H4AR is a device for assisting hard-of-hearing people in their everyday lives, by visually alerting them about the direction of important sources of sound. The device is a wearable pair of glasses, equipped with an array of microphones and augmented reality through a heads up display. This microphone array will identify human speech, and determine the direction of the sound. This information will be displayed as a visual cue in the form of an arrow on the heads up display, greatly increasing the individual’s peripheral awareness by allowing them to identify and turn towards people talking to them.

You can watch our final video here

Tasks :

• Full integration complete
• Ran compass tests (measure at known angles, check whether prediction within error bounds)
• Ran latency tests (difference between hear sound and symbol appears)
• Prepared for final presentation

Deliverables:

• Video
• Blog Post
• Final Report

Tasks Accomplished

• Finished assembling the display module and battery/compute modules
• 
  
• Put together the wire harness for the mic array to interface with the other modules

Deliverables for upcoming week

• Finish integrating with the mic array
• Figure out a way to flash new firmware versions onto the board without taking apart the module

This week saw some good progress on the software side. Most of this was done with pair-programming with Ram.

• Input audio form correction. Previously each mic input was biased from 0 by a certain amount. We adjusted each audio waveform such that it is centered around 0. This means that as far as the acousticSL library is concerned, the audio activation level would be consistent for all waveforms (i.e no 1 waveform is arbitrarily larger or smaller than the other).
• Integrated the 4 microphones into acousticSL (through multiple library and callback instantiations). We also derived an ad-hoc implementation for reconciling the output angles of TDOA on two pairs of microphones into a single output angle (through a mix of adjusting to a common reference angle and then taking the measurements that are consistent from both microphones). The ad-hoc implementation is taken with reference to a parallel line going through one of the mics so this further refinement.
• Augmented the square sums smoothing for both the raw input angles and also the converged angle from multiple measurements.
• Rough latency test is promising; precision test is still worrisome

To-Dos for this week:

• Robustness to noise and adjustments for precision (at least get to 30 degrees for the OLED screen)
• Increasing the sampling frequency; balancing smoothing and latency (i.e how much to smooth, how long to smooth)
• Add a third pair of mics (possibly adjust the physical arrangement) to improve angle resolution
• Move data processing logic gradually into callbacks — this might allow us to be more lock-step when interacting with acousticSL

Some geometry of interest:

This week, we made some good progress on the software side (Will and I pair-coded for most of this).

• We did some code clean-up that made transitioning to 4 mics easier
• Geometry: we did the math to figure out how to reconcile the output angles of TDOA on two pairs of mics, and implemented this
• We debugged the existing solution for smoothing, and improved it some more, to balance the trade-off between latency and smoothness
• Did a rough overall latency test which was promising
• We still do not have 15 degree precision based on some rough tests, so we will need to work on that
• Unrelated: Found a potential fix to my laptop issue

To-Dos This Week:

• Make it more robust to noise – if there are very loud and close-by keyboard noises, it currently does not work
• Adding a third pair of mics – either an actual third pair in between the current ones, or a “virtual” third pair that uses data from two mics that are currently part of different pairs
• Increasing the sample frequency – we don’t really need to yet, but we anticipate that we might need to do this soon (so that we have more breathing room in terms of latency, and less lag when transitioning)
• Move a lot of the code currently in the main while loop into callbacks – the advantage of this is that these callbacks can’t be interrupted so the volatile variables that we change will definitely not be modified in the middle (currently, this isn’t a problem but it could be so it’s probably good to do this move)
• Smoothing Improvements
  • Account for 0 essentially being the same as 360 – so that the smoothing algorithm takes into account the fact that changes from e.g. 3 to 359 are actually fine, and these are still smooth
  • Implement a feature where if we haven’t seen 10 angles in a row, we just cut off all our prior sample knowledge – because those priors are likely unreliable

Tasks accomplished this week:

• Ironed out some data acquisition bugs about resolution and sampling speed (our current design that marks data as ready in callbacks but only does data copying in the run loop will result in poor predictions). As such we will likely do data read/copying in the callback and de-prioritize the OLED update: similarly, we will probably use this to lockstep calling of multiple acousticSL instances.
• Tried solving Ram’s debugger issue 🙁
• Investigated into some counters. This paper discusses probabilistic counter updates which argues that probabilistic updates can match other counters with less bit-width (https://zilles.cs.illinois.edu/papers/probabilistic.hpca2006.pdf). The paper also conveniently lists other counters we can use. We will still need some more discussion about how we exactly want to do this based on the # states our OLED can display and counter resilience so our device’s reaction time remains within the target.

Unfortunately couldn’t get as much done as liked during this week. Got a flu shot on Wednesday and then proceeded to get a high fever shortly after. Still recovering so Ram and I will proceed remotely once we do an equipment hand-off.

Tasks for next week:

• Fully implement 4 mics and correct the audio angle
• Test to see whether we meet the target metrics

My debugger program (Cube Monitor) stopped working this week so a looot of time was sunk into debugging it with no success yet. So the plan is, now that I’ve pushed all the relevant code to the remote repo, we’ll work from Will’s computer and pair-code until/unless I figure this out.

Also, we figured out how to structure our code to get 4 mics to work, and we need to implement that.

I also debugged some code involving removing the arbitrary mic offsets we observe for different mics – not 100% sure if it works yet though (it was in the middle of debugging this when I realized my Cube Monitor was broken).

One of my mics wasn’t working properly in stereo – spent some time and eventually realized that it was because the stereo connection wasn’t cut properly so cut that and it was better.

To-Do:

• Fix this annoying software bug whose source is still extremely mysterious to me
• Get 4 mics working properly, and test it out on Will’s computer
• Test, see if we meet our metrics next week – if we do, then we can begin to think about stretch goals like classification, if not we’ll need to work on getting latencies down or accuracies up…

Tasks Accomplished

This week we saw the most progress in our hardware, namely we now have physical fabricated versions of our designs for us to devleop with and iterate upon. We also saw some integration work in the display side, with a working driver for the display unit. <More stuff here…>

Potential Risks

Time is again our biggest risk, especially since these weeks are extremely tough with large workloads from other classes. Physical design of the harness and assembly takes time, and so does the printing of the design, making it one of our bigger risks for the upcoming weeks. Integration is looming ahead, and although most of the individual pieces are coming together well, integration requires all components to be ready in order to start. Also, since we have yet to do work on the battery unit, power consumption remains a potential issue down our road, as it requires us to optimize code to save on current draw.