Just another Carnegie Mellon University: ECE Capstone Projects site
Input: uniform white noise
Output: attenuation of ~30 dB with noticeable distortion
Input: speech babble noise, directional in east and south-east directions
Output: attenuation of ~6-8 dB, minimal noticeable distortion
Performance here suffers because the voice activity detector has poorer performance in babble noise.
This week, I implemented and tested the beamforming algorithm described last week in Matlab. After finishing the matlab code, I was able to run preliminary tests to approximately see what signal attenuation we got. With one noise signal, we were able to achieve approximately 30 dB attenuation. I also worked with Bobbie on translating it to C.
This week I have put our old mic input problems to rest and now have new and exciting mic input problems.
I found that some of the input selection functions actually returned status codes; we were getting I2C write errors (status code 4: other unknown error). In a long story short, after much de- and re-soldering and purchasing a new Teensy in case our first one had some flaw, I resolved the I2C error, but mic input still did not work.
This entire search for audio input in the last few weeks came to an abrupt halt when I read all the Teensy documentation I could find and realized that the audio jack we were trying to use on the audio shield was never meant to take input (only headphone-level output).
On this realization I bought a few breakout boards for splitting out audio jack line-in into individual pins. While I did not get the lapel mic to work, using the boards and the line-in holes on the audio shield I was able to use my computer as an audio source to send audio to the Teensy.
The resulting output audio had interesting properties: it worked perfectly on a pure sine wave besides being inaudible for all but the few highest ticks of my computer’s output volume; for speech, microphone loopback, and music it was completely inaudible; and for sine wave + other audio, it produced a recognizable sine wave with added unrecognizable distortion from the other audio.
This suggested to me that the voltage ranges that the Teensy was looking for were not correct for the mic-level input. Line-out at headphone level is a far higher voltage range than line-in at mic level. However, when I tried to adjust the audio chip to a smaller voltage range using sgtl_5000.lineInLevel(N), this produced the same results.
To summarize the current mic issues:
I have not resolved the input issues yet and if I don’t in the next few days, I will probably pivot to working on code for moving 4-channel audio on and off the board via USB or SD card, to run the signal processing there and output the result via the (working) headphone jack.
I also worked with Kevin on translating his Matlab code to C.
I spent a significant amount of time, along with Bobbie, trying to get four-channel audio through the board using the audio shields. Bobbie discovered that the Shields weren’t meant to take input through the audio jack,
This week I hooked up mobile power using a battery to the Teensy. I did this by connecting a lithium ion battery to an Adafruit micro-USB charging port, then connecting the charging port to the external power pins on the Teensy.
This week I worked on getting SDW-MWF to work with the Teensy Audio library. This required changing many constants like the buffer length in each audio frame. However, this ran into problems (segfaults) when I tried to actually integrate it. I spent a good amount of time trying to debug the generated C code (from Matlab Coder), when Professor Sullivan suggested just rewriting it directly.
So I rewrote the code for the Teensy Audio library as a new object (mwf.h and mwf.cpp). This also required implementing a voice activity detector, which I did using the algorithm described in “A Simple But Efficient Real-time Voice Activity Detection Algorithm” (Moattar and Homayounpour, 2009). I tested the voice activity detector against some WAV inputs, and it appears to work with good accuracy. The main time spent here was learning the bit of C++ needed to integrate with the library, and familiarizing myself with the CMSIS DSP Software Library (“arm_math.h”), which is a DSP library optimized for the ARM architecture. For example, a few times I wrote a loop to calculate some simple property like the mean, only to find that I could have just called arm_max_q15(buf, len, &max_size).
I also talked to Kevin about material choice for the microphone rig and decided on mounting the microphones to Lexan polycarbonate, as a material which is strong, stiff and slightly flexible. This is decided over acrylic because it will not shatter under load, and over wood because it stronger. We also decided against 3D-printed ABS because it is less flexible, and our geometry is not complex enough to warrant the time and expense.
This week I worked on converting the MATLAB code for beamforming to C code. I am currently working on a delay sum beamformer for our microphone array. I am still currently working on a working C implementation partially delayed because this weekend was spent in Ohio. I currently have code for microphone array initialization and microphone delay calculations. I have added screenshots of sample code below.
I received and tested the PDM microphones this week, but unfortunately translating the data uses too much CPU power. Reading from 1 mic used about 37% CPU and 2 mics used 81%, so we won’t be able to translate data from 4 microphones and process all the audio. Therefore, we’re going forward with using I2S communication via the Teensy audio shields. I found some left/right mono to stereo audio adapters, as well as a microphone/ headphone splitter so we can record and output data on the same I2S port. This setup will allow us to have 4 input and 1 output with only 2 audio shields attached to the Teensy.
I rewrote my Teensy startup code to take in I2S inputs and connect to an empty filter, which will eventually become Kevin’s and Bobbie’s algorithms. I also implemented a simple program that records for 5 seconds, then plays that audio back. I am using this to test different microphone setups.
Next week, I want to build the mic setup and test multiple input and single output via I2S. I hope Bobbie and I will be able to implement and test his noise cancelling algorithm on the Teensy board. We will initially try this with only one microphone input.
I started this week off by looking into converting our MATLAB code into C/C++. I utilized MATLAB coder, however quickly realized that the coder would not provide direct code that we could utilize. The biggest issue that I came across was that a number of built-in MATLAB functionalities could not be directly converted. After changin a lot of the MATLAB built-ins to hand-coded functions, I ended up with one final error which was in coder’s conversion of our frostbeamformer.
While waiting for microphones, I also decided to do more research on beamforming. More specifically, I have not read too much literature on how reverberant noise affects beamforming algorithms. Below I included some of the notes I took on different online literature in order to keep notes not only for the progress of this project but also for the future report we will have to write.
NOTES:
GOALS FOR NEXT WEEK
This week I worked primarily on documenting our existing Matlab code and using Matlab Coder to convert my SDW-MWF implementation into C code. Matlab Coder has certain limitations that make it different from running Matlab directly. For example, system objects cannot take variable length inputs, and variable types must be explicitly declared and cannot change.
The resulting code (excerpt above) is also ugly. This means that it’s more important than ever to have properly documented Matlab source code before generating to C (as comments are preserved).
I also wrote a quick Makefile for actually compiling code which calls into the shared library.
This took an embarrassing amount of time (a few hours) to figure out.
I also tweaked the HINT runner to interactively prompt as if a real HINT test is being run, i.e. with binary search on the example tests to find the proper range of SNRs to run it at.
On the chest mount, since our second PCB arrived and was also not correct, I did not have mics to mount. However, the chest mount we ordered did arrive and I took a look at it; it works very well for our purposes and should only require drilling a couple holes to bolt on a rigid surface like polycarbonate to get the microphones on.
I ordered a few different microphones to try some simple audio input. I ordered some I2S microphones with breakout boards so we can easily get inputs to the board. These microphones have been tested with the Teensy board and there are tutorials on how to set the up with the board. I also ordered breakout board for the current PDM microphones we have, in case we want to go forward with that.
I soldered the board together and got audio output working through the audio shield this week. I also was able to generate a low-jitter 2 MHz clock on the board to a digital pin, which we can use to drive the microphone data. I finished most of the set up code for outputting the clock, left/right select lines, data in lines, and audio output through the audio shield. I’ve also started on processing 4 audio inputs at once.
Next week, I need to get an audio signal into the board through our microphones. I will test both types of microphones after they come in, and continue to work on C code adaptations of the Matlab algorithms in the meantime. The microphone integration is vital, and I need to get that done as soon as possible after they arrive.
I will also be working on multithreading the audio input on the Teensy board. There are a few ways this can be done, and I’m currently deciding which option to pursue.
I successfully reduced the distortion in the SDW-MWF implementation. During lab on Monday, Professor Sullivan helped identify the problem as a windowing issue: previously there was no overlap between windows, which caused boundary effects at the edge of each window. Using a 50% window overlap drastically reduced the distortion. This is at the tradeoff of now needing to process the next (overlapping) frame before producing output, which adds 5-9 ms delay to the output. This is within our timing budget, so it is okay. The new output is below.
For the wearable mic array mount, I ordered this phone chest mount from Amazon: https://www.amazon.com/Samsung-Holder-Record-Awesome-Action/dp/B00MYN0CGI/ref=cm_cr_arp_d_product_top?ie=UTF8
This will serve as the basic model of our wearable mount. The main problem we anticipate from this mount is rotation about the central mounting point. Ideally there should be two solid pieces upon which the mics are mounted, with straps connecting as well. This will consist of:
I also wrote code to generate the HINT test recordings. However, this is not completely practical to use yet, because there is not an interface for quickly switching between various SNRs – this is needed because the HINT is an adaptive test, switching SNR levels based on the subject’s success rates.
One of my goals for this week was to come up with a finalize an initial physical setup for the microphone array. I decided that for a linear array of n omnidirectional microphones with equal inter element spacing, the distance between each microphone element should be d=c/(2*f0)where c is the speed of sound and f0 is the midband frequency. Because the human voice frequency in telephony ranges from approximately 300 Hz to 3400 Hz, I chose 1800 Hz to be the midband frequency. As a result, the distance between each microphone was chosen to be 9.5cm apart. This means with a 4 microphone setup, we would require an array of 38 cm in length.
Additionally, regarding how each microphone input would be processed, the delay Tn between the desired signal arriving at the nth microphone and the signal arriving at the first microphone (at the origin of the coordinate system) is Tn =(n-1)*d*sin(ø)/c where d=9.5 cm and c is the speed of sound (343 m/s).
For signal processing, I have an initial MATLAB program which implements Frost Beamforming to try on our microphone setup. I have not done extensive testing yet on the adaptive beamforming code. This following week will be spent testing and modifying the beamforming code with (hopefully) real microphone input signals.
On the hardware side of things, I have worked in lab with Sean to create a temporary microphone setup for testing using microphones that were available from the past. The microphones we used for now are PM: WM-55D103. The specific circuit we made for each condenser microphone is shown in the figure below for testing.
This week, we put together a circuit for the microphones Professor Sullivan gave us and successfully got an analog signal read on the Teensy board. I just used simple analog read functions from the Arduino library to read sound levels. I also starting writing code using the PDM library to process the input from our surface-mount microphones
We decided to go forward with a rechargeable battery pack for external power, pictured below. In order to prevent damaging the board from supplying USB power and external power, I cut the trace between the two pads, to isolate the voltages. I will also cut the power line of the USB cord, just in case the trace isn’t completely cut.
I found tutorials for outputting high speed clocks from the Teensy, and will talk to Professor Sullivan more about these methods.
This week I implemented speech-distortion weighted multi-channel Wiener filtering as described in “Multichannel Filtering For Optimum Noise Reduction In Microphone Arrays” (Florêncio and Malvar, 2001). The results are very strong in reducing noise, but also introduce a very noticeable distortion in the output signal. See the attached audio for an example.
This implementation processes the audio frame-by-frame, writing results to a buffer. For now, the buffer is simply written to file at script completion, but in an actual program it would be read from continuously for output. The 38 seconds of input were processed in ~3 seconds including file reading and writing; this implies that processing a frame takes at most about 70 milliseconds, which is within our time budget. The result shown involved some tweaking of the parameter P as well as the parameters for the least mean squares adaptive filter. I don’t see any glaring errors in my implementation of the algorithm, but the distortion levels are currently quite high and distracting. I will continue looking at this and trying to reduce the distortion, because the noise is reduced by about 24 dB.
Original input:
Post-MWF output:
I also generated some working C code using Matlab Coder. This was quite straightforward to do; Matlab Coder was able to automatically detect the types based on a script which used the relevant functions. We have the option of using Matlab Embedded Coder to generate directly for a ARM Cortex-M target, which may run faster because it will optimize for the particular processor we are using. However, we are not over on our timing budget yet, and it is useful to be able to see the C source code, so we will just use source code generation for now.
This week I looked into how our team would go from our basic Matlab implementation, which consisted of LMS noise cancellation using a reference and primary signal, to using a microphone array. After looking through research papers, adaptive beamforming seemed to be a very promising.
The basic principle of adaptive beamforming is as follows. Beamforming microphone arrays are spatial filters that take multiple microphone signals as input and combine them to a single output signal. Usually the combined output is calculated by filtering each microphone signal through a digital FIR filter and summing the output of all filters as shown in the figure. The filters are designed so that their output add constructively when sound is coming from a specific direction (main lobe) and add destructively when sound is coming from all other directions. This creates the spatial filtering effect of focusing at the sound that is coming from the main lobe direction while attenuating sounds coming from all other directions.
This week I fixed the PCB board so that we can order it through PCBway. This included manually adding layers that weren’t generated by the Eagle software and changing file encodings. I will order that as soon as I hear back from Quinn.
I have starting writing code for the Teensy to get simple inputs from the digital pins, where we will soon be inputting a signal from the microphones given to us by Professor Sullivan.
I did research on outputting a low-noise clock from the Teensy board, which can be done by clock-dividing the 16MHz I2S master clock. I also researched pulse density modulation on the Teensy board, which I found is included in the Audio library for the Teensy. Finally, I looked into power options for the Teensy. For now, we will use USB power, but the final product will need to have a portable battery pack, which we can set up with 2 or 3 AA batteries. Ideally, we’ll use a lower-profile, rechargeable battery for convenience and wearability.
We have just missed a couple milestones on our schedule.
We have not sent a signal through the mics into the hardware system yet. The main delay here has been the very small mics we initially ordered and follow-on issues with PCB design. The PCB issues have been fixed and we should be able to order this week (waiting to hear back from Quinn). In the meantime, on Monday Sean will put together the larger microphones that Professor Sullivan lent us and get an actual input signal by then.
We also have not yet created a HINT Test which we can run on our test environment yet. This is mainly because there is not much point in creating one when the microphone-to-hardware connection to receive the signal does not exist yet. Bobbie will do this over the coming week.
Although these milestones were missed, we are not far behind in the project overall. This is because other non-blocked tasks in signal processing made progress when the tasks for the milestones above were blocked.
As another schedule-related concern, Sean will be travelling most of this week for interviews, and that may put his work slightly behind schedule. He will work diligently while he is gone to try to keep the project moving forward.
This week I generated test audio samples which would simulate the delays involved in sound arriving at our microphone array, using the Matlab script I wrote last week.
I also explored alternative signal processing techniques beyond adaptive nulling which could also be useful in our project. The most promising of these were Speech-distortion weighted multi-channel Wiener filtering (Ngo 2011), and the acoustic rake receiver (Dokmanic, Scheibler, and Vetterli 2015).
Over the course of this research I identified a few common useful utilities and prepared them for use in our project. This includes implementing the short-time Fourier transform and its inverse in Matlab, and finding the documentation for the Matlab “snr” function.
This week I explored the specific use case of white-noise adaptive cancellation with the implementation currently in place. More specifically, trying to pinpoint the conditions that cause the filtering to work well vs. not well.
Additionally, I worked with Bobbie to come up with a quantitative way to analyze our results/ANC output and compare it to our desired signal. We settled on using the Matlab “snr” function to evaluate the signal-to-noise ratio in our algorithm output relative to the original signal.
Also, I emailed professor Cameron Riviere to meet as a group and discuss his previous experience with adaptive noise cancelling. He is a professor from the Robotics department with whom I am currently taking a sensors and sensing class with. We believe he will be a good resource for creating an effective actual microphone array as well as being able to receive feedback on our ANC implementation and figuring out how to integrate our current basic MATLAB implementation with a microphone array. His previous ANC experience does not involve audio-signal inputs, however we should still be able to receive helpful feedback.
This week I noticed some crossing over on the routing microphone surface-mount PCB that I designed in Eagle. It looks like it might not be necessary, so I want to take a look at these and change the routing to avoid crossing over if possible.
I also did some research into audio on the Teensy board and found the Teensy Audio System Design Tool, which provides a GUI for drawing audio block diagrams and exporting Arduino code.
While our signal processing is very likely to be too complex to model graphically using these blocks, this will still be a useful tool that we can use to generate code and simplify getting audio in and out of the Teensy board.
This week we made progress on planning the schedule, ordering parts, working on the testing setup, and background research on signal processing.
While a lot of the work for our project will be in the signal processing side, we wouldn’t be able to verify that it works without having the required hardware and a way to test that it is working. Therefore, in the beginning of our schedule we are focusing on getting the hardware and testing environment correct.
We are currently on schedule. One potential risk is the TOC early next week, and interviewing season soon after, which could lower the amount of work that we accomplish in the two weeks.
We assembled a list of parts needed for hardware and test setup. These represent the minimal working set of materials we need to get started with actually getting input in a controlled test environment. The parts include the Teensy microcontroller board as well as audio jacks for I/O, speakers and power supply for the test environment, and a torso mannequin to mount the mic array to.
We intend to administer the Hearing In Noise Test (HINT) using a circle of speakers to express the directionality.
Over the course of this week, we contacted institutions which administer the HINT. Some offered stereo recordings for administration with headphones, but unfortunately that would not be helpful for our test setup; many did not respond.
Because of this, we are planning to generate our own 7.1 sound recordings using Audacity and FFmpeg. This week we created an example of a HINT recording. However, the HINT test is an adaptive test, where signal-to-noise ratio is dynamically adjusted based on the test subject’s responses. Therefore, we are also looking into ways to automate the generation of many HINT recordings with varying signal-to-noise ratios.
We also continued research into signal processing algorithms, mostly in adaptive nulling. We found that it has been tried in the context of hearing aids unsuccessfully, at least in part because the space constraints of traditional hearing aids were too limiting.
Hi everyone! For our ECE capstone project, we are working on a directional hearing aid, using a wearable microphone array.
The inspiration for this project came from meeting a dance instructor, Dan, who is legally deaf and relies on a hearing aid to communicate with his students. Many conventional hearing aids only amplify all nearby sound, making hearing in a noisy environment like a dance studio with loud music and multiple conversations near impossible. As a result, Dan usually resorts to lip reading or turning his earpiece to a higher volume, which only damages his hearing further.
To improve on conventional hearing aids in noisy environments, our group proposes a directional hearing aid.
Our implementation will consist of a microphone array worn by a hearing-impaired person. The microphones will be connected to a microcontroller, which then applies signal processing algorithms to focus on the sound from the area in front of the user. We assume that the user’s torso is facing in the direction that they want to hear.
We chose the form of a microphone array worn on the torso because this gives us the space to place more microphones with greater separation. This allows for certain signal processing methods which are not effective in traditional small in- and on-ear hearing aids.
We picture a specific use case here, where the user is trying to hear a person in front of them in various noisy conditions like a crowded room. In particular, we do not aim to help users listen to things like music or public speakers in a crowded venue – this is because the standard solution for these is to use a hearing loop system to transmit the sounds wirelessly, directly to a user’s hearing aid.
And also, we are not trying to handle cases where the sound source is moving around the user – it would be very difficult to predict which sounds a person actually wants to listen to in a whole soundscape. With our project, we envision that the user can simply turn their body to look at whatever they want to hear.
We will create an array of multiple (likely 4) microphones, mounted somehow onto the front of the user’s torso (for example, with a vest). These microphones will then interface with the Teensy 3.6 microcontroller – we chose this particular model because it is the base component for the Tympan open-source hearing aid board.
After signal processing is performed on the mic input, we will send the output to a pair of headphones. This is because we do not have the time or resources to build our own in-ear hearing aid or reverse-engineer a commercial one.
Our goal is to reduce the background noise and improve the directionality of what the user can hear in real-time, especially for speech. We will implement algorithms in C and run them on the Teensy microcontroller to transform the sounds from our microphones into a better version that the user can listen to on the headphones. Potential methods for doing this include multi-channel Wiener filtering and adaptive nulling, though we have not yet decided on any particular algorithm.
We’re working on an interesting project with the potential to help countless hearing-impaired people around the world. If you would like to help, please contact us at seanande@andrew.cmu.edu, bobbiec@andrew.cmu.edu, and kmsong@andrew.cmu.edu!