Just another Carnegie Mellon University: ECE Capstone Projects site
Input: uniform white noise
Output: attenuation of ~30 dB with noticeable distortion
Input: speech babble noise, directional in east and south-east directions
Output: attenuation of ~6-8 dB, minimal noticeable distortion
Performance here suffers because the voice activity detector has poorer performance in babble noise.
This week, I implemented and tested the beamforming algorithm described last week in Matlab. After finishing the matlab code, I was able to run preliminary tests to approximately see what signal attenuation we got. With one noise signal, we were able to achieve approximately 30 dB attenuation. I also worked with Bobbie on translating it to C.
This week I have put our old mic input problems to rest and now have new and exciting mic input problems.
I found that some of the input selection functions actually returned status codes; we were getting I2C write errors (status code 4: other unknown error). In a long story short, after much de- and re-soldering and purchasing a new Teensy in case our first one had some flaw, I resolved the I2C error, but mic input still did not work.
This entire search for audio input in the last few weeks came to an abrupt halt when I read all the Teensy documentation I could find and realized that the audio jack we were trying to use on the audio shield was never meant to take input (only headphone-level output).
On this realization I bought a few breakout boards for splitting out audio jack line-in into individual pins. While I did not get the lapel mic to work, using the boards and the line-in holes on the audio shield I was able to use my computer as an audio source to send audio to the Teensy.
The resulting output audio had interesting properties: it worked perfectly on a pure sine wave besides being inaudible for all but the few highest ticks of my computer’s output volume; for speech, microphone loopback, and music it was completely inaudible; and for sine wave + other audio, it produced a recognizable sine wave with added unrecognizable distortion from the other audio.
This suggested to me that the voltage ranges that the Teensy was looking for were not correct for the mic-level input. Line-out at headphone level is a far higher voltage range than line-in at mic level. However, when I tried to adjust the audio chip to a smaller voltage range using sgtl_5000.lineInLevel(N), this produced the same results.
To summarize the current mic issues:
I have not resolved the input issues yet and if I don’t in the next few days, I will probably pivot to working on code for moving 4-channel audio on and off the board via USB or SD card, to run the signal processing there and output the result via the (working) headphone jack.
I also worked with Kevin on translating his Matlab code to C.
I spent a significant amount of time, along with Bobbie, trying to get four-channel audio through the board using the audio shields. Bobbie discovered that the Shields weren’t meant to take input through the audio jack,
This week I hooked up mobile power using a battery to the Teensy. I did this by connecting a lithium ion battery to an Adafruit micro-USB charging port, then connecting the charging port to the external power pins on the Teensy.
This week, I looked through numerous examples of source code for beamformers and attempted to adapt them to our use. I also spent a large amount of time reading through a large portion of text from Professor Sullivan’s Adaptive Nulling textbook. The textbook helped connect a number of conceptual aspects that I did not clearly understand previously, which has led me to decide to write the beamformer from scratch.
Breakdown the exact parts of the Beamformer I am in the process of writing:
I am currently working on a matlab implementation of this specific frost-beamformer so that I can run tests to pick exact specs such as number of taps,samples, etc. This will also help verify that the algorithms I am running are working. I am also going to verify this with Professor Sullivan during Monday’s lab.
This week I spent many hours messing with getting four-channel audio from the microphones working. While four-channel output worked fine with some white noise on channels 3 and 4, I was unable to get any input at all from any of our microphones. This set off a long debugging process:
At this point I ran out of ideas and decided to try plugging things in and out and running old code in hopes that something interesting would happen. In fact, two things did:
Because of this latest discovery I have resigned myself to desoldering the boards entirely and doing it again, hopefully cleaner this time. I will do this tomorrow (Sunday).
I have also acquired Lexan for use with the microphones. The lapel mics are quite small and light so this will be easy. These have not yet been assembled because of the problems getting input mentioned above
This week I soldered the second audio shield onto the board for four-channel audio input/output. Unfortunately, they aren’t working. Output works fine on all channels, but Input is noisy and unreliable at best. Hopefully bobbies debugging skills are better than mine and he can get to the bottom of this.
I also rewrote audio recording code to work with four-channel input. The main challenge with this was changing to using multiple audio queues to hold all the data. Going forward, processing four blocks of audio simultaneously may possibly issues that don’t come up with a single buffer.
Next week, I hope to fix the input on the board. We need to get this fixed in order to be able to use our algorithms as intended.
Also, I plan to help Kevin finish up with his beamforming code, adding the Arduino- and Teensy-specific components to his C++ code.
This week I worked on getting SDW-MWF to work with the Teensy Audio library. This required changing many constants like the buffer length in each audio frame. However, this ran into problems (segfaults) when I tried to actually integrate it. I spent a good amount of time trying to debug the generated C code (from Matlab Coder), when Professor Sullivan suggested just rewriting it directly.
So I rewrote the code for the Teensy Audio library as a new object (mwf.h and mwf.cpp). This also required implementing a voice activity detector, which I did using the algorithm described in “A Simple But Efficient Real-time Voice Activity Detection Algorithm” (Moattar and Homayounpour, 2009). I tested the voice activity detector against some WAV inputs, and it appears to work with good accuracy. The main time spent here was learning the bit of C++ needed to integrate with the library, and familiarizing myself with the CMSIS DSP Software Library (“arm_math.h”), which is a DSP library optimized for the ARM architecture. For example, a few times I wrote a loop to calculate some simple property like the mean, only to find that I could have just called arm_max_q15(buf, len, &max_size).
I also talked to Kevin about material choice for the microphone rig and decided on mounting the microphones to Lexan polycarbonate, as a material which is strong, stiff and slightly flexible. This is decided over acrylic because it will not shatter under load, and over wood because it stronger. We also decided against 3D-printed ABS because it is less flexible, and our geometry is not complex enough to warrant the time and expense.
This week I worked on converting the MATLAB code for beamforming to C code. I am currently working on a delay sum beamformer for our microphone array. I am still currently working on a working C implementation partially delayed because this weekend was spent in Ohio. I currently have code for microphone array initialization and microphone delay calculations. I have added screenshots of sample code below.
I received and tested the PDM microphones this week, but unfortunately translating the data uses too much CPU power. Reading from 1 mic used about 37% CPU and 2 mics used 81%, so we won’t be able to translate data from 4 microphones and process all the audio. Therefore, we’re going forward with using I2S communication via the Teensy audio shields. I found some left/right mono to stereo audio adapters, as well as a microphone/ headphone splitter so we can record and output data on the same I2S port. This setup will allow us to have 4 input and 1 output with only 2 audio shields attached to the Teensy.
I rewrote my Teensy startup code to take in I2S inputs and connect to an empty filter, which will eventually become Kevin’s and Bobbie’s algorithms. I also implemented a simple program that records for 5 seconds, then plays that audio back. I am using this to test different microphone setups.
Next week, I want to build the mic setup and test multiple input and single output via I2S. I hope Bobbie and I will be able to implement and test his noise cancelling algorithm on the Teensy board. We will initially try this with only one microphone input.
I started this week off by looking into converting our MATLAB code into C/C++. I utilized MATLAB coder, however quickly realized that the coder would not provide direct code that we could utilize. The biggest issue that I came across was that a number of built-in MATLAB functionalities could not be directly converted. After changin a lot of the MATLAB built-ins to hand-coded functions, I ended up with one final error which was in coder’s conversion of our frostbeamformer.
While waiting for microphones, I also decided to do more research on beamforming. More specifically, I have not read too much literature on how reverberant noise affects beamforming algorithms. Below I included some of the notes I took on different online literature in order to keep notes not only for the progress of this project but also for the future report we will have to write.
NOTES:
GOALS FOR NEXT WEEK
This week I worked primarily on documenting our existing Matlab code and using Matlab Coder to convert my SDW-MWF implementation into C code. Matlab Coder has certain limitations that make it different from running Matlab directly. For example, system objects cannot take variable length inputs, and variable types must be explicitly declared and cannot change.
The resulting code (excerpt above) is also ugly. This means that it’s more important than ever to have properly documented Matlab source code before generating to C (as comments are preserved).
I also wrote a quick Makefile for actually compiling code which calls into the shared library.
This took an embarrassing amount of time (a few hours) to figure out.
I also tweaked the HINT runner to interactively prompt as if a real HINT test is being run, i.e. with binary search on the example tests to find the proper range of SNRs to run it at.
On the chest mount, since our second PCB arrived and was also not correct, I did not have mics to mount. However, the chest mount we ordered did arrive and I took a look at it; it works very well for our purposes and should only require drilling a couple holes to bolt on a rigid surface like polycarbonate to get the microphones on.
I ordered a few different microphones to try some simple audio input. I ordered some I2S microphones with breakout boards so we can easily get inputs to the board. These microphones have been tested with the Teensy board and there are tutorials on how to set the up with the board. I also ordered breakout board for the current PDM microphones we have, in case we want to go forward with that.
I soldered the board together and got audio output working through the audio shield this week. I also was able to generate a low-jitter 2 MHz clock on the board to a digital pin, which we can use to drive the microphone data. I finished most of the set up code for outputting the clock, left/right select lines, data in lines, and audio output through the audio shield. I’ve also started on processing 4 audio inputs at once.
Next week, I need to get an audio signal into the board through our microphones. I will test both types of microphones after they come in, and continue to work on C code adaptations of the Matlab algorithms in the meantime. The microphone integration is vital, and I need to get that done as soon as possible after they arrive.
I will also be working on multithreading the audio input on the Teensy board. There are a few ways this can be done, and I’m currently deciding which option to pursue.
This week I wrote a program for interactively running HINT tests. It requires pre-generated audio at varying SNRs, but this helps make the program more portable (as opposed to generating noise at runtime using Matlab).
The chest harness did not arrive until very late in the week, so I did not make any progress on that front.
Last week we had issues with getting a usable and relatively clean signal out of the electret microphones from Professor Sullivan. I worked on an amplification circuit to bring the voltage values to the range of the Teensy’s ADC, but the signal was incredibly noisy so this was unsuccessful. I did consider building a more complex circuit to try and filter out the largest frequency component of the noise (60 Hz), but I thought this would be not worth the time, as Sean put in the redesigned PCB order so we should have it early next week.
I also walked Kevin through some of the utility code for generating test signals that I had written earlier in the semester. This highlighted the fact that a lot of the code I had written was not very well documented, so that is something I will work on going forwards.
This week I worked on refining the matlab code for adaptive beamforming. The matlab implementation uses frost beamforming and is now successfully able to take a linear, uniform array of microphones of arbitrary elements n and implement frost beamforming on multiple voice inputs. The code in the link given below is able to take a specific “steering angle”, in our case 0 degrees because the signal of interest is targeted to be directly in front of our user, and reduce the signal inputs coming from other directions.
The link to the matlab file is the following: https://drive.google.com/drive/folders/1E8JMs5zqOZoTGYU2UY8XqLxLb3x19juc?usp=sharing
If you download and run the file, you should be able to hear two different audio outputs. The first one is the sound from all three audio files. You will notice that the laughter noise drowns out almost all other noise, making the speech files inaudible. The second audio output is after frost beamforming is applied. You will notice that the speech file is very audible.
There are several elements of the code that need to be further explored. First, I utilized diagonal loading which is a MATLAB built-in that I read can help robustness of beamforming. I noticed that utilizing this helped make the output audios more clear and understandable. My goal is to look deeper and understand what diagonal loading does and how it improves robustness. Additionally, we have yet to receive our new PCB boards, so the goal is to test this code on real microphone inputs once we build our microphone circuits.
This week, we realized our PCBs for our microphones wouldn’t work due to some of the traces overlapping with a ground pad, so I redesigned the PCB to have a via through to the other side of the board to avoid this issue. I soldered the audio shield to the Teensy board, and got audio output working through the shield to headphones.
I tried to get audio working with the mics professor Sullivan gave us, and the audio input on the Teensy worked, but the mics were too noisy to get any meaningful audio from them.
I also got a 3.072 MHz clock output from the Teensy that we can use to drive the data input from the surface mount microphones.
Once we get our second iteration of the PCBs, I will start to get audio input from them onto the Teensy.
I will work with Kevin and Bobbie to convert their working audio processing functions into C/Arduino code that we can put on the Teensy.
I successfully reduced the distortion in the SDW-MWF implementation. During lab on Monday, Professor Sullivan helped identify the problem as a windowing issue: previously there was no overlap between windows, which caused boundary effects at the edge of each window. Using a 50% window overlap drastically reduced the distortion. This is at the tradeoff of now needing to process the next (overlapping) frame before producing output, which adds 5-9 ms delay to the output. This is within our timing budget, so it is okay. The new output is below.
For the wearable mic array mount, I ordered this phone chest mount from Amazon: https://www.amazon.com/Samsung-Holder-Record-Awesome-Action/dp/B00MYN0CGI/ref=cm_cr_arp_d_product_top?ie=UTF8
This will serve as the basic model of our wearable mount. The main problem we anticipate from this mount is rotation about the central mounting point. Ideally there should be two solid pieces upon which the mics are mounted, with straps connecting as well. This will consist of:
I also wrote code to generate the HINT test recordings. However, this is not completely practical to use yet, because there is not an interface for quickly switching between various SNRs – this is needed because the HINT is an adaptive test, switching SNR levels based on the subject’s success rates.
One of my goals for this week was to come up with a finalize an initial physical setup for the microphone array. I decided that for a linear array of n omnidirectional microphones with equal inter element spacing, the distance between each microphone element should be d=c/(2*f0)where c is the speed of sound and f0 is the midband frequency. Because the human voice frequency in telephony ranges from approximately 300 Hz to 3400 Hz, I chose 1800 Hz to be the midband frequency. As a result, the distance between each microphone was chosen to be 9.5cm apart. This means with a 4 microphone setup, we would require an array of 38 cm in length.
Additionally, regarding how each microphone input would be processed, the delay Tn between the desired signal arriving at the nth microphone and the signal arriving at the first microphone (at the origin of the coordinate system) is Tn =(n-1)*d*sin(ø)/c where d=9.5 cm and c is the speed of sound (343 m/s).
For signal processing, I have an initial MATLAB program which implements Frost Beamforming to try on our microphone setup. I have not done extensive testing yet on the adaptive beamforming code. This following week will be spent testing and modifying the beamforming code with (hopefully) real microphone input signals.
On the hardware side of things, I have worked in lab with Sean to create a temporary microphone setup for testing using microphones that were available from the past. The microphones we used for now are PM: WM-55D103. The specific circuit we made for each condenser microphone is shown in the figure below for testing.
This week, we put together a circuit for the microphones Professor Sullivan gave us and successfully got an analog signal read on the Teensy board. I just used simple analog read functions from the Arduino library to read sound levels. I also starting writing code using the PDM library to process the input from our surface-mount microphones
We decided to go forward with a rechargeable battery pack for external power, pictured below. In order to prevent damaging the board from supplying USB power and external power, I cut the trace between the two pads, to isolate the voltages. I will also cut the power line of the USB cord, just in case the trace isn’t completely cut.
I found tutorials for outputting high speed clocks from the Teensy, and will talk to Professor Sullivan more about these methods.
This week I implemented speech-distortion weighted multi-channel Wiener filtering as described in “Multichannel Filtering For Optimum Noise Reduction In Microphone Arrays” (Florêncio and Malvar, 2001). The results are very strong in reducing noise, but also introduce a very noticeable distortion in the output signal. See the attached audio for an example.
This implementation processes the audio frame-by-frame, writing results to a buffer. For now, the buffer is simply written to file at script completion, but in an actual program it would be read from continuously for output. The 38 seconds of input were processed in ~3 seconds including file reading and writing; this implies that processing a frame takes at most about 70 milliseconds, which is within our time budget. The result shown involved some tweaking of the parameter P as well as the parameters for the least mean squares adaptive filter. I don’t see any glaring errors in my implementation of the algorithm, but the distortion levels are currently quite high and distracting. I will continue looking at this and trying to reduce the distortion, because the noise is reduced by about 24 dB.
Original input:
Post-MWF output:
I also generated some working C code using Matlab Coder. This was quite straightforward to do; Matlab Coder was able to automatically detect the types based on a script which used the relevant functions. We have the option of using Matlab Embedded Coder to generate directly for a ARM Cortex-M target, which may run faster because it will optimize for the particular processor we are using. However, we are not over on our timing budget yet, and it is useful to be able to see the C source code, so we will just use source code generation for now.
This week I looked into how our team would go from our basic Matlab implementation, which consisted of LMS noise cancellation using a reference and primary signal, to using a microphone array. After looking through research papers, adaptive beamforming seemed to be a very promising.
The basic principle of adaptive beamforming is as follows. Beamforming microphone arrays are spatial filters that take multiple microphone signals as input and combine them to a single output signal. Usually the combined output is calculated by filtering each microphone signal through a digital FIR filter and summing the output of all filters as shown in the figure. The filters are designed so that their output add constructively when sound is coming from a specific direction (main lobe) and add destructively when sound is coming from all other directions. This creates the spatial filtering effect of focusing at the sound that is coming from the main lobe direction while attenuating sounds coming from all other directions.
This week I fixed the PCB board so that we can order it through PCBway. This included manually adding layers that weren’t generated by the Eagle software and changing file encodings. I will order that as soon as I hear back from Quinn.
I have starting writing code for the Teensy to get simple inputs from the digital pins, where we will soon be inputting a signal from the microphones given to us by Professor Sullivan.
I did research on outputting a low-noise clock from the Teensy board, which can be done by clock-dividing the 16MHz I2S master clock. I also researched pulse density modulation on the Teensy board, which I found is included in the Audio library for the Teensy. Finally, I looked into power options for the Teensy. For now, we will use USB power, but the final product will need to have a portable battery pack, which we can set up with 2 or 3 AA batteries. Ideally, we’ll use a lower-profile, rechargeable battery for convenience and wearability.
We have just missed a couple milestones on our schedule.
We have not sent a signal through the mics into the hardware system yet. The main delay here has been the very small mics we initially ordered and follow-on issues with PCB design. The PCB issues have been fixed and we should be able to order this week (waiting to hear back from Quinn). In the meantime, on Monday Sean will put together the larger microphones that Professor Sullivan lent us and get an actual input signal by then.
We also have not yet created a HINT Test which we can run on our test environment yet. This is mainly because there is not much point in creating one when the microphone-to-hardware connection to receive the signal does not exist yet. Bobbie will do this over the coming week.
Although these milestones were missed, we are not far behind in the project overall. This is because other non-blocked tasks in signal processing made progress when the tasks for the milestones above were blocked.
As another schedule-related concern, Sean will be travelling most of this week for interviews, and that may put his work slightly behind schedule. He will work diligently while he is gone to try to keep the project moving forward.
This week I generated test audio samples which would simulate the delays involved in sound arriving at our microphone array, using the Matlab script I wrote last week.
I also explored alternative signal processing techniques beyond adaptive nulling which could also be useful in our project. The most promising of these were Speech-distortion weighted multi-channel Wiener filtering (Ngo 2011), and the acoustic rake receiver (Dokmanic, Scheibler, and Vetterli 2015).
Over the course of this research I identified a few common useful utilities and prepared them for use in our project. This includes implementing the short-time Fourier transform and its inverse in Matlab, and finding the documentation for the Matlab “snr” function.
This week I explored the specific use case of white-noise adaptive cancellation with the implementation currently in place. More specifically, trying to pinpoint the conditions that cause the filtering to work well vs. not well.
Additionally, I worked with Bobbie to come up with a quantitative way to analyze our results/ANC output and compare it to our desired signal. We settled on using the Matlab “snr” function to evaluate the signal-to-noise ratio in our algorithm output relative to the original signal.
Also, I emailed professor Cameron Riviere to meet as a group and discuss his previous experience with adaptive noise cancelling. He is a professor from the Robotics department with whom I am currently taking a sensors and sensing class with. We believe he will be a good resource for creating an effective actual microphone array as well as being able to receive feedback on our ANC implementation and figuring out how to integrate our current basic MATLAB implementation with a microphone array. His previous ANC experience does not involve audio-signal inputs, however we should still be able to receive helpful feedback.
This week I noticed some crossing over on the routing microphone surface-mount PCB that I designed in Eagle. It looks like it might not be necessary, so I want to take a look at these and change the routing to avoid crossing over if possible.
I also did some research into audio on the Teensy board and found the Teensy Audio System Design Tool, which provides a GUI for drawing audio block diagrams and exporting Arduino code.
While our signal processing is very likely to be too complex to model graphically using these blocks, this will still be a useful tool that we can use to generate code and simplify getting audio in and out of the Teensy board.
This week we received the parts for the test environment (minus the mannequin) and got the test environment set up and outputting. This required some research into getting IEC958 (S/PDIF) devices working with Ubuntu, but I got it after a little trial and error (images on next page).
In the images you can see that the speakers are quite close to each other. We might need some extension cables to get the speakers farther apart, but I think we should wait until we see if that’s a problem or not. Until we get the mic hardware set up, we can use the Matlab script I wrote last week to simulate sounds from this speaker setup.
This week we started by collecting speech files to test in a basic MATLAB simulation. We now have approximately 100 test files of different people speaking syllables/words/phrases that we can use for not only our MATLAB simulation but also our final hearing test. Additionally, we have started with our basic MATLAB simulations to narrow down the specific adaptive noise cancellation methods we will use. After attempting and tweaking multiple different variations of ANC, we decided to start with a simple Least-Means-Squared (LMS) ANC algorithm that would be both fast and simple to implement.
Below are two examples of our algorithm on two different test signals.
We found that this algorithm works much better when noise (reference) does not greatly affect desired signal (pretty obvious). Additionally, the two greatest factors in accuracy of our cancelled signal depended on both magnitude and period of the noise signal (ex. If reference signal has much smaller period than desired, you get cleaner cancelled signal).
This week we received our surface-mount microphones. They are extremely small, so we need to create a PCB that will allow us to attach wires to it for testing. This week I designed a first draft of the board in Eagle with a premade solder footprint of the INMP621 microphone and attached it to screw-in wire pins. This will make it easy for us to try different wirings in the building phase. Eventually, I will design a similar PCB with solder pads so we can make the device smaller.
Sam said he could get me an Altium license, and I would like to try designing the PCB on there. With Eagle, I can’t rearrange the pins in the design, which is fine for the build testing phase, but may become a problem when we try to design a low-profile device.
I also would like to start programming on the Teensy board with the Arduino IDE. I will start out with simple audio input/output through the Teensy Audio Shield. I will also start researching libraries we can use to make audio processing easier and faster on the Teensy. Kevin and I will work together to implement an algorithm that is a good balance of effective and efficient. The LMS algorithm he is currently testing is fast, but may have some trouble working in our desired testing environment, a loud room, since the frequencies of the desired signal and the noise will be similar, since both will be human voices.