Kevin

Done this week

This week, I looked through numerous examples of source code for beamformers and attempted to adapt them to our use.  I also spent a large amount of time reading through a large portion of text from Professor Sullivan’s Adaptive Nulling textbook.  The textbook helped connect a number of conceptual aspects that I did not clearly understand previously, which has led me to decide to write the beamformer from scratch.  

 

Breakdown the exact parts of the Beamformer I am in the process of writing:

• INPUT:
  • taking in a 4-channel input (each channel buffer is uint16_t[128])

• Build Sample Matrix
  • We want to get a 9-tap filter for each input channel (picked odd-number so there is a center of matrix)
  • Build a matrix of 9 time samples from each input channel and put into vertical matrix
    • [x1(n),x2(n),x3(n),x4(n),x1(n-1),x2(n-1),x3(n-1),x4(n-1), …, x4(n-9)]
    • 9*4 = 36 degrees of freedom
• Find covariance Matrix
  • QR decomposition of sample matrix (I have C code for this)
  • R will give us the Covariance Matrix (80×80 matrix)
• Multiply Inverse of covariance matrix with steering vector.  
  • Steering Vector: [0,0,0,…,0,x1(n-4),x2(n-4),x3(n-4),x4(n-4),0,…,0]
  • 36×1 Matrix
  • The result gives you the coefficients for each of the tapped-delay lines (FIR filters)
• Now take each channel output and put through FIR filter designed in previous step
• Sum up results of each channel

 

I am currently working on a matlab implementation of this specific frost-beamformer so that I can run tests to pick exact specs such as number of taps,samples, etc.  This will also help verify that the algorithms I am running are working. I am also going to verify this with Professor Sullivan during Monday’s lab.

Bobbie

Done this week

This week I spent many hours messing with getting four-channel audio from the microphones working. While four-channel output worked fine with some white noise on channels 3 and 4, I was unable to get any input at all from any of our microphones. This set off a long debugging process:

1. Check to see that the microphones were working on my computer (they were)
2. Check to see that the microphones worked with all the splitters we were using (they did)
3. Check to see that it is actually a lack of input and not some disconnect between the input and the output (it was)
   1. This was done using a variation on the PeakMeter audio example.
4. Check to see all the code was configured correctly
   1. I actually caught a couple spots where the code was taking line in instead of mic, but fixing this still did not give me proper input
5. Check the documentation for assembling the quad channel Teensy: https://forum.pjrc.com/threads/41371-Quad-channel-output-on-Teensy-3-6/
   1. I actually caught a capacitor which should have been removed but had not been. Removing it also did not fix it.
   2. While messing with this, I also broke some of the solder connections which had been fine before, so I had to resolder those.

 

At this point I ran out of ideas and decided to try plugging things in and out and running old code in hopes that something interesting would happen. In fact, two things did:

1. I was able to get noise through the system by half-plugging in a mic jack and twisting it against the port.
   1. This noise registered on PeakMeter, indicating that the input noise was in fact reaching the board.
   2. But, when I plugged in the mic fully, nothing happened.
2. I was able to get complete audio (!) through the system by half-plugging in an audio source (my computer) into the headphone/mic splitter. Beautiful perfect clean audio playing from the speakers.
   1. Nothing good came out of this actually. On testing I found that it only worked with the headphone/mic splitter in particular, which was suspicious. It took a while to figure out, but eventually I narrowed it down to a quirk of the splitter where half-plugging in the mic would send the signal directly to the headphone side, bypassing the board entirely. So really, this was useless.
3. I found that on writing to SD card, the speakers would play a weird tone.
   1. I interpreted this as meaning that there was some short between some SPI-related pins and the ones used to transmit microphone data.
   2. On the diagram, RX (mic input) and SCLK (SPI clock) are in fact right next to each other (towards the top right of the diagram)

 

Because of this latest discovery I have resigned myself to desoldering the boards entirely and doing it again, hopefully cleaner this time. I will do this tomorrow (Sunday).

 

I have also acquired Lexan for use with the microphones. The lapel mics are quite small and light so this will be easy. These have not yet been assembled because of the problems getting input mentioned above

Goals for next week

• Resolder these stupid boards and get input working (pls)
• Test SDW-MWF Teensy Audio code
• Mount the mics

 

Sean

Done this week

This week I soldered the second audio shield onto the board for four-channel audio input/output. Unfortunately, they aren’t working. Output works fine on all channels, but Input is noisy and unreliable at best. Hopefully bobbies debugging skills are better than mine and he can get to the bottom of this.

 

I also rewrote audio recording code to work with four-channel input. The main challenge with this was changing to using multiple audio queues to hold all the data. Going forward, processing four blocks of audio simultaneously may possibly issues that don’t come up with a single buffer.

Goals for next week

Next week, I hope to fix the input on the board. We need to get this fixed in order to be able to use our algorithms as intended.

 

Also, I plan to help Kevin finish up with his beamforming code, adding the Arduino- and Teensy-specific components to his C++ code.

Bobbie

Done this week

This week I worked on getting SDW-MWF to work with the Teensy Audio library. This required changing many constants like the buffer length in each audio frame. However, this ran into problems (segfaults) when I tried to actually integrate it. I spent a good amount of time trying to debug the generated C code (from Matlab Coder), when Professor Sullivan suggested just rewriting it directly.

So I rewrote the code for the Teensy Audio library as a new object (mwf.h and mwf.cpp). This also required implementing a voice activity detector, which I did using the algorithm described in “A Simple But Efficient Real-time Voice Activity Detection Algorithm” (Moattar and Homayounpour, 2009). I tested the voice activity detector against some WAV inputs, and it appears to work with good accuracy. The main time spent here was learning the bit of C++ needed to integrate with the library, and familiarizing myself with the CMSIS DSP Software Library (“arm_math.h”), which is a DSP library optimized for the ARM architecture. For example, a few times I wrote a loop to calculate some simple property like the mean, only to find that I could have just called arm_max_q15(buf, len, &max_size).

I also talked to Kevin about material choice for the microphone rig and decided on mounting the microphones to Lexan polycarbonate, as a material which is strong, stiff and slightly flexible. This is decided over acrylic because it will not shatter under load, and over wood because it stronger. We also decided against 3D-printed ABS because it is less flexible, and our geometry is not complex enough to warrant the time and expense.

Goals for next week

• Acquire scrap Lexan from my shop and cut/drill it to the sizes we need
• Work with Sean to get the 4-mic input to the MWF working on the Teensy board

 

Kevin

Done this week

This week I worked on converting the MATLAB code for beamforming to C code.  I am currently working on a delay sum beamformer for our microphone array.  I am still currently working on a working C implementation partially delayed because this weekend was spent in Ohio.  I currently have code for microphone array initialization and microphone delay calculations.  I have added screenshots of sample code below.

Goals for next week

• Frostbeamforming code conversion to C-code
• Finish building microphone array on test environment
• Test microphone inputs on frost beamformer

Sean

Done this week

I received and tested the PDM microphones this week, but unfortunately translating the data uses too much CPU power. Reading from 1 mic used about 37% CPU and 2 mics used 81%, so we won’t be able to translate data from 4 microphones and process all the audio. Therefore, we’re going forward with using I2S communication via the Teensy audio shields. I found some left/right mono to stereo audio adapters, as well as a microphone/ headphone splitter so we can record and output data on the same I2S port. This setup will allow us to have 4 input and 1 output with only 2 audio shields attached to the Teensy.

I rewrote my Teensy startup code to take in I2S inputs and connect to an empty filter, which will eventually become Kevin’s and Bobbie’s algorithms.  I also implemented a simple program that records for 5 seconds, then plays that audio back. I am using this to test different microphone setups.

Goals for next week

Next week, I want to build the mic setup and test multiple input and single output via I2S. I hope Bobbie and I will be able to implement and test his noise cancelling algorithm on the Teensy board. We will initially try this with only one microphone input.

Kevin

Done this week

I started this week off by looking into converting our MATLAB code into C/C++.  I utilized MATLAB coder, however quickly realized that the coder would not provide direct code that we could utilize.  The biggest issue that I came across was that a number of built-in MATLAB functionalities could not be directly converted.  After changin a lot of the MATLAB built-ins to hand-coded functions, I ended up with one final error which was in coder’s conversion of our frostbeamformer.

While waiting for microphones, I also decided to do more research on beamforming.  More specifically, I have not read too much literature on how reverberant noise affects beamforming algorithms.  Below I included some of the notes I took on different online literature in order to keep notes not only for the progress of this project but also for the future report we will have to write.

 

NOTES:

• What is beamforming
  • Beamforming is achieved by filtering the microphone signals and combining the outputs to extract (by constructive combining) the desired signal and reject (by destructive combining) interfering signals according to their spatial location.
• How to model reverberant noise
  • MSS (multichannel source separation): process of estimating signals from N unobserved sources, given M microphones

• Si and xj are the source and mixture signals respectively, hji is a P-point Room Impulse Response (RIR) from source i to microphone j, P is the number of paths between each source-microphone pair and ∆ is the delay of the pth path from source j to microphone i
• Question: how to model each different path given an unknown room?

• Time-domain vs. frequency-domain beamforming
  • Broadband speech signals can utilize either beamforming techniques

GOALS FOR NEXT WEEK

• Frostbeamforming conversion to C-code
• Start building rig for microphones

Bobbie

Done this week

This week I worked primarily on documenting our existing Matlab code and using Matlab Coder to convert my SDW-MWF implementation into C code. Matlab Coder has certain limitations that make it different from running Matlab directly. For example, system objects cannot take variable length inputs, and variable types must be explicitly declared and cannot change.

The resulting code (excerpt above) is also ugly. This means that it’s more important than ever to have properly documented Matlab source code before generating to C (as comments are preserved).

I also wrote a quick Makefile for actually compiling code which calls into the shared library.
This took an embarrassing amount of time (a few hours) to figure out.

 

I also tweaked the HINT runner to interactively prompt as if a real HINT test is being run, i.e. with binary search on the example tests to find the proper range of SNRs to run it at.

On the chest mount, since our second PCB arrived and was also not correct, I did not have mics to mount. However, the chest mount we ordered did arrive and I took a look at it; it works very well for our purposes and should only require drilling a couple holes to bolt on a rigid surface like polycarbonate to get the microphones on.

Goals for next week

• Work with Sean to get audio input from our newly ordered mics through the Teensy audio shield.
• Process the input on the SDW-MWF C code, and evaluate maximum sampling rate for real-time operation.

 

Sean

Done this week

I ordered a few different microphones to try some simple audio input. I ordered some I2S microphones with breakout boards so we can easily get inputs to the board. These microphones have been tested with the Teensy board and there are tutorials on how to set the up with the board. I also ordered breakout board for the current PDM microphones we have, in case we want to go forward with that.

I soldered the board together and got audio output working through the audio shield this week. I also was able to generate a low-jitter 2 MHz clock on the board to a digital pin, which we can use to drive the microphone data. I finished most of the set up code for outputting the clock, left/right select lines, data in lines, and audio output through the audio shield. I’ve also started on processing 4 audio inputs at once.

Goals for next week

Next week, I need to get an audio signal into the board through our microphones. I will test both types of microphones after they come in, and continue to work on C code adaptations of the Matlab algorithms in the meantime. The microphone integration is vital, and I need to get that done as soon as possible after they arrive.

I will also be working on multithreading the audio input on the Teensy board. There are a few ways this can be done, and I’m currently deciding which option to pursue.

Bobbie

Done this week

This week I wrote a program for interactively running HINT tests. It requires pre-generated audio at varying SNRs, but this helps make the program more portable (as opposed to generating noise at runtime using Matlab).

 

The chest harness did not arrive until very late in the week, so I did not make any progress on that front.

 

Last week we had issues with getting a usable and relatively clean signal out of the electret microphones from Professor Sullivan. I worked on an amplification circuit to bring the voltage values to the range of the Teensy’s ADC, but the signal was incredibly noisy so this was unsuccessful. I did consider building a more complex circuit to try and filter out the largest frequency component of the noise (60 Hz), but I thought this would be not worth the time, as Sean put in the redesigned PCB order so we should have it early next week.

 

I also walked Kevin through some of the utility code for generating test signals that I had written earlier in the semester. This highlighted the fact that a lot of the code I had written was not very well documented, so that is something I will work on going forwards.

Goals for next week

• Build mount for microphones to attach to the purchased chest harness
• Document existing Matlab utility code
• Convert SDW-MWF Matlab code to C using Matlab Coder and adjust for inputs and outputs to be taken in real-time

Kevin

Done this week

This week I worked on refining the matlab code for adaptive beamforming.  The matlab implementation uses frost beamforming and is now successfully able to take a linear, uniform array of microphones of arbitrary elements n and implement frost beamforming on multiple voice inputs.  The code in the link given below is able to take a specific “steering angle”, in our case 0 degrees because the signal of interest is targeted to be directly in front of our user, and reduce the signal inputs coming from other directions. 

 

The link to the matlab file is the following:  https://drive.google.com/drive/folders/1E8JMs5zqOZoTGYU2UY8XqLxLb3x19juc?usp=sharing 

If you download and run the file, you should be able to hear two different audio outputs.  The first one is the sound from all three audio files.  You will notice that the laughter noise drowns out almost all other noise, making the speech files inaudible.  The second audio output is after frost beamforming is applied.  You will notice that the speech file is very audible.

Goals for next week

There are several elements of the code that need to be further explored.  First, I utilized diagonal loading which is a MATLAB built-in that I read can help robustness of beamforming.  I noticed that utilizing this helped make the output audios more clear and understandable. My goal is to look deeper and understand what diagonal loading does and how it improves robustness.  Additionally, we have yet to receive our new PCB boards, so the goal is to test this code on real microphone inputs once we build our microphone circuits.

 

Sean

Done this Week

This week, we realized our PCBs for our microphones wouldn’t work due to some of the traces overlapping with a ground pad, so I redesigned the PCB to have a via through to the other side of the board to avoid this issue. I soldered the audio shield to the Teensy board, and got audio output working through the shield to headphones.

 

I tried to get audio working with the mics professor Sullivan gave us, and the audio input on the Teensy worked, but the mics were too noisy to get any meaningful audio from them.

 

I also got a 3.072 MHz clock output from the Teensy that we can use to drive the data input from the surface mount microphones.

 

Goals for Next Week

Once we get our second iteration of the PCBs, I will start to get audio input from them onto the Teensy.

 

I will work with Kevin and Bobbie to convert their working audio processing functions into C/Arduino code that we can put on the Teensy.