Status Report 7 – Team A2: Project LAKE – Logging of Acoustic Keyboard Emanations

Kevin

- This week I began the final revision of our PCB, named Snorlax.
- This revision has all of the header pins for I/O removed as well as all the header pins for the Vesper wake up mic removed. We decided to power the Vesper at 3.3V since this is within its capabilities and it allows us to remove a linear regulator and related peripherals. I fixed the issue from our previous board by connecting the feedback pin on the buck/boost converter to its Vout pin. Lastly I removed all the LEDs except the one for main power to save space and energy.
- Right now I am aiming to have to board fit on a 2”x1.5” board, which is smaller than our previous 2”x2.5” board. Reducing the size is important because it increases portability.
- I have also been researching ways to reduce non-stationary noise. I have mainly been looking into a technique called Power Level Difference (PLD). PLD is based on having at least two microphones. The general idea is that the signal we want to record is closer to the two microphones that sources of background noise. This means there will be a perceptible difference in power level from close audio source and background sources will have the same power level.
- The papers I am reading take the PLD concept and created Weiner filters based on the difference of power levels between the two signals.

- Next week I will be finishing the final PCB and reviewing the design. This needs to be completed in order have the PCB ready for the final demo
- I will also start looking into how to implement the PLD noise reduction algorithm if I have time after completing the PCB. This task is secondary because we can demo our project with a less noisy background if necessary. However, the better noise reduction, the more widely applicable the final device will be.

- I worked with James to support multiple clients(listening devices) in the network stack.
- I worked on reducing the power consumption of the PCB by having the board go into deep sleep mode to further reduce power consumption. In this state, the wifi antenna is powered down and the oscillator is turned off. unfortunately , this also means that the gpios are powered off.
- The mode pin on the vesper microphone has to be set high in order for it to output the digital wakeup signal.
- This means that that the PCB has to have a pullup resistor on the mode pin so that it is set high even when the processor goes to sleep.
- I tried optimizing the naive bayes approach, for decoding the substitution cipher. It now terminates 2 around 1 hour 30 mins for texts ~500 words, but this time seems to go up linearly with the amount of noise. Will have to test more.

The focus at this stage is purely on machine learning. I have some small tasks left in the esp32, but that should be manageable.

- I made a few more modifications to the server responsible for receiving sound data from the sensor devices. Now, the connection can be terminated from the server side.
- We collected some recordings using two of our sensor boards placed at about 3ft apart, with the keyboard placed in the middle. Unfortunately, we discovered that data left in the buffer would pollute future transmissions. The image below shows this occurring. The data before the red line from microphone 1 corresponded to the previous recording. Everything after was the new recording. We will need to correct this by clearing all DMA and TCP buffers at the end of a transmission.
- Using the data above, I also began working on extracting TDoA data from individual keystrokes. Each keystroke found in one recording is matched to a keystroke within a 40 ms window in the second recording. The two keystrokes are then cross correlated to determine the TDoA. The time differences are visualized in the plot below for each keystroke.

- In the next week, I will try to help Ronit ensure that buffers are completely cleared after each transaction.
- I will begin incorporating TDoA data into clustering the keystrokes as an additional feature. Notably, there is a currently a risk that the sampling rates are not exactly the same between two boards. This will cause drift in the TDoA values. We may need to use the APLL clock to produce a more accurate clock signal if we find that this is an issue.

- The major accomplishments this week were to begin work on the final PCB design. Many of the debug features were removed, making for a smaller design. All of the final bugs have been worked out regarding the Vesper microphone and the power supply.
- We have begun work on incorporating TDoA data, including modifying the existing data collection server, and processing data collected on multiple microphones in Matlab.
- Lastly, we have worked on lowering power consumption of the device by incorporating the Vesper mic and sleeping.

In the upcoming weeks, we will need to focus on fully refining the signal processing and machine learning portions of the project. We currently have many individual parts, but have not integrated each component together.

We are still currently behind schedule. We will be making use of much of the slack time we originally allotted to work toward completing the project on time.