Status Report 3/2 – Team A2: Project LAKE – Logging of Acoustic Keyboard Emanations

Kevin

This week I spent time on preparing for our Design Review presentation on Monday. Our presentation will server a good beginning for the content in our Design Document. I have also begun to work on the Design Document. I generated the BOM for our PCB and ordered the necessary parts. I ordered enough to make 5 complete boards. This is mitigate problems with components breaking and we will also have most of the parts on hand for our next revision.
I made progress on detecting and separating keystrokes. We have found the Matlab Voice Activity Detector was not sufficient in finding keystrokes, especially with background noise. Instead we looked at a few other strategies. The first we tried was using xorr, a correlation algorithm. The goal here is to find every place in the waveform that has the signature pattern of a keystroke. This was not effective and resulted in poor results. The current method is to take a window in the past and find the mean amplitude. We then compare this with the mean of a smaller window looking forward. If the difference is significant, we consider this a keystroke and take a 100ms snippet. Below is an example of a keystroke found this way.
While this method appears to be effective, it is subject to problems with background noise, especially voices. James has been working on a new method which focuses on power instead of amplitude and looks for the characteristic pattern of a keystroke, two humps spaced ~80ms apart.

For this upcoming week, I will primarily be focused on the bring-up of the PCB. We are expecting the PCBs to arrive this week and I would like to have one assembled before Spring break, which starts on Friday 3/8/2019. The plan for this was outlined in the previous weeks progress report. I will be contacting the post-doc we have been working with for some help with assembly. I will also be researching methods of reducing background noise. We may to use both mics in order to complete this task without losing too much data from the keystrokes.

We are currently on track with the PCBs. It would have been nice to receive them this past week but unfortunately we were not lucky with the timing. We are slightly behind with the signal processing. Fortunately, the keystroke detection built this week is good enough to efficiently grab keystrokes from long audio clips. From here we can start extracting features and begin using real data for our machine learning side. James is helping out with the next version of the keystroke separation to help speed things up while I move onto noise reduction.

Last week, we were having trouble with getting the ESP32 to sample I2S at the rate we wanted it to. This turned out to be because we were performing several small TCP transactions instead of one large TCP transaction. As a result we were getting bottlenecked in terms of RAM, we were not sending samples out fast enough and we were dropping samples and it seemed as though we were sampling data at a lower rate that we were actually.
To fix this, we switched to large streaming transfers and we send raw bytes to the laptop rather than sending ascii text. This required some reconfiguration of the network stack to accept bytes rather than ascii text. Now We are able to record data remotely and replay it back on the laptop in matlab.
We did some preliminary testing with music, there was no audible difference between the recorded track and the real track. We also compared the fft spectrums, there was a little bit of noise in the recorded version but we do not anticipate there being any issues with this.

With this, all the embedded code has been written for now, I will now be moving on to writing the deep learning model to translate each of the clusters to letters based on a language model.

There are no changes required, I am on track. I will spend some time next week to collect data using the ESP32 in order to help bring Kevin and James back on schedule.

In parallel with Kevin, I implemented and researched an alternate method of keystroke identification and separation. In order to identify individual keystrokes, the entire recording of keystrokes was temporally divided into non-overlapping 10ms bins. Subsequently, the FFT coefficients of each bin was used to calculate the energy of each bin, which was then normalized between 0 and 1. A sequence of delta vectors was then calculated by subtracting each bin’s energy with the energy of the previous bin. The delta vectors were then amplitude-thresholded to determine the presence of each feature of keystrokes: two surges of energy followed by rapid falls in energy within a 40ms window signify the key-press sequence. A later surge in energy within a 100ms window then signifies the key-release.

Next week, I will continue to refine the keystroke separation, as well as to be able to isolate the different parts of each individual keystroke, as indicated by the waveform above.
Now that we have also fixed the sampling rate issue on the ESP32, we can begin collecting good data from the actual sensor package. In the upcoming weeks, and over Spring Break, I will begin to apply the keystroke separation to larger portions of this data and begin to work with Kevin to extract FFT, cepstrum, and TDoA features.
Now that the risk of not being able to obtain good data from the ESP32 over wifi has been eliminated, the biggest risk to the project has shifted to

I have fallen behind schedule on extracting features from keystrokes. However, now that we have the sensor package up and running, we can begin collecting large amounts of data to work with. I will make use of the data we collect to work on the feature extraction over Spring Break, in order to catch up to schedule.

The major accomplishments this week we were to fix the issues with obtaining data using the ESP32. We were also able to implement a fairly reliable keystroke separator.
The largest task completed this week was the preparation for the Design Review presentation and the writing of the Design Report.

In the next few weeks, we will be moving forward with the signal processing and machine learning portions of the project, as the initial PCB design is mostly complete.

We have fallen slightly behind schedule due to challenges in obtaining data from the ESP32 over wifi as well as the time spend on composing the Design Report. We will make use of the extra time over Spring Break to catch back up to schedule.

No major changes have been made to the project as of now. We are, however, exploring modifications to the noise reduction and keystroke separation methods.