Author: jameszha

Kevin

Accomplishments

• • This week I helped james build a soundproof box he designed. The inside of the box is lined with two layers of a sound deadening material. In the images below this is the silver layer. It is a dense material which is made to absorb sounds for vehicles.
  • There is an acrylic window on top, so the user can see the keyboard, which is two layers thicks. There is an air gap between the layers to try and further increase sound isolation.
  • The box is closed on all sides except where the user inserts their hands. The closed sides are tightly held together to improve sound isolation.
  • The sound proofing material creates significant echoes inside the box since the sound cannot leave easily. To fix this we padded the inside with sound absorbing foam. This is very light material and often used in home studios help with reducing reflecting sounds
  • The sound proofing box is quite effective. We found it can reduce the noise level by 15dB, using an app on my phone to test. We hope this box will help increase our accuracy by significantly reducing the noise in the loud demo environment.

Upcoming Work

Next week is the final demo and report. We will be spending our time finalizing our demo and writing the report.

James

Accomplishments

• • This week, I designed a sound proofing box to help keep the noise levels at moderate, quiet office levels during the demo. The box is 26inches x 18inches x 6inches and can house full sized keyboards. An acrylic window was cut into the top to allow easy viewing of the keyboard.
  • I wrote a simplified version of the code for the demo, allowing for quick input of a recording of an audience member’s typing.

Upcoming work

The only upcoming work is the demo and the final report. We will also record a demonstrative video in case the noise levels in the demo room make effective classification impossible.

Ronit

Accomplishments

• Built the sound proof box and evaluated its effectiveness by collecting data and trying to infer passwords.
• We found that the box reduces outside noise by about 20 dB.
• Setup the esp32 for demo, temporarily disabled power saving mode. (the demo room will be too noisy for the esp to go to sleep).
• Tested the project against a mechanical keyboard. We found that our algorithm works better for membrane keyboards, but this might be because we tuned our parameters for a membrane keyboard.
• Performed user tests(with people other than me kevin and james).
• Worked on video for demo, in case te demo does not work in the noisy demo room.
• Worked on final report.

Upcoming work

• Work on the final report.

Team Status

Accomplishments

• • This week, the team worked closely together to construct a soundproofing box to try to maintain a moderate sound level during the demo.
  • Slightly simplified versions of the signal processing and ESP32 code were written to allow for a more demonstrable project.

Upcoming work

In the next week, we will be demoing our project and writing the final report.

 

Kevin

Accomplishments

• • • • • The new PCBs and parts came in this week. We assembled and tested all of them. There appear to be no major issues with any of the assembled boards. We were able to flash all of them successfully. Additionally, we can power them via batteries and charge the batteries in the manor intended. Both microphones are are functioning properly and we are receiving good audio quality.
        • The does appear to be a minor issue with the battery management unit however, on two of the board when powered only through the 5V pin, the output voltage sometimes drops below 3.3V, causing the ESP32 to brown-out. The issue is likely caused by imperfect solder joints, however it not not severe enough to warrant re-work.
        • I tested the current draw of the board during the different power modes. The board is pulling about 120mA when in normal use and 0.7mA when in deep sleep mode.

Upcoming Work

• • • Next week we will be giving our final presentation, given by James. We are finishing up the Final Presentation and practicing the talk.
    • We will mostly be focused on getting ready for the demo; practicing how everything will work during the live demo.

James

Accomplishments

• • • This week, I worked toward improving the machine learning algorithms toward labeled data, as we were unsuccessful in fully eliminating dropped data.
    • By using labeled data, I was able to achieve a leave-one-out cross validation accuracy of 16%, with a sample size of 1107.
    • Moving forward, using this trained classifier, unlabeled 10-character random passwords were able to be retrieved with fairly decent accuracy. We met our original requirement of guessing 80% of 10-character random passwords in 75 tries or less. We also achieved a successful guess rate of 50% in 5 tries or less, and 40% in 1 try.

Upcoming work

• In this next week, we will need to work toward preparing for the demo. I will further tune the parameters the classifier to attempt to improve accuracy. I will construct a sound-proofing box to lower the loud background noise we expect in the gym during the final demo.

Ronit

Accomplishments

• The ESP32 is still randomly dropping large chunks of data. As such we have been unable to collect accurate tdoa data.
• I have tried switching to the auxiliary peripheral clock, an external, more accurate oscillator on the esp32, however this issue still persists.
• For a time we thought that the issue may be because the dma buffers are filling up, we tried increasing the dma buffer and switching to udp. However, this did not seem to fix the issue.
• We have so far been unable to ascertain the reason for the dropped data.
• I finally tested the esp32’s current draw in deep sleep mode. It come to about 0.7mA. Which is very small compared to its nominal current draw of 0.1-0.3mA during normal data collection and transmission.
• At this point we have to rely on frequency features alone. I worked with James to get a fresh new clean set of data. With leave one out cross correlation, we were able to get about a 20-30% error rate.
• We generated some confusion matrices and used a breadth first approach to generate the top 75 possible guesses for the password.

Upcoming work

• I will continue to investigate the dropped data, but my major focus will be on integration and getting a working demo.

Team Status

Accomplishments

• • • This week, the team worked closely together to assemble the final revision of the PCB. Three working boards were assembled.
    • Different options were explored to eliminate dropped packets to aid in TDoA localization and clustering. However, because we were unable to, we have decided to move forward with labeled data.
    • We have achieved good accuracy in classification using labeled data and have met our original requirement for password accuracy.

Upcoming work

In the next week, we will need to focus on preparing for the final demo.

Kevin

Accomplishments

• • This week I reviewed the last PCB revision. I placed the order for the PCB through PCBway, same as the previous boards. I also ordered more parts so we have enough to build out 3 of the new PCBs. This should be our last order. We won’t have enough time to buy new parts so I ordered extras of some components that could easily get damaged or lost.
  • I worked on improving the keystroke detection using the delta method from earlier. I was able to tune the parameters to get a slightly improved result. However, we switched to using a thresholding method as it seems to perform better.
  • I worked with James on collecting longer samples of data. In order to test our clustering performance, we collected about 30 samples from each key on the keyboard. We then experimented with different clustering techniques and features. Kmeans with euclidean distance gave us the best results.
  • While collecting the data, we used two boards so that we can also experiment with TDoA data. The TDoA appeared to be working until about one third into the audio clip. At that point the keystrokes moved to 85ms apart from each other, which does not make sense. There may be an issue of adding or dropping samples while transmitting.
  • We believe using Cepstral features and TDoA will give us decent clustering results.

Upcoming Work

• • Next week I will be focusing on supporting the effort of training and clustering. This will involve collecting data tuning parameters.
  • The new PCB should arrive this week, so once that arrives I will be putting it together and verifying its functionality.

James

Accomplishments

• • This week, we worked on collecting long samples of 3-way TDoA data. TDoA between the PCB boards was found have high degrees of separation for non-adjacent keys.
  • However, when moving to a much longer audio recording (6 minutes), we found that the audio signal between the two microphones were becoming misaligned, with the same keystroke appearing on one microphone over 80ms before the other. This should not be possible, as that would require a distance difference of 27 meters based on the speed of sound through air. We suspect that one or both of the sensor packages is dropping samples.
  • We found a more faster, more noise-resistant method of cracking the substitution cipher problem, using quadgram probability data from http://practicalcryptography.com/. This method was able to decipher a 5500 word cipher within 10 minutes. Noise and unknown word boundaries had minimal effect.

Upcoming work

• I will need to fine tune the clustering parameters to improve clustering accuracy.

Ronit

Accomplishments

• In order to increase the resolution in the TDOA data, we increased the sampling rate from 40kHz to 60kHz. There is an average separation of about 1cm between each key switch on the keyboard. With a higher sampling rate, the number of samples.
• We were able to get good separation of keystrokes using tdoa and cepstral features.
• We tried using 3-way tdoa, however we were not able to collect good data. The ESP32 dev board with the external mic was not properly impedance matched.
• We found an  efficient means of solving the substitution cipher using ngrams. We are now able to crack substitution ciphers in less than 10 mins for passages under 400 words.

Upcoming work

• We need to collect 3 way tdoa data to get better separation. So we need to build a 3rd PCB.
• We need to start integration.

Team Status

Accomplishments

• • This week put out the order for our final PCB.
  • We found a way to efficiently decoded the substitution cipher.
  • We were able to get good clustering using cepstral and tdoa data.

Upcoming work

In the following weeks, we will need to divert more attention to the machine learning aspects of the project, as well as to fine tune much of the signal processing algorithms in order to be more robust and effective.

Kevin

Accomplishments

• This week I finished the placement and routing of the new PCB. It is 2”x1.5” which should allow for portability as well as easy integration into any package size. While routing, I used many pours this time to connect power between the two power management pins and from external power. This should help to produce cleaner power, leading to better signal integrity.
• The top layer is a ground pour, similar to the last revision, to make routing much simpler. This time, however, the bottom layer is a 3.3V pour, which allowed me to much more easily route power the devices that needed it.
• I also continued researching the Power Level Difference method of sound reduction. I found several descriptive papers that I can possible recreate the method from if we deem it necessary.

Upcoming Work

• Next week I will be working on integrating our signal processing code together to make it much easier to use, and make it a complete system. Ronit, James and I will be spending most of our time working on tuning our design and code to get the results we need.
• I will also be placing the order for our last PCB and any parts we need.

James

Accomplishments

• This week, I rewrote much of the Matlab code in order to allow the different components (filtering, keystroke detection, clustering, etc.) to fit and interface together. This will allow us to test the full system and see how it performs at the current stage in development
• I optimized the keystroke separation and feature extraction algorithms to run much faster, allowing us to process 10-minute recordings within 30 seconds.
• Lastly, I modified the current data receiving server and TDoA algorithm to accept timestamps taken from an NTP server, allowing for much more accurate determination of the start time of each audio clip.

Upcoming work

• This upcoming week, we will begin to fully integrate the different components of the system. We will need to work toward fine tuning the signal processing and machine learning portions of the project.

Ronit

Accomplishments

• We had  a bug whereby the esp32 would collect data via DMA even when the device was not connected to the laptop, as a result we were not starting the recordings of the two mics at the same time and thus we were unable to collect accurate TDOA.
• The fix was making the mic sleep before collecting data and clear the buffers via a soft reset before reconnecting to the laptop.
• To further improve the tdoa data, we now have the the nodes synchronise time with network time protocol . The laptop then tells them to start collecting data some time in the future, this ensures that the tdoa data is being recorded from the same period in time.
• Machine learning is still still a challenge, i explored the naive bayes approach more as per professor Mai’s advice.
• We are also looking at existing substitution cipher solver and seeing if we can leverage them.

Upcoming work

• The next week will be more machine learning. We need to find an efficient method of cracking substitution ciphers.

Team Status

Accomplishments

• This week, we have completed most of the work for the final revision of our PCB. We have also began finalizing the software running on our ESP32, allowing for NTP synchronization and removing bugs like failing to clear buffers between clips.
• We are in the process of organizing the code in order to allow for integration of the individual components of the system.

Upcoming work

• In the following weeks, we will need to divert much more attention to the machine learning aspects of the project, as well as to fine tune much of the signal processing algorithms in order to be more robust and effective.

Changes to schedule description

• There are no major changes to the schedule.

Kevin

Accomplishments

• • This week I began the final revision of our PCB, named Snorlax.
  • This revision has all of the header pins for I/O removed as well as all the header pins for the Vesper wake up mic removed. We decided to power the Vesper at 3.3V since this is within its capabilities and it allows us to remove a linear regulator and related peripherals. I fixed the issue from our previous board by connecting the feedback pin on the buck/boost converter to its Vout pin. Lastly I removed all the LEDs except the one for main power to save space and energy.  
  • Right now I am aiming to have to board fit on a 2”x1.5” board, which is smaller than our previous 2”x2.5” board. Reducing the size is important because it increases portability.
  • I have also been researching ways to reduce non-stationary noise. I have mainly been looking into a technique called Power Level Difference (PLD). PLD is based on having at least two microphones. The general idea is that the signal we want to record is closer to the two microphones that sources of background noise. This means there will be a perceptible difference in power level from close audio source and background sources will have the same power level.
  • The papers I am reading take the PLD concept and created Weiner filters based on the difference of power levels between the two signals.

Upcoming Work

• • Next week I will be finishing the final PCB and reviewing the design. This needs to be completed in order have the PCB ready for the final demo
  • I will also start looking into how to implement the PLD noise reduction algorithm if I have time after completing the PCB. This task is secondary because we can demo our project with a less noisy background if necessary. However, the better noise reduction, the more widely applicable the final device will be.

Ronit

Accomplishments

• • I worked with James to support multiple clients(listening devices) in the network stack.
  • I worked on reducing the power consumption of the PCB by having the board go into deep sleep mode to further reduce power consumption. In this state, the wifi antenna is powered down and the oscillator is turned off. unfortunately , this also means that the gpios are powered off.
  • The mode pin on the vesper microphone has to be set high in order for it to output the digital wakeup signal.
  • This means that that the PCB has to have a pullup resistor on the mode pin so that it is set high even when the processor goes to sleep.
  • I tried optimizing the naive bayes approach, for decoding the substitution cipher. It now terminates 2 around 1 hour 30 mins for texts ~500 words, but this time seems to go up linearly with the amount of noise. Will have to test more.

Upcoming work

• The focus at this stage is purely on machine learning. I have some small tasks left in the esp32, but that should be manageable.

James

Accomplishments

• • I made a few more modifications to the server responsible for receiving sound data from the sensor devices. Now, the connection can be terminated from the server side.
  • We collected some recordings using two of our sensor boards placed at about 3ft apart, with the keyboard placed in the middle. Unfortunately, we discovered that data left in the buffer would pollute future transmissions. The image below shows this occurring. The data before the red line from microphone 1 corresponded to the previous recording. Everything after was the new recording. We will need to correct this by clearing all DMA and TCP buffers at the end of a transmission.
• 
  • Using the data above, I also began working on extracting TDoA data from individual keystrokes. Each keystroke found in one recording is matched to a keystroke within a 40 ms window in the second recording. The two keystrokes are then cross correlated to determine the TDoA. The time differences are visualized in the plot below for each keystroke.

Upcoming work

• • In the next week, I will try to help Ronit ensure that buffers are completely cleared after each transaction.
  • I will begin incorporating TDoA data into clustering the keystrokes as an additional feature. Notably, there is a currently a risk that the sampling rates are not exactly the same between two boards. This will cause drift in the TDoA values. We may need to use the APLL clock to produce a more accurate clock signal if we find that this is an issue.

Team Status

Accomplishments

• • The major accomplishments this week were to begin work on the final PCB design. Many of the debug features were removed, making for a smaller design. All of the final bugs have been worked out regarding the Vesper microphone and the power supply.
  • We have begun work on incorporating TDoA data, including modifying the existing data collection server, and processing data collected on multiple microphones in Matlab.
  • Lastly, we have worked on lowering power consumption of the device by incorporating the Vesper mic and sleeping.

Upcoming work

• In the upcoming weeks, we will need to focus on fully refining the signal processing and machine learning portions of the project. We currently have many individual parts, but have not integrated each component together.

Changes to schedule description

• We are still currently behind schedule. We will be making use of much of the slack time we originally allotted to work toward completing the project on time.

Design Review Presentation

Kevin

Accomplishments

• This week I spent time on preparing for our Design Review presentation on Monday. Our presentation will server a good beginning for the content in our Design Document. I have also begun to work on the Design Document. I generated the BOM for our PCB and ordered the necessary parts. I ordered enough to make 5 complete boards. This is mitigate problems with components breaking and we will also have most of the parts on hand for our next revision.
• I made progress on detecting and separating keystrokes. We have found the Matlab Voice Activity Detector was not sufficient in finding keystrokes, especially with background noise. Instead we looked at a few other strategies. The first we tried was using xorr, a correlation algorithm. The goal here is to find every place in the waveform that has the signature pattern of a keystroke. This was not effective and resulted in poor results. The current method is to take a window in the past and find the mean amplitude. We then compare this with the mean of a smaller window looking forward. If the difference is significant, we consider this a keystroke and take a 100ms snippet. Below is an example of a keystroke found this way.
• While this method appears to be effective, it is subject to problems with background noise, especially voices. James has been working on a new method which focuses on power instead of amplitude and looks for the characteristic pattern of a keystroke, two humps spaced ~80ms apart.

Upcoming Work

• For this upcoming week, I will primarily be focused on the bring-up of the PCB. We are expecting the PCBs to arrive this week and I would like to have one assembled before Spring break, which starts on Friday 3/8/2019. The plan for this was outlined in the previous weeks progress report. I will be contacting the post-doc we have been working with for some help with assembly. I will also be researching methods of reducing background noise. We may to use both mics in order to complete this task without losing too much data from the keystrokes.

Changes to schedule description

• We are currently on track with the PCBs. It would have been nice to receive them this past week but unfortunately we were not lucky with the timing. We are slightly behind with the signal processing. Fortunately, the keystroke detection built this week is good enough to efficiently grab keystrokes from long audio clips. From here we can start extracting features and begin using real data for our machine learning side. James is helping out with the next version of the keystroke separation to help speed things up while I move onto noise reduction.

Ronit

Accomplishments

• Last week, we were having trouble  with getting the ESP32 to sample I2S at the rate we wanted it to. This turned out to be because we were performing several small TCP transactions instead of one large TCP transaction. As a result we were getting bottlenecked in terms of RAM, we were not sending samples out fast enough and we were dropping samples and it seemed as though we were sampling data at a lower rate that we were actually.
• To fix this, we switched to large streaming transfers and we send raw bytes to the laptop rather than sending ascii text. This required some reconfiguration of the network stack to accept bytes rather than ascii text. Now We are able to record data remotely and replay it back on the laptop in matlab.
• We did some preliminary testing with music, there was no audible difference between the recorded track and the real track. We also compared the fft spectrums, there was a little bit of noise in the recorded version but we do not anticipate there being any issues with this.

Upcoming work

• With this, all the embedded code has been written for now, I will now be moving on to writing the deep learning model to translate each of the clusters to letters based on a language model.

Changes to schedule description

• There are no changes required, I am on track. I will spend some time next week to collect data using the ESP32 in order to help bring Kevin and James back on schedule.

James

Accomplishments

• In parallel with Kevin, I implemented and researched an alternate method of keystroke identification and separation. In order to identify individual keystrokes, the entire recording of keystrokes was temporally divided into non-overlapping 10ms bins. Subsequently, the FFT coefficients of each bin was used to calculate the energy of each bin, which was then normalized between 0 and 1. A sequence of delta vectors was then calculated by subtracting each bin’s energy with the energy of the previous bin. The delta vectors were then amplitude-thresholded to determine the presence of each feature of keystrokes: two surges of energy followed by rapid falls in energy within a 40ms window signify the key-press sequence. A later surge in energy within a 100ms window then signifies the key-release.

Upcoming work

• Next week, I will continue to refine the keystroke separation, as well as to be able to isolate the different parts of each individual keystroke, as indicated by the waveform above.
• Now that we have also fixed the sampling rate issue on the ESP32, we can begin collecting good data from the actual sensor package. In the upcoming weeks, and over Spring Break, I will begin to apply the keystroke separation to larger portions of this data and begin to work with Kevin to extract FFT, cepstrum, and TDoA features.
• Now that the risk of not being able to obtain good data from the ESP32 over wifi has been eliminated, the biggest risk to the project has shifted to

Changes to schedule description

• I have fallen behind schedule on extracting features from keystrokes. However, now that we have the sensor package up and running, we can begin collecting large amounts of data to work with. I will make use of the data we collect to work on the feature extraction over Spring Break, in order to catch up to schedule.

Team Status

Accomplishments

• The major accomplishments this week we were to fix the issues with obtaining data using the ESP32. We were also able to implement a fairly reliable keystroke separator.
• The largest task completed this week was the preparation for the Design Review presentation and the writing of the Design Report.

Upcoming work

• In the next few weeks, we will be moving forward with the signal processing and machine learning portions of the project, as the initial PCB design is mostly complete.

Changes to schedule description

• We have fallen slightly behind schedule due to challenges in obtaining data from the ESP32 over wifi as well as the time spend on composing the Design Report. We will make use of the extra time over Spring Break to catch back up to schedule.

High-level Project Changes

• No major changes have been made to the project as of now. We are, however, exploring modifications to the noise reduction and keystroke separation methods.

Kevin

Accomplishments

• This week I finished the PCB design for revision 0. I completed the routing for power, trying to use wide traces and keep them short to reduce noise. These design features were suggested in datasheets for parts we will be using as well as many online tutorials. I routed the data lines and bus lines for our two microphones.  I added a few resistors and capacitors on the I2S bus to improve signal reliability. I did my best to avoid sharp turns with the analog signal from the Vesper mic to avoid filtering the data in ways that are undesirable.
• I also fixed some small issues with part footprints in Eagle. I double checked the footprints for other parts as well, especially the charging circuit ICs since they are small QFN packages. Lastly, I standardized all of the capacitors and resistors, of the type, to have the same package and library in Eagle.
• By the end of the week, the PCB was ordered. We ordered 5 for 1$ each, plus shipping, approximately $17. In the best case scenario, we may be able to get them before the end of next week. If not they should arrive during the following week.
• I also helped out Ronit and James with collecting audio signals from the ESP32. We were able to collect recognizable voice clips, but are not able to set the correct sampling frequency.

Upcoming Work

• Once the PCB arrives, we can begin testing. We will test the board itself first, making sure the traces are connected properly. After this is complete we can begin assembly. If the boards arrive next week, I should be able to test them before the end of the week.
• We also need to order the parts for the PCB. We currently have the LM1117 and the ESP32, but are lacking the power management circuits and microphones. I will be purchasing those early next week.
• The largest risk with the PCB is that they do not work once assembled. I expect debugging to be challenging. Finding the issues as quickly as possible will be critical so we can get another revision of boards ordered. If problems do arise, we will look for help in debugging, likely from the post-doc we spoke with last week.
• Considering it is very likely we will not have the PCBs back next week, I will also be working on signal processing. Keystroke detection has already begun, but by the end of this week I am aiming to have the system automated.

Ronit

Accomplishments

• Getting I2S working on the ESP32 was particularly challenging. Even though the mems microphone is a 24bit device, the clock has to be driven for 32 clock ticks.
• The I2S bus also expects two I2S devices, so instead I set it up to double sample the one device.
• Set up the I2S device to use DMA to increase sample throughput.
• Set up port forwarding within the cmu network to connect the ESP32 to the laptop over CMU’s wireless network.
• Transferred the I2S data from the MEMS microphone to the laptop over TCP
• Although we were getting data on the laptop, it was extremely noisy, to overcome this we first lowered the sample rate until we could debug the sounds more easily. By listening carefully to the sound, I realised that the sounds were being scrambled. I.e- the samples were not playing in the correct order. I was querying only single values from the DMA buffers instead of the whole buffer and under the covers the ESP32 was swapping the buffers and the sounds were playing out of order. I set it to read the entire buffer and increased the number of DMA buffers.

The image below is the sound of human speech that was recorded on the ESP32 and sent over wifi being plotted on Matlab.

Changes to schedule description

• Right now we are able to play back sound on the laptop that has been sampled on the i2s device. However, we are not able to increase the sampling rate to the desired 44.1KHz. At a mere 10kHz, we will not be able to pick up any desired features. This is a problem we did not anticipate would happen and do not have time budgeted in the schedule and as such must move things around. Since we are ahead on the PCB, Kevin will be helping me with the i2s sampling as he has some slack.

Upcoming work

• Not being able to sample at the desired frequency is currently the biggest problem we have. The strange thing is that, the device clock is set to the right rate, and we are able to observe this on the oscilloscope, but for some reason, the device simply does not give us the values we need. I anticipate that this will take no more than 4 days to fix. After which, most of the embedded software stack will be complete and I will move to signal processing.
• Per Professor Mai’s advice, we will be using multiple microphones, this required a slight modification in the network code, but was easily manageable.

James

Accomplishments

• Began work on the TDoA algorithm for multiple microphones. Because we plan on having 2 microphones per board, one high quality and one low power for waking the processor up, we can apply TDoA to a single board as well as to multiple devices. I was able to achieve accuracy on the order of centimeters by using recorded samples of gunshots. The output time difference will simply be used as an additional variable toward the feature-to-cluster algorithm.
• Attempted to build TensorFlow from source in order to enable AVX instructions. Unfortunately, I was not able to get TensorFlow with AVX up and running. I will continue to work on installing the missing dependencies in the following weeks. In the meantime, I will continue to refine the machine learning algorithms using the pre-built PIP TensorFlow package. Although this will result in slower training, we can still proceed with the rest of the project unhindered.
• Helped Ronit get recorded sound transmitted over wifi from the ESP32. This past week, we were struggling to get recorded data transmitted correctly from the ESP32. Because the rest of the signal processing depends on this data, I helped Ronit get back on schedule. We ultimately fixed the issues regarding the clock and data protocols between the ESP32 and I2S microphone, and reading from buffers in the wrong order. We will still need to debug the sampling rate of the microphone, as we are finding that it seems to be stuck at around 8kHz, despite a correct frequency on the I2S clock line.

Upcoming work

• Currently, we are still working on collecting good data from the ESP32 module with the I2S microphone. Once this is complete, Kevin and I can make further headway into extracting features from recorded keystrokes (FFT, cepstral, and TDoA).
• Again, I plan to continue working on getting TensorFlow built from source in the upcoming weeks.
• Next week, I will work with Kevin to refine the keystroke separation algorithm.
• Currently, one of the biggest challenges and risks we face is the fact that we are still unable to obtain good data from the ESP32 over wifi. As a contingency plan, we will move forward with cell-phone data if necessary, until we explore more microphone and data transmission options.

Changes to schedule description

• I am currently a little behind schedule in terms of getting building TensorFlow from source. Fortunately, this is not critical for the other components of the project, so I will work on this in parallel with other work in the upcoming weeks.

Team Status

Accomplishments

• The PCB design was finalized and vetted by Sam.
• We made some last minute changes like adding additional debug headers just to make the debugging process a little easier. Currently the PCB has lots of jumpers and debug gpio ports that will be removed in future iterations, but as this is our first foray into PCB design we deemed it necessary to build in as much fail safe into the PCB as possible.
• Even if the processor on the PCB does not work, we have set it up so that it can still act as a breakout board for all our components.
• A little bit of work was needed to get the network connectivity working exactly how we wanted it to. From our previous blogpost you would know that we were having some trouble sending data from the ESP32 to the laptop but that issue has been resolved.
• We are able to obtain I2S data from the microphone, however there are still some issues in that area.
• We are in the process of creating a means of using time of delay arrival as another data point. Since we already have two microphones, a low power wakeup mic and a good quality MEMs mic, we are attempting to use them to determine the delay in the sound arrival to the two mics. The are spaced 2 inches apart and therefore, the time delay in arrival between the two of them is in the order of tens of milliseconds ,which we will be able to discern once we reach our desired sampling frequency. If we find that using two mics on the same board do not give us the desired results, we will move to using two sensor packages on two PCBs. The network stack has been written so that additional devices can be trivially added.

Upcoming work

• Next week, Ronit will be working on fixing the I2S sampling frequency.
• Kevin and James will be working on separating keystrokes.

Changes to schedule description

• Not being able to sample i2s data and have real data is a setback, however, we are have reordered the schedule so that things dependent on real data have been pushed back.

High-level Project Changes

• The only update to the project at a high level is the inclusion of a second microphone. We have seen some literature (https://www.sigmobile.org/mobicom/2015/papers/p142-liuA.pdf?fbclid=IwAR2ZJi4b4LWak6W2l5iZk5fF9sDwKGeZHDAlDtUJiNDDYE62bAhzhfUjmZE) regarding using two microphones positioned at known locations relative to a keyboard to obtain millimeter-level localization accuracy using TDoA. Because our project’s aim is to create a practical device that an adversary can quickly and discreetly place, we will be able to determine and utilize the exact locations of the microphones in relation to the keyboard, as was done in the research. However, we will incorporate TDoA into our project as an extra data point in the feature-to-cluster machine learning algorithm.

Here are our first week’s progress reports:

Kevin PR #1
Ronit PR #1
James PR #1