{"id":51,"date":"2019-02-24T00:42:42","date_gmt":"2019-02-24T00:42:42","guid":{"rendered":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/?p=51"},"modified":"2019-02-24T00:42:42","modified_gmt":"2019-02-24T00:42:42","slug":"status-report-2-23","status":"publish","type":"post","link":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/2019\/02\/24\/status-report-2-23\/","title":{"rendered":"Status Report 2\/23"},"content":{"rendered":"<h2>Kevin<\/h2>\n<h3><span style=\"font-weight: 400\">Accomplishments<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">This week I finished the PCB design for revision 0. I completed the routing for power, trying to use wide traces and keep them short to reduce noise. These design features were suggested in datasheets for parts we will be using as well as many online tutorials. I routed the data lines and bus lines for our two microphones. \u00a0I added a few resistors and capacitors on the I2S bus to improve signal reliability. I did my best to avoid sharp turns with the analog signal from the Vesper mic to avoid filtering the data in ways that are undesirable. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">I also fixed some small issues with part footprints in Eagle. I double checked the footprints for other parts as well, especially the charging circuit ICs since they are small QFN packages. Lastly, I standardized all of the capacitors and resistors, of the type, to have the same package and library in Eagle.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">By the end of the week, the PCB was ordered. We ordered 5 for 1$ each, plus shipping, approximately $17. In the best case scenario, we may be able to get them before the end of next week. If not they should arrive during the following week. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">I also helped out Ronit and James with collecting audio signals from the ESP32. We were able to collect recognizable voice clips, but are not able to set the correct sampling frequency.<\/span><\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-54 size-full\" src=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed.png\" alt=\"\" width=\"1393\" height=\"1600\" srcset=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed.png 1393w, http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-261x300.png 261w, http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-768x882.png 768w, http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-892x1024.png 892w\" sizes=\"(max-width: 1393px) 100vw, 1393px\" \/><\/p>\n<h3><span style=\"font-weight: 400\">Upcoming Work<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Once the PCB arrives, we can begin testing. We will test the board itself first, making sure the traces are connected properly. After this is complete we can begin assembly. If the boards arrive next week, I should be able to test them before the end of the week.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">We also need to order the parts for the PCB. We currently have the LM1117 and the ESP32, but are lacking the power management circuits and microphones. I will be purchasing those early next week. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The largest risk with the PCB is that they do not work once assembled. I expect debugging to be challenging. Finding the issues as quickly as possible will be critical so we can get another revision of boards ordered. If problems do arise, we will look for help in debugging, likely from the post-doc we spoke with last week.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Considering it is very likely we will not have the PCBs back next week, I will also be working on signal processing. Keystroke detection has already begun, but by the end of this week I am aiming to have the system automated. <\/span><\/li>\n<\/ul>\n<h2>Ronit<\/h2>\n<h3><span style=\"font-weight: 400\">Accomplishments<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Getting I2S working on the ESP32 was particularly challenging. Even though the mems microphone is a 24bit device, the clock has to be driven for 32 clock ticks.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The I2S bus also expects two\u00a0I2S devices, so instead I set it up to double sample the one device.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Set up the I2S device to use DMA to increase sample throughput. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Set up port forwarding within the cmu network to connect the ESP32 to the laptop over CMU\u2019s wireless network.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Transferred the I2S data from the MEMS microphone to the laptop over TCP<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Although we were getting data on the laptop, it was extremely noisy, to overcome this we first lowered the sample rate until we could debug the sounds more easily. By listening carefully to the sound, I realised that the sounds were being scrambled. I.e- the samples were not playing in the correct order. I was querying only single values from the DMA buffers instead of the whole buffer and under the covers the ESP32 was swapping the buffers and the sounds were playing out of order. I set it to read the entire buffer and increased the number of DMA buffers.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">The image below is the sound of human speech that was recorded on the ESP32 and sent over wifi being plotted on Matlab.<\/span><\/p>\n<p><span style=\"font-weight: 400\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-55\" src=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-1.png\" alt=\"\" width=\"1600\" height=\"442\" srcset=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-1.png 1600w, http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-1-300x83.png 300w, http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-1-768x212.png 768w, http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-content\/uploads\/sites\/17\/2019\/02\/unnamed-1-1024x283.png 1024w\" sizes=\"(max-width: 1600px) 100vw, 1600px\" \/><\/span><\/p>\n<h3><span style=\"font-weight: 400\">Changes to schedule description<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Right now we are able to play back sound on the laptop that has been sampled on the i2s device. However, we are not able to increase the sampling rate to the desired 44.1KHz. At a mere 10kHz, we will not be able to pick up any desired features. This is a problem we did not anticipate would happen and do not have time budgeted in the schedule and as such must move things around. Since we are ahead on the PCB, Kevin will be helping me with the i2s sampling as he has some slack.<\/span><\/li>\n<\/ul>\n<h3><span style=\"font-weight: 400\">Upcoming work<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Not being able to sample at the desired frequency is currently the biggest problem we have. The strange thing is that, the device clock is set to the right rate, and we are able to observe this on the oscilloscope, but for some reason, the device simply does not give us the values we need. I anticipate that this will take no more than 4 days to fix. After which, most of the embedded software stack will be complete and I will move to signal processing.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Per Professor Mai\u2019s advice, we will be using multiple microphones, this required a slight modification in the network code, but was easily manageable.<\/span><\/li>\n<\/ul>\n<h2>James<\/h2>\n<h3><span style=\"font-weight: 400\">Accomplishments<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Began work on the TDoA algorithm for multiple microphones. Because we plan on having 2 microphones per board, one high quality and one low power for waking the processor up, we can apply TDoA to a single board as well as to multiple devices. I was able to achieve accuracy on the order of centimeters by using recorded samples of gunshots. The output time difference will simply be used as an additional variable toward the feature-to-cluster algorithm.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Attempted to build TensorFlow from source in order to enable AVX instructions. Unfortunately, I was not able to get TensorFlow with AVX up and running. I will continue to work on installing the missing dependencies in the following weeks. In the meantime, I will continue to refine the machine learning algorithms using the pre-built PIP TensorFlow package. Although this will result in slower training, we can still proceed with the rest of the project unhindered.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Helped Ronit get recorded sound transmitted over wifi from the ESP32. This past week, we were struggling to get recorded data transmitted correctly from the ESP32. Because the rest of the signal processing depends on this data, I helped Ronit get back on schedule. We ultimately fixed the issues regarding the clock and data protocols between the ESP32 and I2S microphone, and reading from buffers in the wrong order. We will still need to debug the sampling rate of the microphone, as we are finding that it seems to be stuck at around 8kHz, despite a correct frequency on the I2S clock line.<\/span><\/li>\n<\/ul>\n<h3><span style=\"font-weight: 400\">Upcoming work<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Currently, we are still working on collecting good data from the ESP32 module with the I2S microphone. Once this is complete, Kevin and I can make further headway into extracting features from recorded keystrokes (FFT, cepstral, and TDoA).<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Again, I plan to continue working on getting TensorFlow built from source in the upcoming weeks. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Next week, I will work with Kevin to refine the keystroke separation algorithm. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Currently, one of the biggest challenges and risks we face is the fact that we are still unable to obtain good data from the ESP32 over wifi. As a contingency plan, we will move forward with cell-phone data if necessary, until we explore more microphone and data transmission options.<\/span><\/li>\n<\/ul>\n<h3><span style=\"font-weight: 400\">Changes to schedule description <\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">I am currently a little behind schedule in terms of getting building TensorFlow from source. Fortunately, this is not critical for the other components of the project, so I will work on this in parallel with other work in the upcoming weeks.<\/span><\/li>\n<\/ul>\n<h2>Team Status<\/h2>\n<h3><span style=\"font-weight: 400\">Accomplishments<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The PCB design was finalized and vetted by Sam. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">We made some last minute changes like adding additional debug headers just to make the debugging process a little easier. Currently the PCB has lots of jumpers and debug gpio ports that will be removed in future iterations, but as this is our first foray into PCB design we deemed it necessary to build in as much fail safe into the PCB as possible.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Even if the processor on the PCB does not work, we have set it up so that it can still act as a breakout board for all our components. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">A little bit of work was needed to get the network connectivity working exactly how we wanted it to. From our previous blogpost you would know that we were having some trouble sending data from the ESP32 to the laptop but that issue has been resolved.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">We are able to obtain I2S data from the microphone, however there are still some issues in that area.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">We are in the process of creating a means of using time of delay arrival as another data point. Since we already have two microphones, a low power wakeup mic and a good quality MEMs mic, we are attempting to use them to determine the delay in the sound arrival to the two mics. The are spaced 2 inches apart and therefore, the time delay in arrival between the two of them is in the order of tens of milliseconds ,which we will be able to discern once we reach our desired sampling frequency. If we find that using two mics on the same board do not give us the desired results, we will move to using two sensor packages on two PCBs. The network stack has been written so that additional devices can be trivially added.<\/span><\/li>\n<\/ul>\n<h3><span style=\"font-weight: 400\">Upcoming work<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Next week, Ronit will be working on fixing the I2S sampling frequency.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Kevin and James will be working on separating keystrokes.<\/span><\/li>\n<\/ul>\n<h3><span style=\"font-weight: 400\">Changes to schedule description<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Not being able to sample i2s data and have real data is a setback, however, we are have reordered the schedule so that things dependent on real data have been pushed back.<\/span><\/li>\n<\/ul>\n<h3><span style=\"font-weight: 400\">High-level Project Changes<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The only update to the project at a high level is the inclusion of a second microphone. We have seen some literature (<a href=\"https:\/\/www.sigmobile.org\/mobicom\/2015\/papers\/p142-liuA.pdf?fbclid=IwAR2ZJi4b4LWak6W2l5iZk5fF9sDwKGeZHDAlDtUJiNDDYE62bAhzhfUjmZE\">https:\/\/www.sigmobile.org\/mobicom\/2015\/papers\/p142-liuA.pdf?fbclid=IwAR2ZJi4b4LWak6W2l5iZk5fF9sDwKGeZHDAlDtUJiNDDYE62bAhzhfUjmZE<\/a><\/span><span style=\"font-weight: 400\">) regarding using two microphones positioned at known locations relative to a keyboard to obtain millimeter-level localization accuracy using TDoA. Because our project\u2019s aim is to create a practical device that an adversary can quickly and discreetly place, we will be able to determine and utilize the exact locations of the microphones in relation to the keyboard, as was done in the research. However, we will incorporate TDoA into our project as an extra data point in the feature-to-cluster machine learning algorithm. <\/span><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Kevin Accomplishments This week I finished the PCB design for revision 0. I completed the routing for power, trying to use wide traces and keep them short to reduce noise. These design features were suggested in datasheets for parts we will be using as well as many online tutorials. I routed the data lines and &hellip; <a href=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/2019\/02\/24\/status-report-2-23\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Status Report 2\/23<\/span><\/a><\/p>\n","protected":false},"author":46,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/posts\/51"}],"collection":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/users\/46"}],"replies":[{"embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/comments?post=51"}],"version-history":[{"count":2,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/posts\/51\/revisions"}],"predecessor-version":[{"id":57,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/posts\/51\/revisions\/57"}],"wp:attachment":[{"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/media?parent=51"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/categories?post=51"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama2\/wp-json\/wp\/v2\/tags?post=51"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}