This week I worked on writing and creating slides with Caleb and Sanjana for the upcoming Design Presentation. Concurrently, I looked more into the specifications of the Google Board to double check if the board can handle our intensive processing. Our system needs to be able to process real time audio and align it with our MIDI file using Dynamic Time Working. Our system also needs to process real time eye-tracking coming in from the Tobii Eye Tracker Camera. During our presentation practice with Cynthia, she expressed some concerns about the processing power of our Google Board. After reading over the datasheets and specification documents of the Google Coral Dev Board, I found out that it can perform 4 trillion operations per seconds, using 0.5 watts for each TOPS. I also looked into past computer vision projects based on this board, and this told me that the Google Board will be more than enough as our compute for our system.
Additionally, I did some more research on how to implement Dynamic Time Warping, by looking at existing research based python implementations of this algorithm. Dynamic Time Warping is essential in implementing Audio Alignment for our system. This website that I looked at gave me a great starting understanding of how this algorithm works:
https://builtin.com/data-science/dynamic-time-warping
It goes into great detail about how to accurately segment two audio streams and then perform complex matrix calculations to align the two signals. It even gave some starting python code for some of the math behind this algorithm.
The Classes I have taken to assist my team in building SoundSync:
- 18-290 Signals & Systems: This class explored signal processing and signal transformations for processing image and sound signals. It delved into the math behind signal processing. This class gave a strong foundation for assisting us build the Dynamic Time Warping algorithm.
- 18-220 Electronic Devices & Analog Circuits: This class covers many topics within designing analog electrical engineering systems. Specifically, I will be utilizing the skills I picked up about analog signal design and filtering which we will need to properly implement our instrument frequency calibration.
- 18-349 Introduction to Embedded Systems: This class gave a deep dive introduction to designing the software and hardware for real-time Embedded Systems. It also taught me how to properly manage several concurrent tasks within an Embedded System. This class gave me a good basis for when I need to write the low-level software when programming our Google Coral Dev Board. For example, we need to use many GPIO ports for various buttons, and this class taught me how to program GPIO ports into a microcontroller.
Currently, my progress is on schedule, and I hope to make significantly more progress when our parts come in.