Category: Aarushi’s Status Reports

• DTCWT PVOC adjusting matrix dimensions for each level of transformation. Goal was so that DTCWT PVOC could time stretch & shrink signals by any factor. Current implementation ONLY allows for time stretching by 1/x where x is a whole number. – Spoke to Jeffrey Livingston about his implementation.
• Adapted DTCWT PVOC functionality to be referenced from function that performs signal pre-processing for the app.
• ran numerous experiments on STFT PVOC & DTCWT PVOC to compare their performance. STFT is by far faster & more accurate. Experiments varied music choices, and speed of music change.
• Ran numerous experiments on STFT PVOC alone with varied speed of music change to compare to our system requirements music tempo range, and computational speed. Not completely sure how to interpret the results.
• Completed final presentation slide deck

I completed porting the matlab code to C++ for the phase vocoder & stft & istft files – I had to refactor parts of the original matlab code and variable types for this conversion to be compile successful. I also created documentation for how the C++ files should be integrated / used.  This documentation was for communication between our team members, for later reference for myself, and for reference for any future user:

(DTCWT is still a work in progress.)

In working on the time-scale audio modification program component for this project, Prof Sullivan suggested I start in Matlab and write a program that can read an audio file, break the signal into its wavelet components, and reconstruct the signal from its wavelet components. I played around with this. In the wavelet decomposition at levels 2 and 3, the reconstruction sound was faster, thinner, and of considerably higher pitch. Wavelet decomposition at level 1, however, allowed for a reconstructed signal that sounded the same as the original. This is true for ‘db2’ and ‘db1’ wavelet types – we will be using ‘db2’ since it is biorthogonal and our design report details why we chose a biorthogonal implementation. I completed this task this week.

In addition, I worked on a STFT based phase vocoder in Matlab. I was able to successfully test time-scale audio modification through this method with simple one layered musical signals. There were no interfering audio artifacts after the modification of the signals. Having this done means that 1. I can use this phase vocoder implementation as an example as to how to integrate the DTCWT in place of the STFT, and 2. the backup plan for how we would modify the signals is complete in Matlab.

Next steps are to test these on more complex signals, and implement the phase vocoder with DTCWT.

This week involved further group discussions addressing how our project would move forward in our remote environments. Discussions with Prof Sullivan and Jens provided insights on how we could break our project down into milestones that may be safely achievable, to those that would target goals, to those that would be stretch goals. Working through this conceptualization with the team was helpful. It made us more comfortable with the current situation. These were detailed in our Statement of Work, another document we worked on this week.

We also decided on individual tasks for the next week that would help prove that our planned milestones are feasible. While Akash and Mayur are working to collect step count data, my week’s task is to create a skeleton for the time warping algorithm (create a design), and be able to import and visualize mp3 files into this skeleton.

To be clear, our group has decided that our week’s will start/end on Wednesdays to allow enough time and flexibility for us to get our individual tasks done, and for communication/coordination.

p.s. I apologize for this report being one day late. As a result of fracturing my hand mid last week and the new COVID-19 ‘stay at home’ sanctions in New Jersey, I last minute had to spend the weekend moving from Pittsburgh to home. I would really appreciate the flexibility, here.

This report consists of an update over the last two weeks — the latter week was Spring Break.

In the week before Spring Break, I got badly sick  with the flu. As a result, I spent the first half of the week resting and allocating all my ‘work time’/energy working on the design report. This included completing my parts which included the architecture, conclusion, future work, and all the audio modification sections. It also included formatting, and cohesive revisions since the design proposal presentation. Through this work, I realized that we were missing a key design requirement: the audio files that our product would support. After starting and completing research on what file types are best for signal processing, music, and popular audio uses, I completed this section in the design report as well. Considering my illness, and that I spent half of the week on this document.

The following week was Spring Break. Our group did not have plans to work this week. I was supposed to be traveling on a CMU sponsored brigade. Since this was cancelled, I decided to be slightly productive. I scheduled meetings with 2 professors, as previously identified, who would be able to advise on the phase vocoder I plan to implement. Once classes were officially moved online, our project was paused as we were waiting on how to proceed in working on this project remotely. Once our expectations are solidified, our team members will plan how to complete what is needed. For now, we have already identified three individual parts of the project that can be worked on independently. How these components will be integrated and tested is TBD based on forthcoming expectations and group conversations as classes resume this week.

This past week, I heavily focused on the Design Presentation – ensuring a few slides, in particular, were clean, consistent, and logical. These few slides were complex and contained a lot of information that was difficult to aesthetically fit (i.e. slides 2,  3, 7). Additionally, I was the speaker for the design presentation. Thus, I spent considerable time rehearsing since our presentation was on Wednesday which allowed for more practice time.

Additionally, after working with the group to delegate portions of the design report, I have been working through my portion of the document. This will continue past Saturday night.

Other than meeting tasks assigned by the class, I took this week to really understand the benefits and drawbacks of the ‘wavelet transform based’ phase vocoder in relation to other time-scale modification methods we could use. This involved reading (very slowly, haha) numerous research papers and their findings on tradeoffs between the various methods. I now understand the high-level processes behind each of these methods, what distinguishes them from each other, and how exactly each of the more advanced/more accurate techniques builds off of the closest simpler/less accurate method considered. (These methods were explained in the design presentation & will be in the design report).

Next steps, following the completion of the design report, will be to complete a base portion/’method’ of the described phase vocoder.

While last week involved preliminary testing and information gathering for requirements for the design proposal, this week proved to offer drastic and unexpected changes.

1. Software Decisions – Last week we decided that the best integration method between the wavelet transform and the mobile app would include the Wavelet transform implemented in python, and integrated to android studio via Jython. However, this method would have required a different library to be used on the Java side that would not interface with the phone’s step counter data. While researching and discussing this issue, Professor Sullivan suggested using C++ for the wavelet transform. Since I will be working on the wavelet transform, I took this decision particularly personally. I have little experience with C++. In fact, even upon initially playing with C++ to familiarize myself with the language, I was still uncomfortable. Despite my distaste for this language, it was important to note that C++ offered DWT & IDWT (discrete wavelet transform) methods AND these were well documented: http://wavelet2d.sourceforge.net/#toc-Chapter-2. In fact, these implementations and example use cases I found provided more customization and flexibility with the input/output signals than Python’s libraries and examples proved to show. As a result, I decided to bite the bullet to favor using C++ for its easier integration with the mobile app, its flexibility with signal processing, and its sufficient, clear examples/documentation of wavelet transform use cases.
2. Device Decisions – NO watch, ONLY phone based on step counter data we measured and acquired since the watch’s step counter was the least accurate despite the fact that it is of a recent generation.
3. Scoping Step Detection – As of last week, we decided our target audience to be runners. However, after performing our own ‘runner’s’ test, we realized that the discrepancy between our target scope of bpm was because our searches were targeting runners, but we were actually referencing joggers. Additionally, our measured pace/desired bpm of 150-180 bpm actually matched up well with many songs I do use to fast ‘jog’ to. Thus, we adjusted our target bpm/pace accordingly to match this pace.
4. Music Choice – During our proposal presentation, we received feedback to narrow the scope of inputs – AKA the scope of songs that could be used to run to with warped conditions. With our new target pace, we will allow only songs of 150-180bpm. Additionally, when choosing a song from a defined, we will apply a scoring algorithm. This scoring algorithm will give a song a score depending on how many times its played, and how close the song’s natural bpm is to the jogger’s current pace. The algorithm will choose the song with the best score. This will ensure one song is not constantly on repeat, and a song of decent bpm is played. Both factors will be weighed and adjusted relatively based on the outcome of our algorithm.
5. Wavelet Transform vs Phase Vocoder metrics and tradeoffs were searched for and validated as expected. Additionally a plan to accomplish the wavelet transform has been made: code and test for a sin wave without changing inputs, do the same and test on simple music, account for tempo variations, account for pitch, & measure artifacts throughout this process. I have additionally researched additional resources on campus in case I need guidance in applying the wavelets for our specific use cases (i.e. Prof Aswin, Stern, Paul Heckbert, Jelena Kovačević)