- DTCWT PVOC adjusting matrix dimensions for each level of transformation. Goal was so that DTCWT PVOC could time stretch & shrink signals by any factor. Current implementation ONLY allows for time stretching by 1/x where x is a whole number. – Spoke to Jeffrey Livingston about his implementation.
- Adapted DTCWT PVOC functionality to be referenced from function that performs signal pre-processing for the app.
- ran numerous experiments on STFT PVOC & DTCWT PVOC to compare their performance. STFT is by far faster & more accurate. Experiments varied music choices, and speed of music change.
- Ran numerous experiments on STFT PVOC alone with varied speed of music change to compare to our system requirements music tempo range, and computational speed. Not completely sure how to interpret the results.
- Completed final presentation slide deck
Team Status Report for 4/25
We have two weeks left before the end of the semester. At this point, most of our project is wrapping up, although we still have things to aim for. Each component of our project works individually, and 2/3 of the pieces are integrated. We are able to demonstrate that the inputs/outputs of each components match our expectations, and would flow as desired if integrated properly.
The final presentation is on Monday, and will be done by Akash. Accordingly, our group spent time this week running tests for our components, working on the slide deck, and generating graphs to include in the powerpoint. We finalized our decision to move to the phase vocoder as well in consideration of the fact that the wavelet transform experienced greater loss and took longer to process music.
Before “demo day”, we have the following goals:
- Finish testing, and possibly re-test components post-integration
- Attempt to integrate the final two portions
- Implement the -15/+10 BPM requirement for Song Selection
- Create the Video
- Final Report
Akash’s Status Report for 4/25
This week I worked on testing our app and collecting some running data. I ran into some issues with Android Studio, but with the help of Mayur, I was able to download the app onto my phone (the testing device). However, even after that, my phone was not able to collect data with an issue we have still not figured out. I then switched over to using an older Android phone (Galaxy S6) and it worked fine. We got good data that we can use to show the time-scale warp.
I also started working on the slides for our final presentation, that will be given this week. Our presentation is mostly focused on our actual implemented system and our metric validation.
Mayur’s Status Report 4/25
Very short update for this week. We are paradoxically both wrapping up and trying to maximize the amount we finish. I continued attempting to integrate without much success. Namely, my Android phone was not excepting USB connections, which made it difficult to actually physically try out the app. The issue was eventually solved by removing the battery from my phone and putting it back in. Otherwise, I worked with Akash to get the App working on his phone. There were a few problems that we solved, which he will explain in his report. Finally, the presentation is next week, so I am working on the slides. I am still hopeful in getting the integration done, but it is pretty difficult to do.
Team Status Report for 4/18
This week we focused on integrating our independent components.
The matlab code was ported to C++. Documentation for how to integrate the C++ files/functions were created. Upon integration, we ran into issues with (1) typing input variables from Java to C++, (2) translating some matlab functions to Java functionalities — namely, audioread. This is to be done by the Android OS. However, inputting a signal from Java to C++ causes a challenge in typing and storing variables.
To work around this, we are considering creating a C++ function manually to “audioread” wav files. This implementation, however, would then require our java functionality, that initiates music modification every 60 seconds, to be written and integrated in C++. In parallel, we are experimenting with performing “audioread” in Java through the Android OS and storing, typing, and transforming the signal matrices as necessary.
Details about the high-level approach for JNI usage can be found in Mayur’s status report, and details about the time-warping integration and audioread function can be found in Aarushi’s.
Mayur’s Status Report for 4/18
While I believed that the Song Selection Algorithm had been fully implemented with the code, it was not the case. My code was building, but I had not actually attempted to test it on my phone (which was a mistake). As it turns out, I needed to adjust memory settings within the gradle files of the App so that I could increase the allotted memory to hold more songs at runtime. Afterwards, I started to work on integrating the time warping code. This is extremely complicated for several reasons. First, the code requires passing arguments via the JNI. So, code needs to be adjusted so that data is properly serialized/typecast when it is sent between portions of the code. Secondly, the audioread function from matlab needs to be rewritten for C/C++. Finally, several more files need to be “included” with the project, which requires working with CMake and its associated files.
At the moment, I have about 20 tabs open on my computer trying to figure out how to make this work. For now, I believe that the best approach is to send a string with the file name to the C++ code, warping it within C++, writing a new file with C++, and then returning the location of the new file to be played to the Java code. The number one tip for using the JNI according to the official docs in regards to speed is minimizing the amount of data that is sent across it. The JNI needs to marshall/unmarshall all data, which takes a long time. For this reason, doing it the way described would most likely be the fastest approach. Next week I am hoping that we will have finished completely integrating the app and possibly begun testing certain parts. The presentation is the week after!
Aarushi’s Status Report for 4/18
I completed porting the matlab code to C++ for the phase vocoder & stft & istft files – I had to refactor parts of the original matlab code and variable types for this conversion to be compile successful. I also created documentation for how the C++ files should be integrated / used. This documentation was for communication between our team members, for later reference for myself, and for reference for any future user:
This is for IF we modify a song only once, not every minute. It’ll be easier to do that after.
I think the android class we need is AudioTrack (linked).
What you want to do is create a function as follows (i’m gonna write this in pseudo python/java/matlab/comments code sos):
def TSM(song_name, original_bpm, desired_bpm):
ratio = desired_bpm / original_bpm
n = 1024
# audioread is only a matlab function. doesn’t port to C++
# based on my google searches so far…
# we want an analogous JAVA function that is native to androids OS like AudioTrack fns
[original_signal, sampling_rate] = audioread(song_name)
# can be broken up into
audioTrack.getSampleRate() of song_name
# reading the song needs to output a matrix / array of the audio signal
read song_name to signal (not sure which fn in AudioTrack does this) (this link may help)
modified_signal = pvoc(original_signal, ratio, n)
return (modified_signal, sampling_rate)
# calling the function
desired_BPM = 160 # equivalent to running pace
original_BPM = 140 # avg of eye of the tiger bpm got from online
playback_song, sampling_Rate = TSM(“IntroEyeoftheTiger1.wav”, original_BPM, desired_BPM)
audioTrack.play() of playback_song
So far, in actually working with integration, we may not actually be able to use AudioTrack. An idea we are tossing around is writing “audioread” in C++. I am skeptical about this because matlab Coder creates C++ files for all matlab built in functions. Coder does not do this for audioread, however. This being a missing basic functionality of Coder leads me to believe that what we are searching for is NOT a basic functionality. Which is why, blog posts I have read suggest performing “audioread” from the integrated device’s OS. However, there are complicated typing and inputting signal matrices from Java to C++ with this method as well. Both not ideal methods will need to continue to be experimented with.
(DTCWT is still a work in progress.)
Akash’s Status Report for 4/18
This past week I worked on converting the Python implementation of the song selection algorithm to Java. This way we can easily integrate this feature into the app. I had a little difficulty with this as I haven’t coded in Java in a long time, but I was able to use the Python skeleton I wrote to help guide me. Now that it is all in Java, Mayur is going to work on integrating it into the app. The next feature we could add to this would be user preference of songs so that scores could be modified based on how much the user likes specific songs.
Team Status Report for 4/12
After our demo, our team decided that we would work on finishing up loose ends on our individual components. This included (1) porting the song selection algorithm from python to java, (2) porting the audio modification component to C++, and (3) writing code that would break up a song into 60 second chunks before warping. This would allow us to start integration the following week.
We made this decision based on our conversation with Professor Sullivan and Jens as they heavily reemphasized our goals for the end of the project. Additionally, we realized that our work progress was behind in relation to our Gantt chart. According to the plans, we would have completed integration this week and would move on to extended features in our following two weeks of slack time.
Thus, we will take what we have so far, integrate, and continue advancing on our individual parts in the remainder of our slack time, and then integrate again.
Mayur’s Status Report for 4/11
This week, Akash rewrote his code in Java and sent it to me. On my end, I added the song selection algorithm he sent into the code. This week highlighted one source of time sink in the future; up until now, I have only been writing code with a “proof of concept” mindset. Unfortunately, this means that the other two parts of the project are much more difficult to integrate, as the code is not designed well for this to happen. In the coming week, I will be refactoring the code so that it will be [hopefully] as simple as dragging-and-dropping Arushi’s code into the app. It’s pretty obvious integrating will be more complicated, but I want to get the code to as close as that state as possible so that it will be less of a hassle in the future.