Posted on

Eli Status 2/25

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I have started using PyAudio to read/write audio files and do some experimentation with sound generation just to see what its capabilities are.

I referenced this article:

https://plainenglish.io/blog/making-a-synth-with-python-oscillators-2cb8e68e9c3b

To see what simple signal processing/signal manipulation techniques I can use to create interesting sounds. I have the capability to read in guitar audio. I still need to ensure my preprocessing steps can be efficiently accomplished, but I do not foresee any issues with preprocessing.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I have given myself a week for preprocessing audio and a week for testing preprocessing. This means I am not too far behind, but I am hoping to spend much more time next week and have preprocessing and preprocessing testing done early next week so I can start on actual pitch detection. I did not accomplish as much I was hoping to this week (due to other work and sickness). What deliverables do you hope to complete in the next week?

Fully working and tested script to read in guitar audio, perform any noise reduction if necessary, and return the signal with note onset detection.

Posted on

Team Status 2/25

Enumerate how you have adjusted your team work assignments to fill in gaps related to either new design challenges or team shortfalls (e.g. a team member missing a deadline).

To be able to effectively utilize the capabilities of the audio/esp32 board, Sachit and Eli are making time to develop a workflow for working with the board.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk of this project is the generative music model. If the model doesn’t generate pleasant music that would significantly affect the user experience. To mitigate these risks the first approach to building the model will closely follow the research paper  https://arxiv.org/abs/1809.04281. After we have a baseline performance with this model we will modify and optimize the performance.

Since pitch detection is being done on a platform we haven’t used before, there is a risk doing ground up pitch detection will not work as well as we hope. There are a lot of other pitch detection options if this does not work.

Since none of us have used the audio/esp32 board we got, there could be unforseen challenges with using it. If we can’t get it working this week, we will likely opt to record directly to the cloud with cell phones.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

The design is the same, no changes made so far.

Provide an updated schedule if changes have occurred.

No changes.This is also the place to put some photos of your progress or to brag about a component you got working.

 

Posted on

Team Status Report 2/18

Please enumerate one or more principles of engineering, science and mathematics that your team used to develop the the design solution for your project. 

 

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk of this project is the generative music model. If the model doesn’t generate pleasant music that would significantly affect the user experience. To mitigate these risks the first approach to building the model will closely follow the research paper  https://arxiv.org/abs/1809.04281. After we have a baseline performance with this model we will modify and optimize the performance.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We decided to use a transformer based architecture instead of a GAN based architecture, as this would allow more user influence over the music generated. At this stage it did not incure significant costs

Provide an updated schedule if changes have occurred.

This is also the place to put some photos of your progress or to brag about a component you got working.

Posted on

Vedant’s status report for 02/18/2023

Please list the particular ECE courses (if any) that covered the engineering science and mathematics principles your team used to develop your design? If there are no courses please tells us how you learned those principles over the last week?

They aren’t specifically ECE courses but 11-785 (intro to deep learning) and 10-301 (intro to machine learning) covered the engineering principles I’m using in my project.

\What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Since finalizing  the model last week, I have been researching other aspects of the optimization set up that could help our generative music model perform well. I discovered a new optimizer called Lion (EvoLved Sign Momentum) Published in a paper by Google and UCLA researchers less than a week ago The paper is linked Here.  Apart from this I also found libraries that can tokenize the MIDI files which our model takes as input, these tokenized inputs can better facilitate the learning of the generative model here.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I feel that I am on track. I have a good idea of the model I want to implement and how I’m going to implement it.  I have also found a bunch of helpful resources which would provide guidance in design choices

What deliverables do you hope to complete in the next week?

In the coming week I hope to accomplish the following things:

  • Read about different tokenization strategies for MIDI files and the different advantages/disadvantages for each.
  • Understand the LakhMIDI dataset better to understand ways in which I can employ feature engineering to enhance the model
  • Experiment with a pre-built transformer to see the kind of results we can acheive
Posted on

Eli’s Status Report 2/18

Please list the particular ECE courses (if any) that covered the engineering science and mathematics principles your team used to develop your design? If there are no courses please tells us how you learned those principles over the last week?

I am currently taking Digital Signal Processing (18491). Along with an understanding of the basic signal processing ideas I already have, I researched common ways to do pitch detection. My research is below.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I researched my component of transcribing a monophonic guitar into midi. I researched two avenues of this, real time conversion of guitar audio to midi and doing the transcription from a recording (non-real time).

Real time pitch tracking methods and analysis:

http://www.music.mcgill.ca/~ich/classes/mumt611_05/Presentations.old/mumt614pitchtrack.pdf

https://www.cycfi.com/2017/10/fast-and-efficient-pitch-detection/

https://ccrma.stanford.edu/~pdelac/research/MyPublishedPapers/icmc_2001-pitch_best.pdf

I also spent time looking into how to make midi more expressive. Namely supplying meaningful velocity data. Roger Dannenberg has a great paper on this with comparisons of different synth’s velocity and a breakdown of a technical approach for velocity detection.

https://www.cs.cmu.edu/~rbd/papers/velocity-icmc2006.pdf

I also found an interesting master’s project which turns the voice into a midi controller. It implements this via a VST plugin. This seems like an interesting route, but does not apply for non-real time audio to midi conversion:

https://scholarworks.uark.edu/cgi/viewcontent.cgi?article=1013&context=eleguht

and a guide for VSTs:

https://integraudio.com/best-tools-to-develop-vst-plugins/

Another PyAudio midi resource:

https://python.plainenglish.io/build-your-own-python-synthesizer-part-3-162796b7d351

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am on track, I have a good idea of the use case/design requirements and how I will attempt to meet them in creation of a non-real time audio to midi transcription program using the resources I have collected above.What deliverables do you hope to complete in the next week?

I hope to order an amp and adc and start development of a PyAudio program which can record a guitar input (keeping in mind the program will need to run on a rasberry pi.

Posted on

Team Status – 2/11

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Based on the first few weeks, we had not been able to come together as a team for a concept that makes sense and works for all of us until today. To manage risk of further delay, making sure to make and keep short term goals to keep the project on track will be imperative.

Additionally, One of the key components of this project is the generative music model. The goal is to implement a transformer based model from scratch in pytorch, but due to complications such as convergence issues the option to import models is present as a contingency plan. Imported models can be instantiated with or without pre trained weights and even pre trained models can be fine-tuned according to project specifications.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We have decided to change our project to an ML enhanced loop pedal for guitarists. The change was necessary because there was not enough user interaction with the ML model in our previous project idea.

Provide an updated schedule if changes have occurred.

This is also the place to put some photos of your progress or to brag about a component you got working.

 

Posted on

Vedant’s Status Report for 02/11/23

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week for me, was centered around research. The key takeaway from our discussions with faculty were that we need to make our product more interactive for our user. Since I am responsible for the development of the generative model, the interactivity is defined by what the model can take as input and how that can affect the output. The previous model I had decided on (GAN) does not really offer much interactivity for the user. Therefore I researched a variety of different approaches and compiled a document summarizing the strengths and weaknesses of different models, after a team meeting we finalized the approach to move forward with.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Due to the fact that we had to modify our idea, I believe that we are about a week behind schedule. However now that we have a clear vision and purpose for the project I believe we will make up ground quickly and will be back on track by the design review.

Now that we have a model and use case finalized I can start developing prototypes and solidify the workflow. In the next few weeks my plan is to import a transformer model and develop an end-to-end workflow which includes everything from data processing to inference. Once I am confident in the workflow I will move on to model development and experiment with different hyperparameters. right now I am planning to have a workflow ready by spring break, and use the break for model development.

What deliverables do you hope to complete in the next week?

In the coming week I hope to accomplish the following things:

  • Work with magenta API to better understand the flow of data through a transformer based generative model
  • Read the 2018 research paper by Huang et. al which describes the chosen model, and design choices
  • Design and code a simple, basic transformer in pytorch to understand the inner workings of the model

 

Posted on

Eli – 2/11

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Developed project proposal with group. Presented draft project proposal. Met as a team today and decided on new direction for project. Following discussion, I researched options for audio to midi conversion to be able to better redefine user group and scope of project when group is next able to meet:

Found a good source for midi input to Raspberry PI:

https://www.samplerbox.org/makeitelectronic

https://github.com/josephernest/SamplerBox

Audio to MIDI Products:

https://www.sweetwater.com/store/detail/G2MV3–sonuus-g2m-v3-universal-guitar-to-midi-converter/reviews

https://www.sweetwater.com/store/detail/i2M–sonuus-i2m-musicport/reviews

https://www.perfectcircuit.com/kenton-pro-cv-to-midi.html?gclid=CjwKCAiAlp2fBhBPEiwA2Q10Dz8AVSQs6e9ATXALiwCIpGEk7iqlD8WwO-GY1Y1E6YM8pyXWVEK6FhoCIPQQAvD_BwE

Audio to MIDI Software resources:

https://migic.com/

https://www.jamorigin.com/

https://github.com/tiagoft/audio_to_midi

Also found and read the following paper on audio to MIDI conversion:

http://amber.feld.cvut.cz/user/vass/papers/Vass_Poster04.pdf

Which references the following paper which is a very very good resource for signal processing if I end up writing custom audio to MIDI code:

https://www.ee.columbia.edu/~dpwe/papers/Brown91-cqt.pdf

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am behind schedule. We are coming together as a group and almost have a clear collective idea of what the project will be. Once we nail down specifics of who the user group will be and what the components of the project will be, I will have the ability to start actually prototyping/developing my component. Meeting with the group to nail down the user group and define user/design requirements needs to be happening this coming week.

What deliverables do you hope to complete in the next week?

Hope to work with group to define precise design/user requirements. Hopefully this will happen early enough to give me time to research my component. Minimum goals for this week: precise design/user requirements. My component should be well researched, ideally so I am in a good position for outlining how my component will function/prototyping the following week.