Posted on

Eli Status 2/25

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I have started using PyAudio to read/write audio files and do some experimentation with sound generation just to see what its capabilities are.

I referenced this article:

https://plainenglish.io/blog/making-a-synth-with-python-oscillators-2cb8e68e9c3b

To see what simple signal processing/signal manipulation techniques I can use to create interesting sounds. I have the capability to read in guitar audio. I still need to ensure my preprocessing steps can be efficiently accomplished, but I do not foresee any issues with preprocessing.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I have given myself a week for preprocessing audio and a week for testing preprocessing. This means I am not too far behind, but I am hoping to spend much more time next week and have preprocessing and preprocessing testing done early next week so I can start on actual pitch detection. I did not accomplish as much I was hoping to this week (due to other work and sickness). What deliverables do you hope to complete in the next week?

Fully working and tested script to read in guitar audio, perform any noise reduction if necessary, and return the signal with note onset detection.

Posted on

Team Status 2/25

Enumerate how you have adjusted your team work assignments to fill in gaps related to either new design challenges or team shortfalls (e.g. a team member missing a deadline).

To be able to effectively utilize the capabilities of the audio/esp32 board, Sachit and Eli are making time to develop a workflow for working with the board.

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk of this project is the generative music model. If the model doesn’t generate pleasant music that would significantly affect the user experience. To mitigate these risks the first approach to building the model will closely follow the research paper  https://arxiv.org/abs/1809.04281. After we have a baseline performance with this model we will modify and optimize the performance.

Since pitch detection is being done on a platform we haven’t used before, there is a risk doing ground up pitch detection will not work as well as we hope. There are a lot of other pitch detection options if this does not work.

Since none of us have used the audio/esp32 board we got, there could be unforseen challenges with using it. If we can’t get it working this week, we will likely opt to record directly to the cloud with cell phones.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

The design is the same, no changes made so far.

Provide an updated schedule if changes have occurred.

No changes.This is also the place to put some photos of your progress or to brag about a component you got working.

 

Posted on

Vedant’s status report for 02/25/2023

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I consolidated resources for building a GPT from scratch. These include:

  • This video by Andrej Karpathy:  https://www.youtube.com/watchv=kCc8FmEb1nY&ab_channel=AndrejKarpathy
  • This article with code and theory:  https://habr.com/en/company/ods/blog/708672/
  • This Jupyter notebook implements a similar workflow as a reference, I will follow a similar process for experimentation and prototyping:  https://github.com/Natooz/MidiTok/blob/main/colab-notebooks/Full_Example_HuggingFace_GPT2_Transformer.ipynb
  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I feel that this week I am a little behind schedule due to midterms and other work that was brought forward. This next week will be relatively light and I intend to put more time into the project implementing an end-to-end workflow.

  • What deliverables do you hope to complete in the next week?

The main deliverable I want to achieve next week is having an end-to-end workflow ready using a pre-built GPT

Posted on

Sachit’s Status Report for 2/25

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I attended the design review presentations and learned more about each group. Additionally, I kept working with Eli on the website dashboard design on Figma and trying to improve the interface and ease of use for users. Below is the link for the Figma design: https://www.figma.com/file/NgYIERzxk8RFzgk5p14ZKj/Untitled?node-id=0%3A3&t=IFlWXSzODiPX6mkB-0.

Moreover, I brushed up on my knowledge of distributed systems and networking to better understanding how the multiple software components should interact with each other.

Eli and I acquired the esp32 board and started interfacing with it to understand how its Wifi and audio recording capabilities were.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I believe we are on schedule with the only component needed for the project already acquired. Even though, I don’t believe in being complacent and plan to finish understanding the esp32 this week.

What deliverables do you hope to complete in the next week?

I want to start learning and using Bootstrap and Django to build the web app and also finish testing the capabilities of ESP32 along with Eli. I also am to complete the design report to better understand the small details of our project.

Posted on

Team Status Report 2/18

Please enumerate one or more principles of engineering, science and mathematics that your team used to develop the the design solution for your project. 

 

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk of this project is the generative music model. If the model doesn’t generate pleasant music that would significantly affect the user experience. To mitigate these risks the first approach to building the model will closely follow the research paper  https://arxiv.org/abs/1809.04281. After we have a baseline performance with this model we will modify and optimize the performance.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We decided to use a transformer based architecture instead of a GAN based architecture, as this would allow more user influence over the music generated. At this stage it did not incure significant costs

Provide an updated schedule if changes have occurred.

This is also the place to put some photos of your progress or to brag about a component you got working.

Posted on

Vedant’s status report for 02/18/2023

Please list the particular ECE courses (if any) that covered the engineering science and mathematics principles your team used to develop your design? If there are no courses please tells us how you learned those principles over the last week?

They aren’t specifically ECE courses but 11-785 (intro to deep learning) and 10-301 (intro to machine learning) covered the engineering principles I’m using in my project.

\What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Since finalizing  the model last week, I have been researching other aspects of the optimization set up that could help our generative music model perform well. I discovered a new optimizer called Lion (EvoLved Sign Momentum) Published in a paper by Google and UCLA researchers less than a week ago The paper is linked Here.  Apart from this I also found libraries that can tokenize the MIDI files which our model takes as input, these tokenized inputs can better facilitate the learning of the generative model here.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I feel that I am on track. I have a good idea of the model I want to implement and how I’m going to implement it.  I have also found a bunch of helpful resources which would provide guidance in design choices

What deliverables do you hope to complete in the next week?

In the coming week I hope to accomplish the following things:

  • Read about different tokenization strategies for MIDI files and the different advantages/disadvantages for each.
  • Understand the LakhMIDI dataset better to understand ways in which I can employ feature engineering to enhance the model
  • Experiment with a pre-built transformer to see the kind of results we can acheive
Posted on

Sachit’s Status Report 2/18

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Having finalized our use case to be a jam tool alongside a piano accompaniment for guitar players, I moved on to designing the software components and tech stack.

  1. I worked with my team members to understand how the users should interface with the dashboard and designed the website in Figma alongside with the user workflow.
  2. The tech stack will comprise of a frontend in bootstrap with a backend in Django running a cloud container for the signal processing as well as ML inference.
  3. I developed the block diagram for interaction between software components namely the Raspberry pi, frontend and the backend.

I also extensively worked on the design review presentation mainly the implementation and requirements.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I believe my progress is on track with the design for the frontend done and the software components ideated.  As a team we have also caught up after pivoting from our proposal presentation.

What deliverables do you hope to complete in the next week?

Next thing I need to start working on is learning to work with the raspberry pi and interfacing it with the cloud, speakers and microphone. I also need to finalize the user interface experience based on feedback. Moreover, also work on the design report and detail the software components more in depth.

Posted on

Eli’s Status Report 2/18

Please list the particular ECE courses (if any) that covered the engineering science and mathematics principles your team used to develop your design? If there are no courses please tells us how you learned those principles over the last week?

I am currently taking Digital Signal Processing (18491). Along with an understanding of the basic signal processing ideas I already have, I researched common ways to do pitch detection. My research is below.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I researched my component of transcribing a monophonic guitar into midi. I researched two avenues of this, real time conversion of guitar audio to midi and doing the transcription from a recording (non-real time).

Real time pitch tracking methods and analysis:

http://www.music.mcgill.ca/~ich/classes/mumt611_05/Presentations.old/mumt614pitchtrack.pdf

https://www.cycfi.com/2017/10/fast-and-efficient-pitch-detection/

https://ccrma.stanford.edu/~pdelac/research/MyPublishedPapers/icmc_2001-pitch_best.pdf

I also spent time looking into how to make midi more expressive. Namely supplying meaningful velocity data. Roger Dannenberg has a great paper on this with comparisons of different synth’s velocity and a breakdown of a technical approach for velocity detection.

https://www.cs.cmu.edu/~rbd/papers/velocity-icmc2006.pdf

I also found an interesting master’s project which turns the voice into a midi controller. It implements this via a VST plugin. This seems like an interesting route, but does not apply for non-real time audio to midi conversion:

https://scholarworks.uark.edu/cgi/viewcontent.cgi?article=1013&context=eleguht

and a guide for VSTs:

https://integraudio.com/best-tools-to-develop-vst-plugins/

Another PyAudio midi resource:

https://python.plainenglish.io/build-your-own-python-synthesizer-part-3-162796b7d351

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am on track, I have a good idea of the use case/design requirements and how I will attempt to meet them in creation of a non-real time audio to midi transcription program using the resources I have collected above.What deliverables do you hope to complete in the next week?

I hope to order an amp and adc and start development of a PyAudio program which can record a guitar input (keeping in mind the program will need to run on a rasberry pi.

Posted on

Team Status – 2/11

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Based on the first few weeks, we had not been able to come together as a team for a concept that makes sense and works for all of us until today. To manage risk of further delay, making sure to make and keep short term goals to keep the project on track will be imperative.

Additionally, One of the key components of this project is the generative music model. The goal is to implement a transformer based model from scratch in pytorch, but due to complications such as convergence issues the option to import models is present as a contingency plan. Imported models can be instantiated with or without pre trained weights and even pre trained models can be fine-tuned according to project specifications.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We have decided to change our project to an ML enhanced loop pedal for guitarists. The change was necessary because there was not enough user interaction with the ML model in our previous project idea.

Provide an updated schedule if changes have occurred.

This is also the place to put some photos of your progress or to brag about a component you got working.

 

Posted on

Vedant’s Status Report for 02/11/23

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week for me, was centered around research. The key takeaway from our discussions with faculty were that we need to make our product more interactive for our user. Since I am responsible for the development of the generative model, the interactivity is defined by what the model can take as input and how that can affect the output. The previous model I had decided on (GAN) does not really offer much interactivity for the user. Therefore I researched a variety of different approaches and compiled a document summarizing the strengths and weaknesses of different models, after a team meeting we finalized the approach to move forward with.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Due to the fact that we had to modify our idea, I believe that we are about a week behind schedule. However now that we have a clear vision and purpose for the project I believe we will make up ground quickly and will be back on track by the design review.

Now that we have a model and use case finalized I can start developing prototypes and solidify the workflow. In the next few weeks my plan is to import a transformer model and develop an end-to-end workflow which includes everything from data processing to inference. Once I am confident in the workflow I will move on to model development and experiment with different hyperparameters. right now I am planning to have a workflow ready by spring break, and use the break for model development.

What deliverables do you hope to complete in the next week?

In the coming week I hope to accomplish the following things:

  • Work with magenta API to better understand the flow of data through a transformer based generative model
  • Read the 2018 research paper by Huang et. al which describes the chosen model, and design choices
  • Design and code a simple, basic transformer in pytorch to understand the inner workings of the model