Posted on

Vedant’s status report for 02/25/2023

  • What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I consolidated resources for building a GPT from scratch. These include:

  • This video by Andrej Karpathy:  https://www.youtube.com/watchv=kCc8FmEb1nY&ab_channel=AndrejKarpathy
  • This article with code and theory:  https://habr.com/en/company/ods/blog/708672/
  • This Jupyter notebook implements a similar workflow as a reference, I will follow a similar process for experimentation and prototyping:  https://github.com/Natooz/MidiTok/blob/main/colab-notebooks/Full_Example_HuggingFace_GPT2_Transformer.ipynb
  • Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I feel that this week I am a little behind schedule due to midterms and other work that was brought forward. This next week will be relatively light and I intend to put more time into the project implementing an end-to-end workflow.

  • What deliverables do you hope to complete in the next week?

The main deliverable I want to achieve next week is having an end-to-end workflow ready using a pre-built GPT

Posted on

Sachit’s Status Report for 2/25

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I attended the design review presentations and learned more about each group. Additionally, I kept working with Eli on the website dashboard design on Figma and trying to improve the interface and ease of use for users. Below is the link for the Figma design: https://www.figma.com/file/NgYIERzxk8RFzgk5p14ZKj/Untitled?node-id=0%3A3&t=IFlWXSzODiPX6mkB-0.

Moreover, I brushed up on my knowledge of distributed systems and networking to better understanding how the multiple software components should interact with each other.

Eli and I acquired the esp32 board and started interfacing with it to understand how its Wifi and audio recording capabilities were.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I believe we are on schedule with the only component needed for the project already acquired. Even though, I don’t believe in being complacent and plan to finish understanding the esp32 this week.

What deliverables do you hope to complete in the next week?

I want to start learning and using Bootstrap and Django to build the web app and also finish testing the capabilities of ESP32 along with Eli. I also am to complete the design report to better understand the small details of our project.

Posted on

Team Status Report 2/18

Please enumerate one or more principles of engineering, science and mathematics that your team used to develop the the design solution for your project. 

 

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk of this project is the generative music model. If the model doesn’t generate pleasant music that would significantly affect the user experience. To mitigate these risks the first approach to building the model will closely follow the research paper  https://arxiv.org/abs/1809.04281. After we have a baseline performance with this model we will modify and optimize the performance.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We decided to use a transformer based architecture instead of a GAN based architecture, as this would allow more user influence over the music generated. At this stage it did not incure significant costs

Provide an updated schedule if changes have occurred.

This is also the place to put some photos of your progress or to brag about a component you got working.

Posted on

Vedant’s status report for 02/18/2023

Please list the particular ECE courses (if any) that covered the engineering science and mathematics principles your team used to develop your design? If there are no courses please tells us how you learned those principles over the last week?

They aren’t specifically ECE courses but 11-785 (intro to deep learning) and 10-301 (intro to machine learning) covered the engineering principles I’m using in my project.

\What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Since finalizing  the model last week, I have been researching other aspects of the optimization set up that could help our generative music model perform well. I discovered a new optimizer called Lion (EvoLved Sign Momentum) Published in a paper by Google and UCLA researchers less than a week ago The paper is linked Here.  Apart from this I also found libraries that can tokenize the MIDI files which our model takes as input, these tokenized inputs can better facilitate the learning of the generative model here.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I feel that I am on track. I have a good idea of the model I want to implement and how I’m going to implement it.  I have also found a bunch of helpful resources which would provide guidance in design choices

What deliverables do you hope to complete in the next week?

In the coming week I hope to accomplish the following things:

  • Read about different tokenization strategies for MIDI files and the different advantages/disadvantages for each.
  • Understand the LakhMIDI dataset better to understand ways in which I can employ feature engineering to enhance the model
  • Experiment with a pre-built transformer to see the kind of results we can acheive
Posted on

Sachit’s Status Report 2/18

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Having finalized our use case to be a jam tool alongside a piano accompaniment for guitar players, I moved on to designing the software components and tech stack.

  1. I worked with my team members to understand how the users should interface with the dashboard and designed the website in Figma alongside with the user workflow.
  2. The tech stack will comprise of a frontend in bootstrap with a backend in Django running a cloud container for the signal processing as well as ML inference.
  3. I developed the block diagram for interaction between software components namely the Raspberry pi, frontend and the backend.

I also extensively worked on the design review presentation mainly the implementation and requirements.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I believe my progress is on track with the design for the frontend done and the software components ideated.  As a team we have also caught up after pivoting from our proposal presentation.

What deliverables do you hope to complete in the next week?

Next thing I need to start working on is learning to work with the raspberry pi and interfacing it with the cloud, speakers and microphone. I also need to finalize the user interface experience based on feedback. Moreover, also work on the design report and detail the software components more in depth.

Posted on

Eli’s Status Report 2/18

Please list the particular ECE courses (if any) that covered the engineering science and mathematics principles your team used to develop your design? If there are no courses please tells us how you learned those principles over the last week?

I am currently taking Digital Signal Processing (18491). Along with an understanding of the basic signal processing ideas I already have, I researched common ways to do pitch detection. My research is below.

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I researched my component of transcribing a monophonic guitar into midi. I researched two avenues of this, real time conversion of guitar audio to midi and doing the transcription from a recording (non-real time).

Real time pitch tracking methods and analysis:

http://www.music.mcgill.ca/~ich/classes/mumt611_05/Presentations.old/mumt614pitchtrack.pdf

https://www.cycfi.com/2017/10/fast-and-efficient-pitch-detection/

https://ccrma.stanford.edu/~pdelac/research/MyPublishedPapers/icmc_2001-pitch_best.pdf

I also spent time looking into how to make midi more expressive. Namely supplying meaningful velocity data. Roger Dannenberg has a great paper on this with comparisons of different synth’s velocity and a breakdown of a technical approach for velocity detection.

https://www.cs.cmu.edu/~rbd/papers/velocity-icmc2006.pdf

I also found an interesting master’s project which turns the voice into a midi controller. It implements this via a VST plugin. This seems like an interesting route, but does not apply for non-real time audio to midi conversion:

https://scholarworks.uark.edu/cgi/viewcontent.cgi?article=1013&context=eleguht

and a guide for VSTs:

https://integraudio.com/best-tools-to-develop-vst-plugins/

Another PyAudio midi resource:

https://python.plainenglish.io/build-your-own-python-synthesizer-part-3-162796b7d351

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am on track, I have a good idea of the use case/design requirements and how I will attempt to meet them in creation of a non-real time audio to midi transcription program using the resources I have collected above.What deliverables do you hope to complete in the next week?

I hope to order an amp and adc and start development of a PyAudio program which can record a guitar input (keeping in mind the program will need to run on a rasberry pi.

Posted on

Team Status – 2/11

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Based on the first few weeks, we had not been able to come together as a team for a concept that makes sense and works for all of us until today. To manage risk of further delay, making sure to make and keep short term goals to keep the project on track will be imperative.

Additionally, One of the key components of this project is the generative music model. The goal is to implement a transformer based model from scratch in pytorch, but due to complications such as convergence issues the option to import models is present as a contingency plan. Imported models can be instantiated with or without pre trained weights and even pre trained models can be fine-tuned according to project specifications.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We have decided to change our project to an ML enhanced loop pedal for guitarists. The change was necessary because there was not enough user interaction with the ML model in our previous project idea.

Provide an updated schedule if changes have occurred.

This is also the place to put some photos of your progress or to brag about a component you got working.

 

Posted on

Vedant’s Status Report for 02/11/23

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week for me, was centered around research. The key takeaway from our discussions with faculty were that we need to make our product more interactive for our user. Since I am responsible for the development of the generative model, the interactivity is defined by what the model can take as input and how that can affect the output. The previous model I had decided on (GAN) does not really offer much interactivity for the user. Therefore I researched a variety of different approaches and compiled a document summarizing the strengths and weaknesses of different models, after a team meeting we finalized the approach to move forward with.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Due to the fact that we had to modify our idea, I believe that we are about a week behind schedule. However now that we have a clear vision and purpose for the project I believe we will make up ground quickly and will be back on track by the design review.

Now that we have a model and use case finalized I can start developing prototypes and solidify the workflow. In the next few weeks my plan is to import a transformer model and develop an end-to-end workflow which includes everything from data processing to inference. Once I am confident in the workflow I will move on to model development and experiment with different hyperparameters. right now I am planning to have a workflow ready by spring break, and use the break for model development.

What deliverables do you hope to complete in the next week?

In the coming week I hope to accomplish the following things:

  • Work with magenta API to better understand the flow of data through a transformer based generative model
  • Read the 2018 research paper by Huang et. al which describes the chosen model, and design choices
  • Design and code a simple, basic transformer in pytorch to understand the inner workings of the model

 

Posted on

Sachit – 2/11

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I personally am in charge of the software development of the project. Initially, we were deciding to create a sound experience where I was focused on a distributed speaker system. However, after more in-depth research we have pivoted to a more user-centric project which allows budding musicians have a real time music accompaniment with ML generated music. I will focused on developing embedded code for the ras pi to communicate with speakers, cloud and a cloud microservice to allow users to control the device easily through their device and also upload a custom MIDI file for accompaniment.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Due to the pivot, I believe we are about a week behind a schedule. However we have a week for the design review and we believe we can redefine our use case requirements and schedule much faster after doing it once and catch up in term of a thorough design review.

What deliverables do you hope to complete in the next week?

Have a thorough specification of the dashboard and cloud microservice and moreover research more into chip specification which allows for cloud interaction and ml inference with ease of use peripherals.

Posted on

Eli – 2/11

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

Developed project proposal with group. Presented draft project proposal. Met as a team today and decided on new direction for project. Following discussion, I researched options for audio to midi conversion to be able to better redefine user group and scope of project when group is next able to meet:

Found a good source for midi input to Raspberry PI:

https://www.samplerbox.org/makeitelectronic

https://github.com/josephernest/SamplerBox

Audio to MIDI Products:

https://www.sweetwater.com/store/detail/G2MV3–sonuus-g2m-v3-universal-guitar-to-midi-converter/reviews

https://www.sweetwater.com/store/detail/i2M–sonuus-i2m-musicport/reviews

https://www.perfectcircuit.com/kenton-pro-cv-to-midi.html?gclid=CjwKCAiAlp2fBhBPEiwA2Q10Dz8AVSQs6e9ATXALiwCIpGEk7iqlD8WwO-GY1Y1E6YM8pyXWVEK6FhoCIPQQAvD_BwE

Audio to MIDI Software resources:

https://migic.com/

https://www.jamorigin.com/

https://github.com/tiagoft/audio_to_midi

Also found and read the following paper on audio to MIDI conversion:

http://amber.feld.cvut.cz/user/vass/papers/Vass_Poster04.pdf

Which references the following paper which is a very very good resource for signal processing if I end up writing custom audio to MIDI code:

https://www.ee.columbia.edu/~dpwe/papers/Brown91-cqt.pdf

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I am behind schedule. We are coming together as a group and almost have a clear collective idea of what the project will be. Once we nail down specifics of who the user group will be and what the components of the project will be, I will have the ability to start actually prototyping/developing my component. Meeting with the group to nail down the user group and define user/design requirements needs to be happening this coming week.

What deliverables do you hope to complete in the next week?

Hope to work with group to define precise design/user requirements. Hopefully this will happen early enough to give me time to research my component. Minimum goals for this week: precise design/user requirements. My component should be well researched, ideally so I am in a good position for outlining how my component will function/prototyping the following week.