Posted on

Lee’s Status Report for 3/18/2023

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on the CV code to segment out black keys. I first started by obtaining test images from online that were just top down view to start off. Then, I created functions to input those test images from my hard drive to run the algorithms on them. I first started off with changing it to grayscale. Then I put a gaussian blur on it to prepare it for putting edge detection onto it. Lastly, I decided to put a sobel filter for horizontal and vertical edge detection to obtain the image below:

Furthermore, the camera attatchment for the laptop and phone came in and we put it on the laptop with the phone on it to see if it worked and it was able to point down at the direction we desired. We did not take a photo of it during class but an image of what this looks like can be found on the amazon store:

I also did some further research with finding out several ways to look at doing warp perspective since the view will not always be top-bottom view. I came up with 3 ideas. 1. Have the student choosing (clicking) points on the streaming video to choose where the piano is. 2. Use feature matching by passing in an image of the top-down view of the piano. 3. Add a red border so the CV algorithm recognizes the 4 corner points of the piano. After speaking with Tamal, he suggested changing the actual attachment itself to a goose neck, phone holder, tri-pod ring light mechanism. This would give it the flexibility to reach over and look down from a top-down view making our lives easier. Additionally, the ring light was a great suggestion since he said that lighting conditions could greatly affect the CV algorithm so it was importatnt to make this a control element rather than a variable. For corner points,  he said red border would be best, or you even make the corners plus signs and just do feature matching for the plus signs to easily identify it. I will certainly take these into considerations.

Lastly, I worked on doingthe ethics assignment this week too. This includes reading through the two articles and completing tasks 1 and 2 of the assignment.

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is a bit behind as the black keys need to be segmented. However, all I need to do is put a binary threshold filter on it to filter out white vs black keys. Since we have edge detection achieved, I will use a segmentation function to add distinct values to the binary values of pixels and borders of the edge detection. So I will be working on that tomorrow before the start of the week on Monday.

“ What deliverables do you hope to complete in the next week?

By next week according to the schedule, I will have the white keys also segmented too. Additionally, I plan to have choose the goose neck phone holder ring light tri-pod amazon product on Monday during class to have it sent to the order form by Tuesday to get that in early. This will not stunt by progress since I am relying on test images still. I also will talk to my group mates on the task 3 of the ethics assignment where we discuss that and take ntoes. I also want to work on adding the corner point plus signs to the test images to see how well the warp perspective performance will do.

Posted on

Caroline’s Status Report for 3/18/23

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I set up a simple circuit to test the sensitivity of the pressure sensors we purchased. At the time of this status report, I am currently writing the code to test the sensors, so that when more than 50g of pressure is sensed, the light will turn on. This is the initial test to see if the sensors work correctly.

In addition to this, I have downloaded and set up Kivy and Xcode so that we are able to start writing the interface for the app next week. I also familiarized myself with the platforms so that I am comfortable to start writing code for it next week.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I believe that we are slightly behind schedule because the pressure sensor testing should have been done before/during spring break. However, because this step isn’t crucial to the progression of our project, I am able to continue it on the side while also working on the interface with Nish.What deliverables do you hope to complete in the next week?

Next week, I will continue to stress test the pressure sensors and create a series of LED lights, each corresponding with a range of pressures (50g to 2g) to see how sensitive the sensors are.

I also hope to write some basic code in Kivy and Xcode so we can have an app interface complete before interim demos.

Posted on

Nish’s Status Report 3/18

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I evaluated the ethics of our project in the Ethics assignment. We feel that overall, our project does not have negative ethical dilemmas. However, it solidified our ideas that we only want to target experienced piano players (to some degree), as we don’t want to cause bad hand posture and bad technique in younger/beginner users, if they attempt to play the piano on a flat surface for a long time.

 

I also worked for a bit in trying to get our oscillators and synth sounds. However, this work is now obsolete given that we will just use an existing package. I’ve been looking through and testing different audio packages and MIDI packages available in Python. pypiano and fluidsynth are the ones we are now considered, and will narrow down when we work on audio processing again later in our timeline.

 

I also spent a large amount of time learning how MIDI protocol works, and working through tutorials to practice sending and receiving MIDI messages and writing them to a file. I used a bit of an arrangement of Dandelions that I had been working on and wrote out MIDI messages and tried to play them back in a .mid file, using a few resources such as this that uses MATLAB to actually write the messages to the file:

https://www.mathworks.com/help/audio/ug/convert-midi-files-into-midi-messages.html

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

After evaluating our risks (as seen in the group report), we feel a little behind as a lot of the work that I was trying to do this week is kind of obsolete now. To get back on schedule, we are focusing on having something working for our interim demo. That will require our interface to be up and running, so both Caroline and I will be working together next week to build up the structure of the app in Kivy, and hopefully be able to launch it through Xcode as well.

What deliverables do you hope to complete in the next week

Next week, I hope to have been able to launch our app from our iPhone, such that it shows a calibration page and accesses the camera.

Posted on

Team Status Report 3/18

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

I think our most significant risks in how we are going to be integrated our different components and subsystems. In order to be able to work, we have different branches in our repo for CV, Interface, and Audio, and plan on integrating them together in the master branch. However, when CV and Audio are integrated together, we will probably have to write in some threading capabilities so that CV can continually run  while MIDI timing is being taken with  the pressure sensor, so we will require a timing mechanisms.

In order to mitigate this, we will take a few steps. Firstly, we will edit points in our schedule so that we merge in each CV milestone with our interface. This will require us to set up the bare bones of the interface rather quickly, so we will be pushing the audio details later. In our interface, we will set up a software timer that runs/is checked as necessary, with a minimum BPM of 30, so we will always merge the CV milestones with a timer in place and software interrupts are triggered by a note being played via the pressure sensors to check the timer and send the MIDI message.

Our contingency plan is to have a MIDI message sent on each beat. This is not ideal, as we may have a note that starts on a half beat, but given that we are focusing on the note accuracy + volume and interface in this project, we feel that it would be okay to have this as a contingency plan.

 

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Given the feedback on our design report, we have decided to edit our implementation of the audio processing flow. We will not spend time trying to figure out the frequency spectrums of each note, but rather use a piano package with MIDI available through Python to render our sounds. We are currently exploring a few different options for these packages. This will reduce a lot of workload by using work already available to us rather than starting from scratch and trying to build deep sounds. Although we received feedback that suggested we use a soundcard as we are under budget, 1) we want to play the sounds through the phone for the portability aspects, as we expect our user to be using their laptop for actual score writing, and 2) we want to minimize cost to the user, as outlined in our use case.

Posted on

Nish’s Status Report 3/11

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

In the past week, I worked on the design report. Particularly, I worked on the Design Trade, Architecture, and System Overview sections. I made the block diagrams and also created the audio processing flow, as we switched our focus from PCB design (hardware) due to choosing the Arduino Nano into audio processing. I worked a lot on researching how synthesizers are made and finding the software we will need to implement them, such as pygame midi and the Oscillator packages, and understanding how to shape our sound waves to sound like a piano.

 

Here are some of the diagrams I created: 

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I think we are on schedule. Most of our work is now in the software/audio realm, so we are able to focus on prototyping and understanding those applications next week even though we haven’t gotten our Arduino Nano yet.

What deliverables do you hope to complete in the next week?

Next week, I hope to complete running a dummy Kivy app that is able to make all 49 pitches that we are hoping to include in our piano, and be able to shape the waves using the Sustain and Release modulators talked about in our design report, experimenting with what gets the most realistic sound.

Posted on

Caroline’s Status Report for 3/11/2023

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

In the past week, I worked with the rest of my group to finalize our design report document. I was originally tasked to design the PCB that would connect the pressure sensors with the Bluetooth and microcontroller, but we changed our design to use the Arduino Nano BLE instead. So, in the past week, I helped research the documentation behind the Arduino Nano and how to best connect it within our system. I helped draw up an example diagram of how to best implement the gloves so that when our Arduino Nano BLE arrives, I can quickly assemble them together so that we can test them as quickly as possible.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Currently, I believe that our progress is on schedule. We have shifted our GANTT chart to account for our spring break. We will begin the bulk of our project when we return from spring break.

What deliverables do you hope to complete in the next week?

In the next week, I hope to research and download XCode and write a dummy file to test my abilities and the different functionalities that XCode has. In addition, if our Arduino Nano BLE arrives, I will be assembling the components on the gloves.

Posted on

Team Status Report for 3/11/2023

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Some of the significant risks that could jeapordize the success of the project are edge cases if the user presses inbetween two keys. How are ties broken? What if they are closer to one but still on the other? We mitigated this risk by simply taking the higher note (in pitch) to break ties.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We made  several changes to the existing design our system requirements. The first being identifying a battery to charge the gloves + micro-controller. We decided to settle on using AAA batteries in order to power this since they are small and lightweight and can provide enough to power the device at 3.3v for about roughly 3 hours.

Another change we made to the system is coloring the white keys of the keyboard layout to differentiate between octaves. We know that the camera field of view cannot see the entire length of the keyboard, so if it is only fixed on viewing two octaves at once (our current requirement), distinguising which octaves via different colors will significantly aid the CV algorithm.

For identifying fingers, we dicussed that we would add another color to the finger by printing a color out and attaching it to the top of the gloves in the view of the camera. This would help the CV algorithm identify what in the image are fingers. In addition, we are considering attaching fiducials of that color to these fingers, so the algorithm can further differentiate the fingers from one another, which will be integral when it gets the data from the micro-controller about which finger it is that is pressing a key from the pressure sensor.

As you’ve now established a set of sub-systems necessary to implement your project, what new tools have your team determined will be necessary for you to learn to be able to accomplish these tasks?

We’ve established that the iphone or android camera will need to interface with the app. This sometimes poses a technical challenge since making our product for both apple and android phones is tricky. We will be using and need to learn to use Xcode since it will be used as a wrapper over the python kivy interface to accomondate for apple products.

For interfacing the cameras of apple phones to Lee’s windows laptop for testing and streaming was also a challenge. He learned to use a tool called Iriun webcam to help accomplish having a streaming video feed from his iphone camera to work on his windows laptop. Android phones have an easier time doing this since the apple app store has may have more restrictions.

 

Posted on

Lee’s Status Report for 3/11/2023

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week, I worked on researching the capabilities of OpenCV to see the extent of what functions to use to identify and differentiate piano keys and fingers from one another. This is still a work-in-progress and still needs to be more fleshed-out. Some other additional things I helped out with was brainstorming with my team some more details on our project that was not mentioned during the design review presentation but are integral. Some of these was researching a battery to be used to power the gloves + micro-controller.

Another one was for the CV component was coloring the octaves on the piano printout to aid the software side to identify what octave of notes are being played. Since we know that the full layout of the piano keys cannot fit inside the range and view of the camera, it is paramount to distinguish these octaves from one another since the camera view would not be able to know without seeing the full plane on the piano keyboard layout.

On the idea of coloring components for the CV, we also discussed adding colored fiducial markers to each of the fingers that would face up on the glove towards the camera when being played. This would help the CV calgorithm identify based on that color that it is a finger in the image, and then the fiducial would differentiate which finger it is that is pressing a key. All of these 10 fiducials would be different than the others so as to identify the 10 distinct fingers.

I also worked this week to set up a camera stream between my iPhone camera and my laptop for the purposes of testing. Although it is not necessary, since it is more preferrable to start out with testing the algorithm with images and frames of a video, but our end phase requires that we take out frames to process of a live streaming video feed. This step is still important and therefore not trivial. It relies on a software called Iriun which can be downloaded as an app on the app store and on my laptop. It then connects as a webcam to my laptop through the same wifi. CMU blocks this though since it is a public connection, so it currently only works in my house, which should suffice for testing still if need be. If not however, I can always take images with my phone, send them over to my laptop, and conduct image processing on them there. This is what it looks like below.

Lastly, I worked alongside my team on the design review report document. I worked on the introduction, use-case, and design requirements.  It was tricky differentiating between use-case and design requirements, but thanks to the design template and its directions, as well as Prof Gary’s clarification on the matter, I was able to come up with both qualitative and quantitative requirements that accurately describe the details of the project.

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is on schedule as of right now. Nish and I will also be working on some Kivy related items in conjunction with OpenCV according to the schedule as well. Spring Break also takes a chunk of a week away from us, but our schedule accounts for this already.

“ What deliverables do you hope to complete in the next week?

Next week is Spring Break, so for the week we come back after, I would like to have my computer vision algorithm be able to detect the location of white keys by having bounding boxes over them. I would also like to create and printout the colored fiducials for the fingers. Fiducials is a common technique used for identification in compter vision, so there is likely a website that makes creating these fiducials easily since this concept of being able to identify essentially numbers 1-10 with a camera has already been done before. There will most certainly be documentation about this out there.

Posted on

Nish’s Status Report 2/25

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I worked for a few hours on making the slides for the presentation. I think spent another few hours practicing and delivering the presentation. Following that, I’ve been working on putting together the materials we need to order, such as sourcing the wrist strap and bluetooth module. We are also meeting to order the PCB this week, so it will arrive during spring break.

 

I made the block diagrams and testing plans, as well as further extensions.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

 

I think we are slightly behind, but once we can order the PCB and parts I think we will be on track, and we can start working on setting up the code base.

What deliverables do you hope to complete in the next week?

 

I want to create a dummy app that creates a Kivy Interface from an Xcode wrapper. Hopefully, I will also be implementing camera and bluetooth access in the app, to make sure we can connect to the iPhone infrastructure.

Posted on

Caroline’s Status Report for 2/25/2023

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

In the past week, I worked with the rest of my group to finalize our design presentation. In addition to this, I ordered a pair of pressure sensors to test on: 

We chose these sensors because they are lightweight and aren’t bulky. We hope that they will not interfere with the user’s finger movements and will not seem noticeable. We hope to test for its reactiveness and functionality to see if they will be the best sensors to use. Although these pressure sensors might not be the exact model of sensors we are going to use for our final product, we are using these to test our components and CV.  This allows for modularity in our system, so if one sensor breaks, it is easy to replace, without having to disrupt the whole system. I will design our PCB around this so it is convenient to replace.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Currently, I believe that our progress is on schedule. We aimed to have all of our parts ordered by this weekend and we are finalizing and ordering any remaining parts by tomorrow.

What deliverables do you hope to complete in the next week?

In the next week, I hope to create the breadboard for our PCB and order the PCB before spring break starts. This would allow us to not waste time waiting for the PCB to arrive.