Posted on

Lee’s Status Report for 3/25/2023

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on the CV code to segment out both black and white keys. I made several changes to my code and I will go through them here:

  1. Resize the image to make things consistent (used some factor so that every image would have the same number of pixels while keeping the same aspect ratio).
  2. Add a gray border to the image to help with thresholding. Can’t be black nor white because then the border would be thresholded with either the white or black keys which we don’t want
  3. Do the grayscale, blurring, and edge detection. Then do binary thresholding on the edge detection. This gives us areas dividing all keys to the borders seen here:
  4. Then we do segmentation on the image so that each of the white spaces are assigned enumerated labels. I used the cv.connectedComponentsWithStats function for this task. These labels were converted by me to greyscale values between 0 and 255 normalized with an even distribution. The output image has layers of shades of gray each value essentially representing a segmented region for each piano key or as I define them as “sectors”:
  5. I then used a mouse cursor window function called cv.setMouseCallback that allows me to create a window for the image where my mouse can read information on the 2D matrix. By getting its value I can find it’s respective sector information for that segmented region from a list I created to store that information. When the user hovers over a key, the enumerated number of that sector will show up.  Here is some of my code below and the terminal output showing this:

Per last week, last Monday I ordered the phone mount that i decided to go with. It has arrived to the ECE corridor so I can try it out this coming Monday.

Lastly, I worked on finishing the ethics assignment this week too. This includes having the task 3 discussion with my team in addition to going to the ethics lecture for task 4.

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is on schedule. I need to do some small adjusting for the image sizes to make that more test images are more consistent. I will test this with the new phone holder stand to start testing with real images with the ring light to make further adjustments there.

“ What deliverables do you hope to complete in the next week?

By next week according to the schedule, I plan on incorporating the hands in there. How I will do that is essentially have a picture shot of the piano with no fingers on it to have it as a background. Then the colored tips of the gloves will hover over a piano key, the CV will easily be able to tell that the colored region overlapping with a segmented region. So it will be able to determine which piano key the finger is over then.

Posted on

Team Status Report 3/25

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Our most significant risks is sourced from our ethics discussions this past week. Some groups gave some ethical critiques on our design and how if they were not addressed in a worst-case scenerio, could certainly jeapordize the success of the project for prospective users. For computer vision, having a camera filming could introduce privacy concerns. If our design doesn’t simply work the user may just go back to using an actual piano. If for some reason the battery shorts/explodes, it could injure the user too. If played for extensive periods of time without breaks, it could bring about fatique and physical pain to the user. Lastly, if the user wasn’t aware of some information before using the product, it may fail in environments that isn’t a flat surface or with low lighting. The user may also wear the glove backwards/weirdly or can’t pair with their phone interface if they aren’t familar with the device. They may not be able to pair at all if there are too many bluetooth devices as well.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We did not make any changes to the existing design of the system since last week.

Posted on

Caroline’s Status Report for 3/25/2023

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I finished testing the pressure sensors. They are able to differentiate between the low, medium, and high pressure that we want for our project. They are relatively accurate and work well for our project.  We will be ordering them from Amazon in the next week after Nish and Lee test out the pressure sensors and see how they work for themselves.

Video of pressure sensors: https://drive.google.com/file/d/1XjYCct6e8iyqOXaQRiQNeGvuUmjMI3tv/view?usp=sharing

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I think that we are on track. We are making significant progress and will have a working project by interim demo.

What deliverables do you hope to complete in the next week

Next week, I will focus on getting Kivy and OpenCV up and ready to go. On Monday, Nish and I will try to be able to open the camera and receive BLE information in our interface. If we don’t finish on Monday, we will use Wednesday as our buffer.

Posted on

Nish’s Status Report 3/25/2023

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, we met a few times to discuss our progress. I was working on the interface mainly. I started to build the kivy app. I built it in separate parts and working on integrating these features together. Below is a screenshot of some of the output with configured Camera properties to run a square on our  We will only be able to test this once we wrap it in Xcode (the next step).  Individually, we are able to access the Camera and the speakers with our code We also test our Arduino Nano to make sure it can send connect to our iPhones.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Yes, I think we are on track at this point. For our interim demo, we will be showing our CV algorithm. As long as I continue to make progress, we might even be able to integrate our CV into the interface before the demo date.

What deliverables do you hope to complete in the next week

On Monday, I will be able to show the Kivy interface with the speakers, camera, and BLE transmission working. After that, by Friday I hope to create the calibration screen, and then work on hiding the camera as we begin to create the keys interface will using CV in the background.

Posted on

Lee’s Status Report for 3/18/2023

What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on the CV code to segment out black keys. I first started by obtaining test images from online that were just top down view to start off. Then, I created functions to input those test images from my hard drive to run the algorithms on them. I first started off with changing it to grayscale. Then I put a gaussian blur on it to prepare it for putting edge detection onto it. Lastly, I decided to put a sobel filter for horizontal and vertical edge detection to obtain the image below:

Furthermore, the camera attatchment for the laptop and phone came in and we put it on the laptop with the phone on it to see if it worked and it was able to point down at the direction we desired. We did not take a photo of it during class but an image of what this looks like can be found on the amazon store:

I also did some further research with finding out several ways to look at doing warp perspective since the view will not always be top-bottom view. I came up with 3 ideas. 1. Have the student choosing (clicking) points on the streaming video to choose where the piano is. 2. Use feature matching by passing in an image of the top-down view of the piano. 3. Add a red border so the CV algorithm recognizes the 4 corner points of the piano. After speaking with Tamal, he suggested changing the actual attachment itself to a goose neck, phone holder, tri-pod ring light mechanism. This would give it the flexibility to reach over and look down from a top-down view making our lives easier. Additionally, the ring light was a great suggestion since he said that lighting conditions could greatly affect the CV algorithm so it was importatnt to make this a control element rather than a variable. For corner points,  he said red border would be best, or you even make the corners plus signs and just do feature matching for the plus signs to easily identify it. I will certainly take these into considerations.

Lastly, I worked on doingthe ethics assignment this week too. This includes reading through the two articles and completing tasks 1 and 2 of the assignment.

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is a bit behind as the black keys need to be segmented. However, all I need to do is put a binary threshold filter on it to filter out white vs black keys. Since we have edge detection achieved, I will use a segmentation function to add distinct values to the binary values of pixels and borders of the edge detection. So I will be working on that tomorrow before the start of the week on Monday.

“ What deliverables do you hope to complete in the next week?

By next week according to the schedule, I will have the white keys also segmented too. Additionally, I plan to have choose the goose neck phone holder ring light tri-pod amazon product on Monday during class to have it sent to the order form by Tuesday to get that in early. This will not stunt by progress since I am relying on test images still. I also will talk to my group mates on the task 3 of the ethics assignment where we discuss that and take ntoes. I also want to work on adding the corner point plus signs to the test images to see how well the warp perspective performance will do.

Posted on

Caroline’s Status Report for 3/18/23

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I set up a simple circuit to test the sensitivity of the pressure sensors we purchased. At the time of this status report, I am currently writing the code to test the sensors, so that when more than 50g of pressure is sensed, the light will turn on. This is the initial test to see if the sensors work correctly.

In addition to this, I have downloaded and set up Kivy and Xcode so that we are able to start writing the interface for the app next week. I also familiarized myself with the platforms so that I am comfortable to start writing code for it next week.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I believe that we are slightly behind schedule because the pressure sensor testing should have been done before/during spring break. However, because this step isn’t crucial to the progression of our project, I am able to continue it on the side while also working on the interface with Nish.What deliverables do you hope to complete in the next week?

Next week, I will continue to stress test the pressure sensors and create a series of LED lights, each corresponding with a range of pressures (50g to 2g) to see how sensitive the sensors are.

I also hope to write some basic code in Kivy and Xcode so we can have an app interface complete before interim demos.

Posted on

Nish’s Status Report 3/18

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week, I evaluated the ethics of our project in the Ethics assignment. We feel that overall, our project does not have negative ethical dilemmas. However, it solidified our ideas that we only want to target experienced piano players (to some degree), as we don’t want to cause bad hand posture and bad technique in younger/beginner users, if they attempt to play the piano on a flat surface for a long time.

 

I also worked for a bit in trying to get our oscillators and synth sounds. However, this work is now obsolete given that we will just use an existing package. I’ve been looking through and testing different audio packages and MIDI packages available in Python. pypiano and fluidsynth are the ones we are now considered, and will narrow down when we work on audio processing again later in our timeline.

 

I also spent a large amount of time learning how MIDI protocol works, and working through tutorials to practice sending and receiving MIDI messages and writing them to a file. I used a bit of an arrangement of Dandelions that I had been working on and wrote out MIDI messages and tried to play them back in a .mid file, using a few resources such as this that uses MATLAB to actually write the messages to the file:

https://www.mathworks.com/help/audio/ug/convert-midi-files-into-midi-messages.html

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

After evaluating our risks (as seen in the group report), we feel a little behind as a lot of the work that I was trying to do this week is kind of obsolete now. To get back on schedule, we are focusing on having something working for our interim demo. That will require our interface to be up and running, so both Caroline and I will be working together next week to build up the structure of the app in Kivy, and hopefully be able to launch it through Xcode as well.

What deliverables do you hope to complete in the next week

Next week, I hope to have been able to launch our app from our iPhone, such that it shows a calibration page and accesses the camera.

Posted on

Team Status Report 3/18

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

I think our most significant risks in how we are going to be integrated our different components and subsystems. In order to be able to work, we have different branches in our repo for CV, Interface, and Audio, and plan on integrating them together in the master branch. However, when CV and Audio are integrated together, we will probably have to write in some threading capabilities so that CV can continually run  while MIDI timing is being taken with  the pressure sensor, so we will require a timing mechanisms.

In order to mitigate this, we will take a few steps. Firstly, we will edit points in our schedule so that we merge in each CV milestone with our interface. This will require us to set up the bare bones of the interface rather quickly, so we will be pushing the audio details later. In our interface, we will set up a software timer that runs/is checked as necessary, with a minimum BPM of 30, so we will always merge the CV milestones with a timer in place and software interrupts are triggered by a note being played via the pressure sensors to check the timer and send the MIDI message.

Our contingency plan is to have a MIDI message sent on each beat. This is not ideal, as we may have a note that starts on a half beat, but given that we are focusing on the note accuracy + volume and interface in this project, we feel that it would be okay to have this as a contingency plan.

 

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Given the feedback on our design report, we have decided to edit our implementation of the audio processing flow. We will not spend time trying to figure out the frequency spectrums of each note, but rather use a piano package with MIDI available through Python to render our sounds. We are currently exploring a few different options for these packages. This will reduce a lot of workload by using work already available to us rather than starting from scratch and trying to build deep sounds. Although we received feedback that suggested we use a soundcard as we are under budget, 1) we want to play the sounds through the phone for the portability aspects, as we expect our user to be using their laptop for actual score writing, and 2) we want to minimize cost to the user, as outlined in our use case.

Posted on

Nish’s Status Report 3/11

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

In the past week, I worked on the design report. Particularly, I worked on the Design Trade, Architecture, and System Overview sections. I made the block diagrams and also created the audio processing flow, as we switched our focus from PCB design (hardware) due to choosing the Arduino Nano into audio processing. I worked a lot on researching how synthesizers are made and finding the software we will need to implement them, such as pygame midi and the Oscillator packages, and understanding how to shape our sound waves to sound like a piano.

 

Here are some of the diagrams I created: 

 

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I think we are on schedule. Most of our work is now in the software/audio realm, so we are able to focus on prototyping and understanding those applications next week even though we haven’t gotten our Arduino Nano yet.

What deliverables do you hope to complete in the next week?

Next week, I hope to complete running a dummy Kivy app that is able to make all 49 pitches that we are hoping to include in our piano, and be able to shape the waves using the Sustain and Release modulators talked about in our design report, experimenting with what gets the most realistic sound.

Posted on

Caroline’s Status Report for 3/11/2023

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

In the past week, I worked with the rest of my group to finalize our design report document. I was originally tasked to design the PCB that would connect the pressure sensors with the Bluetooth and microcontroller, but we changed our design to use the Arduino Nano BLE instead. So, in the past week, I helped research the documentation behind the Arduino Nano and how to best connect it within our system. I helped draw up an example diagram of how to best implement the gloves so that when our Arduino Nano BLE arrives, I can quickly assemble them together so that we can test them as quickly as possible.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Currently, I believe that our progress is on schedule. We have shifted our GANTT chart to account for our spring break. We will begin the bulk of our project when we return from spring break.

What deliverables do you hope to complete in the next week?

In the next week, I hope to research and download XCode and write a dummy file to test my abilities and the different functionalities that XCode has. In addition, if our Arduino Nano BLE arrives, I will be assembling the components on the gloves.