Author: lpoirier

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week, I worked during class on Monday to unpack the amazon package that arrived of the goose neck phone holder tripod desk stand mechanism with the ring light. We found that the ring light made insignificant changes to the lighting of the surface but the lighting in Ansys Techspark was something that couldn’t be completely representative of all lighting conditions.

But it worked great, and could be lifted high enough to take a photo of a bird’s eye surface down below that it could show the entirety of the four octaves. This is good since the entire image captured can simply display all the octaves and we don’t have to do extra work to accomodate for a selection of octaves while the user is playing, rather we can do all of them at once.

I created a printout of what I wanted the design of the piano keyboard printout should look like going through multiple iterations. I wanted to make sure that the corners had a distinct enough visual to do feature matching so that i could find its coordinates and then do a warp perspective to get the four corners of the piano from the overhead camera view. This proved tricky but first the piano layout can’t fit on an 8.5 x 11 sheet of paper, so I had to print out two and then meticulously tape them together to combine a total of four octaves on the keyboard layout.

Feature matching proved to not be accurate enough, so I decided to experiment with doing red border contour identification using the cv.findcontours function. This took some time to figure out in application, but I was able to get the moments of the contours to identify the centroid areas of the thresholded objects found in the image, which was the 4 corners. This center points of these centroids was good enough to get the 4 corners to then do a warp perspective function. Since it wasn’t a close bounding border to the piano and there still some extra slack since it was only the center of the centroid, not the other ones, I ended up just cropping the post warped image and then adding the gray border that I used in the segmentation algorithm from last week.

I tried this out on test images from online and this has worked to a good degree so far but it requires some additional polishing. My hope is that there shouldn’t be too many issues doing this with real photos in person.

” Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is behind schedule, as I did not take into account warp perspective into my schedule. I could just take the easier route if necessary to make sure demoes go smoother by having the user select the points themselves in the interface, removing most potential for error. I also need to rearrange the order of the segmentation to map to the order of the actual keynotes including sharps (i.e. C key and C# key). Lastly, for the interim demo, they want me to show they can threshold the entire hand on the image which shouldnt be too difficult since I’m not focusing on specific fingers for right now.

” What deliverables do you hope to complete in the next week?

By next Wednesday for the interim demo I hope to have the following above completed. As a recap this is polishing up the warp perspective, ordering the segmentation keys, thresholding an entire hand on the image, then lastly having this work on real life photos taken from the top down view phone holder.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on the CV code to segment out both black and white keys. I made several changes to my code and I will go through them here:

1. Resize the image to make things consistent (used some factor so that every image would have the same number of pixels while keeping the same aspect ratio).
2. Add a gray border to the image to help with thresholding. Can’t be black nor white because then the border would be thresholded with either the white or black keys which we don’t want
3. Do the grayscale, blurring, and edge detection. Then do binary thresholding on the edge detection. This gives us areas dividing all keys to the borders seen here:
4. Then we do segmentation on the image so that each of the white spaces are assigned enumerated labels. I used the cv.connectedComponentsWithStats function for this task. These labels were converted by me to greyscale values between 0 and 255 normalized with an even distribution. The output image has layers of shades of gray each value essentially representing a segmented region for each piano key or as I define them as “sectors”:
5. I then used a mouse cursor window function called cv.setMouseCallback that allows me to create a window for the image where my mouse can read information on the 2D matrix. By getting its value I can find it’s respective sector information for that segmented region from a list I created to store that information. When the user hovers over a key, the enumerated number of that sector will show up.  Here is some of my code below and the terminal output showing this:

Per last week, last Monday I ordered the phone mount that i decided to go with. It has arrived to the ECE corridor so I can try it out this coming Monday.

Lastly, I worked on finishing the ethics assignment this week too. This includes having the task 3 discussion with my team in addition to going to the ethics lecture for task 4.

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is on schedule. I need to do some small adjusting for the image sizes to make that more test images are more consistent. I will test this with the new phone holder stand to start testing with real images with the ring light to make further adjustments there.

“ What deliverables do you hope to complete in the next week?

By next week according to the schedule, I plan on incorporating the hands in there. How I will do that is essentially have a picture shot of the piano with no fingers on it to have it as a background. Then the colored tips of the gloves will hover over a piano key, the CV will easily be able to tell that the colored region overlapping with a segmented region. So it will be able to determine which piano key the finger is over then.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on the CV code to segment out black keys. I first started by obtaining test images from online that were just top down view to start off. Then, I created functions to input those test images from my hard drive to run the algorithms on them. I first started off with changing it to grayscale. Then I put a gaussian blur on it to prepare it for putting edge detection onto it. Lastly, I decided to put a sobel filter for horizontal and vertical edge detection to obtain the image below:

Furthermore, the camera attatchment for the laptop and phone came in and we put it on the laptop with the phone on it to see if it worked and it was able to point down at the direction we desired. We did not take a photo of it during class but an image of what this looks like can be found on the amazon store:

I also did some further research with finding out several ways to look at doing warp perspective since the view will not always be top-bottom view. I came up with 3 ideas. 1. Have the student choosing (clicking) points on the streaming video to choose where the piano is. 2. Use feature matching by passing in an image of the top-down view of the piano. 3. Add a red border so the CV algorithm recognizes the 4 corner points of the piano. After speaking with Tamal, he suggested changing the actual attachment itself to a goose neck, phone holder, tri-pod ring light mechanism. This would give it the flexibility to reach over and look down from a top-down view making our lives easier. Additionally, the ring light was a great suggestion since he said that lighting conditions could greatly affect the CV algorithm so it was importatnt to make this a control element rather than a variable. For corner points,  he said red border would be best, or you even make the corners plus signs and just do feature matching for the plus signs to easily identify it. I will certainly take these into considerations.

Lastly, I worked on doingthe ethics assignment this week too. This includes reading through the two articles and completing tasks 1 and 2 of the assignment.

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is a bit behind as the black keys need to be segmented. However, all I need to do is put a binary threshold filter on it to filter out white vs black keys. Since we have edge detection achieved, I will use a segmentation function to add distinct values to the binary values of pixels and borders of the edge detection. So I will be working on that tomorrow before the start of the week on Monday.

“ What deliverables do you hope to complete in the next week?

By next week according to the schedule, I will have the white keys also segmented too. Additionally, I plan to have choose the goose neck phone holder ring light tri-pod amazon product on Monday during class to have it sent to the order form by Tuesday to get that in early. This will not stunt by progress since I am relying on test images still. I also will talk to my group mates on the task 3 of the ethics assignment where we discuss that and take ntoes. I also want to work on adding the corner point plus signs to the test images to see how well the warp perspective performance will do.

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week, I worked on researching the capabilities of OpenCV to see the extent of what functions to use to identify and differentiate piano keys and fingers from one another. This is still a work-in-progress and still needs to be more fleshed-out. Some other additional things I helped out with was brainstorming with my team some more details on our project that was not mentioned during the design review presentation but are integral. Some of these was researching a battery to be used to power the gloves + micro-controller.

Another one was for the CV component was coloring the octaves on the piano printout to aid the software side to identify what octave of notes are being played. Since we know that the full layout of the piano keys cannot fit inside the range and view of the camera, it is paramount to distinguish these octaves from one another since the camera view would not be able to know without seeing the full plane on the piano keyboard layout.

On the idea of coloring components for the CV, we also discussed adding colored fiducial markers to each of the fingers that would face up on the glove towards the camera when being played. This would help the CV calgorithm identify based on that color that it is a finger in the image, and then the fiducial would differentiate which finger it is that is pressing a key. All of these 10 fiducials would be different than the others so as to identify the 10 distinct fingers.

I also worked this week to set up a camera stream between my iPhone camera and my laptop for the purposes of testing. Although it is not necessary, since it is more preferrable to start out with testing the algorithm with images and frames of a video, but our end phase requires that we take out frames to process of a live streaming video feed. This step is still important and therefore not trivial. It relies on a software called Iriun which can be downloaded as an app on the app store and on my laptop. It then connects as a webcam to my laptop through the same wifi. CMU blocks this though since it is a public connection, so it currently only works in my house, which should suffice for testing still if need be. If not however, I can always take images with my phone, send them over to my laptop, and conduct image processing on them there. This is what it looks like below.

Lastly, I worked alongside my team on the design review report document. I worked on the introduction, use-case, and design requirements.  It was tricky differentiating between use-case and design requirements, but thanks to the design template and its directions, as well as Prof Gary’s clarification on the matter, I was able to come up with both qualitative and quantitative requirements that accurately describe the details of the project.

“ Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is on schedule as of right now. Nish and I will also be working on some Kivy related items in conjunction with OpenCV according to the schedule as well. Spring Break also takes a chunk of a week away from us, but our schedule accounts for this already.

“ What deliverables do you hope to complete in the next week?

Next week is Spring Break, so for the week we come back after, I would like to have my computer vision algorithm be able to detect the location of white keys by having bounding boxes over them. I would also like to create and printout the colored fiducials for the fingers. Fiducials is a common technique used for identification in compter vision, so there is likely a website that makes creating these fiducials easily since this concept of being able to identify essentially numbers 1-10 with a camera has already been done before. There will most certainly be documentation about this out there.