Posted on

Lee’s Status Report for 4/29/2023

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on the slides for the final presentation. In addition to that, since I was the one presenting I had to write a script and practice it before I went. To also prepare for the presentation, I worked on modifying the lighting environments thresholding but it didn’t work as well since the various amounts of lighting environments were too vast to account for. Creating the manual points subsystem where users can choose their points to zoom in is just much easier now. The hand and finger detection also fully worked now so that wherever my fingers were, the algorithm would tell me the sector it would think it is in. There is also a red bounding box so that the user knows where on the camera they should keep their piano within the bounding box when playing.

” Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is currently on schedule as I am done with my CV subsystem.

” What deliverables do you hope to complete in the next week?

This coming week I plan work on the Poster, Final Report, as well as help with testing and collecting data for that.

Posted on

Team Status Report for 4/29/2023

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?  

The most significant risk is how fragile the pressure sensors are. We ordered extras, but currently have broken 3, which means that if we break any more, we might be down a glove. Because we have broken so many, we have gone ahead and ordered more sensors, although they might not get here in time. In addition, we are handling the pressure sensors with a lot of care so that they won’t break in the future. The backup plan is to have one fully working glove with the pressure sensors, and the other glove with as many pressure sensors as possible. While it might not be receiving pressure inputs, the CV code can still detect which key is being pressed.

 

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?  

We have not made any changes to the existing system design.

List all unit tests and overall system tests carried out for experimentation of the system. List any findings and design changes made from your analysis of test results and other data obtained from the experimentation.

Note accuracy -> We tuned each note that is being outputted to an online tuner and ensured that they were all correct.

Key segmentation -> We used multiple test images and assigned values to each key and saw that they were all different and correct.

Color thresholding -> We tested the paper piano printout in various different lighting and ensured that they all work.

Key Identification -> We tested each finger on each key and ensured that the correct note and finger were being detected.

Warp Consistency -> We tested the paper piano at different angles to ensure that it would work even if the piano wasn’t straight.

Posted on

Caroline’s Status Report for 4/29/2023

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week, I worked on assembling the gloves to the wrist strap that we ordered. This involved fully soldering the wires and resistors to the flexible breadboard, as well as connecting the battery pack to the gloves. Currently, everything is taped down because we want to verify that all of the components work seamlessly together before securing them down.

In these photos, it may appear that the gloves don’t fully fit my hand, but that is because these gloves are designed around Nish/Lee’s hands, which are slightly larger than mine.

In addition, I was able to fully send pressure values over Bluetooth, which can be received by the interface: https://drive.google.com/file/d/19v-FNZ2IqrRwa5q09mDdh1ZcA8WSYXcP/view?usp=sharing

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is slightly behind schedule because I hoped to produce both gloves by the end of the weekend. I will be working a lot on Monday to catch up to where I wanted to be.

What deliverables do you hope to complete in the next week?

This next week, I will be finishing up the gloves and working closely with Nish to execute all of our tests and verifications.

Posted on

Nish Status Report 4/29

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I’ve been working on redoing the App interface on a Desktop version for easier CV + Bluetooth integration. I also spent a lot of time learning how to do to the Bluetooth integration and getting that set up to pass to Caroline, who is working on the actual glove.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

Slightly behind because I got sick and wasn’t able to get out of bed for a few days! But getting back on track, don’t really have a choice since the demo is due in a week. Should have the CV + app integrated by Sunday night or Monday morning, and then spend the rest of the time testing and integrating bluetooth.

What deliverables do you hope to complete in the next week?

Finish up the CV + App integration and then bluetooth integration and testing.

Posted on

Lee’s Status Report for 4/22/2023

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on Monday and Tuesday to the mediapipe package to show the hands on the webcam screen. I differentiated the hands from left to right and used print statements in a time package to print out the location of each of the fingers on each hand where its location was on the screen.

Later I integrated this feature with using the finger location to determine what key it is placed on. To do this, I took the coordinate of the finger and did another warp perspective to get the coordinates of the segmented image. This is basically just translating between the two sets of images, the segmented one being warped. Once I had the coordinates in the segmented image, I used my mouse-cursor function and modified its basic functionality to print out the number sector for what key the finger is hovering over.

I also improved this feature by adding a red box that indicates to the user in print statements if the hand placement is outside the boundaries of where the piano would be placed normally.

” Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is currently on schedule as I finished doing the warping under real lighting environments. And I am already working on the hand and finger detection which is the next and last task for me before integration.

” What deliverables do you hope to complete in the next week?

This coming week I plan to present for the final presentation as well as work with my team with integration steps to prepare for our May 8th Demo.

Posted on

Team Status Report for 4/22

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk currently is still the integration of our entire project. Our entire CV portion is written in Python, while our interface is done in C++. We know that there are packages that are available that can use the code we wrote from Python and integrate it into our interface. However, we aren’t sure of the speed of using these packages as they are going to be slower than if we ran the CV using C++. We will be testing out this in the next week to see if the difference in speed is crucial for our project. The backup plan is to translate the existing code from Python into C++.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are currently no significant changes made to the existing design of the system.

Posted on

Caroline’s Status Report for 4/22/2023

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

In the past week, I was able to flesh out all of the issues with the Arduino BLE Nano. One key thing that I didn’t account for was that the Arduino Uno that I was testing with previously ran on 5V, while the Arduino Nano runs on only 3.3V, which was affecting the numbers that we were receiving. However, I changed the resistor values so that we are able to read the inputs to the Arduino Nano now.

In addition, I have half-produced the gloves for our project. I connected and soldered all of the wires to the Arduino BLE Nano and connected them to the breadboard temporarily. We wanted to use a flexible breadboard for the final design, but we were waiting on that.

In addition, I have also been working on writing up the code for the interface that accesses the phone’s Bluetooth and connects to the Arduino BLE Nano. I have been following this video, which has been giving me a lot of helpful tips: https://www.youtube.com/watch?v=UolJHmZ0RFk

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I think that my progress is behind schedule. I have been away for most of the past week, so a lot of the work that I wanted to do on producing the gloves was paused. I will be putting a lot more work into this in the next week in order to get my part back on schedule.

What deliverables do you hope to complete in the next week?

In the next week, I will spend all of my time finishing up the gloves and also finishing the interface Bluetooth connection.

Posted on

Nish Status Report

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I tried to work on soldering the glove fully so we would be able to show it. It did not work, as I am really bad at soldering, and soldering a flexible protoboard is terrible. I spent a long time doing it and then at the end snipped the whole thing and have asked to order a new arduino because I ruined it pretty badly. I spent a long time trying to desolder it but then figured we have the budget anyway.

^^ failed soldering that was painstaking.

” Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is currently behind. To catch up, I will have the full integration between the apps done tonight and have the Python interpreter in the package tomorrow.

” What deliverables do you hope to complete in the next week?

This coming week, on Tuesday I will have the two halves of the app merged, and on Friday I would have gotten all the Python code for the CV running in the App. We recognize that there may be a delay with the frame processing, but hopefully the interpreter will still run in under 100 ms.

 

Posted on

Lee’s Status Report for 4/8/2023

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week I worked on Monday and Tuesday to work on preparing for the interim demo. From last week I fixed some things with changing the coordinates of the warp to make them alligned as well as fixing some things that were hard-coded to adjust for outlier test cases.

I printed out the piano layout sheet with the red dots to start practicing on a real live photo when I was finished on the digital images. I printed out two types, one with smaller red dots and another with larger red dots. As I went through with testing, I found that the larger red dots were better so that there were more red pixels to threshold with which was helpful. Therefore, I am no longer using the smaller dots. I practiced on different lighting environments to adjust for the RGB colors in the threshold range for red to get that to work under most environments.

After the interim demo I worked on the hand and finger thresholding. As I was doing my research I came across an external package created by google engineers that would detect not only hands in an image, but also detect certain landmarks in the hands. It took me a while to work on this to get the installed package to work since I need python 3.7 and the relative numpy and opencv editions that are compatible with that. Once I got that working I tested the code out on my computer and so far I am able to get my hand and the landmarks printed on the screen.

” Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is currently on schedule as I finished doing the warping under real lighting environments. And I am already working on the hand and finger detection which is the next and last task for me before integration.

” What deliverables do you hope to complete in the next week?

I hope to continue analyzing how to utilize the landmarks to identify specific fingers to then get their coordinates and then check what region they are located in. Basically working on the next task in my schedule for this upcoming week.

Posted on

Caroline’s Status Report for 4/9/2023

What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This past week, I was able to get Bluetooth connection from the Arduino Nano BLE onto my phone.

In addition, I was trying to transfer the circuit design of the pressure sensors onto the Arduino Nano BLE instead of the Arduino Uno. However, when I tried to do this, the outputs that the pressure sensors were giving me were random, unusable outputs. I have been trying to debug this problem, but I have not successfully found a solution. My theories as to why this is happening are that the specific Arduino Nano BLE that I have is not working perfectly or that the connections to the Arduino Nano BLE aren’t secure, since I haven’t soldered the pins onto the Nano.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I think that my progress is behind schedule. I have been having lots of difficulties with the Arduino Nano BLE and its connections, which makes it hard to test and move on with my part of the project. I will be spending a lot of time in the next week debugging the Arduino and getting it to work.

What deliverables do you hope to complete in the next week?

In the next week, I hope to figure out everything that is wrong with the Arduino Nano BLE and fix the issues so that I can work on integration in the next week. In addition, if I have time, I will be writing up the code for the interface to allow for the app to talk to the Arduino instead of another phone app that I downloaded.