Author: digonzal

General updates:

• Team trip to Home Depot to buy parts to connect the webcam gooseneck to the tripod. The connecting part of the gooseneck was a wider tube (⅝”) and not threaded while the connecting part of the tripod was a smaller threaded screw (¼”) so it was difficult to find a connector for the 2. However, with assistance from the Home Depot staff we landed on using multiple spacers. We screwed the tripod connecting part with the help of a heat gun to create threads in the spacers. In the future, we might have to glue the spacers to the gooseneck tube to make sure it stays in there.
• We met with Dueck’s students and tested our newly built stand. Images are attached below. The stand worked great! It was stable (didn’t require a counterweight) and was tall enough to capture all the keys on the keyboard. Additionally, the gooseneck was able to stretch enough horizontally such that the pianist’s head did not block the view of their hands. The webcam’s USB cable was a bit short so it was inconvenient to hold the laptop close to the camera stand. Additionally, the USB cable kept bumping into the pianist’s head. We have ordered velcro for the purpose of wire management and a USB extension cable. 
  • We also gather video of scales & Hanon specifically to test with. More details can be found in Shaye’s report 

• As per Joshna and Prof Bain’s suggestion we met with Varun to ask for advice on using the KV260. More details can be found in Jessie’s report. 

Design Changes: 

Based on the feedback we received from Varun, we have decided to drop using the KV260 and commit to pivoting to the RPi with an accelerator to accelerate our hand landmark model. 

Updated Schedule:

We are slightly behind our originally planned schedule of finishing the CV and FPGA integration by the coming Wednesday; additionally, our plans have shifted to using the RPi instead of the FPGA. Here is our updated schedule with the RPi:

Due 11/01 – 

• Jessie: finish moving the model to RPi and research how to connect the buzzer and display and web app hosting RPi using UART 
• Shaye: write many versions of the tension detection algorithm for RPi (see Shaye’s status report)

Due 11/06 – (next meeting with Professor Dueck’s students)

• Shaye: finalize tension detection algorithm using output of RPi model (see Shaye’s status report for more)

Due 11/08 –

• Jessie: finish implementing buzzer feature and set up UART connection with web app hosting RPi
  

(11/08-11/11) – buffer time

Due 11/11 – start full system integration

General updates:

• Worked on and finished the design report (the vast majority of our time this week was spent here).
• Worked with musicians on 10/9 to test out rough camera setup & tension detection algorithm—see Shaye’s status report for more info.

Product solution considerations: 

A was written by Jessie, B was written by Danny and C was written by Shaye.

Part A: 

We hope our product will allow pianists of all skill levels, from hobbyists and beginner students to professionals, to protect themselves from hand and wrist injuries related to piano playing– all players can benefit from injury prevention. The product’s utility, though intended for students with access to a teacher, could also be applied to those who are self-taught. However, the self-taught student would also have to self-learn how to identify correct positioning in order to properly set the initial calibration. Additionally, our system does not rely on a laptop to host our system, so users with only access to a phone or a tablet can also protect themselves. The system will also have simple and intuitive features, so it does not require the user to be technologically advanced. 

Part B:

Our product will hopefully encourage people to either learn or continue playing the piano. Whether you are completely new to the piano or perhaps have been injured in the past, our product will ensure that these players are avoiding positions that can cause injury. We hope that the peace of mind our product will provide will lead to an increase in piano players. If the amount of pianists increases due to our product then we believe that we are contributing to a growing and deeper culture. While we are targeting piano players, we believe that a more musically inclined population is able to both better appreciate the culture we currently have but also make contributions to the culture we all share. 

Part C: 

We account for environmental factors with our system in two main ways: by using less power-hungry hardware, and by decreasing overall medical interventions for wrist strain injuries. In terms of hardware, our FPGA-based system would use less power than a Jetson/ GPU based system, minimizing power consumption during longer practice sessions. If our proof-of-concept becomes more widely adopted, this difference in power consumption would have a large impact on reducing overall energy waste from our system. 

Additionally, by preventing wrist strain injury in pianists, our product will decrease the amount of medical resources spent attending to those injuries, thus reducing the environmental impact of those resources. This includes a variety of healthcare items, ranging from single-use wrist wraps to energy spent on wrist imaging. 

General update: 

• We gave the design report presentation. This involved considering some use case requirements and making design decisions regarding our camera setup. We also fleshed out the testing for each of the different components.
• We had a couple of practice sessions over the weekend to strengthen Danny’s presentation skills. 
• We obtained the parts for our stand. We did a rough setup to see if the stand was able to capture the full keyboard. Our stand was tall enough that it was able to capture the entire keyboard. We plan to build a connector between the gooseneck and the stand before we meet with Professor Dueck’s students again on Wednesday. 
• We ran through the basic setup tutorial for Vitis AI on the Kria KV260. We were able to set up a model and detect their example image. More details can be found in Danny’s status report. We also ran through the Vitis AI workflow tutorial for quantizing and compiling a model on the FPGA. More details can be found in Jessie’s Status report. 
• We looked into how to implement kinematics on the FPGA. More details can be found in Jessie’s Status report. 
• We worked on printing out tension vs not tension CV pipeline—see Shaye’s status report
• We tested kinematics in CV pipeline—see Shaye’s status report
• We further looked into the different webapp examples that are found in Django. 

Risks: 

• Weight distribution of the camera stand being off-center leading to the stand tipping over. 
  • If this happens, we plan to add a counterweight. 
• FPGA not working—many complications in general 
  • Order RPI to work with over break for contingency. 
  • May order RPI ai kit if we end up switching over 100%.

Changes to schedule/ design: 

• No changes were made, we’re still on track of our schedule.

Stand setup:

 

Screenshot of “rough setup” we did

At the start of the week, we prepared slides for the proposal presentation and did a couple of practice runs during the weekend.  We also placed inventory orders for a (temporary) camera and KR260.  Discussed more concrete requirements for the camera tripod. Shaye got a basic CV pipeline up.  Danny looked into different backends for the web app and started the Django tutorial.  Jessie began Varun’s tutorial for Kria KR260 setup.

One risk that came up this week is camera compatibility with FPGA. We are waiting for Varun to test the compatibility with the 1080p webcam and are hoping to hear back by the end of this weekend. If the compatibility is confirmed, we’ll decide on and order a camera next week.  We are also concerned with the camera field of view in relation to capturing the whole keyboard.  We plan to either use a taller camera tripod or place the camera in a taller position, or get a more expensive camera with a larger field of view.  We will weigh the decision out by the end of next week.  A general worry we still have is porting to the FPGA. We’ll hold off on FPGA porting until the CV pipeline is fully finalized. We may start working in parallel on RPi in addition to the FPGA if we’re unable to see a path forward by October 12th. Shaye will focus on working with either an RPi4 or RPi5 while Jessie continues with the FPGA. If we’re unable to get the FPGA working on a basic level by mid October then we will give up on the FPGA.

No major changes happened—we have more concrete ideas on how to position the camera. This is included in Jessie’s status report for more detail; the diagram also included there.

We’re still on schedule. For next week we want to finish up CV pipeline and FPGA setup and hopefully start CV and FPGA integration. We will meet with Professor Dueck’s students on Wednesday, where we will test the CV pipeline to detect different angles on the keyboard. We will use a loaned camera from the inventory that will be handheld temporarily before we order one.

Link to video of current CV pipeline: link