Posted on

Team Status Report for 09/28

General accomplishments:

  • We worked on the design report and slides.
  • Jessie looked into the AMD workflow for putting a model on the FPGA. We now have a more fleshed-out idea of the tasks that need to be done. More details can be found in Jessie’s status report. 
  • Danny finished the Django tutorial. 
  • We asked Dueck’s students on their preferences on different WebApp features. More details can be found in Danny’s status report.
  • We met with Dueck’s students to preliminarily test system 
    • Need to track angle variation rather than consistent angle position—more details in Shaye’s status report
  • We have better specified the setup measurements and requirements based on additional measurements of Shaye playing piano. (Shaye not pictured 🙁 )

Risks: 

  • We are slightly concerned about converting the currently used MediaPipe model (TFLite) to PyTorch or TensorFlow, the only 2 formats compatible with Vitis AI. We plan to look into 
    • existing scripts that convert between the 2 formats or 
    • find different models that are in the compatible format 
  • We are also worried about the MediaPipe model losing accuracy once compiled on the FPGA due to over-optimization and bad inputted samples during the quantization phase
    • We plan to try multiple data sets and ways to convert samples to tweak accuracy
  • We need to look into how to implement kinematics on FPGA. We are not sure how to interact with the output of the model
  • Contingency: switch to RPi—will request on 10/8 if needed so we have time to work on it before fall break

Updates on the system:

  • We figured out how to port model & communicate between FPGA & RPI
    • Using Vitis AI workflow for model conversion 
    • Using UART for board communication 

Schedule updates:

  • Still on schedule, no updates

Additional questions:

Part A is written by Jessie, Part B is written by Danny, and Part C is written by Shaye

Public health/ safety/ welfare factors:

Our product aims to reduce injury among piano players by identifying and correcting harmful playing positions. Injury among piano players is extremely common (50-70%), with many injuries being related to the hand and wrist. The effects of being injured can have mental health impacts additionally, as it may prevent players from being able to practice for extended periods. Our system will provide players with real-time and post-processed feedback on the player’s hand position. This will help them correct their position, even without the guidance of a piano teacher. We realize that providing bad feedback can lead to more injury; thus, we’re taking special care to focus on the testing and verification metrics we’ve mentioned throughout this process to ensure system accuracy. 

Social Factors:

Our product will change the dynamic between the piano teacher and the student. Instead of having the piano teacher focus on the health of the student and harmful techniques, they can focus on music-related content. This product could also help encourage beginners to get into piano playing without the potential worry of becoming injured, lowering the barrier to entry. This could potentially lead to an increase in piano players. 

Economic factors: 

Our project will be the first product pianists can use to monitor their technique while practicing. Thus, although the total cost of our current system is high (FPGA cost, camera, stand, etc), creating a proof of concept using more general hardware will allow for more projects down the line to decrease the cost and create more accessible & commercialized products. With more specific hardware/ boards dedicated to running our system, the cost will decrease, allowing the product to be available for all piano players. For now, even just parts of this tool (CV pipeline, video saving features) can help save pianists from injury and incurring more personal costs. 

Posted on

Jessie’s Status Report for 09/21

  1. At the beginning of the week, I worked with my group to prepare for our proposal presentation. In addition to practicing with my group, I went through the slides a couple of times myself to better prepare for the presentation.

  2. We are trying to buy a camera soon so we can start collecting data from Professor Dueck’s students. Since the camera depends on the camera stand, I fleshed out the requirements for the camera and the camera stand. These diagrams/drawings and calculations are necessary to justify which camera we purchase.
    • The stand should be easy to use, and portable. 
    • The stand should capture the entirety of the keyboard length for different types of pianos.
      • For upright pianos, the camera stand can be placed on the flat top of the piano. 
      • However for grand pianos, if they are played open then there is no flat top; therefore we are looking into creating a clip to attach the camera to the music stand. We should be wary of the risk of the camera falling backward and into the piano, thus causing damage to the strings.

      • There is more variance when it comes to electric keyboards. At the time, I could not think of many ways to create a camera stand to position the camera. Currently, I envision keyboard players will be responsible for providing a flat surface (e.g. table) behind the keyboard to place the camera stand on. 
        • Many electric keyboards have very flimsy music stands, so it could be difficult to clip a camera onto it. They are also sometimes free-standing (no flat surface nearby to place it on). We don’t want to create a camera stand that would rest on the ground since it would have to be very tall to capture the full length of the keyboard from overhead. It would be difficult to create a stand that is very tall, adjustable, strong enough to hold the camera, and meet our use case requirements of being quick to set up and portable. 
    • Since the camera stand needs to be fairly tall and attached to a clip, it should be as light as possible. For this reason and to minimize costs, we decided to opt for a webcam. 
    • Another concern is camera compatibility with the KR260, which we are waiting for Varun to verify. We know that the camera must have a USB output to connect to the FPGA though. 

    I did some calculations using the field of view of variously priced cameras and the length of a piano keyboard to get a feel of how tall the stand would need to be and which camera we should opt for.

  3. I also started Varun’s KV260 setup tutorial. I estimate that I am ⅓ of the way through.

I am currently still on schedule. 

Next week I plan to finish the KV260 setup guide and install the packages necessary for media pipe.