General accomplishments:
- We worked on the design report and slides.
- Jessie looked into the AMD workflow for putting a model on the FPGA. We now have a more fleshed-out idea of the tasks that need to be done. More details can be found in Jessie’s status report.
- Danny finished the Django tutorial.
- We asked Dueck’s students on their preferences on different WebApp features. More details can be found in Danny’s status report.
- We met with Dueck’s students to preliminarily test system
- Need to track angle variation rather than consistent angle position—more details in Shaye’s status report
- We have better specified the setup measurements and requirements based on additional measurements of Shaye playing piano. (Shaye not pictured 🙁 )
Risks:
- We are slightly concerned about converting the currently used MediaPipe model (TFLite) to PyTorch or TensorFlow, the only 2 formats compatible with Vitis AI. We plan to look into
- existing scripts that convert between the 2 formats or
- find different models that are in the compatible format
- We are also worried about the MediaPipe model losing accuracy once compiled on the FPGA due to over-optimization and bad inputted samples during the quantization phase
- We plan to try multiple data sets and ways to convert samples to tweak accuracy
- We need to look into how to implement kinematics on FPGA. We are not sure how to interact with the output of the model
- Contingency: switch to RPi—will request on 10/8 if needed so we have time to work on it before fall break
Updates on the system:
- We figured out how to port model & communicate between FPGA & RPI
- Using Vitis AI workflow for model conversion
- Using UART for board communication
Schedule updates:
- Still on schedule, no updates
Additional questions:
Part A is written by Jessie, Part B is written by Danny, and Part C is written by Shaye
Public health/ safety/ welfare factors:
Our product aims to reduce injury among piano players by identifying and correcting harmful playing positions. Injury among piano players is extremely common (50-70%), with many injuries being related to the hand and wrist. The effects of being injured can have mental health impacts additionally, as it may prevent players from being able to practice for extended periods. Our system will provide players with real-time and post-processed feedback on the player’s hand position. This will help them correct their position, even without the guidance of a piano teacher. We realize that providing bad feedback can lead to more injury; thus, we’re taking special care to focus on the testing and verification metrics we’ve mentioned throughout this process to ensure system accuracy.
Social Factors:
Our product will change the dynamic between the piano teacher and the student. Instead of having the piano teacher focus on the health of the student and harmful techniques, they can focus on music-related content. This product could also help encourage beginners to get into piano playing without the potential worry of becoming injured, lowering the barrier to entry. This could potentially lead to an increase in piano players.
Economic factors:
Our project will be the first product pianists can use to monitor their technique while practicing. Thus, although the total cost of our current system is high (FPGA cost, camera, stand, etc), creating a proof of concept using more general hardware will allow for more projects down the line to decrease the cost and create more accessible & commercialized products. With more specific hardware/ boards dedicated to running our system, the cost will decrease, allowing the product to be available for all piano players. For now, even just parts of this tool (CV pipeline, video saving features) can help save pianists from injury and incurring more personal costs.