Jessie’s Status Report for 09/28

This week’s tasks:

1. After finding a webcam that we plan to use and talking to Jim Bain and Joshna more about the camera stand design, I finalized the necessary measurements. Since we have chosen a webcam, I used that webcam’s FOV to calculate the distance between the camera and the keyboard. Additionally, I worked with the group to measure the distance between the camera and the base of the stand (how much does the stand have to bend over the player). The measurements and calculations can be found in the team status report. From these calculations, the stand will have to be very long (about 7.5’ or 8’ if we want some wiggle room). I’m starting to look into attaching a gooseneck (various flexible plumbing materials) to a mic stand or tripod. 
2. We were able to acquire the KV260, so I stopped working on Varun’s Vivado tutorial since I’m not sure if it’s still applicable. Instead, I started looking into Vitis AI. 
   1. The KV260 came without any accessories, so we had to scavenge to find them. We were able to find all the necessary components but discovered the 
3. I learned more about the flow of Vitis AI and what would be required from us. My findings are largely based on the Xilinx documentation/tutorials but also from https://github.com/PeterQuinn396/KV260-Pose-Commands/tree/main and https://www.hackster.io/AlbertaBeef/accelerating-the-mediapipe-models-with-vitis-ai-3-5-9a5594#toc-model-inspection-7 who successfully put a MediaPipe model on the KV260.
   1. Vitis AI takes in models in either a TensorFlow or PyTorch format. However MediaPipe is in a TFLite format. The hackster post mentions conversion scripts from other people, but they had little success using these scripts. We might have to write a conversion script or look into finding another model that is in a compatible format.
   2. Vitis AI first takes the model and inspects it. From the hackster post, even if the model doesn’t pass the inspection, the Custom OP flow can be used as a workaround. Then Vitis AI will optimize the model (prune it). Then the model gets quantized. This step requires several 100s to 1000s of samples. 
      1. Since we don’t have access to the data set used to train the MediaPipe model, we will have to find a data set and convert it to samples. Both Peter Quinn and the author of the hackster post write scripts to convert data sets to samples that we can reference. 
      2. A concern I have right now is the quality of the samples and how that might influence the accuracy of the model. The author of the hackster post experienced degraded model accuracy as a result of using bad samples. We will likely have to experiment with different data sets or sample generation scripts to maintain a high model accuracy.
   3. Vitis AI then compiles the mode. I realized I’m not sure how to interact with the output of the model on the FPGA, so I will have to look into that in order to successfully calculate the kinematics on the FPGA.
4. I worked with Danny to follow this quick guide from xilinx https://xilinx.github.io/Vitis-AI/3.0/html/docs/quickstart/mpsoc.html. We downloaded Vitis AI and ran the scripts, but had to stop at the setting up the target step since we discovered the power supply we found was not the right size for the board.

Schedule:

I’m still on schedule.

Next week’s deliverables:

1. Once the power supply comes in, successfully run Vitis AI on the example models provided. 
2. Either determine it’s not possible/feasible to run kinematic calculations on the FPGA, or have a plan for a way to execute it. 
3. Finalize plans for a camera stand so that we have a rudimentary stand the next time we meet with Prof Dueck’s students on 10/9.