This week’s tasks:
- I started the week with the intention of putting the model onto the KV260. I downloaded the model and dataset (for the quantization step) suggested by Shaye. I parsed through the dataset and selected a subset that I thought would be best for the quantization step. Note that the model we chose is not same the MediaPipe model Shaye has been using. This is because the Vitis AI workflow is only compatible with Pytorch and TensorFlow files while the MediaPipe model is a TFLite file. For now, we opted to use the Blaze model instead which is in the form of a PyTorch file, but planned to look into conversion scripts in the future if the accuracy of the Blaze model was too low.
- I met with Varun to discuss our concerns and difficulties while using the KV260. We learned that using the KV260 is unfavorable to the accelerated Raspberry Pi for our specific application.
- The Raspberry Pi has a more powerful accelerator and processor. The RPi can do 13 tera-operations per second (TOPS) while the FPGA can do around 1 TOP.
- The Raspberry Pi is easier to work with because there are more resources online. We found someone using the RPi with our exact model application who achieved a frame rate of 22-25 fps. The KV260 would take a lot of effort to reach our target frame rate (if it’s even possible). Varun said that the frame rate of processing the Restnet 18 model (from the quickstart tutorial), which I found to be 15-24 FPS from doing the tutorial a previous week, is likely the maximum capability of the KV260.
- Varun mentioned the following benefits of using the KV260; however, none of them apply to our project
- The KV260 is more power efficient than the RPi. We plan to have users plug the device into wall power (no battery necessary), so this is not a concern for us.
- The KV260 has more I/O. It has more pins that can be used for anything while the RPi just has a small amount of GPIO pins and some USB ports. However our application has a limited number of I/O (display, buzzer, webcam, web app hosting RPi), so the additional I/O pins that the KV260 provides are not necessary for us.
- The KV260 can be useful if there are multiple things that need to be accelerated. However, our application only requires the model to be accelerated. Varun believes that software should be fast enough to run the tension detection algorithm.
- I read through the documented RPi workflow for putting MediaPipe’s hand landmark model onto the accelerated RPi to prepare to do it myself next week.
- I also worked on the ethics assignment that was due this week.
Schedule:
I failed to complete last week’s assigned deliverable because we decided to shift from putting the model onto the KV260 to putting the model on an accelerated RPi. I’m currently behind schedule as Shaye and I originally planned to finish the FPGA/CV integration by the coming Wednesday; however, due to the pivot from KV260 to Raspberry Pi, our schedule has changed as noted in the team status report.
Next week’s deliverables:
Next week I plan to integrate the MediaPipe model onto the RPi. I should also have a rough idea of how to implement the buzzer and display feature. I will also look into how to set up UART to communicate with the web app hosting RPi.