This week’s tasks:
- I started the week by setting the RPi so I could ssh into it from my laptop. I had trouble doing this on my hotspot or the school wifi, but I could do it on my home wifi. Additionally, we needed to connect the RPi to some I/O (monitor, keyboard, and mouse) to connect the RPi to wifi. This is not feasible in the future when we need to use the RPi on campus. We plan to register the device to the CMU device wifi and get that set up next week. More long term, we plan to look into incorporating our code into the booting so that the program is automatically running when the RPi is booted up. We hope that if we do this, the user will not have to ssh into the RPi to run our code and therefore not even need to worry about connecting the RPi to wifi.
- I tried to go through the tutorial mentioned in last week’s status report for putting the MediaPipe model on the RPi that was mentioned; however, I was unable to get their script to work since the accelerator component hasn’t arrived yet. We believe the part will come early next week.
- As a fallback plan in case the accelerator doesn’t work and out of curiosity about the speed of the model without an accelerator, I followed this tutorial to run the MediaPipe model on the RPi. It seems to consistently run at 10 fps, even when there is movement in the frame. For some reason, the model can only detect one hand at a time. The accelerated tutorial seemed able to detect 2 hands at a time, so we can reevaluate this issue once the accelerator comes in.
- As a fallback plan in case the accelerator doesn’t work and out of curiosity about the speed of the model without an accelerator, I followed this tutorial to run the MediaPipe model on the RPi. It seems to consistently run at 10 fps, even when there is movement in the frame. For some reason, the model can only detect one hand at a time. The accelerated tutorial seemed able to detect 2 hands at a time, so we can reevaluate this issue once the accelerator comes in.
- I shopped around for displays for the calibration step. Most were fairly big (7 inches), advanced (unnecessary features like touch screen), and expensive (around $60). I ended up landing on this one because it is smaller and cheap. We were slightly worried about the 2-inch display being too small; however, we thought it was reasonable as it’s bigger than the screen of a smartwatch. Once the display has arrived, I will look into how to interface it with the RPi.
- Shaye was able to find a buzzer. I tried to write some basic test code to interface with it. The code I wrote can be found in the accelerated_rpi folder of this Github repo. You can look at the commit history to see the different iterations of code I tried. The final version of the code is on the jessie_branch (I forgot to branch my code earlier). Shaye informed me that this is a passive buzzer, so it takes in a square wave and buzzes at the frequency of the square wave. I tried to use different libraries with the GPIO pins and struggled to get the buzzer to buzz at a loud volume (you can see my struggles in the multiple iterations of code). However, with the help of Shaye, we were able to max out the volume of the passive buzzer by tinkering with the frequency. The max volume of the passive buzzer is still somewhat quiet and we’re not sure if the user will be able to hear it well over piano playing, so Shaye plans to acquire an active buzzer. I will have to write different code to interface with this active buzzer in the coming weeks.
- Here is a link to some video of the buzzer with piano in the background: https://drive.google.com/file/d/1GE6zB6sEb07_hdlClQzQO-LpfjCWQq9t/view?usp=sharing. In the video, the camera is closer to the buzzer than the piano and the keyboard is set to a low volume.
- I attempted to interface UART between the 2 RPis. You can find the sample code on jessie_branch in the same Github repo. The sender code is in the accelerated_rpi directory and the receiver code is in the host_rpi directory. I was unable to get this sample UART code to work– the code doesn’t print anything so the receiver isn’t receiving anything. I suspect this is because the host_rpi is running on Linux OS (for Danny’s web app needs), so the UART setup is a little different. I’m not sure if I set up UART correctly on the receiver RPi because when I look for serial ports, none of the common ones show up; therefore I don’t think I’m using the right port in the receiver code. Next week I plan to look into other ways to do UART.
- Currently, I’m using the tx and rx pins, but Shaye has suggested I also try using the dedicated UART JST pins. For these pins, I won’t be able to use jumper cables, so I’ll need to acquire JST pins.
- Another possible option we can explore is using the USB port instead. I’ll need to acquire a USB-to-USB cable for this approach.
Schedule:
I am still roughly on schedule as though I haven’t finished the accelerated MediaPipe on RPi tutorial (waiting to receive the accelerator), I did get the model on the RPi and I have made progress with interfacing I/O on the RPi (UART, buzzer, display).
Next week’s deliverables:
- Get the RPi working on the school wifi.
- Finish the tutorial and accelerate the model on the RPi.
- Write new code for the active buzzer.
- Get UART working between the 2 RPis.
Some future tasks that are less time-sensitive:
- Look into adding code to the boot so we don’t have to connect to wifi
- Connect fans to RPi
- Connect the display to RPi