We all put in a ton of work this week (~60 hrs) towards our MVP. Most of the individual components are completed, and we will integrate our system next week.
OpenPose and Xavier board
We were able to successfully run OpenPose on the Xavier board at 30 FPS after a few optimizations. Additionally, we were able to setup the Xavier board with the CMU network, so we can ssh into our board to work on it anytime.
Gesture Recognition
We were able to start classifying our 3 easiest gestures, left hand up, right hand up, and both hands up with heuristics using images and video examples. We set up the camera we purchased and we are able to classify gestures live at 30FPS. We also developed heuristics for tele-op commands, but are going to try model based approaches next week.
Webserver
After our tests with OpenPose on the board showed that it only used less than 20% CPU, we decided to host our webserver on the Xavier board. This lets us reduce the latency for communication to AWS. We also tested python flask and node.js express servers. We decided on using node.js because it was asynchronous and could handle concurrent requests better. We also tested the latency to the server, and it took around 70ms, so we are still on track for being under our 1.9s response time requirement.
Raspberry Pi
We successfully setup the Raspberry pi for the robot module. Raspbian was installed, which might be helpful programming the robot in the early stage before we move on to the headless protocol. Rama also setup a web server for the RPi so we can control it without a physical connection.
Roomba
We setup the Roomba and are able to communicate with it to pyserial. Sean was able to experiment with the Roomba Open Communication Interface to send motor commands and read sensor values.
Design Review Presentation
We all worked on the design review presentation next week. The presentation made us discuss our solution approaches to different problems. Now that we have tested many of our individual components, we had a much better perspective on how to tackle many of our problems.
Drive-to-Point
It is still early to think about this functionality since it is beyond our MVP, but we spent some time this week discussing about potential solution approaches to it. We will try out both methods and determine which is the right way to proceed.
-
- Method 1: Using multiple images from different perspectives to draw a line from the user’s arm to the ground.
- Using trig with angles from core to arm, arm to shoulder, and feet position to determine ground.
- Method 2: Using neural network classifier to predict the position in the room using keypoints as input
- Collect training data of keypoints and proper bin
- Treat every 1ft x 1ft square in the room as a bin in a grid
- Regression model to determine x and y coordinate in the grid
- Method 1: Using multiple images from different perspectives to draw a line from the user’s arm to the ground.
Deliverables next week
We are on track to complete our MVP and we hope to finish our tele-op next week!