Posted on

Jerry’s Status Update for 02/29 (Week 3)

Progress

I started this week by finishing up the design presentation. I hooked up the camera system, image processing system, webserver, and robot to finish the first version of our MVP. We are now able to control the robot using teleop and also gesture for the robot to go home.

However, there are some issues with recognizing gestures using heuristics when the user is not facing the camera directly. So, I have built a data collection pipeline to gather training data to train a SVM classifier for if a teleop gesture means a person is pointing left, straight, or right. The features are normalized distances from each keypoint to the person’s chest, and the labels are the gesture. We started collecting around 350 datapoints, and will test models next week.

There are also some issues with the Roomba built in docking command, so we will have to overwrite its command to use our mapping to navigate close to home before activating the Roomba command that slowly connects to the charging port.

In addition,  I installed our second camera. Processing two camera streams halves our FPS, since we have to run OpenPose on twice the frames. So, we are rethinking using multiple cameras and seeing what we can do with two cameras. Ideally we can still have a side camera so there is no one angle where you can gesture perpendicular to all the cameras in the room. I will have to keep experimenting with different angles.

I also have been working on the design report.

Deliverables next week

1. First pass of the gesture classification models

2. Finalize camera positioning in the room

Schedule

On Schedule.

Posted on

Sean’s Status Update for 02/29 (Week 3)

Progress

This week, I worked on the building a mapping algorithm and improving the Roomba control protocols. For the mapping algorithm, I initially intended to use Roomba’s internal “clean” function to scan the room. However, it turns out that I cannot directly control or get sensor reading once it is in the clean mode. Thus, it was necessary to build our own mapping algorithm. First method I plan to test is to make the robot 1) follow the wall, 2) move back and forth in the area in between. This seems to be the method iRobot is using as well. This would be done autonomously. During this process, it became clear that some kind of threading would be necessary. Since the data packet is being streamed constantly, having a thread dedicated to processing the odometry was necessary. I was able to control the Roomba using threading with my local computer, but haven’t tested on the RPi yet.

Deliverables next week

Next week I would focus on 1) completing the mapping algorithm and 2) making sure the control protocol is robust.

Schedule

On Schedule.

Posted on

Team Status Update for 02/22 (Week 2)

We all put in a ton of work this week (~60 hrs) towards our MVP. Most of the individual components are completed, and we will integrate our system next week.

OpenPose and Xavier board

We were able to successfully run OpenPose on the Xavier board at 30 FPS after a few optimizations. Additionally, we were able to setup the Xavier board with the CMU network, so we can ssh into our board to work on it anytime.

Gesture Recognition

We were able to start classifying our 3 easiest gestures, left hand up, right hand up, and both hands up with heuristics using images and video examples. We set up the camera we purchased and we are able to classify gestures live at 30FPS. We also developed heuristics for tele-op commands, but are going to try model based approaches next week.

Webserver

After our tests with OpenPose on the board showed that it only used less than 20% CPU, we decided to host our webserver on the Xavier board. This lets us reduce the latency for communication to AWS. We also tested python flask and node.js express servers. We decided on using node.js because it was asynchronous and could handle concurrent requests better. We also tested the latency to the server, and it took around 70ms, so we are still on track for being under our 1.9s response time requirement.

Raspberry Pi

We successfully setup the Raspberry pi for the robot module. Raspbian was installed, which might be helpful programming the robot in the early stage before we move on to the headless protocol. Rama also setup a web server for the RPi so we can control it without a physical connection.

Roomba

We setup the Roomba and are able to communicate with it to pyserial. Sean was able to experiment with the Roomba Open Communication Interface to send motor commands and read sensor values.

Design Review Presentation

We all worked on the design review presentation next week. The presentation made us discuss our solution approaches to different problems. Now that we have tested many of our individual components, we had a much better perspective on how to tackle many of our problems.

Drive-to-Point

It is still early to think about this functionality since it is beyond our MVP, but we spent some time this week discussing about potential solution approaches to it. We will try out both methods and determine which is the right way to proceed.

    • Method 1: Using multiple images from different perspectives to draw a line from the user’s arm to the ground.
      • Using trig with angles from core to arm, arm to shoulder, and feet position to determine ground.
    • Method 2: Using neural network classifier to predict the position in the room using keypoints as input
      • Collect training data of keypoints and proper bin
      • Treat every 1ft x 1ft square in the room as a bin in a grid
      • Regression model to determine x and y coordinate in the grid

Deliverables next week

We are on track to complete our MVP and we hope to finish our tele-op next week!

Posted on

Rama’s Status Update for 02/22 (Week 2)

Progress

We got OpenPose running on the Xavier board with the USB camera, and got a static IP for the Xavier board so we can develop on it remotely as needed. I started work on the webserver that the Xavier board will be running to bridge the gap between the gesture recognition and Roomba control. We also got a lot of the specifics of implementations regarding gestures nailed down.

Posted on

Jerry’s Status Update for 02/22 (Week 2)

Progress

Rama and I got OpenPose running on the Xavier board this week.  This allowed me to start playing around with the results OpenPose provides.

Gestures:

After first running OpenPose on images we took manually and videos, I was able to classify our 3 easiest gestures with heuristics: left hand up, right hand up, and both hands up. After we got our USB webcam, installed the webcam, and got the camera to work with the Xavier board (required a reinstall of OpenCV), I started working on classifying teleop drive gestures (right hand point forward, left, and right). I implemented a base version using heuristics, but the results can be iffy if the user is not standing directly at the camera. I hope to try to collect some data next week and build a SVM classifier for these more complex gestures.

OpenPose Performance on Xavier board:

Rama and I worked on optimizing the performance of the OpenPose on the Xavier board. With the default configuration, we were getting around 5 FPS. We were able to activate 4 more CPU cores on the board that brought us to 10 FPS. We also found an experimental feature for tracking that limited the number of people to 1, but brought us to 30 FPS. (0.033s per image) We were anticipating OpenPose to take the longest amount of time, but optimizations made it much faster than we expected. It should bring us below our time bound requirement of 1.9s for a response.

Openpose running with the classified gesture: “Both hands raised”

Deliverables next week

1. Help the team finish teleop control of the robot with gestures. Integrate the gestures with the webserver.

2. Collect data for a model based method of gesture classification.

3. Experiment with models for gestures classification.

Schedule

On schedule to finish the MVP!

Posted on

Sean’s Status Update for 02/22 (Week 2)

Progress

I had a chance to actually work and play around with Roomba this week. We purchased Roomba 671, which can be controlled using serial communication with a help of their Open Interface.

Understanding Roomba Open Interface:

After inspecting the Open Interface documentation, I was able to figure out the basic serial communication protocol the Roomba expects. There are 3 modes the Rooba can be at: passive, safe, and full mode. Each mode has different limitation to how much control the user has. For instance, the robot is in the passive mode by default. In this mode, we cannot interrupt the movement of the robot (i.e. have control over the motor system). Once we switch to the safe/full mode, we are able to directly drive the robot.

Motor control:

To drive the motor, we send 2-bytes of serial command to the robot. 1 byte for the velocity and 1 byte for the radius of rotation. It is interesting iRobot decided to make their drive method this way instead of more conventional left-right-velocity control.

Issue:

Currently, the sensor reading is unstable. The first issue I encountered was that the sensor reading becomes unreliable when the mode changes. For instance, the robot returns a reasonable encoder value in the safe mode. However, once it switches to another mode (e.g. “Dock” command automatically switches the mode to passive) the encoder value jumps to unexpected value, making the data unusable. I suspected that there is some internal method that resets encoder/sensor reading once the mode changes. So I attempted to work around it by requesting a constant stream of data packets from Roomba. Luckily, this got rid of the first issue. However, I found out that there seems to be some data corruption during the streaming. Sometimes, it returns an unreadable packet with the incorrect header and check-sum byte. I attempted to first ignore the corrupted data, but it seems like they are considerable portion of the returned data. I will look into this problem further in the following week.

Deliverables next week

We were able to accomplish some basic functionalities of the robot. Now, we must integrate the system all together and test it. As of the moment, my goal next week is to:

1. Finish tele-op control
2. RPi headless control of Roomba
3. Integrate Gesture recognition with Roomba control

Schedule

On Schedule.

Posted on

Team Status Update for 02/15 (Week 1)

We started off this week going over what materials we needed to purchase for our project. We compared pros and cons of different product options, looking at their price and their specs. We ordered:

  • Roomba 671 – documentation for our Roomba API showed it worked with this model
  • Cable to connect roomba to USB – recommended by documentation of Roomba API
  • Raspi – To receive data from system and relay commands to the Roomba
  • Raspi power supply – To power the Raspi
  • Wide angle camera – We found other people using wide angle cameras with OpenPose for gesture recognition.

We built a spreadsheet to track our costs and manage our budget.

We continued the design process by finalizes what gestures our system will handle. We decided on:

  • Tele op control with 4 gestures – Tele op commands set with left arm on right elbow
    • Forward – Right arm with elbow tilted forward
    • Backward – Right arm with elbow tilted back
    • Rotate left – Right arm with elbow tilted left
    • Rotate right – Right arm with elbow tilted right
  • Save home – Right hand up
  • Go home – Left hand up
  • Drive to pointed location – Right hand point while left hand up
  • Drive to user – Both hands up

We also drew out a design for what we want the robot to look like. (with cookies!) This helped us finalize how many more parts we need to buy to build the robot.

We also wanted to experiment with possibly using multiple cameras for our project, with one to identify gestures mounted at desk level and one overhead camera used for localization of user and robot. We want to experiment with OpenPose before finalizing this design. For similar issues, we created an issues tracking doc for project risks. We put up solutions we thought of now, but we want to experiment more with our parts before we can make plans around them.

We started testing with installing OpenPose locally. It ran super slow on our machine (without GPU), 30s per frame. Additionally, we ran on a Google Cloud instance with a NVIDIA Tesla K80 GPU (5000 CUDA cores), which took 0.42s per frame. We don’t need to process every frame our camera gets for our system, but we believe 0.42 is still slow. The example video we ran it on had up to 12 people to run OpenPose on, which could have also increased the latency of it. We hope we can get the runtime down running it on our Xavier board. (500 cores) There are specific methods to install OpenPose on the Xavier board, so we hope we can install an optimized OpenPose that runs faster.

Later in the week, we received our Xavier board, so we have been working to try to run OpenPose on it. The default method for Xavier setup was to use a Ubuntu 16 or 18 machine with 20GB free space, but our computers did not have that much space or the specific linux version. We tried many different ways to install CUDA packages on it, and ended up finding success installing the OS with L4B and using SSH to install jetpack on it. We are still working to get OpenPose to run on the board.

Posted on

Rama’s Status Update for 02/15 (Week 1)

Progress

Installed OpenPose on my laptop to test out what kind of information we can expect to receive. It ran very slow with around 30 seconds per frame, which was around what we expected from CPU-only execution.

Started the installation progress on the Xavier board, and after much difficulty trying to operate within the strict confines of NVIDIA’s JetPack SDK installer. I ended up creating an Ubuntu VM on my laptop through VirtualBox, and we were able to flash the OS and install CUDA, OpenCV, and other dependencies.

Installing OpenPose was very difficult and is not yet completed. All of the provided installation scripts are outdated and the process required extensive hacking. Unfortunately, there were immediate runtime errors so we will likely have to do some research. From a cursory investigation, I suspect our CUDA versions are to blame, so a first step will be a clean reinstall of CUDA.

Schedule

On schedule with the board, but gesture recognition will take longer than expected.

Posted on

Jerry’s Status Update for 02/15 (Week 1)

Progress

I started this week by researching what components we need to buy, figuring out what hardware the Roomba SDK and OpenPose supported. I submitted the purchase requests by Tuesday, and hopefully we can get our components ASAP so we can start testing and finalizing our design.

Since our hardware components have not arrived, I spent more time with the team finalizing what gestures we were going to have in the system and discussing what pathing and localization strategies we were going to use for the system.

OpenPose on Google Cloud

In addition, I worked with Rama to begin testing with OpenPose. I got OpenPose to run on Google Cloud GPU (Tesla K80 5000 CUDA cores), getting 0.4s per frame of processing. This is too long for the requirements of our system, but I believe alot of time is in the file IO of writing the sample results onto video. OpenPose also has documentation the website for how to optimize the time of OpenPose. AWS will serve as a backup for our system if OpenPose does not run fast enough on the Xavier board.

Running OpenPose also helped us get a better understanding of what keypoint information to get to identify gestures. We want to start with a heuristic classification approach for gestures, and use a classifier like SVM or neural net if necessary.

OpenPose on Xavier board

When we got the Xavier board, I worked with Rama to get OpenPose running on it. We faced a roadblock since we did not have a computer that fit the requirements from the SDK, so we had to use a VM to flash the L4T OS onto the Xaivier board. We also installed the Jetpack SDK (a collection of NVIDIA CUDA software for ML) onto the board.

We faced a few issues getting OpenPose to run on the board. We had to modify their existing scripts, fix C++ code headers, and add paths to library path to load. At the end, we ended up with multiple Caffe, CUDA, OpenCV versions and an installed OpenPose. However, OpenPose had CUDA errors during runtime, so we are trying to do another clean install.

Schedule

We are on schedule to get OpenPose to run on the board. However, we are more behind schedule on the gesture recognition front since our local computer cannot run OpenPose at a reasonably fast rate.

Posted on

Sean’s Status Update for 02/15 (Week 1)

Progress

This week’s progress for me was mostly doing researches. Since the Roomba hasn’t arrived yet, I had enough time to give some thoughts to the implementation of the robot.

Path finding algorithm:

Since the goal is fixed throughout the path finding, and the environment is for the most part unchanged, Any-time search such as Anytime A*/D* won’t be necessary. Instead, we can incrementalize the C-space and perform some variation of a simple A* search. We will test different algorithms to see how much path-finding computation RPi can handle. If the computation takes too long, we can perhaps take advantage of the Xavier board.

Robot configuration:

We have previously found some SDK for mapping and Python wrapper library for Roomba Open Interface. These seem promising, but after some close examination of the Python wrapper library, I have some doubts about it working accurately as-is. We will begin testing these by implementing a simple motor controls.

Deliverables next week

Hopefully we will get the Roomba next week, then we will have opportunities to test our ideas on the actual robot. As of the moment, my goal next week is to:

1. Configure the Roomba with RPi
2. Import SDK/Libraries
3. Implement simple motor control (turning, moving back and forth)

Schedule

On Schedule.