FinalReport

Joseph:

This week I worked on fine tuning the movement of the bot and dealing with edge cases. There are still a few more that we have to deal with so ideally, by next week we will have addressed all of them.  We also created a 4 x 4 grid in one of the rooms we have been using and measured the accuracy of the movement so far. For the final we anticipate demoing the bot on a 10 x 10 so we realized that we will have to ensure that we have enough time for set up before hand as well as having enough obstacles ready to show obstacle avoidance. The second half the week was spent working on the final presentation slides since our presentation is on Monday.

For next week, I would like to have hit all edge cases and tested the integrated system before the demo. I would also like to test the system with more than 3 people so that we can ensure that we are ready for variations in environment.

This week I completed and tested my pipeline script. The script scp’s images that were saved from Joseph’s UI to the aws instance. It then runs the inference and pushes the detection images to the local computer. This week we also had our demo on Wednesday. On Friday, Joseph and I worked to integrate the roomba UI/ script with the model script.  This weekend we will be working on the project poster and the final presentation slides.

Next week we will be meeting to test the integration and to identify edge/trouble cases for the model. From these results, I will determine whether I should train the model for a few more epochs given the budget or if I should simply play around with the bounding box threshold.  I will also be modifying the inference script to produce a count of humans to supplement the detection images.

Joseph:

Next week I plan on implementing the sensors as well as work with Karen to finish obstacle avoidance. I will also be working with Manu when the model is ready to implement that with the UI

Next week, I would like to aid Joseph in incorporating the sensors on the bot and testing the obstacle detection script. Ideally we would like to get obstacle detection completed by next week so that the following week we can fine tune obstacle avoidance.

One issue I ran into this week was with the memory capacity of my ec2 instance and the compute time required to retrain the entire model. In order to train the model with 8 GPU’s and have enough memory per GPU for the given model and data size I need to upgrade to a P3.xlarge. However the cost for this machine is very expensive. I am currently doing some calculations to identify a way to serve our needs while staying within budget.

Joseph:

This week I was able to connect make a connection between the UI and the robot. I am able to send coordinates to the raspberry pi via a script that sends commands through ssh. I run the movement script with coordinates as the inputs and the bot is capable of moving.

Next week I hope to have some form of picture transfer from the robot to the laptop working. I will also work with Karen to fine tune movements as well as installing the ultrasonic sensors for obstacle avoidance.

Karen:

This week was mostly spent fine tuning the movement of the bot again. I have noticed that there seems to be an accumulation of error in the angle when we use the turn_angle function from the pycreate module. Because of this, usually our first two coordinates are accurate, but when the bot re-orients itself, it seems to overshoot. The offset is not consistent so we cannot adjust accordingly. I am now thinking of creating my own turn_angle function that will instead use time to control the angle of rotation. Hopefully this will give us more accurate results. However, because we have given ourselves a margin for error in the accuracy of movement, I do not anticipate this being an incredible issue. I have also basically finalized my portion of the movement script with localization and will hand it off to Joseph for him to incorporate sensors.

This week we will be doing our demo. By Wednesday I hope to have the UI completely connected to our script so that when points are clicked on the GUI, this transfers to the command line and runs the movement script. Then, the bot will go to those points and rotate in a manner that if a camera were attached, a picture could be taken. I hope to finalize movement and help Joseph with the sensors this week as well.

Manini

This week I pivoted from the CPU version of the Faster RCNN model to a GPU based version. I was running into too many issues with the integration set up and module builds. I decided that the time I was spending on getting this version of the code to work was not worth it. I therefore launched an AWS instance with GPU capabilities and a Deep Learning AMI. I finally got the new GPU based Faster RCNN up and running and ran multiple tests using both VGG 16 and ResNet101 architectures. I then had to re-add the bias with the new model and figure out how to switch the model from multi-way classification to binary classification. Over the weekend I will be training this new modified model so that for the demo we can hopefully use this new version. Now that the inference will be completed on AWS, we will also need to create a script to send the images from the pi to AWS.  This pivot put me slightly behind schedule, but  I hope I can get back on track in the upcoming week by putting in more hours outside of class.

This weekend I also helped Karen with the bot’s movement. We ran multiple tests and recognized a  fundamental inconsistency with the roomba’s angle turns. In order to accomplish accuracy with the bot’s basic movements, we will need to utilize a different approach (time-based turning) to enable the roomba to turn the amount we would like.

TEAM STATUS:

• What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The current risk is that we are slightly behind in schedule. We do have a slack week that we are currently using up but it would be preferable if we could somehow catch up back on schedule incase any other issues arise. Our contingency plan is to meet more outside of class work on the project

• Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

Instead of running inference of the model on the local laptop, we are going to send the images to AWS as we were unable to get the CPU version of the model working properly. This requires us to write a script to send and receive images from the cloud but overall it is not a huge change.

C9

Joseph:

This week I worked on creating the presentation for the design review as well as the design document that followed it. I also began looking for different Faster-RCNN repositories on Github as we decided that reimplementing the entire model ourselves would be too tedious of a task and we would not gain much from that experience. Our main task is to have an existing model to tune and bias. I was able to come across a model that seemed promising as it provided the option to run on both gpu and cpu, which is exactly what we need.

For next week, I would like to have this model running and then I will begin helping Karen with obstacle avoidance when the sensor come in.

 

Karen:

This week I focused on the design review and putting together  the official write up for this. After the presentation, we reviewed the immediate feedback and have tried to incorporate this into our future plans. I am currently trying to come up with more benchmarks for the path planning and obstacle avoidance part of the project. This week we also just received the SD card and reader, so now we can finally upload to the pi. The time it took for this to arrive has caused a significant amount of delay and we hope to address that by putting in more time outside of class. I have downloaded the Raspbian image onto my laptop and have started setting up the pi for use.

Hopefully, now that we ownership of all the different parts, we can finally start getting code uploaded onto the pi and have it running on the roomba. I do think that there might need to be some adjustments with how the commands are serially sent, so I will have to add some additional testing to see if they are being received properly. Currently, the networking protocol I have implemented is based off of example scripts. However, I think we might need adjustments, regarding time and frequency. So, once the code is uploaded, I will have to adjust for that.  Overall, I think there has been overestimation in how quickly we would be able to get the Roomba to move. So ideally I would like to have successful movement by the end of next week- assuming all class time will be used for working.

Manini

This week I focused on the design review and my presentation.  Joseph, Karen and I had to fine tune some design specifications including metrics and implementation decisions.  During our design review, the Professor suggested that it might be a good idea to try using both YOLO and Faster RCNN and compare the performance of both. Thus, for the later part of the week I began searching for a good YOLO implementation and learning more about the internals of the model. In terms of Faster RCNN, I downloaded one version on my laptop and am currently trying to get it up and running with Pytorch.  I am also currently trying to sift through the COCO dataset and limiting the size of the data we will be using. This was another suggestion by Professor Savvides to improve the training time for our model and to stay on track.

In this upcoming week I would like to begin training our Faster RCNN model and get some baseline performance metrics so we can plan for how we want to bias/ fine tune the Faster RCNN model. The design review took up a little bit more time than we had anticipated so this week we plan on putting in extra hours outside of class to catch up and finish our first sprint before spring break.

TEAM STATUS:

• What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The biggest risk right now is still staying on schedule. We did not account for the time it takes to make presentations and reports which has really limited the amount of time we have to do other things.

 

• Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

One change that we have made is that we will also be trying to use the YOLO model for human detection. This change is so that we can compare the performance of both Faster RCNN and YOLO to achieve the best combined accuracy and inference time.

C9