Posted on

Status Update: Week 10

Joseph:

This week I worked on integrating my UI with Manini’s model so that it can now send images to the cloud where inference is happening. I also worked on cleaning up the code for the UI, making it more robust. I also worked with Karen on further fine tuning the movements of the robot as well as work out how we are going to approach demo day. I also worked on the poster and the presentation slides.

Next week the goal is to test everything. There are a few known bugs with movement that can be fixed relatively easily and the hope is that we find all of them before the day of the demo.

Karen:

This week I worked on fine tuning the movement of the bot and dealing with edge cases. There are still a few more that we have to deal with so ideally, by next week we will have addressed all of them.  We also created a 4 x 4 grid in one of the rooms we have been using and measured the accuracy of the movement so far. For the final we anticipate demoing the bot on a 10 x 10 so we realized that we will have to ensure that we have enough time for set up before hand as well as having enough obstacles ready to show obstacle avoidance. The second half the week was spent working on the final presentation slides since our presentation is on Monday.

For next week, I would like to have hit all edge cases and tested the integrated system before the demo. I would also like to test the system with more than 3 people so that we can ensure that we are ready for variations in environment.

Manini:

This week I completed and tested my pipeline script. The script scp’s images that were saved from Joseph’s UI to the aws instance. It then runs the inference and pushes the detection images to the local computer. This week we also had our demo on Wednesday. On Friday, Joseph and I worked to integrate the roomba UI/ script with the model script.  This weekend we will be working on the project poster and the final presentation slides.

Next week we will be meeting to test the integration and to identify edge/trouble cases for the model. From these results, I will determine whether I should train the model for a few more epochs given the budget or if I should simply play around with the bounding box threshold.  I will also be modifying the inference script to produce a count of humans to supplement the detection images.

TEAM STATUS:

  • What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

As of right now, movement is still a little bit of an issue but hopefully after further fine tuning we will be able to fix everything. We have a list of known bugs and it is now just a matter of sitting down and fixing them. Another issue we are facing is the communication between the robot and the cloud. This is relatively untested right now but we have plans to test it fully.

  • Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?
Posted on

Status Update: Week 9

Joseph:

This week I worked with Karen on obstacle detection and avoidance. We were able to get it to work. Then we were able to make a basic avoidance and we were able to avoid obstacles of size one 1ft by 1ft. The bot was able to successfully avoid the obstacle and continue down it’s original path. We also worked on creating a box for all the stuff to be secured in on the bot.

Next week the goal is to create a smarter avoidance system and account for the edge cases (what happens then the bot needs to avoid but its at the edge of the room, etc) and also further finetune the movement. I also would like to be able to integrate the ui with Manini’s model by then as well.

Karen:

This week I helped Joseph with obstacle detection and avoidance. We were able to borrow bread boards and create the small circuit that was needed to attach the sensors to the pi. From there we moved on to basic detection and fine tuning the best stopping distance. After this, we worked on avoidance which was more difficult because this entailed not only avoiding the object completely but also continuing on the original path. This also brought up many edge cases such as when the avoidance caused the bot to go past the point of interest. Overall we were able to get basic detection and avoidance completed, but we are currently trying to make sure that we have taken care of all cases so that there are no surprises. After this, we made a box to put the peripherals – the pi, circuit, camera and sensors- in, on top of the Roomba. This was made to give the camera a higher position as well as to secure the objects so that they did not slide around when the bot moved.

Next week, I would like to start working on integration. We have the different subsystems working, but we need to make sure the entire pipeline is put together. This includes connecting the pictures taken by the bot to the S3 bucket so that the model can run and produce results. After integration is taken care of I want to go back to fine tuning the different parts.

Manini:

This week I completed re-training our faster rcnn model. I tested the model’s performance and the model identifies only humans in images. After a few iterations of testing, I noticed that many times the model would group together multiple humans into a large bounding box. The individual humans were being identified, but the group was being identified too. One solution that would possibly alleviate this problem was removing the largest anchor box dim. However, this assumption I realized would not work well with our use case. Since our robot takes images from a close angle, the larger anchor box would be needed to identify humans in close proximity. The second solution would be to increase the classification threshold to .75 (originally .5). This solution worked well because it removed the large group classification and also removed erroneous classifications of chair legs and other objects as humans.

This week I also wrote a script to pull images from s3 buckets, connect to an ec2 instance, and dump the resultant images into different s3 buckets. I used boto3 to complete this script. This script completes the pipeline between the hardware and software. This weekend I will be testing the pipeline script to make sure it actually works and finetuning the model if necessary.

TEAM STATUS:

  • What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

At this point the most important risk is localization in terms of the edges of the grid. As of right now we have the movement in terms of different points down, however when the bot moves to the edges we are having a hard time keeping track of bounds. In order to take care of this, we will have to expand our localization scheme to keep track of edge cases.

  • Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We have had to put some requirements on which points of interest can be picked on the UI. This is because of the size of the Roomba- it is larger than we thought, so when it avoids obstacles, we have to give it a larger berth than originally thought. This then requires that the points of interest are farther apart from each other so that we do not miss them. This is a minor change- so there are no other changes that need to be made.

Posted on

Status Update: Week 8

Joseph:

This week I worked with Karen to fine tune the robots movements as well as implement the picture taking. We are able to take multiple pictures at each interest point although we are finding that they do not give full coverage of the room that we are in. I was also able to implement software that is able to take the pictures taken and display them on the UI.

Next week I plan on implementing the sensors as well as work with Karen to finish obstacle avoidance. I will also be working with Manu when the model is ready to implement that with the UI

Karen:

As explained above, this week I focused on the image taking portion of the bot. We are able to connect the camera and incorporate the commands need so that a still picture could be taken. Originally, we had specified that 3 images would be taken- however it seems that this does not provide full coverage. So I decreased the turn angle and am now taking six images of the room. The taken images are saved to the raspberry pi Desktop, allowing the remote UI to access the files. The next step regarding the images is to mount a pole of some sorts and place the camera module on this, so that the images taken are elevated to account for varying human heights.

Next week, I would like to aid Joseph in incorporating the sensors on the bot and testing the obstacle detection script. Ideally we would like to get obstacle detection completed by next week so that the following week we can fine tune obstacle avoidance.

Manini

This week I was able to download all of the required COCO data (train, validation) to the ec2 instance. I was also able to modify the script to only identify  humans and background classes (binary classification). I also was able to train the model for one epoch. This weekend I will complete the training for all epochs and evaluate the performance of the model and retrain the model with the bias.

One issue I ran into this week was with the memory capacity of my ec2 instance and the compute time required to retrain the entire model. In order to train the model with 8 GPU’s and have enough memory per GPU for the given model and data size I need to upgrade to a P3.xlarge. However the cost for this machine is very expensive. I am currently doing some calculations to identify a way to serve our needs while staying within budget.

TEAM STATUS:

  • What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk currently is ensuring that the trained model only identifies human objects and that we make good headway on obstacle avoidance. We are working extra hours outside of class and with regards to the model training, we have spoken to Nikhil who helped us out a lot with this. Another risk is that the multiple pictures we take do not give us full coverage. This can be fixed by just taking more pictures, or at least enough such that no human is missed.

  • Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There has been no changes since last update.

Posted on

Status Update: Week 7

Joseph:

This week I worked with Karen to fine tune the robots movements. We were able to get a very accurate turning mechanism down but we are now finding that there may be some issue with the distance traveled mechanism. While the robot travels the correct distance, the turns it makes to take the pictures causes it to move backward an inch. We will work to fix this next week. This week I was also able to get file sharing working. I am now able to remotely access files on pi. Next week I will help Karen further fine tune movements and work on the ultrasonic sensors.

Karen:

This Week I finalized the bot’s movements. With the new space that we were given we were able to conduct more precise unit testing. Using normal masking tape we measured out a grid to test for accuracy. The final result is that our bot comes within half an inch of the final destination. For now I think that we will stop here for fine tuning and move on to integrating the sensors. We have also attached the camera and found example code in which we plan on to use as guidance

For next week I plan to incorporate the camera so that when the bot  goes  to the three points of interest it will take 3 still pictures for full coverage. I also want to build the circuit for the sensors so we can start testing code for obstacle avoidance.

Manini

This week we prepared for our demo on Wednesday afternoon. For our demo we presented the roomba’s basic movement to three points entered through the UI and the baseline model working on AWS GPU resources on test images. This week I was also able to speak with Nikhil who helped me identify a way to create a COCO dataset with just the human class so that this newly created data could be used to train our own faster rcnn model.

This weekend and next week I hope to fine tune the model and have our own version up and running by the end of the upcoming week.  We also will need to test the performance of the model and see how it works with real world images. Finally, we need to start planning and writing the script to connect the images taken from the camera module to AWS since the model’s inference will be completed on AWS.

TEAM STATUS:

  • What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk currently is ensuring that the trained model only identifies human objects and that we make good headway on obstacle avoidance. We are working extra hours outside of class and with regards to the model training, we have spoken to Nikhil who helped us out a lot with this.

  • Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

C9