For our interim demo last Wednesday, I worked on the UI a bit more. It was not exactly where it needed to be, but I will work on that and integrating all of the pieces of our project in the next few days. Besides that, since we just had Carnival, I have made no other progress.
Team Status Report for 4/10
Raymond has been progressing on his project components and has begun part of the integration step. However, he still needs more data to form a more accurate network. Therefore, the Gantt chart is labelled as 90% complete.
Joseph needs to change the type of inverse kinematics he used due to issues with the Matlab library outlined in the detailed design writeup. He will now be using another inverse kinematics library as one of the risk mitigation options.
Jasper is behind in his progress in the software UI component, as he did not properly test out the success of his user interface interaction with camera.
As of the demo, the most recent updated gantt schedule is included in this post. Catch up may be needed, and some slack time will have to be used up to get the system working.
Raymond’s status report for 4/9
In the days before the demo, I modified the pixel to inch parameter to fit the size of the meat within the range of error. This value is hardcoded instead of dynamically determined because the meat is anticipated to be a fixed distance from the camera in the final product, so having fixed values instead of a reference point is better.
Before the demo, I was able to download the necessary libraries and files onto the Jetson, however I lacked enough adaptors to get the keyboard and the camera both working at the same time, so I was unable to get that working before the demo. I was able to get the necessary adaptors to get the Computer Vision working on the Jetson during Carnvial.
Joseph Jang’s Status Report for 4/2
This week, I was able to hardcode some movements for the robotic arm. I am using the I have taken the measurements of the robotic arm, so I have put it in the Matlab code. First I had to specify the Rigid Body Tree Model. I could modify the model by using the addBody. Finally, I used the inverseKinematics function to try to have the robot arm moving. I’ve tried two different IK models, which are called BFGSGradientProjection and LevenbergMarquardt. However, since there are multiple solutions that are possible for an (x, y, z) position, the angles of each joint of the robotic arm need to be specified somehow. We have to choose the right arm joints by making sure the robot arm links are not close to the grill. Although the picture below is not a picture of the robotic arm I’m using, it is the GUI that I am using to control the robotic arm.
Jasper Lessiohadi’s Status Report 4/2
As I wrote in the team status report, this week I finished work with the UI and cooking time algorithms. The UI is hopefully clear enough to provide a clean and intuitive experience for the user. It displays information about the detected thickness of a new piece of meat, how long the system decides it needs to cook, and which section of the grill it will go on. There is also an overhead camera feed of the grill which displays which section is which, enabling the user to be confident that the system of working as intended. I plan to add the capability of changing the remaining cooking time for any section that they select, but I do not have that working yet.
Team Status Report 4/2
This week, the team has been diligently working towards being ready for our project demo on Wednesday 4/6. We seem to be in a very good spot with all of our subsystems in a presentable point.
Joseph can demonstrate the robotic arm we plan to use to flip and transport the meats being cooked. The inverse kinematics of it have been difficult to implement, but at least we will be able to show that it moves with the range of motion that we need for our end goal. We have the wiring and other heat-sensitive parts insulated so that they will be protected from the high temperatures of the stove. This has been tested using a soldering iron which did not have any effect on the more delicate internals of the arm.
Raymond has blob detection working with the CV able to detect meats with decent accuracy. He has switched from image classification to object recognition to prevent scenarios where a user places 2 different types of meat on a plate. He has changed parameters of the original blob detection algorithm to be better at identifying meats instead of other objects in the scene, along with improving performance in low light conditions.
Jasper has finished work with the UI and cooking time algorithms. The UI is hopefully clear enough to provide a clean and intuitive experience for the user. It displays information about the detected thickness of a new piece of meat, how long the system decides it needs to cook, and which section of the grill it will go on. There is also an overhead camera feed of the grill which displays which section is which, enabling the user to be confident that the system of working as intended. We plan to add the capability of changing the remaining cooking time for any section that they select, but we do not have that working yet.
Overall, the project is keeping up with the proposed schedule from earlier in the semester, and the team has made great progress.
Raymond Ngo’s status report 4/2
In the previous week, I combined the 3 different computer vision algorithms into one program that invokes all of them. Along the way, I found out the blob detection was not as effective as I thought under dim lighting conditions. As a result, I had to make modifications to the blob detection by changing several parameters regarding eroding excess lines.
You may have noticed the image is a result of an object recognition and not of an image cassification network. There are several reasons for this. One, the dataset we collected had multiple types of meat strewn together, and at that moment, our team realized that if someone placed multiple types of meat on a plate in front of the robotic arm, a classification system is not robust enough to know that and may cause undercookng of meat. Another reason is that if the blob detection fails to work in a way we want, object recognition algorithm is our backup mitigation technique. While object recognition is slower than blob detection, it’s still sufficiently fast enough for our desired metrics.
What you see below is the result of YOLOv5 trained on 150 epochs on a tiny data set (only 20 images) augmented to be 60 images in total. The batch size each epoch was 12 images each. YOLOv5 was selected due to its speed advantage and its active community support online.
Currently on track to complete by the completion date indicated on the schedule, which is Monday, at least I am mostly done. The hesitation is because the integration period will provide a chance to add to the dataset, and that would require more training on the network.
By next week, I hope to begin the integration of the subsystems by having the files uploaded onto the Xavier. Hopefully that would lead to an improvement in detection time also.
