Category: Leland’s Status Reports

This week, I focused on making some optimizations to the camera settings to reduce latency. Specifically, I worked on the following:

Improved Camera Latency

• Collaborated with the team to measure latency from user input to system output by switching to MJPEG compression for the stereo camera 
• Reduced latency to a range of 200 – 250 ms which is about a 200 ms improvement from the previous camera configuration

 

Project Status
I am currently a little behind with the project timeline. The team was busy this week because of the semester coming to a close, and we we’re not able to get an early start to our last additions such as the z-axis. However, we met over the weekend and made a plan for success next week. We have allotted sufficient time next week to finish course assignments and add the final touches.

Goals for Next Week

1. Integrate Z-axis into the gantry system and implement depth calculation in OpenCV
2. Conduct final use case tests to include in the final report (water pouring and more object manipulation

This week, I focused on finalizing and testing key system components in preparation for our final presentation. Specifically, I worked on the following:

1. System Testing and Validation
   • Collaborated with the team to conduct tests on the project, gathering critical information on use-case and design requirements for the final presentation
2. Made a New Gantt Chart
   • This outlines the final tasks we have planned and when we plan to do them
     
3. Researching Camera Latency
   • During testing this week, we measured a lot of latency from the camera capturing frames and performing OpenCV calculations. We measured latency through video taping which had some human error. I did some research online to measure CV latency directly in software

https://www.dlology.com/blog/how-to-measure-the-latency-of-a-webcam-with-opencv/

• I also found official OpenCV documentation for measuring performance and possible optimizations I could use to reduce latency

https://docs.opencv.org/4.x/dc/d71/tutorial_py_optimization.html

Project Status
I am currently on track to meet the project timeline. I have high expectations for how our final presentation will go. I’m also confident in the team completing the rest of our tasks, as we planned in the Gantt chart. 

Goals for Next Week

1. Update the current main project script to include Z-axis computations and send Z-axis commands to the Gantry system
2. Create the force lookup table. I will create the table by squeezing objects with the gripper and measuring the distance of compression of the foam on the gripper and the force the gripper is exuding on the object
3. Update the gripper control software such that the gripper will squeeze with variable force instead of binary open or close

New Knowledge and Learning Strategies

The most important knowledge I learned for the part of the project I worked on was the OpenCV library, as this is the crux of the camera system. The hardest part was working with the software that was available for the camera I picked out. The majority of time I spent on the project was with my teammates because system integration was a big challenge. I learned a lot from my teammates, as I had to understand their parts of the project to debug our system implementation.

I primarily searched online to find forums or documentation on problems I ran into or questions I had. However, I would say I learned the most through my teammates.

This week, I made significant progress by working collaboratively with team members to integrate various components of the system. Specifically:

• Gantry System Integration: Successfully integrated the camera software with Jack’s gantry system.
• Remote System Integration: Worked closely with Cary to integrate the remote system, enabling the remote to communicate effectively with the ESP32 for the gripper.
• Gripper Control Software: Completed the gripper control software, allowing the gripper to respond quickly to remote inputs.
• Camera Stand: Jack constructed a camera stand, which has greatly improved the stability and usability of the camera setup for testing and integration.

These contributions have allowed us to achieve a fully integrated system.

Currently, I am on track with the revised project timeline. The integration work completed this week ensures that we are well-positioned to achieve our MVP for the Interim Demo on 11/18.

My primary goals for the coming week are:

1. Address the issues that are presented during the Interim Demo
2. Start trials for the design and user requirement tests
3. Start integration for depth measurement

Verification for the camera system

• Measure the distance traveled in software and compare to the actual distance moved by the user
  • Record the  xy coordinates of the ArUco tag on the remote calculated by OpenCV and compare them with the actual dimensions of the visible workspace

This week, I completed the tasks related to outputting the XY location and roll orientation of ArUco tags, achieving one of my primary goals for this week. I also started cleaning up the camera code in preparation for the upcoming system integration phase, ensuring that it is optimized and ready to be implemented. I’ve been able to learn more about the camera software packages to understand it better such that I can configure it for an easier integration with the rest of the project. Here’s a link to my document detailing the updated camera code.

While I am still slightly behind the initial schedule, my recent progress has brought me closer to our revised timeline. I will be packaging the computer vision code for integration and initiating the software for PC and ESP32 communication and other important communication protocols over the weekend. This will ensure a strong foundation for the integration process that will take place all of next week.

Goals for Next Week

In the coming week (and Sunday) , I aim to:

• Complete the software for PC and ESP32 communication to support reliable data transfer during integration.
• Develop the control software for the gripper, allowing it to respond accurately to inputs from the remote
• After these are completed, the main goal is completing MVP by 11/18

These deliverables are crucial for integration, and I am committed to meeting these objectives to ensure the project is on track for a successful Interim Demo.

This week I devised a new test plan for our project, and I got a demo for ArUco tag detection working. Here is a link to my work document.

According to our previous schedule, I’m about two weeks behind in work. Originally, our team was supposed to have at least two weeks of total system integration for the Interim demo. As of now, we will have one week to integrate. That being said, we have devised a new schedule that will lead us to a successful Interim demo. I have a lot of work to do, but I have made this plan and devoted more time next week to complete the necessary tasks.

My goals for next week are as follows:

• Track Multiple ArUco Tags
• Track ArUco Tag Orientation (roll)
• Create software for PC and esp32 communication

This week, I followed the directions from this GitHub repository to download the necessary eYs3D libraries, but also some nice wrapper functions. After that, I was able to run a video demo of the depth map and the normal frames. The GitHub repository included these demos to test and calibrate the camera. Here is a link to my work document for this week.

Unfortunately, my progress is not on schedule. I wanted to include the ArUco tag calculation this week, but I wasn’t able to get that done. I have developed a schedule of my total workload for the next coming weeks. This will help me keep up with my other classes and free up time to spend towards this project. This is included in my work document. Despite being behind on schedule, I’m proud of my work this week. I had a lot of other hard class assignments to complete this week, and I got those done. I’m convinced that if I keep up this pace, we’ll be back on track by next week.

I have my deliverables for next week in my work document, but I will reiterate them here. By next week, I hope to write my own python script that utilizes the software packages I downloaded. This script needs to be able to identify and locate ArUco tags. I need the relative XYZ position of the tags and their orientation.

This week I started setting up the software environment to use the eYs3D stereo camera. I have created a Linux workspace on Ubuntu to run the image processing. Also, I have found the necessary SDK (Software Development Kit) to digitally capture frames. In combination of these two, I was able to produce some code that will capture frames with the SDK and convert them to a OpenCV matrix for further image processing. Here is a link to that code snippet.

Overall, I’m still on track for further camera calibration and ArUco marker detection for next week. For next week’s deliverables, I hope to be able to produce some visual examples of capturing and processing the camera frames. Also, I’d like to implement a code example that captures position and orientation of ArUco markers.

This week, I changed my camera design by picking a different type of camera. I made this change because the depth perception of a monocular webcam was not good enough. Also, the stereo cam I want is in the 18-500 inventory which is great for budget. Also, I researched some pseudo-code to start crafting what the OpenCV code will look like when I get the camera.

I’d say my progress is just a little behind where I want to be. I’d really like to get the camera next week and set it up with my laptop. Then, I can set it up with a stand and a dummy remote with Aruco tags to start calibration and computation. In the meantime of getting the camera, I can continue to build the pseudo-code for the image processing. Another thought is I will need to start crafting pseudo-code for the UART protocol of sending and receiving data from the two ESP32s we have. I’d like to include that in my report next week too.

My work this week

This week, I have finalized a design plan for the camera station. I made some calculations to estimate what specifications would be necessary for the camera and the camera stand. I also picked out a specific webcam and camera stand that will satisfy my specification requirements.

I am on track, as I have a complete design and almost complete bill of materials. This is the goal that I had in last week’s status report.

Next week, there will be design review presentations where my team will receive design feedback for our project. By the end of next week, I want to use that feedback to finalize all our design plans and start ordering parts.

My work this week

This week, I researched more thoroughly what will be needed for the computer vision part of the project. I have looked at the pros and cons for different types of cameras like a USB webcam vs a stereo camera. I have also started to look through OpenCV libraries to get comfortable with them. Lastly, I modified the website a little to make it look nicer.

We are on schedule. This week and next week are for design and ordering parts.

I hope to have an almost complete design and list of parts for the camera/laptop station part of our project.