Author: japedraz

These past two weeks, I finished pose estimation that goes with the CV. I also wrote logic to lock onto our target after they gesture that they want the ball, as well as added logic that determines when and where to turn. One issue I ran into is that adding pose estimation and running it multiple times for each person detection within a frame caused a lot of latency. I reduced this latency by reducing how many people we run pose estimation on, as well as reducing the number of frames we run pose estimation as well. Because our CV works and is functional, I’m exploring redoing my method of linking pose estimation to the depthAI detections. Instead of running pose estimation on each detection, I am thinking of running pose estimation once through the entire RGB map and linking it through pixel location to our RGBD map. However, since I am done with my portion of the project, Tomorrow I plan on cycling between working on the new CV, doing further stress testing of my CV to see if I need to implement pixel mapping, and if I do not, helping with other parts of the project, specifically the PID control for rotating and the launcher.

As for progress, I believe that I am nearly done; however, we have one day left and are most definitely behind on launching and rotational control, even if the CV is working. We are confident in our ability to finish our turning, but the setbacks we experienced with our motor control board early this week have hampered our ability to be able to control our physical launching of the ball. This aspect of the progress won’t end up the way we want it, and we didn’t have enough time to do enough integration testing between our CV and rotational control to ensure that it will be the way we had envisioned by demo.

At the beginning of the week, I spent time testing the CV to make sure that the depth values were accurate. Miles and I then put the CV on the pi.  After this, I started focusing on building the robot. I helped drill and assemble the base onto our robot, ensuring our launcher can now be mounted on its base.

We are still behind schedule, but we made good progress and are catching up to where we need to be.  We still have a lot to do, especially in terms of interfacing, but with our Motor Control board and our robot structure nearly fully built besides the ball feeder, we should be able to get a lot done over the next week.

Throughout this project, I was tasked with learning how to work with CV.  As a result, I ended up having to learn about multiple CV algorithms and learned how to use OpenCV and DepthAI.  Before I pivoted to helping with the physical build, I was also learning how to train custom neural networks for depthAI when I was attempting to work on pose estimation. Since I have never worked with a CV of any kind or worked with an AI library, there were a lot of issues I ran into, especially with CMake, as the majority of the semester, I was writing my code in C.

At the beginning of the week, a lot of my attention was dedicated to the Interim demos on Monday and Wednesday. We ran into a few issues, including our motor board blowing up on us the weekend before the demo. Because of this, our group had to pour our attention into trying to get something presentable to demonstrate how the launch would work. Following the demo, I continued trying to obtain another AI model that focused on hand positioning, and I also worked on installing my current model onto the Raspberry Pi. I also contributed to deciding which robot we should use to rotate our base.

Originally, our group was extremely behind because building the base on time, as well as integration, seemed like it would not be achievable. The shift to using a robot kit as our base to turn makes our project much more likely to succeed, although we still have a lot of work to do. My progress through my portion of the project started a little behind this week, but I have caught up to where I am meant to be and can now devote my attention to other portions of the project, as well as finish all my basic CV unit testing.

Next week, I plan on during the first half, doing all the unit testing and putting my CV code onto the Pi. During the later parts of the week, depending on whether the robot comes in on time, help build the robot and help with our turning algorithm, as my CV code will be interacting with it, ensuring that the data I send arrives promptly so that the robot turns accurately.

I have already started the CV unit testing, but I will continue to work on it throughout this week and the weekend while also helping to complete other portions of the project. Since we do not have the rotational capabilities yet, my testing will involve manually moving around the camera to face people and seeing if when my x value reaches 0, that when I draw a line with a tape measure from the center of the camera to the person that the line is within a foot of where the person is standing. I will also test to ensure that the depth value that the camera is within a foot and a half of the distance of the person it is facing. These two tests will be one under different levels of lighting. At first, I will be performing these tests using my laptop, and afterwards, I will be doing them on the Pi. My goal is to measure from a certain spot and premark the spots to ensure that the code works the same on both, to ensure that integration to the Pi was successful as well. My goal is to eliminate as many integration problems as we may have early and thoroughly test the CV before we assemble the base.

This week, I’ve been mostly working on interfacing with the camera and attempting to get depthAI to work. This took much longer than expected because of various integration issues I kept getting. Eventually, I had to abandon my old code and switch to Python, as I had dependency issues that I spent three days and over 15 hours trying to fix. Then I started spending my time attempting to get a Python version of my code up and running, and I am currently in the debug phase. I plan to have a fully functioning CV code by tomorrow for the demo on Monday.

For next week, we will mostly be focusing on the demo, but I plan on cleaning up the CV, potentially revisiting the possibility of doing it in C, while also helping get our launchers’ rotational base working.

All the parts came in this week, and as such, we started orienting the build. We ran into a large issue with our 3D printing plan. We realized that we cannot 3D print our housing unit in one print due to the weight limit of 175g.  This caused us to have to make a few design changes. Specifically, we decided to only 3D print precise portions of our design, such as the gears, and replace the base as well as the support beam in the center of the base with PVC. This will also allow us to make a higher-density print for the gears, ensuring they are more sturdy.

We have fallen behind because we had to wait for parts to come in for our project. Our schedule has been condensed a bit as we now have to put certain software parts of our project, such as the coding and interfacing, ahead in our schedule to free up time later on. As a result, to get back on schedule, I plan to get the camera working with the CV algorithm and hopefully also allow it to send coordinate values to our Pi.

This week, I worked on the ethics assignment and put in all of our orders, and we are starting to build our launcher. We are also starting to interface our Pi with our Camera and attempting to decide on how we want to approach our sensing. We decided that the CV will be done on the Oak-D Pro Camera’s Processor, and that it is going to send out coordinates to our Pi. The Pi will then have code that will determine launch speed and base rotation based on the coordinates that it receives from our Camera. We decided this as the latency involved with sending a full image or depth map to the processor can be bypassed by simply doing our CV on the Oak-D Pro.

We fell a little bit behind schedule as we wait for our orders to come in, but if we can finish the launcher build by next week, we should be back on schedule.

This week, I worked on the Design Report and got started on the CV and depth AI code.  For our design report, I did the introduction, use case requirements, and the Test, Verification, and Validation section. I also worked on parts of the design trade studies centered around our use of CV and the OakD pro camera.  This took a long time; however, over break, I was able to get started on the code for our CV.

Since I didn’t have the Camera with me, I could not actually test any code that I wrote; however, I was able to figure out how DepthAI worked, specifically that DepthAI is run by the camera and simply will output the information it finds to our microcontroller. I found it is possible to get coordinates from depth AI running their general-purpose (mobilenet-ssd) model and using it for person detection, and sending coordinates and depth information. Because of this, I made a first attempt and wrote out our depthAI code that finds coordinates, horizontal displacement, depth, and height, which is to be run on the OakDPro Camera; however, I also wrote simple, incomplete CV code to use on our Pi just in case.  Both of these are simply a start and are unfinished and untested.

Our main goal last week was to complete all the deliverables needed for the report, which we were able to do. The goal for this upcoming week will be to get all of our parts ordered and to start our 3d print so that we can begin assembling and testing our ability to launch as soon as possible. We are a little behind progress with our physical build; however, I am currently still on schedule with the CV portion of the project

This week, I prepared and gave the design presentation. I also reviewed our design to make some changes before the presentation. Specifically, I planned out the removal of the tilting function after determining that the design we had in place wouldn’t function, exploring other designs, and determining that there was no solution where we either got a series of motors that would push our budget, or compromise the predictability of the flight of the ball post launch. Since tilting was not necessary and would strain our budget, we decided to remove it.  For the design Report, I started finalizing the solution approach and have added and expanded upon the tests and testing metrics.

I have not made much progress with the CV, other than deciding to use a HOG(Histogram of Oriented Gradients) and SVM (Support Vector Machine) in order to identify a person, and using depthAI (the depth perception library that pairs with the OAKD camera) for our depth perception. I have a few decisions to make about my code. The algorithm I plan on using normally suggests a resize for faster processing; however, I am hesitant to do so, as I am unaware of how I’m meant to draw depth information out of the depthAI. My concern is being unable to locate where exactly a person is on a depth map, since the image being processed by my CV would not be the same. Because of this, I had to put a pause on my CV code until I figure out how the OpenCV and DepthAI libraries will interact.

This past week, my main responsibility has been putting together our presentation. However, I have also focused on researching materials, helping determine specifics for our calculations, and making design decisions. Since I am the one in charge of the CV, I researched multiple cameras and decided to get the Oak D Pro Robotics Camera. I am also working on learning how to use OpenCV since that’s what we’re using to identify people with our camera. I have no deliverables yet, as the first goal of mine is to get the launcher built and working before the break. I have not finished any code, although I have started interfacing with the Camera.

I think we have fallen slightly behind schedule, primarily because we had to pivot part of our design at the last minute due to unforeseen physical constraints that we discovered with our current design. For next week, I would like to have some code deliverable as I continue to work on the CV, and potentially also have proof that I can successfully interface with the Oak D camera. I would also like to have our motors ordered and parts of our launcher 3d printed.