Varun’s Status Report for 4/27/2024

This week, I presented the final presentation for the group. Additionally, I created the circuit that should be our final motor control circuit for the DC pumps on the Rover Upper Deck. This took a few tries:

For some reason (which I still cannot figure out), the motor driver on the upper deck would not output the desired nominal battery voltage on the motor output, despite the enable pin being set to high on the Raspberry Pi. To try and mitigate this, I attempted to hold the pin at 5V (necessary because the Raspberry Pi uses 3.3V signaling) and activate the pumps. They were still not receiving the voltage required when connected, which is something for which I still do not know the reason- my current speculation is max power on the board.

Eventually, I decided to implement a logic level shifter circuit to work with the motors. It is a simple power transistor circuit, attached to a logic level shifter. Testing it with raw voltage, it works, but I’m about to test to see if it works with motors (If it does, it’ll be on the final Rover.) A picture (we’re currently out of space, I’ll endeavor to fix that soon):

https://drive.google.com/file/d/1S7TOr7209yDcjg8nRn55qBP8nFoxBLN4/view?usp=drive_link

Progress is on schedule with the TechSpark demo in mind, though I will work on the ML side of things in the following days. I would like to finish the entire robot suite by next week.

 

Nathan’s Status Report for 4/27/24

This week, I personally accomplished two main tasks. First, I edited the depth pipeline to finally be able to utilize the ColorCamera ability of the camera. Specifically, using the same mobilenet framework, I was able to make the two StereoDepth inputs come from the color camera and utilize the color camera for input into the spatial network pipeline as well as for display. The updated code can be seen here https://github.com/njzhu/HomeRover/blob/main/depthai-tutorials-practice/1-hello-world/depth_script_color.py

Additionally, a photo demonstrating this color functionality is shown here: We can see a bounding box around Hayden, demonstrating that the pipeline is able to make detections. A closer glimpse into the code can be seen here: https://drive.google.com/file/d/1MTEEsWslS3K_79CAxSCOV2DoH274E5bl/view?usp=sharing

From this photo, we can see the linking together of the pipeline, starting from the color cameras, going through stereo depth, and ending at the spatial detection output.

The second main task I accomplished was that I was able to put a live camera feed onto a locally-hosted website. To achieve this, I followed and adapted a Flask tutorial to setup the website, html pages, and routing actions. We plan on having the controller Pi access this website to display it on the display. One challenge was similar to a previous issue of multithreading and accessing variables in different files. To overcome this, I combined the code of my depth pipeline with the locking and unlocking required to update the frame to be displayed on the website. A video showing the live camera feed updating on the website can be found here: https://drive.google.com/file/d/1kDW8dthBEhdgHHi2DJDfmURngqmv2ULZ/view?usp=sharing

After this week, I believe my progress is on schedule because a point of worry from last week was the ability to broadcast the video and live camera feed for the user, but a major component of that functionality was figured out today, which is good.

In this coming week, I hope to be able to fully finalize the rover in a culmination of all of our combined effort over the semester. In addition, I hope to, together with the team, finalize all the requisite documents as well.

Team Status Report for 4/27/24

We have officially completed all parts of our system so the only risk right now is if we cannot configure the system effectively to meet our pickup accuracy. We have been working hard to tune our system to detect items correctly and position itself to pickup; however, we are yet to find the sweet spot in our configuration that gets us the 80% threshold we want to achieve. The current contingency plan is to keep tuning it until it can detect the item correctly; we are experimenting with a few different item detection models still.

This week we reverted our design back to having the logic level shifters as well as a PMOS switch so we could reach the 7.4V input to the pumps. Through experimentation we proved that 7.4V was necessary to achieve our pickup weight of 700 grams. We will also be adding some extra weight to the rear of our design to make sure it does not tip over. Nathan is also working on developing a web page we can host with the camera feed; we found that this was the most effective way to be able to relay the camera feed back to the user side.

Here is the camera feed working.

Unit Tests Performed:
User Side Latency –> 10 ms Passed (5 trials timed/slomo camera)
Transmission Latency –> 15 ms Passed (5 trials timed/slomo camera)
Rover Side Latency –> 10 ms Passed (5 trials timed/slomo camera)
Lifting Capabilities –> 500 grams Failed –> 850 Grams Passed with 7.4V
Driving Speed –> 0.24 m/s Passed (5 trials measuring drive time across 1 meter)
Lifting time –> 8.4 seconds (calculated by timing lift across 70 trials)
Detection Accuracy –> 64% ish (calculated by 45 successful trials across 70)
Battery Life –> 1.25 hours of capacity (measured how much time we were able to perform tasks like driving/pickup)
Item Detection Range –> 33 cm (placed objects varying distances and found the maximum bound)

Hayden’s Status Report for 4/27/24

This week my workload was divided into three different categories: the first was finishing the final presentation and making sure our metrics for testing were presentable. I spent most of Sunday working on touching up the presentation and helping Varun prepare for presenting. The rest of the week I spent assembling the new pcb as well as the monitor setup for the user side. I also spent time working with Nathan researching how to pipeline the camera feedback effectively; we have made significant progress and I have no doubt it will be done by Wednesday. The latter part of the week has been spent drafting up our poster design and planning for the deliverables for this next week. A lot of the work we have been doing is to make sure our project is presentable this coming week.

Image of the user side setup.

In order to meet our Tuesday deadline we have to finish our poster design; in order to meet our Wednesday deadline we need to get the camera feed working. I will be spending the next three days working closely with Nathan to make sure we get the camera feed transmitted as well as helping Varun wherever debugging and verification is necessary for the kinematics. No schedule changes need to be made and quite frankly we do not have the opportunity to make any.

Nathan’s Status Report for 4/20/24

These past two weeks, I did a lot of collaborative group work in integrating all the requisite parts required for a full implementation of the rover. For instance, one thing I did was help solder the third pump for our suction arm, as seen here. I also contributed to the group’s effort in the foray of investigating multiprocessing in Python.

In addition, I also helped setup a testing suite for analyzing the coordinates and bounding box dimensions in our object detection pipeline, as well as analyzing what these different variables refer to. The testing suite can be seen below. I took the various detection.y, ymin, ymax, and other similar coordinates and measured spatially with a ruler to determine how the camera is calibrated.

I also investigated potentially abandoning a neural network altogether and using openCV thresholding, contouring, and rectangle-creating functions to use the depth color map to extract an object that is close enough. However, this idea fell short because as we would get closer to an object, the floor would also be lower than our desired object’s z-distance threshold, meaning the heatmap would reflect that the floor is just as close as the object. If I were to try and manually create a bounding box, it would also include the floor; there would be no way to differentiate. In addition, as the video frame displays, I saw some blips on the heatmap that showed closeness that wasn’t part of the object. These small blips could potentially cause issues if I were to threshold the frame since they wouldn’t be greyed out.

My progress is flip-flopping between mainly on schedule and a teeny bit behind, mainly because problems arise when performing integration tests that did not appear when doing unit tests, such as the lower accuracy resulting from the object detection pipeline. Since we budgeted this time, I would consider progress to be on schedule.

The main deliverable I hope to complete in the next week is to tweak and adjust the object detection pipeline so when we perform validation tests, we meet our desired accuracy threshold as described in our use case requirements. Additionally, I hope to be able to work with Hayden to successfully transmit the camera feed back to the monitor.

Throughout the semester, the acquisition of new knowledge was a particularly salient undertaking due to the fact that I had to learn and get used to the depthai module, features, and programming paradigm. To be able to accomplish these tasks and successfully utilize the module, I found it necessary to be able to learn quickly, which I tackled through sifting through the wide variety of examples created by depthai’s Luxonis. I utilized their depthai-examples repository, which had example programs for every feature of the camera, from spatial detection, object tracking, and depth. Being able to see example programs really allowed me to learn and quickly get acclimated to the module, and in addition, when searching the internet to help solve my error messages, I found forum posts and discussions that really helped my debugging process, which was especially helpful because the camera is so new and constantly undergoing changes when compared to different versions and models.

Team Status Report for 4/20/24

For the past two weeks, the main thing we as a team have been working on revolves around integration and connecting together all our separate subsystems.  When connecting our object detection/depth map code that instructs the motors to turn and move a specific direction, we ran into difficulty parallelizing the two processes, either using Python’s multiprocessing and socket modules. The two control loops were blocking each other and preventing us from progressing in either program, but as stated in Varun’s status report, the end program is an overarching one acting as a state machine. After fully assembling the rover and running integration tests that involve the full pipeline of navigation -> detection -> pickup, the most significant risk lies on the detection side. Earlier in the week, we were running into an issue with the coordinates that the object detection pipeline was giving for the bounding box it created in that the y-coordinate it outputted wasn’t necessarily the top of of the object, but the center relative to some part of the camera, causing our suction to undershoot because of inaccurate kinematics. After investigating the various dimensions and y-values that exist in the code and comparing them to our hand measurements, we found that the detection.y value it spit out did reflect the top of the object, but regarding its magnitude, it reflected the distance from the bottom of the camera. To mitigate and manage this risk, as well as improve our kinematics in the process, we plan on finetuning a potential hard offset to all y values to ensure that we hit the top of the object as well as tuning with adding a variable offset based on the bounding box dimensions. We have started doing this, but plan on performing a lot more trials next week. Another risk involves the accuracy of the object detection, which meets below our standards as defined in the design and use case requirements. A potential cause for this issue is that since mobilenetSSD has a small label database, there’s a chance it doesn’t encompass all the objects we want to pickup, but since we don’t necessarily need identification, just detection, a potential risk mitigation strategy is to lower the confidence threshold of the detection pipeline to hopefully improve on detection accuracy.

One change that was made to the design of the system was adding a second layer to our rover to house our electronics due to space constraints. The change does not incur major costs since the material and machinery were both provided by Roboclub. This is a quick, lasting fix, with not further costs needing mitigation going forward.

As stated in our individual status reports, we are fairly on track, with no major changes to the schedule needed.

A full run-through of our system can be found here, courtesy of Varun: https://drive.google.com/file/d/1vRQWD-5tSb0Tbd89OrpUtsAEdGx-FeZF/view?usp=sharing

Varun’s Status Report for 4/20/24

These past two weeks, I brought together the Rover’s components.

A week ago, I used the aforementioned kinematics scheme to make the arm move into the correct position when commanded. Shown here:

 (I commanded the arm to go 20 cm away)

By Sunday night, we had all the electronics made but no code to unify the parts. Initially, the thought was to use parallel processing in Python, or maybe sockets, to bring everything together, but I re-evaluated it and I was able to put everything into one while loop and have the Rover function. I created (essentially) a state machine to describe the way the Rover operated.

Ultimately, the Rover (draft 1):

All of the electronics were “assembled”, and it had functionality. I essentially partitioned the motion of the servos to move from Point A to B in many steps, effectively smoothing out the motion and making the arm motion more gentle. One problem it faced was that my smoothened kinematics code would operate arbitrarily, sometimes slamming the arm into the ground at full speed. I realized that I should be checking the absolute value of the difference between the arm’s current position and its last (rather than just comparing the equivalence), and the arm motion was smooth. With the state machine I wrote in Python, the Rover was able to drive from point A to B, and pick up an object once given the coordinates from the camera. Once I verified functionality, I cut out some new base plates on the iDeATe laser cutters and on Friday, we re-assembled the Rover and neatened the wiring.

Rover (v2):

For the first time in awhile, I think that we’re on schedule. By next week, I want to make sure operation is agile and can be set up quickly, and I want to make sure that the pickup sequence is more flushed out. Right now, as I’ll cover in the final presentation, our accuracy is lower than we’d like, and I want to fix that.

I needed to re-learn python once more to make this project work. I did this by just looking up tutorials and using my pre-existing knowledge of programming in general to bring them together to make what I wanted function. I needed to learn how to create perfboard circuits, something I’ve wanted to learn for awhile now, and I was able, thanks to the motivation provided by this project. I needed to learn how to organize power electronics, and I learned that with the help of the people in Roboclub, whose help I asked quite a lot.

Hayden’s Status Report for 4/20/24

These past two weeks I have focused on improving the user experience. I found a monitor and have connected it to the Raspberry Pi. Nathan and I are still trying to find a way to transmit the camera feed back to the monitor but we could just directly SSH into the rover RPi if it comes down to it. I also spent this week redoing the PCB to make it have better keys and I made it more like a controller. I also designed a user terminal that will hold the battery and the Rpi/monitor and allow the user to adjust the angle. On top of this I have spent countless hours with Varun rebuilding the rover and testing it to make sure it still works every time we redo the electronics. I also have crafted some beautiful slides for our final presentation on Monday.
Here are some photos of the new PCB design, the current setup and the CAD for the new terminal I have designed.

We are very much on track to finish things; I do not need to modify the timeline at all, I just need to focus and get the deliverables done.
In the next week I will keep helping Varun calibrate the rover as I have been doing and figure out how to get the video feedback.

As I’ve been designing my side of the project I have had to improve my physical CAD skills with Fusion 360 to be able to create a fully integrated design. I also found it necessary to learn about new libraries in Python and learn about serial and socket communication methods. I have spent a lot of time reading data sheets and library documentation to be able to get a thorough understanding of what is happening in our design and figuring out how to debug it. The learning strategy that has proved to be the most effective has been reading these documents and then trying to implement the concepts into our design and then use previous knowledge to debug.

Nathan’s Status Report for 4/6/24

This week on the project, I helped integrate the Raspberry Pi program that successfully receives button presses serially from the controller and simultaneously transmits the results serially to the Arduino Nano that controls the turning of the motors. In addition, I integrated the camera onto the rover and did minor calibrations to the camera control loop based on realizations after mounting the camera to the rover.

After demo day, I witnessed some other groups have computer vision follower-based ML systems, and one group was using the Yolo v8 architecture. I heavily considered the idea of using a Yolo framework over a MobileNetSSD framework due to the increased variety of objects it can detect (# of labels) as well as potential higher accuracy due to using BGR frames rather than black and white frames. This meant that I had to figure out the longstanding issue of the ColorCamera never being recognized. I finally figured out the solution to this issue: in various example programs, the default configuration of the camera and the socket that is used, called “RGB”, is not present on the Oak D SR. Thus, to fix it, I had to manually set the socket port to CAM_B or CAM_C, and rgb functionality finally worked. However, I was having trouble setting up a Yolo architecture because it required both an rgb socket in use as well as left and right stereo camera. Originally, they were MonoCamera, but this conflicted because we had three different cameras needed but only two sockets. Thus, I researched how the StereoDepth feature worked further on the Oak D SR camera and found a workaround which was to use rgb ColorCameras for the left and right cameras necessary for depth detection, as well as to “align depth map to the perspective of RGB camera, on which inference is done“.  The tentative code that I adapted can be found here: HomeRover Github

The Github also includes the rover control arduino/python code written for intersystem communication.

This photo shows the openCV frame that captures an rgb image output, something that was so elusive to me for so long, along with the corresponding depth heatmap.

My progress is on schedule despite the revamping of our object detection system. It is aligned with the developments made to the kinematics scheme in that the communication between the camera control loop and the arm control loop needs to be established, which is what I hope to complete this coming week. Since there is no cross-device communication for this, I hope it is easier than serial communication between different computers like a few weeks ago.

The tests I am planning to run are twofold and involves testing my individual subsystem and integration testing. For testing my individual subsystem, I aim to keep a few big concepts in mind that define our use case and design requirements, that of versatility and accuracy. I hope to gather a variety of objects, and with the camera mimicking a position like on the rover (lifted off the ground with a similar background setting), achieve accuracy up to our requirements for both object detection and transmission of accurate signals for directing the rover. Further tests to ensure accuracy will be made in conjunction with incorporating the arm in that the coordinates passing and translation constitutes an important part of our system. In addition, I hope to perform latency tests with my control loop code running on the RPi in using slow motion cameras to identify the object detection reaction time to an object appearing on screen to make sure it falls within our requirements for appearing instantaneous.

Team Status Report for 4/6/24

This week we accomplished a lot; we completely finished the driving subsystem with the controller and we were able to start verification of this system. We had to revamp the driving because the motors were under powered and we also revamped the robotic arm to position the camera in a central location to simplify the task of alignment. These decisions to change the arm and the drive train was to mitigate the risk of time. The largest risk we have right now is that we are unsure if we will have enough time to properly verify and validate our design once we complete it. We also broke one of the servos and luckily we have a spare but frying another one would be a major setback when we have to wait for delivery. We updated the Gantt chart last week and we are feeling much more confident about our timeline; as we have been working some things have become complex and others simpler than we expected.

Verification has occurred for driving and controlling and we will begin verification for the arm this week. As far as full system validation we plan on driving the rover this week and trying to pickup items when we align the rover properly using our own vision. We will not be incorporating the camera just yet but we will be driving blindly to work on fine tuning the object detection. For validation of our proposed latency, as Hayden mentioned, we’ll be using a high frame rate camera to check how long signals take to go from button press to robot movement. So far, the movement has been pretty close to instantaneous, so we haven’t needed to worry about it too much. In terms of drive validation, we have validated our drive- the Rover moves in the fashion that we intended it to, and can connect from quite a distance and remain relatively seamless in operation. As Varun mentioned in his report, we’ll be verifying consistency of the arm subsystem by power cycling the rover and determining with a vertical marker if the Rover got to the correct Y and Z positions. After that, we’ll begin working to bring the separate systems together. Considering the simplification that the new arm brings, we should be pretty solid in terms of progress.

Biggest risk right now is the Servos breaking in testing, causing a large lead time to get new ones. The contingency plans are to switch to a smaller, more common, high-torque servo for the second stage in case this servo breaks. Quite a lot of changes were made to the overall system to better facilitate testing- however, the overall block diagram has not changed this week.

(HomeRover’s overarching redesign!)