Posted on

Chris’ Status Report for 04/20/2024

What I did last week

Worked on kinematics and set up testing environment.

What I did this week

My work this week was centered around finishing the kinematics module and installing Linux on an old laptop I have.

Due to some roadblocks, we have had to switch which FPGA we are using in our system.  Originally we were using the AMD Kria which has the ability to run ROS and has reasonably powerful embedded scalar cores.  We have transitioned to using the AMD Ultra96v2 which does not have as powerful scalar cores and does not have the same ROS functionality.  Therefore in order to make the transition we need additional scalar cores and have chosen to use a laptop for this purpose.

The laptop we are using is an x86 Mac which we needed to install Linux on in order to run ROS.  We had previously attempted to use a KVM but the hypervisor calls add too much overhead when we are handling perception data.  Installing Linux on Macs can be non-trivial as Apple includes a T2 security chip which will prevent naive attempts at installing another OS.  Luckily there exists open source Linux distros that have been tweaked to circumvent this.  After some debug related to the WIFI and Bluetooth drivers, I was able to install Linux and the packages we need for our system onto the laptop.  This laptop will be the center of our system integration in the following week.

This week I also continued work on kinematics.  This involved a partial rewrite of the forward kinematics module in order to use a more robust version of rotation and translation matrices.  The inverse kinematics problem is solved analytically by computing angles for 2 link systems and then computing the difference in angles to find the angle of the third link.  This module will be calibrated beginning tomorrow.

What I plan to do next week

System integration, calibration, and debug.

Learning Reflection

Before this project I knew some basic principles of robotics but not to the level of depth that I do now.  In order to learn about this I have read many academic papers on robotics in general as well as in more specific areas like motion planning and kinematics.  In order to implement RRT and A* I was able to use my learning from the research papers as well as pseudocode that is publicly available on Wikipedia.

I have also found Wikipedia to be an extremely helpful tool for reviewing the mathematical and geometric equations that are used in kinematics.  Luckily I am currently taking a course on Discrete Differential Geometry so my general knowledge is up to date.  Textbooks and literature from this course have proved to be invaluable general knowledge and significantly reduced the learning curve.

Kinematics is the area in which I have learned the most.  This was necessary as I have implemented a custom forward kinematics solver, analytic inverse kinematics solver, and a graphical simulation environment.  In order to do so I watched a series of lectures at MIT as well as reviewed the available course notes.  I also found the lecture notes from a robotics class at CMU and this robotics textbook helpful.

Finally I was able to review the lecture notes and other resources from 18-643 in order to get up to speed on FPGAs.

Posted on

Team’s Status Report for 04/20/2024

This past week was spent continuing the integration effort, this time with the Ultra96 instead of the Kria. After discussing with Professor Kim, we decided to pivot to the Ultra96 after it became clear through discussions with AMD that setup for the Kria would require many more steps. Thus, we decided to stick with what we know better given the short timeline that we have left.

Pivoting to the Ultra96 means we will no longer be doing perception nor kinematics on the same device as motion planning/RRT. Perception will be done on a laptop, and perception data will be sent to the Ultra96 over UART. Only RRT will be done on the Ultra96, and the tree data will be sent back to the laptop, will develop a motion plan using A* (our prior troubles with A* will not be an issue here since it will be done in software). The motion plan will then be passed to our kinematics module.

Posted on

Matt’s Status Report for 04/20/2024

The week prior to last week, I been working with an AMD representative, Andrew, and an ECE PhD student at CMU, Shashank, to get the Kria board working. He had sent me three tutorials to work through, all of which I did, and I ran into errors when running two of them. After discussing with Shashank and sharing the logs with him, we determined a few things wrong with how I was using Vitis. First, the scripts that I was using to run Vitis 2022.2 had some minor bugs that Shashank fixed. He also pointed out that I cannot work on Vitis projects on AFS, and so I moved my work to the local scratch directory of the ECE machine that I was working on. After this I was able to run Vitis and all three of the tutorials without failures.

At this stage, Andrew sent me a tutorial on how to specify and build a platform for the Kria. However, after discussion with Professor Kim about the pacing of our project, we decided to fall back to the Ultra96, which had more knowns and a smoother development path than the Kria, which still had a few unknowns, the main one being exactly what modules provided by the Kria we wanted to use. The tutorial that Andrew had sent was required to create a platform file that would specify to Vitis what resources were available to the accelerator that we were building. Doing this would require Vivado, and while I was able to follow the tutorial, I was not confident in my ability to adapt the tutorial and develop my own hardware platform that would suit our project. I did not originally expect having to do this step when planning to use the Kria—I had taken a lot of setup steps for granted, all of which were done by Shashank and the 18-643 staff for the course labs. Thus, that week we decided to move away from the Kria, which sadly tosses out a lot of work that my partners did to set up the Kria as an end-to-end robotics system.

This past week I easily got a working hardware version of RRT built. Due to the complications with A* search that we experienced right before interim demo, we have separated RRT and A* so that RRT alone will be done on the FPGA. Adapting the C version of RRT into a synthesizable hardware version that could be understood by the HLS compiler was difficult—I was running into this cryptic and completely undocumented error “naming conflict of rtl elements” from Vitis. Even after thoroughly searching through Google I could not find anything, so I resorted to reshaping my HLS code in different ways. Namely, I refactored our RRT code so that it essentially lives entirely in one function (this is better for HLS anyways), and I forced many automatic optimizations off (namely loop pipelining). Eventually I got a working version that would compile and give the correct results. What’s left for me to work on is now figuring out which optimizations I can turn back on so that our accelerator can be as performant as possible.

Lastly, on top of working on the RRT kernel itself, I also worked on defining how the FPGA board/SoC would communicate with the laptop (which we are using in place of the Kria board for perception+kinematics). After trying some libraries out, I settled on this very simple UART library (literally) which seemed to suit our needs. With it we are able to send bytes over UART and read/write them into/from C buffers. More importantly, it is very easy to use, consisting of only a .c and .h file pair. This is important because it means I can simply drop it into my Vitis project and compile the host code with the UART library.

Learning Reflection

During this project, I experienced for the first time what it was like to learn how to use a new tool and device (Vitis and the Kria board) by walking through online tutorials as well as through guidance from an expert (Andrew). I had prior experience with Vitis and the Ultra96 through a well-written step-by-step document given by the 643 course staff, but the online tutorials are not written with the same clarity and thoroughness. Thus, I found it useful to ask Andrew many questions, which he was more than happy to answer.

Posted on

Yufei’s Status Report for 04/20/2024

What I did the week before

ROS integration and perception mapping calibration.

What I did this week

Since we ditched the Kria board, the setup now includes a laptop that needs to talk to the Ultra96 board. So having the laptop talking to Ultra96 in addition to all ROS communications became our top priority last week.

I wrote a RRTComms ROS node that sends 3d grid perception data to peripheral via UART for this purpose. It is able to subscribe to the perception node’s output channels, convert perception data to integer arrays, and send that out via UART.

As for perception — I 3D printed a 1cmx1cmx1cm calibration block for calibrating camera mapping. I have also finished coding the whole perception front end. The only thing left to do is to tune the parameters of the perception module once we have a fully set-up test scene.

What I plan to be doing next week

Since perception and RRT comms are both ROS nodes and they are already working and talking to each other, I don’t expect much to do to integrate them with the whole system. However, one uncertainty is getting back the RRT output from Ultra96 and sending that to the kinematics module. Since the kinematics module is not yet packaged as a ROS node, I have yet to code the interface for kinematics yet. So this will be the top priority for next week.

One other thing I want to do is to improve the RRT algorithm’s searching for the nearest neighbor.

Extra reflections

As you’ve designed, implemented, and debugged your project, what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?

Well, I’ve learned quite a few things.

ROS, for one, since I have never used it before and it is necessary for our robotic project. It was much harder than I initially anticipated to learn to use others’ ancient ROS code, port it to ROS 2, and write my own ROS code that works with it. This involved many hours of reading ROS/ROS 2 documentation and searching on GitHub/Stackoverflow/ROS Dev Google Groups.

There are also some other things that I learned and I appreciate. For example, I learned about octomap, a robust and commonly used mapping library. I also learned more about FPGAs and HLS by reading 18-643 slides and setting up a Xilinx Vitis development environment on my local machine and playing with it.

During the journey, I found that the most helpful learning strategy is actually to not get deep into the rabbit holes I encounter. Rather, getting a high-level idea of what a thing is and starting to build a prototype to see if it works as soon as possible proved to be a good idea.

Posted on

Chris’ Status Report for 04/06/2024

What I did last week

Wrote script to control and communicate with robotic arm

Worked on kinematics

What I did this week

This week I implemented forward kinematics for the arm as well as implemented a simulator that allows us to visualize the arm in the 3D plane.  This involves calculating rotation matrices and a lot of hand drawings and geometry to ensure correctness.

I went to Home Depot and bought two 1/4” thick MDF boards.  There boards are about 2′ x 4′ and we were able to use some wood clamps I had to hold them together, creating a 4′ x 4′ state space.  We mounted the robotic arm in the middle and are done with our baseline setup.

Using this setup, I was able to correlate the arm in the simulator with the arm actual arm.  This allowed me to verify that axis, angles, and measurements int he simulator align with what we have in reality.

Next steps include transitioning from forward kinematics to inverse kinematics.  I believe this transition should be relatively smooth because I have already built the simulator and ensured its correctness.  I have already done a lot of the math for the inverse kinematics on paper already as well.

What I plan to do next week

Complete inverse kinematics.

Matt has taken over SMA* but run into some bottlenecks.  When I am done with kinematics I will transition to code optimization.

Testing and Validation

The majority of the work I have done so far has been with regards to the software implementation of our algorithms, the robotic arm control, and the kinematics.

The RRT and A* algorithms are implemented in C but I have created python bindings for them that make it significantly easier to run and test.  Because of this we have been able to run these algorithms on different state spaces and check their ability to converge and to find a correct path.  Now that we have have created our actual setup we have decided on a state space that is 128 * 128 * 128 voxels (~2 million total).  Further testing has showed we must optimize the software implementation more if we want to converge on paths in a reasonable time.  The number of iterations needed for RRT to converge increases as we increase the size of the state space.  With a 64 * 64 * 64 voxel state space we achieved paths in 10,000 iterations but in the 128 * 128 * 128 state space it requires 100,000 (note that each iteration takes two steps and that not every iteration is successful in adding a node to the RRT graph).  This first example completes in roughly 4 minutes while the second example takes about 20 minutes.  We must increase the speed of a single iteration significantly.

Testing with regards to the arm controller and the kinematics has been done via a graphical simulator I developed.  This simulator allows us to compare the supposed angles of the arm in the state space with the positions they are in in actuality.  This testing was completed before the interm demo and was done to ensure the correctness of our implementation.  Things appear to be working for now and there are minimal concerns with regards to the speed of these operations.

Posted on

Team’s Status Report for 04/06/2024

Status Report

This week was spent on getting all the sub-components of the project to work and setting up the physical components of the project.

We’ve assembled a test scene setup with the robot arm mounted in the middle of it. We’ve gotten software RRT* to work on a 1024 x 1024 x 1024 scene (albeit very slowly). We’ve gotten the perception system to map the test scene into a 3d voxel grid.

We are still improving the sub-components this week and next week. Then we will move on to overall integration (assuming we are using the Kria board, everything should be packed as ROS 2 node so there isn’t much physical integration required) and optimizing RRT* on FPGA.

Testing and Validation

We have already set up a scene that will be used for all full system testing. We have the dimensions of the scene measured and fixed and plan to have two sub-grids, one for item pick-up, and the other for item drop-off. There will be an obstacle in the middle of the sub-grids. Accuracy, correctness, speedup, and power efficiency will be measured using the same methods detailed in our design-review report using this scene.

Posted on

Yufei’s Status Report for 04/06/2024

What I did the week before

My last week was spent on getting started on developing the mapping software.

What I did this week

I finished implementing the mapping from 3d point cloud to 3d voxels. Here’s an example:

The mapping software is able to map a 2m x 2m x 2m scene to a voxel grid precisely (+/- 1cm).

The mapping module is named kinect2_map and will be made publicly available as an open-source ROS 2 node.

I then looked into calibrating the camera. So far everything is manual, especially the part that transforms the mapped grid into the RRT* world coordinates. So we need to come up with a solution that can do that semi-automatically (since we aim to support various scenes, not just a static one, it will require a human operator to guide the mapping module).

What I plan to be doing next week

I’ve already started implementing calibration. I expect to get a prototype of it done within the next week. Then the perception module is complete and I could help with porting software RRT to FPGA and optimizations.

Testing and Verification

I have been testing the perception module when developing it and plan to keep on doing it as I add more features to it. The verification is done by measuring the dimensions of the real-world objects and comparing that to the voxelized mapping. Currently, each voxel has a dimension of 1cm x 1cm x 1cm so it is relatively easy to manually verify by using a tape measure and comparing the measured dimensions of the real-world object and the mapped object. The same applies to verification of mapping calibration.

Posted on

Matt’s Status Report for 04/06/2024

This week I wrapped up implementing RRT for the FPGA. Most of this was simply porting the C code we wrote and adapting it slightly for HLS, but we had big changes in our code wherever dynamic memory allocation was done (since memory can’t be dynamically allocated in hardware). The main change had to do with A*, since A* uses a queue to keep track of a frontier of nodes to search next. For the FPGA we swapped out A* for SMA* (Simple Memory Bounded A*), which requires a fixed-size queue. To do this, I implemented a circular, bounded generic queue.

However I was not able to get this implementation done before interim demo, as the documentation for the algorithm is poor. In fact, I am not sure if it is a good idea to pursue this algorithm due to how small its presence is on the internet (the algorithm comes from one paper; the Wikipedia page has the paper as its only reference).

Regarding the Kria board, I also met with Andrew this past week to debug why I am not able to build with Vitis for the Kria. He gave me some tutorials to try, and I connected him with ECE IT to debug the issue further.

Verification & Validation

During our interim demo, running our software RRT* on a real-sized test case (roughly the size of a cube with sides 4 feet long) took 20 minutes. Our current RRT* is heavily unoptimized, and we expect it to run much faster after some changes.

We don’t want to over-optimize because a) the focus of our project is the FPGA version and b) a slower SW version means more speedup from the FPGA version. We will have to verify the correctness of our paths—i.e. that the path generated on the FPGA is close enough to the one generated by our SW version. This will be done by calculating deltas between both paths and making sure they are marginal. For validation, we will be measuring time elapsed for the full robotic arm pipeline, as well as the time elapsed for running just RRT on each system. We can then do comparisons to calculate speedup.

Posted on

Yufei’s Status Report for 03/30/2024

What I did the week before

My last week was spent on getting kinect2_bridge to work with the camera.

What I did this week

This week was spent on developing the mapping software for mapping from 3d point cloud to 3d voxel grid.

I started with installing octomap and developing the mapping software on the Kria board. However, after spending some time doing this, I realized that this is not a good idea due to Kria’s limited processing capability — it’s scalar core is rather slow, which becomes a limiting factor in development, and due to Kria’s graphics driver issues (that I have noted from last week’s report) I could not use octovis to easily visualize the mapping and debug it visually.

So then I moved to my own Linux workstation for software development. Since I have already spent some time learning octomap, everything went smooth initially. Then, as I attempted to integrate calibration for scene mapping, I realized that it is much harder than we expected. We have talked about this in our Wednesday meeting and I have taken the advise from our faculty mentor and am now in the processing of using grids to calibrate mapping.

The mapping software isn’t done yet, but I plan to get the software running stand-alone (i.e. on my workstation at least, or on Kria if I get the chance) before our interim demo.

There are also some issues with packing the software as a ROS 2 node and getting it to receive point cloud data via ROS pipes. I’ll look into it as well.

What I plan to be doing next week

As mentioned above, try to finish the initial developmeng of the mapping software and get it to work on my workstation. Then, figure out how to package it as a ROS 2 node and start integrate it with RRT* (this will probably take another week or so).

Posted on

Team’s Status Report for 03/30/2024

This week was spent working on individual modules of our system.  Time was spent debugging the Kria setup and porting our RRT implementation to the Ultra96.  Progress has been made with regards to the perception system and we should be able to begin working with real perception data in the next week.  The inverse kinematics module is being calibrated and integrated with the arm controller and the perception simulator.

This coming week we have the interm demo.  This means we will need to set up our test area.  We plan on doing so this weekend and on Monday.  We should be able to demonstrate significant portions of our modules although we might not be able to finish integration on time.