Team E1: FPGA Accelerated Fluid Simulation
Carnegie Mellon University Capstone, Spring 2022 – Jeremy Dropkin, Alice Lai, Ziyi Zuo
This week I primarily worked on integration and the poster for our final presentation. I also worked a little bit on debugging the AXI ports with Ziyi. We integrated the different optimizations together and also worked on getting more data for our optimizations.
I am happy with our progress in this project.
This week I worked primarily on optimizing the fluid simulation algorithm on the fabric. This involved iterating upon the algorithm and exploring different ways to restructure the hardware and taking advantage of the HLS pragmas to allow the algorithm to run faster.
This also required figuring out what optimizations would break the implementation, and was a fairly iterative process. I was also working on developing the slides for our final presentation this week.
I think that we are on track in our schedule. Next week I will continue work on optimizing, and also work on our poster and video.
So this week unfortunately I have been sick with COVID, and wasn’t able to make as much progress as I wanted to originally. For the majority of the week I have been trying to get rest and recover quickly, so I’ll be productive next week. Later in the week I was able to put in some work where I worked on the correctness of our algorithm, as we were facing some issues where particles did not behave as we expected them to. The issues turned out to be with how we calculate the bounds of the scene and the voxels that particles are in, but they are mostly solved now.
I feel slightly behind since I did not expect to get sick this week, but I think that we are still in a good spot, and will be able to produce good results from our project. Next week I am ti help with collecting data and continuing to optimize our implementation.
This week we did a lot of work on both some optimizations and synthesizability of our project. There were still many C++ errors from the interactions with the Vitis libraries and the modern C++ codebase that we are using. After initial synthesis attempts, it turned out the amount of floats, there was no way we could meet any sort of reasonable clock period for synthesis, so we began converting the code to use fixed point numbers as outlined in our design. But, the libraries provided by Vitis did not interact well with the already written math libraries, and I spent a while trying to get these libraries to cooperate with each other.
I think that we are on time, and are making steady progress towards the goals we set out to accomplish. We need to ensure that we work consistently in the upcoming weeks, but we have been working well recently. Next week I hope to work more on optimizations with unrolling and pipelining.
This week I was primarily working with Ziyi on getting the code in the C++ build environment to work well with the Vitis HLS build. We spent a while improving the portability of the code as Vitis has a set of specific requirements, and does not work well with very modern C++ code that is generic for many purposes. Vitis HLS has very cryptic error messages, so progress is steady but slow on getting this to work. There is a lot of trial and error involved in figuring out what is causing errors, and it is tricky to figure out how to redesign the code to fix it. We also began to work on optimizations for the fluid simulation, initially by starting to unroll loops, which creates larger parallelism and hardware utilization.
I think that we are on track to complete our project on time, but slightly behind where I would like to be at this point. It is good that we left ample slack in our schedule. Next week I want to continue to work on optimizing the implementation, and also work with Alice on communication between the FPGA and CPU to display the output of our renders.
At the moment, the most jeopardizing issue we have is figuring out how to integrate everything into one package on the board including a display output. As our efforts have been focused on building and accelerating the fluid simulator, we are still unsure about display. We have a variety of backup plans for this issue issue, such as producing a file of the rendered output, and producing the display from somewhere else. This still aligns with our project goals as we are not aiming for a real-time system, and are just trying to accelerate the fluid simulation process.
We decided to make a system change, such that the FPGA platform will only perform the fluid simulation algorithm. In other words, it will take an array of 3D positions and output an array of 3D positions for each frame, and save that to a file such that a host computer that is able to run Scotty3D can load in the simulation file and play the fluid simulation. This system change plays into the category of “request scheduler”, which is shown below.
Jeremy and Alice also swapped responsibilities since Jeremy is more experienced at C++ and digital hardware, and Alice’s knowledge of the Scotty3D codebase is helpful for adding additional features necessary to load the simulation into Scotty3D and getting the FPGA to send over the necessary data to the host computer. Below is the updated Gantt Chart:
This week I spent a lot of time trying to wrestle with C++ compilation errors, which were particularly cryptic in the Vitis HLS build environment. The Scotty3D code that we are basing our project on takes advantage of a lot of modern C++ features, and although these are good practice for general programming, they make stripping down the general codebase to what we need fairly tricky. But, after resolving several build errors and restructuring the code, we are able to build and run in the Vitis HLS environment. I also worked on beginning to implement some loop unrolling and allowing for parallelism in hardware.
I think that we are working at a steady pace, and are on track to complete our goals. Especially now that we have a steady build flow working, I think that we will be able to make smooth progress now. Next week I hope to make more progress on working on the acceleration, and also figure out a preliminary method for displaying our output, and work towards figuring out a final version of displaying output.
This week started by finishing up our design presentation, and working on the slides for that over the weekend. Since both classes this week involved the design presentations, my productivity was lower than usual, as those are good time for getting work done. But the work that I did do this week was some troubleshooting and some design work. As a group, we finalized how we are going to store points and find neighbors. We figured that by partitioning our simulation space, we could use the cube the point is in’s neighbors to find any points relative neighbors. This means that each neighbor lookup would be O(1) time, which is of particular importance as that is something each point needs to do every iteration. Our approach can also be done in parallel. I also was working on getting some of the build flow to work, which is getting to a stable place, but interfacing with hardware and getting programs to build for different architectures is never straight forward.
Next week I will be working on both the design report, as that deliverable is due next week. Additionally, I will be working on starting taking benchmarks of the CPU implementation, and finding how to benchmark our HLS implementation when we get it working.
This week unfortunately we had to spend a while to determine that for a while there was a bug with the version of Vitis installed on the cmu ece number machines. This means that for a long time we were determining why we were facing build errors on the ece machines and were unable to compile any project in Vitis, even basic example ones. After a while we determined that we were facing a bug due to the way Vitis stored integers in a format that broke on Jan 1 2022.
In addition to dealing with compiler errors, we also figured out what portions of the fluid rendering algorithm we are able to pipeline and apply loop unrolling to. I personally was able to gain a much deeper understanding of how the fluid rendering algorithm internally works and structures data. This helped me understand how we will structure the hardware, and be able to massively accelerate the algorithm. I also began thinking about how to schedule requests, and worked on the slides for the design presentation.
I think that we are slightly behind on schedule due to the issue with Vitis, but due to the way we created our schedule, I think that we are in a very recoverable position.
This week was a familiarization week for me. I personally do not have experience with HLS or xilinx FPGAs, so I was trying to get familiar with the platform we are using. To do so I went through the tutorial documents at https://xilinx.github.io/Vitis-Tutorials/2021-2/build/html/docs/Getting_Started/Vitis/Getting_Started_Vitis.html in order to understand the general workflows and structures of applications. This allowed me to gain a better understanding of how we need to restructure the Scotty3D library in order to accelerate it on the hardware.
I also began looking at using the DisplayPort interface on the board, as we want to display our rendered outputs so utilizing the DisplayPort will be necessary. There were several helpful documents I found such as https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842318/ZynqMP+Standalone+DisplayPort+Driver. This page links to several examples of how to use DisplayPort drivers for applications on the Xilinx board. I anticipate getting this to work correctly will be somewhat finicky, which is why I have started to look into how it works early on.
Although initially starting up any project will be somewhat slow due to figuring out workflows, I believe that we are making steady progress and I think that we are on schedule. Next week I would like to make progress on working on getting the data from the CPU on the board to the FPGA.