Posted on

Team Status Report for Feb 22

This week we focused on a few different things – the design presentation, which was mainly covered by Andrew,  UI development, which was covered by Anirudh and I (Amelia), and FPGA drama, which we all dealt with.

We finally got the UI setup scripts to work on other user’s computers – something we’d been working on. Once the set up process was finalized, we worked on a more usable interface that fits our use case requirements as laid out in our design presentation. We want to have the UI set up and run scripts completed soon because we plan to do some preliminary user testing while there is still plenty of time to make changes based on anything we learn.

The FPGA drama of the week was we found out that the Ultra96-V2 board we got is part of a broken batch that has non-functional wifi. Without the wifi, we can’t ssh into the board, so we had two options, either pick a new board, or obtain a wired uart extension board. We decided to try the Kria KV260, since that was another board we initially considered due to its greater memory capacity. Our initial hesitation with using this board was because none of us were familiar with Vitis, but after a little reading, we felt that we can learn how to use those EDA tools well enough to develop a synthesis flow. Now we may also be able to work with a larger model, which will give us more accurate text completions.

Group Goals for the next week:

  1. We have to complete the design review report for Friday
  2. Our new FPGA is arriving soon, so we need to start working on our new synthesis flows.
Posted on

Amelia’s Status Report for Feb 22

This week I focused on getting the UI ready to be tested by users. I learnt the basics of LUA and edited Anirudh’s original scripts to support a text preview and then the user can either accept the auto complete or regenerate the response. The UI now utilizes two hotkeys – cmd G to prompt the model and cmd L to remove the output if it isn’t what the user wants. I wanted to get the UI finished early so that we can start preliminary user testing (likely after spring break).

The other main thing I was focused on this week was watching/reading some Vitis tutorials to learn how to synthesize softcores on our new Kria FPGA.

Preview of generated text

Goals for next week:

  1. make significant progress on the design review report as that is due Friday
  2. Spend more time getting comfortable with Vitis EDA tools
Posted on

anirudhp_status_report_Feb22nd

This week, the focus was on packaging and clearing our user interface system in order to start getting feedback form people that can trial our system.

This week I managed to get the power and timing parameters achieved to be printed on the side of the screen in a location that I thought was unobtrusive. It’s one of the details that I would like to verify in our user feedback form.

Additionally, given that we are moving from an Ultra96v2 FPGA to a Kria based FPGA, we would need to learn how to use a different set of EDA tools. So I spent the past week mainly focussing on how to operate and use Vitis to synthesize our softcores and language models.

Over the next week my goals are:

  1. I want to work out how the full synthesis flow to load our models onto the Kria works.
  2. I want to see how to move data onto the FPGA and pull the results out of the FPGA works, this way I can extend our previous python script to use the FPGA for inference.

After this, I would plan to go for the more advanced power and performance data that we want to monitor on the FPGA.

 

We’re currently well ahead of schedule and on track to reach the iteration and architecture phase within another 2 weeks.

Posted on

Andrew’s Status Report for Feb 15

My main focus this week was to keep looking into the FPGA operation, securing and trying to setup the FPGA.

I was able to successfully secure an Ultra-96 V2 FPGA from Quinn in the parts inventory. The setup was troublesome for two reasons: 1) my laptop was sent to repair and I was unable to flash the sd card image myself to complete FPGA setup. 2) When I was able to flash the sd card and setup the FPGA myself, I find out that the FPGA is faulty out of the box. Its wifi module, the only available way for the FPGA to communicate off-chip, was found to be broken. I later located an errata file that detailed production issues that caused malfunctioning Ultra-96 V2 units, and our FPGA was produced during the affected batch.

The plan for next week is to decide on a replacement FPGA as soon as possible. We have two choices, another Ultra-96 V2 or the Kria KV-260, the latter has more ram for better deployment models, (4GB instead of 2GB on Ultra-96 V2) but suffers from the drawback that it doesn’t have an onbard SoC and would require an instantiation of a soft-core instance, complicating the setup and requiring the use of Xilinx Vistis, a tool that no team member has worked with before at this point.

 

For Status Report Part C:

Deploying modern large language models (LLMs) locally often comes with significant economic challenges. Traditional solutions typically require multiple high-end graphics cards, which are expensive and require non-trivial energy costs. 

Our proposed FPGA-based local inference solution addresses these economic concerns in several key ways. Reduced equipment costs make FPGAs a more affordable alternative to multi-GPU systems, lowering the upfront capital needed for local LLM deployment. Energy efficiency and long-term savings become a significant economic advantage over time. While a typical multi-GPU setup can consume around 2000W during a standard workload, an FPGA implementation operates within a 20-50W range. The drastic reduction in energy consumption translates to considerable cost savings and a more sustainable operational model. Lower maintenance overhead compared to maintaining multiple GPUs means FPGA-based systems typically experience lower failure rates and reduced thermal management requirements, minimizing both downtime and repair costs.

Posted on

Amelia’s Status Report for Feb 15

I started off this week setting up the DeepSeek model I downloaded last week. While very interesting, after using the model and trying to find a way to prompt it to do text completion, I found that it is very much a chatbot with visible chain of thought. Therefore, we decided to stick with our original model.

I then pivoted to getting our UI set up on my computer, which required some debugging. I wanted to make sure the set up scripts were working because I plan to conduct some preliminary user testing next week. I quantified our metrics for usability, deciding on a setup time of at most 15 minutes and a benchmark of 100% success for users to reject or accept text completion suggestions, which is one of our system requirements. Finally, I worked on the design review presentation, making final block diagrams and quantifying technical design requirements.

The specified need our product will meet with consider of social factors is that it levels the playing field in terms of who has access to AI tools that support more efficient workflows for individuals. While many people can use text completion copilots to speed up their work, those working in data sensitive field cannot. This leads to a disparity across groups of who has more time to spend on more skilled aspects of their job, or more free time to spend doing other things. By enabling greater access to copilot tools, our product effectively enables greater access to better uses of time for a greater number of people.

Posted on

Anirudhp status update Feb 15th

This week, I focussed on setting up power and timing infrastructure on my Mac and integrating this into the overall system.

I managed to achieve all of those goals, and evaluate it on a couple of test prompts. This seems to yield some encouraging results:

  1. Mean power dissipation:
    1. CPU — 600–700mW
    2. GPU — 24-40mW
  2. Mean timing:
    1. 1.1 — 1.3 seconds

Which seem to indicate that the FPGA system will effortlessly beat these specifications, so it looks like we’re on the right track in that regard.

A more important aspect now is to be quite thorough in this system, so while the FPGA setup is ongoing I plan to find a dataset to benchmark the power and timing on to find the average performance. I also evaluated the model on truthfulQA and found a score of 30 which is a pretty decent score for a model of this size.

For the next week, I aim to complete the above goals and also extend my python script for WiFi connectivity to the FPGA.

Answering part A:
Most people’s data whenever they wish to leverage large language models or any other AI based systems, get sent into data centres. These data centres process and compute the results. This leads to vulnerability on two ends:

  1. The data may be intercepted and read while in transit.
  2. Without control over the data, you never know what is being done with it after it has been used.
    Which leads to poorer intellectual property protection and personal data safety.

    Additionally, as people become more and more reliant on these systems they will start using it for more critical tasks — like urgent healthcare etc. As a result, in the absence of wireless connectivity, this can cause significant harm.

Our solution aims to provide a fully offline setup for distilled AI systems in order to provide reliable, secure, and offline AI inference to people that want to keep control of their data.

Posted on

Team Status Update Feb 15th

This week we focussed on wrapping up the auxiliary tasks that lead up to the final stage of the project where we’ll focus on iterating on the hardware accelerator. Namely:

  1. Setting up our benchmarking and profiling system for the baseline.
  2. Setting up the FPGA connectivity and synthesis flow.
  3. Evaluating a Chain of thought alternative model for improving model accuracy.

We managed to complete the benchmarking and profiling system, and eventually decided against using deepseek-r1’s smaller variant, however the FPGA system did not end up working as we expected.

We found that the the FPGA system that we used had some flaws in its WiFi connectivity setup, this leads to us not being able to service multiple clients at the same time.

Our goals for next week are:

  1. Run our benchmarking and profiling system on a wide spectrum of input tokens, and collect a comprehensive characterization dataset on our Macs.
  2. Swap to a functional FPGA with WiFi capacity and boot linux as well as our synthesis flow on the board. However, in the duration that the other FPGA takes to arrive, we can still try to synthesize the model onto ours and get that working — but this will only be a viable option if we’re going to use the same FPGA type after changing.
  3. Preemptively prepare the interconnect from FPGA to laptop and begin drawing a block diagram for the accelerated system.

For status report 2: A was written by Anirudh, B was written by Amelia, and C was written by Andrew.

Posted on

Andrew’s Status Report for Feb 8th

In the first half of this week I was looking into softcore IPs for CPU and GPU to be synthesized later for performance benchmark. Then after talking with Prof. Benson we decided that we would instead compare our benchmark on commercial CPUs and GPUs using non-quantized version of the models. I also worked on securing and initial setup of the FPGA. We chose and obtained from the class inventory the Ultra-96 V2 FPGA and I am currently trying to boot linux on the FPGA. In case of insufficient onboard ram, I have also obtained permission to use one of the research FPGA from Professor Ken Mai if needs arise later into the semester.

My plan for next week are:

  1. Complete FPGA setup and look into ways to easily interface into the FPGA
  2. Start the design of overall architecture of the FPGA for ternary llm inference
  3. Setup tool chain on Vivado focusing on data transfer between the FPGA PS and PL

 

Posted on

Amelia’s Status Report for Feb 8th

At the beginning of this week I was focused on finished our proposal presentation slides and practicing for that presentation. In addition to practicing, I spent some time refining/narrowing our use case in order to guide benchmark creation for some of the more subjective elements of our project (like user experience). After finishing the proposal presentation I looked into new models for our text completion assistant. Specifically, we found that someone had quantized DeepSeek, and I wanted to see if it would be possible for us to fit that on our FPGA. I started by downloading the quantized model and getting it set up to run on my computer – to test output quality and ensure the quantized version was still of decent quality. The first roadblock I ran into was that this quantized model requires around 3Gb of memory, and our Ultra96v2 FPGA only has 2Gb of RAM. Unfortunately the group that quantized deepseek did not provide their quantization code, so I reached out to them to see if they would be able to provide it to me. If that happens, I plan to look into quantizing the model further, to see if we could fit in on the RAM in our FPGA

For this coming week, my goals are:

  1. Figure out how to load the model onto the FPGA (using the softcore to scp files)
  2. Get the FPGA to computer UART framework in place
  3. Develop a metric for usability of our UI and conduct some preliminary user testing

 

Posted on

Team Status Report for Feb 8th

We started this week focused on completing the proposal presentation, which included narrowing down our use case to a focus on users who want to use text completion models but are unable to use commercial products due to privacy concerns with sending sensitive information to the cloud. After the proposal presentation, we received some feedback that caused us to change our approach to benchmarking. Instead of synthesizing CPU/GPU cores onto our FPGA to generate timing and power benchmarks, we are now exploring a way to measure those benchmarks on a Mac, which allows us to start developing and synthesizing our architecture sooner than anticipated. In terms of updating our schedule, we now have more room for slack which will be key as we have to do integration more towards the beginning of our project and will likely run into hurdles getting the host computer and FPGA communicating.

We got our FPGA this week – the ultra96v2, and are now in the process of booting Linux on it (and finding a power supply).  We also got a UI working for all text boxes on a Mac as well as a python script that automates the installation process of all libraries required to use the autocomplete feature. The next steps for the UI include finalizing a model that is small enough to fit in the DDR memory on our FPGA but has decent outputs. One risk we have identified is that we haven’t tested the installation process on any computers other than our own, and we may conduct some user testing to ensure it’s a simple installation process for people with and without technical skills.

Our group goals for next week are:

  1. Finalize a model that is small but has a potentially higher output quality than what we are currently working with
  2. Boot linux onto the FPGA
  3. Figure out how to get timing and power data from MacOS
  4. conduct preliminary user testing (and develop a quantifiable metric to benchmark it’s quality)