Author: ameliahe

This past week I did not do anything since it was spring break. The week before last I focused on getting the design review done before the Friday deadline. I did not spend as much time getting comfortable with the Vitis EDA tools as I had planned, however, I’m sure over the course of the next few weeks, having hands on exposure will be enough for me to figure things out. I therefore accomplished one out of my two goals from two weeks ago.  I think I am still on schedule.

Given I have quite a busy upcoming week, I plan to focus on figuring out how to pull power data from our new FPGA and also look into multiuser authentication as an extension to our UI. I expect these to be manageable goals for the upcoming week. I would also like to spend some time organizing our website.

This week I focused on getting the UI ready to be tested by users. I learnt the basics of LUA and edited Anirudh’s original scripts to support a text preview and then the user can either accept the auto complete or regenerate the response. The UI now utilizes two hotkeys – cmd G to prompt the model and cmd L to remove the output if it isn’t what the user wants. I wanted to get the UI finished early so that we can start preliminary user testing (likely after spring break).

The other main thing I was focused on this week was watching/reading some Vitis tutorials to learn how to synthesize softcores on our new Kria FPGA.

Goals for next week:

1. make significant progress on the design review report as that is due Friday
2. Spend more time getting comfortable with Vitis EDA tools

I started off this week setting up the DeepSeek model I downloaded last week. While very interesting, after using the model and trying to find a way to prompt it to do text completion, I found that it is very much a chatbot with visible chain of thought. Therefore, we decided to stick with our original model.

I then pivoted to getting our UI set up on my computer, which required some debugging. I wanted to make sure the set up scripts were working because I plan to conduct some preliminary user testing next week. I quantified our metrics for usability, deciding on a setup time of at most 15 minutes and a benchmark of 100% success for users to reject or accept text completion suggestions, which is one of our system requirements. Finally, I worked on the design review presentation, making final block diagrams and quantifying technical design requirements.

The specified need our product will meet with consider of social factors is that it levels the playing field in terms of who has access to AI tools that support more efficient workflows for individuals. While many people can use text completion copilots to speed up their work, those working in data sensitive field cannot. This leads to a disparity across groups of who has more time to spend on more skilled aspects of their job, or more free time to spend doing other things. By enabling greater access to copilot tools, our product effectively enables greater access to better uses of time for a greater number of people.

At the beginning of this week I was focused on finished our proposal presentation slides and practicing for that presentation. In addition to practicing, I spent some time refining/narrowing our use case in order to guide benchmark creation for some of the more subjective elements of our project (like user experience). After finishing the proposal presentation I looked into new models for our text completion assistant. Specifically, we found that someone had quantized DeepSeek, and I wanted to see if it would be possible for us to fit that on our FPGA. I started by downloading the quantized model and getting it set up to run on my computer – to test output quality and ensure the quantized version was still of decent quality. The first roadblock I ran into was that this quantized model requires around 3Gb of memory, and our Ultra96v2 FPGA only has 2Gb of RAM. Unfortunately the group that quantized deepseek did not provide their quantization code, so I reached out to them to see if they would be able to provide it to me. If that happens, I plan to look into quantizing the model further, to see if we could fit in on the RAM in our FPGA

For this coming week, my goals are:

1. Figure out how to load the model onto the FPGA (using the softcore to scp files)
2. Get the FPGA to computer UART framework in place
3. Develop a metric for usability of our UI and conduct some preliminary user testing

This week I explored a number of trained BitNets that are supported by microsoft’s bitnet inference framework. The goal of this was to find a model that would be small enough to fit on an FPGA, but worked well enough to be repurposed into a viable product.

Initially, we wanted to work with a model that had only 70M parameters, in the hopes that we could fit that model on any FPGA we wanted. However, after trying to chat with it, I found that the low number of parameters contributed to very poor performance as seen in my conversation with it below:

I tested a few more models with larger parameters (up to 10B) from this family of models. While they perform significantly better,  these models are too large to fit on any FPGA we can afford (the 10B parameter model is around 3GB after quantization). I ultimately settled on this model, that has around 700 million parameters and is around 300 MB after quantization. This model is for text completion, as you can see below, so that is likely the direction we will take for our final project.