Author: anirudhp

This week, I mainly focussed on testing as well as wrapping up the project, in the process I patched the following issues:

1. Multi-client was reading the output too early becasue the flag was set well in advance.
2. Output quality was ramped up using a better bitnet model.
3. An interesting bug was identified during user testing: just because you don’t actually print the output in the autocomplete while waiting doesn’t mean that you aren’t reading it. Hence you are capable of reading the other person’s output.
4. This was fixed by making it read only during the non output ready time steps.

Currently, my goals for next week are:

1. Complete the final report
2. I will be presenting the final presentation, so I will need to rehearse for that.

Some things that I specifically learnt over the course of the project are:

1. Access controls and the like on an FPGA hard-core. Have not learnt this before, and user testing allowed me to identify that this would be an issue.
2. The software defined interrupt interface on the mac, this was the key technology that I had used for the keyword prompting.
3. Presentation and specifically not to focus on text that is already stated on the slides, this was done while rehearsing the presentation yesterday.

This week we placed the final touches to our overall project and completed the presentation for the coming week.

We accomplished the following:

1. User testing with experienced tech savvy users, this is to find potentially “broken” things, so far we found a few minor fixes covered in the individual status reports.
2. Testing multi-client and ensuring functionality as well as security.
3. Completing our presentation and report.

Over the next week, as we complete the presentation, we will also aim to do the following:

1. Identify some way of bringing the FPGA to the demo(beyond requiring the ethernet dock that we had requested earlier)
2. Complete the report and poster for teh final demo date.

Currently we are well ahead of schedule with only minimal work required next week.

This week, I wanted to verify that the 700M parameter model would run on the FPGA(given that our primary focus is to get the output quality up and running).

In order to verify this, I had a full analysis of the memory resources that were available on the FPGA, and added a memory profiling system to the CPU based runtime on my laptop.

For now, I believe that the FPGA should not have trouble running the model once we delete PYNQ from the board. However, the always on python script will need to be adjusted.

In terms of the milestones, we are still on track. But I ran into a few other things that need to be changed in order to delete PYNQ and that additional work might put it marginally behind schedule.

Over next week, I want to work out exactly how the interfacing and inference system would work once I delete PYNQ from the board.

This week, my goal was to implement the model inference system on the FPGA.

I ended up running into a large number of issues, and was forced to switch models in order to fix them. For now, the model switch is temporary. I simply switched because the dynamic memory was over the board’s resources, theoretically this should not be happening and it should definitely be only because there is a bug in the inference systems.

We shifted to afrideva/llama-160m-GGUF llama-160m.q2_k.gguf along with the q2_k quantization system.

For now, we are well ahead of schedule given that this model produces decent output quality(37% hallucination rate). For now, changing from this model back to the original one is a much lower priority.

My goals for next week are to increase the performance(it is currently operating at a reading speed level(8-10 tokens/sec) all the way to reading level. It currently is a bit slower than the reading speed given that I notice a slight lag.

This week, now that the FPGA has both been setup and connected to the campus wifi network, we could easily complete all parts individually without having to pass the FPGA around.

We architected an alternate approach to multi-client based response that operates with the scheduler on the client side rather than the server side of things:

1. When the FPGA receives a query, it sets an “in-use” flag on and starts operating on that query.
2. Before the client sends a query to the FPGA, it checks the in-use flag.
3. Waits till the in-use flag turns off before actually sending the query.

This system leaves it vulnerable to race conditions, but we have decided to accept that minor flaw.

Andrew worked on running the model on the FPGA.

Anirudh setup a basic answer system and the in-use flag requirement.

Amelia refined the UI script so that it reads the flag and performs the wait before sending the query across.

For the time being, all individual components have been completed and at this stage we are moving on to the integration step. While we have tested everything, this can only be verified during integration. But it looks like we are well ahead of schedule given how easy integration should be.