This week, I focussed on setting up power and timing infrastructure on my Mac and integrating this into the overall system.
I managed to achieve all of those goals, and evaluate it on a couple of test prompts. This seems to yield some encouraging results:
- Mean power dissipation:
- CPU — 600–700mW
- GPU — 24-40mW
- Mean timing:
- 1.1 — 1.3 seconds
Which seem to indicate that the FPGA system will effortlessly beat these specifications, so it looks like we’re on the right track in that regard.
A more important aspect now is to be quite thorough in this system, so while the FPGA setup is ongoing I plan to find a dataset to benchmark the power and timing on to find the average performance. I also evaluated the model on truthfulQA and found a score of 30 which is a pretty decent score for a model of this size.
For the next week, I aim to complete the above goals and also extend my python script for WiFi connectivity to the FPGA.
Answering part A:
Most people’s data whenever they wish to leverage large language models or any other AI based systems, get sent into data centres. These data centres process and compute the results. This leads to vulnerability on two ends:
- The data may be intercepted and read while in transit.
- Without control over the data, you never know what is being done with it after it has been used.
Which leads to poorer intellectual property protection and personal data safety.Additionally, as people become more and more reliant on these systems they will start using it for more critical tasks — like urgent healthcare etc. As a result, in the absence of wireless connectivity, this can cause significant harm.
Our solution aims to provide a fully offline setup for distilled AI systems in order to provide reliable, secure, and offline AI inference to people that want to keep control of their data.