Category: Sida’s Status Reports

• Updated the LCD UI to reflect the new button layout, ensuring the on-device display stays in sync with the revised hardware interaction model.

• Reworked the synthesis feature so that it now pulls context from relevant apps rather than only the current active app. This is a meaningful improvement to the core value of SPARK — the system can now produce more complete and useful summaries by considering the broader set of open context, not just whatever window happens to be in focus at the moment of the keypress.

• Next up: making the response drafting feature smarter, with improvements planned for next week.

• Completed final integration of the full SPARK system — all features are working end-to-end and the system is ready for testing and demo.
• Decluttered and reorganized the repository to significantly lower the barrier to getting started. This includes one-click setup scripts that automatically configure the SPARK app on a fresh Windows or macOS environment, removing manual dependency setup steps.
• Helped prepare materials and testing results for the final presentation.

What new tools or knowledge did you need to learn?

Repo Management: Clearly defining the file structure of the repo early on can significantly boost clarity, productivity, and readability. Things will get messy really quickly, especially when collaborating with others. Same with Git best practices: do commit and commit often, branch management, commit message style, etc.

Development Environment Management: same ideology. What may work on your machine may not work on your other machine, let alone your teammate’s one. This is especially true for our team: I work mostly in Windows, and Tatyana works mostly in macOS. Carefully selecting mutual toolchains and navigating the OS-specific tools is 100% needed. Sometimes, hardcoding is necessary; most of the time, it should be avoided.

Agile Development for PCB: for early prototypes(which is every capstone project – no project would REALLY pass the early prototype stage), have more flexibility. Have test points, backup vias and pads for manual soldering and rerouting, 3d modeling for spacing, mounting holes for casing. order extra components… All very necessary. When making iterations, one should try to use as much already-bought components as possible.

The learning strategy was mostly hands-on and informal: running the scripts repeatedly on clean environments to catch failures, reading platform-specific documentation and Stack Overflow threads for edge cases, and iterating quickly based on what broke. This trial-and-error approach on real machines was far more effective than any top-down reading would have been for a task this integration-heavy.

 

• Continued system integration work. Attempted to port the UI demo directly onto the current Pico implementation, but the integration is more complex than initially anticipated: the UI cannot be drag-and-dropped onto the existing firmware. Will continue debugging the RAW HID / UI bridge next week.
• Designed a second version of the PCB with several improvements over v1: better mounting holes for more secure physical assembly, clearer silkscreen labels for easier debugging and assembly, and modular connectors for improved adaptability.

 

• My work: Spent significant time debugging and testing the full end-to-end loop: Host app → Pico → Jetson → LLM server → back to Host. Solved multiple issues to when bridging the app-pico connection with the pico-jetson connection, then made the full loop work. The results need tweaking, but the core functionality is completed and ready to demo
• Major architecture refactor: Moved the pico firmware from QMK to Circuit Python. Due to the nature of the pico shifting from a keyboard to a transmission bridge between the host-side software and the LLM, the QMK firmware is capable but redundant. CircuitPython gives better development and debugging experience and is easier to work with. It still keeps the general HID functionality and can type back, acting as a keyboard if needed.
• Quality of Life:
  • Implemented live capture deduplication so the polling feed only appends text when the content actually changes, avoiding redundant entries.
  • Added robustness improvements including automatic device reconnection after unplug/replug on both the Jetson bridge and host HID client, a self-window guard so SPARK doesn’t capture its own window, a fixed-size
    scrollable output window, and a quit button.
• Pending work:
  • moving the host-side database implementation to Jetson
  • Review for the 2nd version of PCB design.

• Works with Tatyana on the communication protocol between the Pi Pico and the host-side software. Still needs refinement and further integration
• Worked on the integration between the Pi Pico and the Jetson. Successfully created a demo that sends prompts and receives responses through UART. The Jetson-side bridge program receives the prompt, makes API calls locally, and sends the response back.
• The delay was reasonable, and it simulates the feature calls from buttons.

• Created a proof of concept desktop app that’s able to communicate with the keyboard using RAW HID reports. The desktop app is able to send a large block of text or data to simulate the context we are sending. The keyboard then types back a simulated summary, which is just the first and last words of the text block. This is to simulate that the LLM inference outputs in a stream, not act like an instant paste of the entire text block.
• The next step is to integrate this communication with Tatyana’s actual desktop GUI app

• tested the newly released Qwen3.5 models, including 0.8B, 2B, 4B and 9B versions(all Q4_K_M). All tested with 8K context length(double of our design requirement. Still need to do more detailed tests on whether the models can accurately perform the tasks, but based on online reviews, they will definitely be better than the Qwen3 models.
  • 9B: read 217, write 10.84
  • 4B: read 341, write 14.88
  • 2B: read 839.2, write 30.99
  • 0.8B: read 1149, write 41.58
• Result: the Qwen 3.5 4B model is decently slower than the Qwen 3 one, which violates our design requirement. Qwen 3.5 2B, on the other hand, is faster than the Qwen3 4B.  We can set both as backups while developing further.

• Making the solution part of the Design Review presentation(system block diagram, implementation plan)
• Presented the Design Review presentation, waiting for the feedback
• Working on finalizing the BOM and starting on the firmware

Tested multiple models that the Jetson AGX Xavier can run (all with llama.cpp and 4-bit quantization).

• Qwen3-4B: output 22 token/s
• Phi-3 Mini (3.8B): output 22 tokens/s
• Llama3.1-8B: output 15 token/s
• Nemotron-3-Nano, GLM 4.7 Flash cannot run due to the lack of memory.

Findings:

• Qwen3-4B and Phi-3 Mini are both able to perform correction tasks correctly and generate output at a similar speed. Llama3.1-8B is slower due to more parameters, and it could be a backup if the first two models cannot perform complex tasks later.
• Reading speed of the prompt is non-linear. For short prompts(<50 tokens), the reading speed is ~60 token/s –> ~0.3s. For long prompt(~500 tokens), the reading speed is ~300 token/s –> 1~2s. I need to test this further so we meet our design requirement.
• Theoretically, Nemotron-3-Nano and GLM 4.7 Flash should be able to fit into the 32GB memory, but they couldn’t unless being significantly tuned smaller, which perform slower and dumber than the above models.

Team Effort(what I have done with other team members):

• Working on and presenting the proposal pitch

Individual Effort:

• Got ShearedLlama-2.7B(4-bit quantization) + NanoLLM + MLC working, but had to reduce the context length to 2048(maximum is 4096) in order to fit into the memory. Even then, it’s unstable and runs out of memory sometimes, and the performance is not very good.
• Acquired the Jetson AGX Xavier from the inventory, which has 32GB of memory, hopefully it will perform much better. Currently working on getting Qwen3 to work on the device.