Posted on

Tatyana’s Status report 3/28

This week my contributions were concentrated in two areas: device-side UI development and software architecture, with additional conceptual contributions to the team’s testing strategy.

On the UI side, I led the design and development of the device-side interface for the 2.4″ LCD display. This included planning the full interface layout, prototyping all screens, and completing button mapping for the physical controls. Given the constraints of a 320×240 display, significant consideration went into ensuring the interface remains functional and readable at scale. This work was deliberately advanced as a parallel workstream to avoid blocking progress while the team continues debugging full system integration.

On the architecture side, I drove the decision to transmit data from the host to the device in the form of SQL queries. This approach was chosen to reduce processing load on the Jetson’s GPU and better leverage the host side’s computational capacity. The decision had meaningful downstream implications for how the host and device communicate and directly shaped the current host-side packet sending and scraping interface.

Finally, I originated the concept for the team’s testing plan, which combines technical output grading with a randomized user evaluation component to assess response quality from both a system and human perspective.

Posted on

Team Status report 3/29

This week the team focused on development in preparation for our upcoming demo, with progress across both software architecture and hardware integration.

On the software side, we moved from a host-side prototyping to the a device-side execution model. The host is now responsible for packet sending and a scraping interface rather than running the full stack. A key decision this week was to send data in the form of SQL queries to the device, which reduces GPU load on the Jetson and better leverages host-side processing power. We are actively working through communication challenges between the Jetson and Pico as part of closing the full development loop.

On the hardware side, the PCB is in hand and system integration has begun. In parallel, we started work on the device-side UI and screen, which presents its own challenges given the constraints of a 2.4″ display. The UI is fully planned and prototyped, and button mapping is complete. While this is not yet connected to live data or the full system pipeline, we made the deliberate call to advance it as a parallel workstream to avoid blocking on system integration debugging.

Our demo objective is to showcase multiple subsystems working in isolation, demonstrating the breadth of what has been built. The critical success metric for the demo is achieving at least one complete end-to-end loop — a single full pass through the system from input to output. Outstanding work before the demo includes finalizing the prompting library and developing robust, dynamic suggestion generation, which remain active areas of development.

Following a meeting with our TA and professor, we finalized a testing plan that covers both dimensions of evaluation. On the technical side, system outputs will be graded against defined criteria. On the qualitative side, we will recruit a random selection of users to evaluate and grade system responses, giving us a human-centered signal on output quality alongside the technical metrics. This dual approach gives us a well-rounded framework for validating the system ahead of final evaluation.

Key decisions this week:

  • Shifted architecture from host-side to device-side execution
  • Adopted SQL query format for host-to-device data transfer to optimize GPU load
  • Started device UI development as a parallel track to unblock progress
  • Finalized dual-track testing plan with TA and professor sign-off

Current blockers:

  • Jetson and Pico communication stability
  • Full system integration and end-to-end data flow not yet complete
  • Prompting library incomplete; dynamic suggestion system still in development
Posted on

Sida’s Status Report for 3/28

  • My work: Spent significant time debugging and testing the full end-to-end loop: Host app → Pico → Jetson → LLM server → back to Host. Solved multiple issues to when bridging the app-pico connection with the pico-jetson connection, then made the full loop work. The results need tweaking, but the core functionality is completed and ready to demo
  • Major architecture refactor: Moved the pico firmware from QMK to Circuit Python. Due to the nature of the pico shifting from a keyboard to a transmission bridge between the host-side software and the LLM, the QMK firmware is capable but redundant. CircuitPython gives better development and debugging experience and is easier to work with. It still keeps the general HID functionality and can type back, acting as a keyboard if needed.
  • Quality of Life:
    • Implemented live capture deduplication so the polling feed only appends text when the content actually changes, avoiding redundant entries.
    • Added robustness improvements including automatic device reconnection after unplug/replug on both the Jetson bridge and host HID client, a self-window guard so SPARK doesn’t capture its own window, a fixed-size
      scrollable output window, and a quit button.
  • Pending work:
    • moving the host-side database implementation to Jetson
    • Review for the 2nd version of PCB design.
Posted on

Leonard’s Status Report for 3/28

This week, I assembled the initial version of the SPARK PCB and began integration. During this process, several issues were identified in the current design, including layout limitations that affected integration. Based on these findings, I reviewed the schematic and PCB layout and implemented necessary design updates. These revisions aim to ensure smoother integration in the next iteration of the hardware.

In parallel, significant progress was made on the system testing and verification plan. I developed a structured set of 20 testing scenarios, aligned with the four main SPARK features: context summary, keyword search, text formatting, and response drafting. Each feature is supported by five realistic, standardized scenarios designed to evaluate system performance based on a rubric of 6 categories.

For the upcoming week, my main focus will be on preparing for the interim demo and also waiting for the arrival of updated PCBs and replacement components.

Posted on

Tatyana’s Status report 3/21

This week I covered two main areas: completing the hardware-software HID bridge and beginning coordination on the Jetson output pipeline.

The primary technical completion this week was implementing and getting the keyboard communication module up and running on the Pico. This closes the hardware-software integration loop that was planned last week. The new module follows the same architectural pattern as the existing hotkey manager, running in a background thread, emitting signals on incoming packets, and wiring directly into the existing capture and release handlers in the host application. With the Pico confirmed working as a physical trigger, the keyboard can now initiate the full capture-process-release pipeline as a hardware input. This shifts the backend logic on to the firmware instead of having it completely on the software side. The communication,in byte format ,is also in place, allowing the host application to send status information back to the keyboard for LED or layer feedback.

The second area is a collaboration with Sida beginning this week on the receiving logic for the Jetson output. Once the Jetson generates a response, that output needs to be received on the host side, routed correctly through the pipeline, and delivered back to the user either displayed in the UI or pasted into the active application. As part of this collaboration the team is also evaluating whether to consolidate the keyboard firmware and the host application into a single unified repository rather than maintaining two separate ones. The goal is to reduce overhead and make the integration between the two sides of the system easier to manage as the project moves into its first demo.

Next week the priorities are investigating text extraction approaches for websites and Google Docs. The current accessibility based capture works well for native desktop applications but does not reliably expose content from browser based surfaces, so this will involve researching browser extension APIs, the Google Docs REST API, and other approaches to identify the most reliable and privacy consistent solution. The other priority is to continue work with Sida on the Jetson output-receiving logic to move toward a complete end-to-end run of the full pipeline.

Posted on

Team’s Status Report for 3/21

This week, the team’s focus is on integrating the connections between each part of the system and building toward a fully functional demo.

Between the host-side software and the Pi Pico, Sida and Tatyana have produced an MVP using the RAW HID reports. The protocol remains to be tested for heaviers load and actual feature calls, but the basic communication is tested to be functional.

Between the Pi Pico and the Jetson, Sida has produced an MVP using a UART connection between the GPIO of the Pi Pico and the 40-pin expansion header of the Jetson.

On the hardware side, most components have arrived, but the PCB has been delayed and is now expected next week. Once the PCB is in hand, Leonard can start with the physical assembly. In the meantime, Leonard is working on the testing suite.

The team also completed the ethical considerations discussion this week, including a worst-case scenario analysis and red-teaming feedback from other teams. We will incorporate the feedback into our interim demo and final showcase.

Next Week: The primary focus will be on hardware and system assembly to bring all parts together for a full demo. We will have separate meetings of two members for the integration tasks.

Biggest open risk: Both the PCB delay and delays caused by personal situations have some impact on the overall progress. We are 2-3 days behind our schedule, but we will make up for it to build the demo.

Posted on

Sida’s Status Report for 3/21

  • Works with Tatyana on the communication protocol between the Pi Pico and the host-side software. Still needs refinement and further integration
  • Worked on the integration between the Pi Pico and the Jetson. Successfully created a demo that sends prompts and receives responses through UART. The Jetson-side bridge program receives the prompt, makes API calls locally, and sends the response back.
  • The delay was reasonable, and it simulates the feature calls from buttons.

Posted on

Leonard’s Status Report for 3/21

This week, the main focus is on the ethical considerations associated with the SPARK system. Worst-case scenario is considered as well as red teaming from other teams.

On the hardware side, additional components have now arrived. However, the PCB has been delayed and is now expected to arrive next week.

In the upcoming week, hardware assembly should begin once the PCB arrives. In parallel, focus should be shifted toward improving the software and testing framework. The plan is to organize smaller group meetings with Tatyana that is dedicated to discuss and develop the code testbenches required for system testing and verification.

Posted on

Team’s Status Report for 3/14

This week, the team focused on reviewing the feedback received during the design review and identifying areas for improvement in the testing and verification plan for the SPARK system. Based on the comments from the reviewers, the team discussed ways to make the validation process more systematic and measurable.

For upcoming week, the team will work  to clarify how system performance will be measured, including accuracy of generated outputs and the effectiveness of the system in supporting user interaction scenarios.

The team will also have a ethics discussion related to the SPARK system. As part of this process, the team will hold a short discussion session to consider potential ethical implications of the product if it were widely used or misused. The results of this discussion will be summarized and shared within the team communication channel, and the ideas will later be refined and incorporated into the ethics section of the final project report.

Posted on

Leonard’s Status Report for 3/14

This week, the hardware development of the SPARK system progressed from the design stage to the implementation stage. The final Gerber files for the PCB were generated using the CAM processor. These files were reviewed and successfully submitted to the PCB manufacturer JCLPCB, and the PCB boards have now been ordered. In parallel with the PCB order, most of the required hardware components for the prototype have already been purchased and delivered. Some remaining components are still in transit and are expected to arrive soon.

The focus for the upcoming week will be to refine the testing and verification plan for the SPARK system based on the feedback received for the design review report. The idea is to use another LLM to judge whether the results can be considered correct or not.