https://drive.google.com/file/d/1i8ojhM9djVCHblSq5gopb_jmhNufqRns/view?usp=sharing
Final Report
Final Video
Team Report 4/25
Unit Tests and Changes
This week we completed formal testing of the system against our quantitative design requirements. Our testing framework consisted of three components: predefined 20-word sentences for latency testing, paragraphs with labeled grammatical errors for accuracy validation, and repeated button-triggered actions across multiple trials to measure response consistency.
Across the four metrics we tracked, the system passed three cleanly. Autocomplete latency came in at a median of 0.661s and an average of 0.943s against a target of under 1 second. Grammar accuracy measured at 91%, clearing the 90% threshold. Context capacity came in at 8192 tokens, well above the 4000 token minimum. Full response time was the one inconsistent result, with a median of 1.861s but an average of 2.417s and a max of 7.161s against a 2 second target, indicating the system meets the requirement most of the time but has variability under certain conditions.
In addition to the quantitative testing, we conducted user testing sessions and collected verbal qualitative feedback from participants. That feedback was documented and summarized into a findings document that was handed off to inform refinement priorities going into this final week.
We decided to change the prompting style as well as the display of the information to help with the context/flow. Now the summary has more context and is more descriptive.
Based on testing findings, Sida updated the LCD UI to reflect a revised button layout to keep the on-device display in sync with the updated hardware interaction model. Additionally, the synthesis feature was reworked so it now pulls context from all relevant open applications rather than only the currently active window, which meaningfully improves the quality and completeness of the summaries the system can produce. This was a direct response to feedback about output relevance during user testing.
Risk
The primary remaining risk is the inconsistency in full response time, particularly the long tail at the high end. The team is aware of this and the focus going into the final week is on reducing that variance through continued prompting refinement and any remaining integration cleanup. No major changes to requirements or the block diagram were made this week.
Next Week
Going into the final week, the team’s focus is on polishing features based on user feedback, cleaning up the repository, completing the final poster and demo video, and consolidating all technical specs and lessons learned into the final report.
Tatyana Report for 4/25
This week my focus shifted toward testing support, qualitative research, and final presentation prep as the project moves into its last stretch.I helped source testers for our final presentation and assisted with troubleshooting some of the prompting and general refinement of the project. After conducting user testing, I collected a lot of verbal qualitative feedback from users and put together a document summarizing what we heard and what we should be improving going into this week. I handed that off to Sida, who used it to help improve the back end LLM prompting.
Since most of my front-end and technical work on the project is wrapped up, I shifted into more of a support role this week, helping Leonard with testing and taking on the layout and design of the final poster presentation.
Going into this week, my goals are to clean up the repository, work on the final demo and video strategy, and once the poster and demo are done, consolidate all of our lessons learned and final technical specs into the final report.
Tatyana Status Report 4/18
This week was mostly a collaborative week as the team worked together to bring the project into its final MVP form ahead of the demo.
On the application side, I finished up the front-end application and got it into its final MVP state. This included removing leftover testing buttons, cleaning up the UI, and integrating a new content scraping library that finally works to strip out headers and unnecessary information that had been getting passed through the pipeline. That extra noise was degrading the quality of our output, so getting clean text extraction working was a meaningful step forward.
Most of my week was focused on testing. I worked with Leonard to create and refine the testing materials, where I took on building the rubric and designing the general testing flow. I created the forms, conducted user tests, and also helped troubleshoot some final integration issues we were running into on Mac.
After testing, I helped with the final presentation by creating the metrics slides. This involved doing some Excel analysis to bucket the form criteria into a cohesive metrics table. I also gathered qualitative insights from the user sessions that will help us going into next week as we look to refine features and improve the MVP for the final demo.
Going into our final week, the goal is to use the user feedback to polish features and make the project as strong as possible for the demo.
Tatyana report 4/4
This week my contributions spanned device-side UI development, system architecture, and the conceptual foundation of the team’s testing strategy. Most of it in preperation for interm demo.
The majority of my time was spent leading the design and development of the device-side interface for the 2.4″ LCD display. This involved planning the full layout across all screen states, prototyping each screen, and completing the button mapping for the physical controls. Working at 320×240 resolution with the constraints of the Pico required a different design approach than the host-side interface, to keep the UI readable at that scale. To avoid blocking overall progress while the team continues debugging Jetson-to-Pico communication, I did this to make a parallel task, seprating it from the live data pipeline until integration is further along.
On the architecture side, I drove the decision to transmit data from the host to the device in the form of SQL queries rather than raw or pre-processed output. The rationale was straightforward: offloading that structure to the host reduces unnecessary GPU load on the Jetson and takes better advantage of the processing capacity already available on the host side.
I worked on the concept for the team’s testing plan. The core idea was to evaluate the system across two dimensions simultaneously: technical output grading against defined system criteria, and a qualitative signal from a randomized pool of users asked to grade system responses directly.
Next Week:
I will focus on improving the web scraping layer, refining the database design, and advancing the test design. On the scraping side, the goal is to improve selector reliability and parsing robustness so the pipeline handles a wider range of real-world page structures cleanly. For the database, I plan to revisit it. And on testing, I will work to making concrete test cases and evaluation criteria ready for execution.
Tatyana’s Status report 3/28
This week my contributions were concentrated in two areas: device-side UI development and software architecture, with additional conceptual contributions to the team’s testing strategy.
On the UI side, I led the design and development of the device-side interface for the 2.4″ LCD display. This included planning the full interface layout, prototyping all screens, and completing button mapping for the physical controls. Given the constraints of a 320×240 display, significant consideration went into ensuring the interface remains functional and readable at scale. This work was deliberately advanced as a parallel workstream to avoid blocking progress while the team continues debugging full system integration.
On the architecture side, I drove the decision to transmit data from the host to the device in the form of SQL queries. This approach was chosen to reduce processing load on the Jetson’s GPU and better leverage the host side’s computational capacity. The decision had meaningful downstream implications for how the host and device communicate and directly shaped the current host-side packet sending and scraping interface.
Finally, I originated the concept for the team’s testing plan, which combines technical output grading with a randomized user evaluation component to assess response quality from both a system and human perspective.
Team Status report 3/29
This week the team focused on development in preparation for our upcoming demo, with progress across both software architecture and hardware integration.
On the software side, we moved from a host-side prototyping to the a device-side execution model. The host is now responsible for packet sending and a scraping interface rather than running the full stack. A key decision this week was to send data in the form of SQL queries to the device, which reduces GPU load on the Jetson and better leverages host-side processing power. We are actively working through communication challenges between the Jetson and Pico as part of closing the full development loop.
On the hardware side, the PCB is in hand and system integration has begun. In parallel, we started work on the device-side UI and screen, which presents its own challenges given the constraints of a 2.4″ display. The UI is fully planned and prototyped, and button mapping is complete. While this is not yet connected to live data or the full system pipeline, we made the deliberate call to advance it as a parallel workstream to avoid blocking on system integration debugging.
Our demo objective is to showcase multiple subsystems working in isolation, demonstrating the breadth of what has been built. The critical success metric for the demo is achieving at least one complete end-to-end loop — a single full pass through the system from input to output. Outstanding work before the demo includes finalizing the prompting library and developing robust, dynamic suggestion generation, which remain active areas of development.
Following a meeting with our TA and professor, we finalized a testing plan that covers both dimensions of evaluation. On the technical side, system outputs will be graded against defined criteria. On the qualitative side, we will recruit a random selection of users to evaluate and grade system responses, giving us a human-centered signal on output quality alongside the technical metrics. This dual approach gives us a well-rounded framework for validating the system ahead of final evaluation.
Key decisions this week:
- Shifted architecture from host-side to device-side execution
- Adopted SQL query format for host-to-device data transfer to optimize GPU load
- Started device UI development as a parallel track to unblock progress
- Finalized dual-track testing plan with TA and professor sign-off
Current blockers:
- Jetson and Pico communication stability
- Full system integration and end-to-end data flow not yet complete
- Prompting library incomplete; dynamic suggestion system still in development
Tatyana’s Status report 3/21
This week I covered two main areas: completing the hardware-software HID bridge and beginning coordination on the Jetson output pipeline.
The primary technical completion this week was implementing and getting the keyboard communication module up and running on the Pico. This closes the hardware-software integration loop that was planned last week. The new module follows the same architectural pattern as the existing hotkey manager, running in a background thread, emitting signals on incoming packets, and wiring directly into the existing capture and release handlers in the host application. With the Pico confirmed working as a physical trigger, the keyboard can now initiate the full capture-process-release pipeline as a hardware input. This shifts the backend logic on to the firmware instead of having it completely on the software side. The communication,in byte format ,is also in place, allowing the host application to send status information back to the keyboard for LED or layer feedback.
The second area is a collaboration with Sida beginning this week on the receiving logic for the Jetson output. Once the Jetson generates a response, that output needs to be received on the host side, routed correctly through the pipeline, and delivered back to the user either displayed in the UI or pasted into the active application. As part of this collaboration the team is also evaluating whether to consolidate the keyboard firmware and the host application into a single unified repository rather than maintaining two separate ones. The goal is to reduce overhead and make the integration between the two sides of the system easier to manage as the project moves into its first demo.
Next week the priorities are investigating text extraction approaches for websites and Google Docs. The current accessibility based capture works well for native desktop applications but does not reliably expose content from browser based surfaces, so this will involve researching browser extension APIs, the Google Docs REST API, and other approaches to identify the most reliable and privacy consistent solution. The other priority is to continue work with Sida on the Jetson output-receiving logic to move toward a complete end-to-end run of the full pipeline.
Tatyana’s Status report 3/14
This past week I focused on two main things: integration testing and cleaning up the page scraping application.
For integration testing, I worked on connecting the host-side pipeline to the Jetson by adding the calles to the the model and verifying that SparkDB is correctly handling reads and writes. Getting the model api call up was a solid step forward since it’s is closer to the mvp and the product offering og our sysetem, and confirming SparkDB behavior gave us more confidence that context is being stored and retrieved the way we expect. That said, there are still some open issues going into next week, so integration is very much still in progress rather than wrapped up.
On the page scraping side, I did a round of cleanup that was overdue. The main things I tackled were fixing some broken selectors and parsing logic that were causing the scraper to miss or misread certain page content, and refactoring the code structure to make it easier to work with going forward. The scraper feeds directly into the context capture pipeline, so having the parsing logic actually reliable matters a lot once real pages start flowing through during integration. The refactor also just made the code more efficient and easier to explain to the rest of the team.
Progress is on schedule overall. Next week I plan to keep pushing on the integration side, specifically resolving the open issues from this week and working toward a more stable end-to-end run of the full pipeline.