Category: Rohit’s Reports

Accomplishments
This week, I made a pivot in our web application approach as I migrated our frontend from Django to React. While Django was fine for initial development, we encountered performance issues with the real-time updates required when new images are captured and processed. The server-rendered approach was causing noticeable page reloads that disrupted the user experience. After analyzing these limitations, I implemented a React-based frontend that communicates with our backend API, allowing for seamless updates through WebSockets without requiring full page refreshes.

Additionally, I worked closely with Will and Dhruv on completing the integration pipeline. I assisted in debugging the MQTT communication between the hardware components and our backend services, particularly focusing on ensuring image metadata is properly transmitted and processed. We identified and resolved several edge cases, such as handling connection interruptions and ensuring data integrity throughout the pipeline.

I also dedicated some time to designing and producing the 3D printed housings for the Raspberry Pi Pico and battery pack. Using Fusion 360, I created custom enclosures that optimize for both protection and ergonomics. The battery pack housing was particularly challenging as it needed to balance size constraints. We are in the process of getting this printed.

Schedule
We’re now back on schedule with the React migration actually accelerating our development timeline. The component-based architecture has made it easier to distribute work among team members and iterate quickly on specific features. The integration pipeline is nearly complete, and we’ve begun comprehensive testing of the entire system. The 3D printed housings have also resolved many of the physical integration challenges we were facing.

Plans for next week
For the final week before our demo, I’ll focus on polishing the React frontend with final styling and UX improvement. I’ll also continue assisting with any final adjustments to the 3D printed housings based on testing feedback

Migrating from Django to React required rapidly acquiring new knowledge in several areas. When I encountered performance issues with server-side rendering, I needed to quickly learn React’s component lifecycle and state management approaches. Rather than following a traditional course, I adopted a project-based learning strategy where I rebuilt our key features in React one by one.

Learnings
For learning React, I primarily relied on the official documentation combined with specific problem-solving on Stack Overflow. I found that reading the conceptual documentation first, followed by practical implementation, helped solidify my understanding.

For the 3D modeling work, I had no prior experience with CAD software, so I utilized YouTube tutorials that specifically focused on enclosure design in Fusion 360. I also did some hands-on experimentation, creating prototypes to test dimensions and fitment.

This week I completed the frontend implementation of our GlassMate web application. I finalized all the UI components including the calendar view, diary entry display, and user settings interface. I also collaborated with Will on 3D printing the casings for all our hardware components, which involved learning CAD design and working with the 3D printer to create properly sized housings for the Arducam.

A significant achievement was implementing the metadata addition functionality for images. This system extracts GPS coordinates from captured images and enhances them with location names using the Google Maps API integration we previously developed. I ensured this metadata is properly stored and associated with each image in our database.

I also worked closely with Dhruv to connect the frontend with the backend pipeline he was developing. We’ve successfully established the connection so that when the camera captures an image, it’s automatically processed and appears in the web interface with the proper metadata and location information. The system now displays captured images in the calendar view and generates diary entries that incorporate the location data.

I’m slightly behind schedule on some aspects of the integration. While the frontend components are largely complete, there’s still work to be done on ensuring communication between all parts of the system. We should have begun more comprehensive testing by now, but the challenges with the GPS module and hardware integration have delayed this phase.

For the coming week, I’ll focus on completing the final integration of the frontend with the manual trigger button that Will has implemented. I will also start testing of the entire pipeline from image capture to web display.

In terms of verification, I’ll be testing the frontend’s response to various image capture scenarios and ensuring that metadata is correctly displayed in the interface. I’ll also conduct usability testing to identify any UX issues before the final demo.

This week I made more progress on the frontend components. I successfully implemented the user auth system, which now allow users to register new accounts and login securely. This includes all the necessary validation, password hashing, and session management. The login and registration pages are now fully functional.

I also integrated the Google Maps API with our vision model. This allows us to extract the latitude and longitude metadata from the captured images and uses the Google Maps Places API to identify nearby points of interest. The system can now automatically generate captions that include recognizable location names instead of just coordinates or addresses to help make the diary entries much more readable for users. As a result, I refined the daily summary view to incorporate these location names into the timeline visualization. When a user views their daily log, they can now see their chronological list of visits with the actual place names.

I am currently on schedule with the web app development. The auth system was completed earlier than expected and the Google Maps API integration was a planned feature that I was able to implement without major obstacles.

Next week, I plan on completed the settings interface that allows users to configure the capture prefs and continue enhancing/adding features to the front end.

As Will and Dhurv make progress with the Blues Notecard and GPS functionality. I’ll prepare to integrate the location data coming from the actual Arducam images.

Following the team’s decision to shift away from AWS, I refactored the Django backend to use SQLite for local database storage instead of the previously planned AWS RDS. I also modified the image storage system to use the local filesystem rather than S3 buckets, implementing a directory structure that organizes images by user ID and capture date.

Additionally, I worked on implementing the server-side components that will receive and process the image data from the Raspberry Pi Pico. Since Dhruv successfully established serial communication between the laptop and the Pico, I created the necessary endpoints to receive this data and integrated it with our local storage solution. I built a simple webhook listener that can accept the camera captures and trigger the necessary processing.

I also made significant progress on the user interface components. I completed the implementation of the calendar view with proper highlighting for days that contain captures, and refined the daily summary view to display images in chronological order. I added location visualization using a local map implementation that doesn’t rely on cloud services.

For the summary generation component, I researched and implemented a solution using a locally deployable LLM instead of the OpenAI API. I tested several options and found that a quantized version of Llama-3 can run on our development machines with acceptable performance for generating the diary summaries based on image metadata and location information.

I am currently on schedule with the web application development, despite the pivot away from AWS. The local development approach actually simplified some aspects of our implementation, removing dependencies on cloud configuration and allowing for faster iteration. The core functionality of the web interface is now working in our local environment, which aligns with our timeline.

Next week, I will focus on integrating with the hardware components as they become available. I plan to complete the communication layer that will receive images from the Pico once Dhruv transitions the serial connection to MQTT. Additionally, I’ll try to implement the location data processing system that will work with the GPS coordinates from the Blues Notecard that William is setting up.

Overall, the shift to local development hasn’t impacted our schedule significantly, and I remain confident that we can deliver a functional proof of concept on time.

This week I made progress on the backend integration of our database models. I completed the implementation of the endpoints that will receive image and location data from the hardware components. I’ve tested these endpoints using mock data to simulate the incoming information from the Blues Notecard and verified that they correctly sore the data in our database models.

Since Will was able to resolve the connection issues, I worked on implementing a queuing system that can process incoming images. Additionally, I added error handling for cases where image uploads might fail.

My progress is now on schedule, catching up on the slight delays from last week after resolving the connectivity issues of the Pico.

For the coming week, I plan to complete the user authentication system, including account creation, login, and profile management.  I also plan to implement the settings interface that would allow users to configure the glass and image taking parameters.

I’ll also be working closely with Dhruv as he transitions to help with the hardware side to ensure that software backend remains compatible with any changes.

This week, I focused on our web app dev. I spent more time researching and implementing alternative cloud solutions after learning about the AWS reimbursement limitations. I’ve explored the Google Cloud Platform as our backup option, setting up an initial testing environment to ensure that we can transition our project easily if needed.

For the web app itself, I’ve completed the initial Django project setup with basic URL routing and view templates. Additionally, I have also completed the first version of the calendar view interface which will be our main entry point for users to view in their daily summaries.

I also started working a bit on the integration with Google Maps API for location display. I’ve successfully implemented a basic map component that can display location markers based on coordinates. This will help us provide context to the captured images in the daily summaries.

During testing image compression, Dhruv found out about the some of the limitations of the Blues Notecard’s file size. Given these challenges, I’m researching on solutions on the server-side to potentially handle fragmented image data and then efficiently reconstruct the complete images and store them into our database.

My progress is slightly behind due to having to spend some time researching a new cloud infra in case we may need to pivot our project there. However, I can catch up by working on the frontend and backend dev simultaneously.

Below I’ve attached a screenshot of our UI mockup overview and database schema designs thus far.

This week, I focused on web application development and cloud infra setup. I was able to build an initial Django backend structure and setup basic database models for storing user data, images, and location information. I did more research and documented our approach for AWS S3 integration for image storage. Additionally, I started to work on the database schema so that we have an efficient design for image retrieval and location data storage.

From a web design standpoint, I created some initial UI mockups using Figma. We plan on reviewing these mockups as a team and make any adjustments as needed. The main struggle here has been creating the calendar view page, as this page will be the most interacted with.  How should the functionality be? What data should be presented with users? Both of these questions play a big role into what the UI will appear like.

According to our schedule, we are slightly behind on the web app development. The delay is primarily due to waiting for the hardware components to arrive for full integration testing. While a lot of the basic setup is ready, I plan to catch up by: creating mock data endpoints to simulate hardware integration, parallelize the frontend and backend dev, and set up the AWS infra ahead of the hardware arrival.

Next week, I plan on completing a basic setup of the AWS S3 bucket, implement the initial version of the calendar view interface, and create some basic API endpoints for our location data processing.

Along with all the above, I also collaborated with Will and Dhruv on the design presentation and documentation.

This week, I focused on researching and planning the web application architecture and database design.

Frontend Research and Planning:
I primarily evaluated Django’s server-rendered HTML/CSS approach for the web interface, researched responsive design patterns to ensure 30-second page load time, started planning the calendar-based interface for daily entries, and began sketching some initial UI wire frames for the daily summary view.

Database Architecture:
Regarding the database, I investigated a schema design for storing the user diary entries, associating images with timestamps, location metadata from Google Maps API, and LLM-generated summaries. I also researched heavily on efficient storage mechanisms and retrieval patterns for image data using AWS S3.

LLM Integration Planning:
I explored AWS Rekognition API capabilities for image processing and started to work on an initial design for the data pipeline for processing daily images and generating summaries. I also looked into approaches for combining images analysis with location data for better context in the summaries.

The main challenge that I am anticipating from my research this week is optimizing the database queries to meet our performance requirements, especially when fetching and displaying multiple images and summaries for a single day’s view. I anticipate we will need to implement some kind of efficient caching system to address.