As of now, I finished integration _4, which houses everything that the pathing code and the LLM model asks for. Right now, I’m waiting for Krrish to finish implementing depth, and seeing if I can improve on the current code.
Opalina’s Status Report 4/26
This week, I created a quantized YOLO model and tuned it in order to increase speed of the model on the Pi. I also rewrote an integration script, making optimizations with the OCR intervals and threading in order to reduce the inference time on each frame when running the end-to-end integration script on the Pi.
For unit tests, I ran the video processing script on a set of manually recorded videos (using printed signs) as well as pre-existing image datasets (117 airport signs) and reached an accuracy of approximately 92% with <100ms of preprocessing and inference time on the Mac. Eventually, the new script yielded a latency of <2s on the Raspberry Pi, which met our initial use case requirements.
Team Status Report 04/26/25
We’re working on integration and trying to get everything running smoothly on the pi. We’ve run many individual tests and so far we’re meeting our specification requirements. We had 90% accuracy for the model on the test image dataset. Our battery life was between 4-5 hours. The device weighs under 1kg. Currently we’re able to run at about 10FPS which meets our 2s latency spec, however, we have some more integration to do and hope that we can continue to meet the specification.
Krrish’s Status Report 04/26/25
I’m working on integration and optimising the script to run locally on the pi. We’re running into some issues as ocr is very computationally heavy. I’m testing various techniques like threading, running it on intervals, etc. I also tested the depth accuracy for the camera system.
Daniel’s Status Report 4/19
I worked on finalizing the integration of the LLM and the Directional Guidance and pathway coding of the project. I worked on the already-existing directional code that we demonstrated in the demo. By this weekend, the LLM (which is done) and the pathway coding (almost done) will be complete. And hopefully, the device will be good enough to demo by the end of tomorrow.
Team Status Report 4/19
The biggest risk that could jeopardize the success of the project is if the AI Hat doesn’t integrate into the project. It is getting pretty late into the project, and we’re a couple days away from all the presenting and final demo. If it doesn’t work, we will have to make sure that threading and the new camera is enough to deal with the lag. Currently, there are no changes to the existing design of the system, but by this weekend we should have a working device, and we’ll be able to showcase at least some of that into the final presentation.
Krrish’s Status Report 04/19/2025
I’ve been working on the AI accelerator and trying to compile our custom models to run on it. I’ve gotten the software and drivers installed but am struggling with the optimisation and compiling steps involved. I’ve also been working on integration of the model and the raspberry pi.
One thing i’ve learnt is that we need constant communication and we need to start integrating early. Learning to communicate and give feedback in a team is important to keep everyone on track. It also ensures everyone is constantly working on integration so it’s not left to the end.
Opalina’s Status Report 4/19
This week, I achieved significant milestones in my project. I successfully integrated the entire software subsystem, which included retraining the YOLO model to recognize “gate text”. Additionally, I developed a new video processing script and integrated it with the speech-to-text interface. A pivotal improvement was transitioning the speech-to-text functionality to leverage the Whisper model by OpenAI, which demonstrated superior performance compared to the initial VOSK model.
Throughout this semester, I have deepened my understanding of machine learning models, their implementation, and their application in real-world projects. This experience has provided me with valuable insights into how different disciplines within Electrical and Computer Engineering come together to create cohesive projects—a perspective that was less emphasized in my earlier coursework. My learning approach evolved to prioritize online documentation, videos, and research papers over traditional textbooks and lecture slides, enhancing my ability to tackle complex technical challenges effectively.
Daniel’s Status Report for 4/12
We as a team met earlier this week to hash out our integration plan. Opalina was to send me a revised YOLO trained model which has updated signage recognition, and I was to integrate that model in my already-working LLM and OCR script. For this week, I finetuned and prepared my scripts so that when the models are finished training the integration will be as smooth as possible.
Team Status Report 04/12
The new camera arrived this week, which meant we had to start the process of hardware integration from scratch. The new AI hat, while easy to set up, is posing compilation issues with the custom ML models. While the text-to-speech and sign detection seems to be acceptably, this week, we hope to make more progress on integrating the camera with the software. Once we have a basic level of functionality, we hope to start user and battery tests to ensure that the device holds up to the initially outlined use-case requirements (accuracy, latency, power, depth perception etc)