As of now, I finished integration _4, which houses everything that the pathing code and the LLM model asks for. Right now, I’m waiting for Krrish to finish implementing depth, and seeing if I can improve on the current code.
Daniel’s Status Report 4/19
I worked on finalizing the integration of the LLM and the Directional Guidance and pathway coding of the project. I worked on the already-existing directional code that we demonstrated in the demo. By this weekend, the LLM (which is done) and the pathway coding (almost done) will be complete. And hopefully, the device will be good enough to demo by the end of tomorrow.
Team Status Report 4/19
The biggest risk that could jeopardize the success of the project is if the AI Hat doesn’t integrate into the project. It is getting pretty late into the project, and we’re a couple days away from all the presenting and final demo. If it doesn’t work, we will have to make sure that threading and the new camera is enough to deal with the lag. Currently, there are no changes to the existing design of the system, but by this weekend we should have a working device, and we’ll be able to showcase at least some of that into the final presentation.
Daniel’s Status Report for 4/12
We as a team met earlier this week to hash out our integration plan. Opalina was to send me a revised YOLO trained model which has updated signage recognition, and I was to integrate that model in my already-working LLM and OCR script. For this week, I finetuned and prepared my scripts so that when the models are finished training the integration will be as smooth as possible.
Daniel’s Status Report 3/29
We met today to start the integration process for the demo next week. I have already finished OCR and the UI, and now we are trying to combine them with the YOLO. I have coded the script with Opalina where the system takes the inputs from the OCR and YOLO, and translates it into directional guidance.
Daniel’s Status Report 3/22
This was a busy week. Throughout the days, I worked on the OCR aspect of the project. So far, I’ve been working on a script that allows for a repeat of words or instructions on the sign in a photograph. The output is a print statement so far, and the next thing to do is to work on the CV aspect of the project. Once both are done, I can integrate my LLM models and actually have the program repeat instructions.
Daniel’s Status Report 3/15
Over the week, I focused on getting OpenCV to work. Now that I’m getting sent the YOLO script for sign detection, I’ll integrate it into what I got, and produce an output that the LLM models can translate. Hopefully, this gets done over the next few days, Tuesday at the latest.
Team Status Report 3/15
Our most significant risk is the RPi not being sufficient enough to handle the camera input, object detection processing, and handling the output. The risk is that there will be latency issues, and the contingency plan as of now is to switch to offline LLM models to reduce the load on the RPi. Currently there have been no changes to the existing design of the system, and no changes to the schedule, as of right now we are on track.
Daniel’s Status Report 3/8
Spent some time focused on finishing up the tweaks on the LLMs that I mentioned that I was working on last status report. Also started to work with OpenCV to help with the object detection part of the project, since that requires a hefty workload which will need multiple people to get working. So far getting OpenCV to work has been difficult but I should make some solid headway through the next few days.
Daniel’s Status Report 2/22
As well as preparing for the slides and presentation for the Design Report, I have been working on the audio feedback part of the system. Audio integration setup has been done, and I’ve been working on implementing VOSK and Google TTS, and testing it myself. I’m in the process of creating a simple script that just repeats whatever I say back to me, which I’ll be able to present to my group on our meeting on Monday.