We as a team met earlier this week to hash out our integration plan. Opalina was to send me a revised YOLO trained model which has updated signage recognition, and I was to integrate that model in my already-working LLM and OCR script. For this week, I finetuned and prepared my scripts so that when the models are finished training the integration will be as smooth as possible.
Team Status Report 04/12
The new camera arrived this week, which meant we had to start the process of hardware integration from scratch. The new AI hat, while easy to set up, is posing compilation issues with the custom ML models. While the text-to-speech and sign detection seems to be acceptably, this week, we hope to make more progress on integrating the camera with the software. Once we have a basic level of functionality, we hope to start user and battery tests to ensure that the device holds up to the initially outlined use-case requirements (accuracy, latency, power, depth perception etc)
Opalina’s Status Report 04/12
This week, I used new training and testing datasets to fine-tune the YOLO model that we previously used for our interim demo. The new model looks for an added set of features in order to reduce the amount of OCR passes needed on each image, subsequently reducing our latency. I ran a few standard tests in order to verify the functionality of this new model, and found a slight drop in accuracy, which I am currently working to reduce. In the coming week, I plan to build new custom test datasets in order to properly identify these points of error.
Krrish’s Status Report 04/12
I assembled the cameras and raspberry pi into it’s 3d printed housing. I have stereo depth working and am now integrating with the YOLO model. I’m having trouble getting the model to compile so it can run on the accelerator. I need to install a lot of drivers on an x86 machine which is proving difficult as I don’t have one and need to go to campus. The software is also prone to a lot of issues which I am in the process of figuring out
Daniel’s Status Report 3/29
We met today to start the integration process for the demo next week. I have already finished OCR and the UI, and now we are trying to combine them with the YOLO. I have coded the script with Opalina where the system takes the inputs from the OCR and YOLO, and translates it into directional guidance.
Opalina’s Status Report 3/29
This week, I managed to run and fully test YOLO and OpenCV in conjunction with OCR, to extract and interpret all the relevant information from an airport sign. This script simply needs to be run with the TTS component in order to complete the end-to-end software subsystem. In the coming weeks, we hope to use camera footage to run the models on the Pi.
Krrish’s Status Report 3/29
I’m running experiments with the camera and YOLO model. It is struggling and working quite slowly with significant delays. I’ve put together a list of new camera options and have narrowed it down to pretty much 2 options. I will place an order on the one i think will work best.
Team Status Report 3/29
We have OCR and the YOLO model working independently. We are working on integrating that and also testing on videos rather than still frames. Our goal is to have this working on our laptops and then integrate it with the Pi. Simultaneously we are testing the camera on the Pi with the YOLO model to see how fast it would run. Currently it looks like the camera is running too slow on the pi and so we’re looking at different options. We’ve narrowed it down to 3 options and will place an order today.
Daniel’s Status Report 3/22
This was a busy week. Throughout the days, I worked on the OCR aspect of the project. So far, I’ve been working on a script that allows for a repeat of words or instructions on the sign in a photograph. The output is a print statement so far, and the next thing to do is to work on the CV aspect of the project. Once both are done, I can integrate my LLM models and actually have the program repeat instructions.
Krrish’s Status Report 3/22
I’m doing research into new camera that we can use since the old camera is not compatible with the AI accelerator. I’ve also made progress on the CAD model.