Category: Richard’s Status Reports

This week, I worked with Eric on the verification of low-confidence matches through the AWS SageMaker platform. This included setting up the AWS service and environment and deploying the models to the cloud. During this process, we also learned about Rekognition, an AWS service specifically for image recognition, so we also looked into this service as a possible better option. Last week, I worked on polishing the website so that the interface was clean throughout and bug-free, especially with the interim demo. I also implemented and debugged the code for our GPS in our pipeline, and that aspect is working smoothly for us.

My progress has been on schedule. By next week, I hope to have made a decision and implemented the cloud verification system, and run tests on our entire pipeline.

This week, I worked on debugging the camera with Eric and verifying the functionality of everything else. We tried many troubleshooting methods on both our camera module 3 and the backup arducam camera. However, no matter what we did we could not get the camera module 3 to detect and take pictures. The backup camera does detect, but would only take extremely blurry pictures of weird colors, such as blue and pink despite facing a white wall. Ultimately, we decided to move forward with a usb webcam Tzen-Chuen already had for demos. Outside the camera, mostly everything else was tested to see if it ran smoothly. The whole pipeline from image capture (simulated with just a downloaded image file on the raspberry pi) to being verified as a match was tested and worked within timing requirements. In addition, I fixed some bugs in our website such as images not displaying in fullscreen properly. The link is the same as the previous week. 

Outside of the camera issues, my progress has been on schedule. Since we don’t think putting the camera into our pipeline is much of an issue once we have a working one, we don’t expect this to delay much. By next week, I hope to add a working camera to our pipeline and then work on the sagemaker implementation with Eric.

This week I focused on finalizing the locally run code. I added the queue functionality where if a match cannot be sent up to the database for whatever reason, such as loss of internet connection, the data will be saved locally, and it will be retried later. I have also written a main function that will take a picture and send matches every 40 seconds, check for database updates every 200 seconds, and retry failed entries to the database every 200 seconds. Due to some problems we have with initializing the camera, that is the only aspect of the locally run code we do not have working right now. The code can be found here. In addition to the locally run code, I have made a prototype for the front-end website using lovable for law enforcement to look for matches. They can see the currently active amber alerts, possible matches sent to the database, and verified matches where the larger models on the cloud confirm the result of the possible matches. It also has useful filtering options such as filtering by confidence level or license plate number. The website can be found here.

My progress is on schedule. By next week, I hope to implement the verification process of possible matches into verified matches with Eric, and have an MVP.

This week I focused on the integration between the Raspberry Pi and the cloud.  I worked on creating the database update code, where a Python function will retrieve the latest version of the Amber Alert database from the supabase and update the locally saved database with it. If the Raspberry Pi does not have an internet connection, it will simply continue using the locally saved version.  I also wrote the code to check for matches in the database. A loop checks the OCRed text with the database and if a match is found, the original image is uploaded to a supabase bucket, and an entry is made into a “possible matches” table for further verification by the cloud models. We will later integrate Eric’s edge function to move more processing to the cloud. The updated code can be found here.

My progress is on schedule. By next week, I hope to finalize the locally run code and work on the edge function that should run when a match is uploaded to corroborate with larger models.

This week I worked on deploying the ML models to the raspberry pi. This consisted of setting up the python environment, converting the Jupyter notebook into a standard python file, and setting up the file structure the raspberry pi will use. Since the notebook displays the bounding boxes and images when inferencing, when converting to a python file, I removed this code for faster performance since the end user will not see this anyway. I tested this implementation with a sample image that had two license plates in plain view. This is the same image used when testing the Jupyter notebook in Google colab. The program ran in just over 23 seconds, which should be plenty fast enough for our 40 second timing requirement. The models I used were the NCNN models but no quantization was used, so this number can be easily lowered further if needed. The code can be found here. When setting up the file system, I put the pictures and models into their own folders to easily switch between models and the images I test. Last week I worked on the design report, where I focused on the system implementation as well as the design trade studies.

My progress is on schedule. By next week I hope to finalize the MVP of the dashcam side of things, and shift focus to setting up the cloud

This week I worked on the presentation with Tzen-Chuen and Eric, especially with regard to the details of the implementation, such as using YOLOv11 and AWS Lambda in our final design. In addition, I worked with Eric on setting up the pipeline of YOLOv11 to PaddleOCR. I made two versions of the pipeline, one that first detects cars and crops those images, then into license plate detection and cropping, then finally PaddleOCR to read the license plate. The second one does not do the initial car cropping and goes straight into license plate detection. The google colabs can be found here and here. I also did some more research on how to deploy the models to the raspberry pi, and found that we should use NCNN models. For our mvp, we will use a python script that I am working on running on a headless os using the optimized models. As soon as the camera arrives, we should be able to make a basic MVP excluding the cloud server.

My progress is on schedule. By next week, we hope to have a dash cam module MVP and get metrics on the initial performance of the device.

This week I worked more on the YOLOv11 model as a possible model for license plate detection. I set up a workflow to make training these models very easy on the ECE machines so that we do not waste our budget. After doing this, I trained the YOLOv11n model for both 100 and 400 epochs to get a good baseline model for testing on our raspberry pi and getting to MVP as quickly as possible. The models can be found here. The precision of the 400 epoch model is 0.984 and the recall is 0.964 on the validation dataset. After training, I researched the best model file types to which I should export my trained model, and I discovered that TensorFlow Lite is a good option, especially for edge devices. After some technical problems, I was able to export the Pytorch-trained model to ONNX and then to TFLite. I tried to export with INT8 quantization to further improve performance on an edge device, however, I faced many difficulties and was not able to do it.

My progress is on schedule. Next week I hope to load the model onto the Raspberry Pi and test its performance metrics on the device, specifically on how long inference takes, so that I can figure out if I want to use a different size model or if I need to get quantization working.

This week I worked on the presentation slides, researching amber alert data/statistics, competitors such as Plate Recognizer, and the requirements for our project, as well as adding detail and polish throughout the presentation. After some research into how others have implemented license plate detection, I worked on investigating the feasibility of using YOLOv11 as a base model for license plate detection, since it is popular in the space, and have looked into ways to fine-tune the model for this use case. I have made a Jupyter Notebook file that will tune the model for license plate detection using an open-source dataset I found online, and I have explored ways to have this training run on sites like Google Colab and Kaggle, as well as on the CMU ece machines. Right now I am learning toward using the ece machines due to the strict GPU restrictions with using google colab and kaggle for free. The link to the jupyter notebook is here.

My progress is on schedule. Next week I hope to have completed training on the YOLOv11 nano model on the dataset for around 100 epochs and get some preliminary data on its performance. I would also like to find out how to load the model on a Raspberry Pi and get some metrics on how fast this trained model will run on the device.