Tag: status report

We finally have a decent replacement camera, not the camera module 3 which is still not here, but an Arducam IMX519 Raspberry Pi camera which meets all the requirements we had except the IR functionality which won’t affect our demo. As such, at this point, the biggest risk is the time crunch and to finish all deliverables on time.

No changes were made to the existing design of the system other than the use of this new camera. It was needed since our raspberry pi camera module 3 is not here and our demo/final deliverables are coming up. The costs are minimal since the camera is easy to replace in our system pipeline, and did not cost too much money.

Currently the next things to do would be to integrate the new camera and fully test its capabilities. In addition to that, work is going to be needed in making our poster and final deliverables. 

Testing we have done:

Unit tests:

• Cooling and power supply
  • We ran the raspberry pi for 55 minutes with the full implementation
  • We saw the temperature stay below 75C, avoiding thermal throttling
  • Power supply kept our device stable the whole time
  • Both met our requirements
• ML Models
  • We evaluated the model pipeline on images at different distances (5m, 10m) and weather/lighting conditions (day, night, rain, night+rain) of ideal image quality
  • We got a 98.6% precision and 79.1% recall, meeting our 90% precision requirement but not our 85% recall requirement
  • We will adjust with cutoff confidence levels to try and trade some of the precision for recall to meet both requirements
• GPS Accuracy:
  • We got GPS location at 10 locations we know the true coordinates of outside (ex. CMU Flagpole) and compared it with the GPS reading at these locations
  • All results were in 31m, well within our 200m requirement
• Edge-Cloud Communication:
  • We ran 15 simulated matches on the raspberry pi, which should show up properly in supabase and 15 updates to the database should be added to the Edge database
  • This should update 100% of the time which we observed in our testing

 

System tests:

• Timing requirement:
  • We ran the raspberry pi for 55 minutes with the full implementation
  • All images were checked within our 40 second requirement
• Full system performance test:
  • We evaluated the model pipeline on images at different distances (5m, 10m) and weather/lighting conditions (day, night, rain, night+rain) of our camera’s quality (the old webcam at the time)
  • 64.7% precision and 23.6% recall observed, well below our 90% and 85% targets respectively
  • We have a new camera to test, which should have much better results

 

As per last week, the camera we ordered still hasn’t arrived. While concerning, our MVP camera will still serve us well, as even though it doesn’t have IR capabilities or a very high resolution sensor, it is still more than good enough to demonstrate the functionality of the project as a whole.

There are no changes to the overall design of the project. However, we have made a slight change with using AWS Rekognition rather than Sagemaker for verifying low confidence matches. This was done since AWS Rekognition is specifically designed for OCR which is our main use case of the cloud model, which also makes it easier to work with. The costs this incurs are minimal since we did not waste time or resources on the Sagemaker portion other than basic testing. As for schedule, we are currently testing both the ML and camera right now, with ML tests ongoing and camera tests taking place tomorrow before the slides are assembled.

During demos, we verified that we already have a working and functional MVP. The risk is that we need to verify that the replacement camera we bought can work with the system properly, and testing for this will begin as soon as the replacement arrives. However, this is a small risk, as the previous one functioned already, and our backup is sufficient for demonstrating functionality. 

There are no changes to the design, as we are in the final stages of the project. Only testing and full Sagemaker integration remain. 

On the validation side, we plan to test the fully integrated system once we receive and install the replacement camera. We’ll run tests using real-world driving conditions and a portable battery setup to simulate actual usage. We will also test in various lighting conditions. 

More specifically, while we have not run comprehensive tests yet, our initial testing on the timing requirements as well as the database matching are all meeting our requirements of 40 seconds and 100% accuracy, respectively. To test these at a more comprehensive level, we will run 30 simulated matches with different images to make sure all are within the timing and match requirements. Once we receive the camera we will use in our final implementation, we will take images of varying distances and weather/lighting conditions, and test the precision and recall of our whole pipeline. These images will also be put into platerecognizer.com, a commercial model, to see if it is our models that need improvement or the camera. The details of these tests are the same as what we have in the design report. Finally, we will either run this system in an actual car, or take a video of a drive with the camera and feed this video into the pipeline to simulate normal use, and make sure it detects all the matches of the video. 

In the last two weeks,  we finalized our MVP and are working on additional features as well as testing.