This week we had a wide variety of tasks that we needed to complete. The first involved the final presentation. We all worked to combine aspects of our previous presentations in addition to our new testing results and design tradeoffs. This involved collection of data and recording videos for the presentation inclusion to capture our team’s current progress thus far.
We also focused our attention on forwarding data from the rpi cameras to the database. We will need to record both the image of the product as well as the scale reading. These recordings will be stored in a MySQL database which we access through the cloud using Nginx cloud deployment. These tasks involved a lot of python code on the rpi and work configuring the rpi to properly write to the database. Since we have only recently focused our efforts on the scale integration, we came up with a comprehensive strategy in regards to its testing. Some calibration was done to ensure the scale provides accurate readings. Another question emerging was whether or not it could easily detect small changes in weight or not. Sensitivity is a big component to consider since we want to minimize latency between weighing and transferring the data to our user interface.
Another area where we concentrated significant effort was in the ML portion of our project. We tested ML classification in both the website end as well as the rpi end. Furthermore, we integrated ResNet-18 instead of GoogLeNet to improve performance. This involved a lot of configuration details that we had a lot of errors in. Likewise, we needed to fine-tune this classification algorithm to make it classify into the four primary groups. In addition to that, we coded up some functions to alter images to reduce the effect of the background. This involved some background subtraction code and experimentation using Mask R-CNN to extract instances of images in the picture. In our case, there is only one image in the photo, so Mask R-CNN will solely extract the pixels for that while ignoring the background. There are still some bugs in the implementation which we are currently debugging, but we hope to finish a working algorithm which would greatly improve the accuracy.
In addition to classifying canned foods and fresh produce, we also implemented 7-segment display OCR to streamline the process of digital capture of the digital scale reading. The camera inside the box had to be displayed in a manner where it could get a clear, unobstructed view of the reading. In a similar process to the previous ML classification, image preprocessing was done including conversion to black and white and cropping the image to focus on the weight reading only. This ensures the images to all be similar besides the actual reading. Currently, we do not have the best accuracy results with using Tesseract but we will conduct further testing with various light conditions to verify that this subsystem correctly interfaces with the web application. A possible solution is to experiment with the different page segmentation modes to treat the images as a single line of text (i.e. ‘–psm 11’). Of course, good lighting will also play an important role in ensuring the text to be as legible as possible without any shadowing and reflections.
Last but not least, we worked a little on the front-end to better display the data. We implemented the infrastructure of the code to access values in the MySQL database and query based of username. We were able to finish the code to create tables, lists, and graphs as needed in the web application. We hope to have all this connected and working by end of next week with the rpi and hardware. We shoot to have a working, fully-integrated product that resembles our MVP by end of next week.