Author: sczeng

Below are some of the tests that our team conducted:

1. Image Classification Accuracy Tests: We had a dataset of 500 compiled images of all forms (white background, normal background, and blurred background) and tested the classification accuracy of it. The results fell into one of four categories: canned food, bananas, oranges, and apples. However, our system will be able to classify into other groups if needed. The purpose of the four categories was to simplify the confusion matrix into a smaller form. The accuracy was around 89%; however, we hope to aim for around 95%. The design changes that we will make to improve the accuracy involve background reduction to make the image consist of only the primary object.
2. Image Classification Latency Tests: We had a dataset of 500 compiled images of all forms (white background, normal background, and blurred background) and tested the classification speed of it. Our goal is for it to take around 2-3 seconds. The tests took 4-5 seconds on average which is not too far from the final goal. Our primary focus is on accuracy, but once we get accuracy to the ideal threshold, we will work on optimizing the classification algorithm (potentially removing layers).
3. Text Extraction Accuracy Tests: We used our sample of 125 canned food images as well as an additional 75 pictures that we took to create a testing set of 200 samples. We had two focuses: calorie extraction and product label extraction. The result indicated a 92% accuracy in obtaining the right calories which we ideally want to be above 96%. The primary issue is that sometimes the text extraction algorithm formats it into a weird way in which the calorie amount does not follow the word “calorie.” As a result, we plan to tweak our text extraction algorithm to take care of edge cases like that. In addition to that, we were able to extract the product label 98% of the time. The issue is the outputted text includes too many extraneous characters. We will work on a parsing algorithm to make the label clear and concise for inventory documentation purposes.
4. Text Extraction LatencyTests: Using the 200 samples, we reproduced a score of 3-4 seconds for the whole calorie and product label extraction process. The results were skewed a little by the fact that some tests took exceptionally long trying to find the calorie amount. We expect to see this value to decrease to 1-2 seconds after tweaking the calorie extraction algorithm. Our goal is 2-3 seconds, so this gives us some buffer in adding a product label sanitization algorithm to make the product label clear and concise rather than a blob of all the text.
5. RPi and MySQL Connector Latency Tests: We used a set of 100 image captures from the RPi (200 in total: 100 for product images and 100 for scale readings) and timed the time it takes from sending to being accessed via the web application. Our ideal time range includes 10-15 seconds. However, the results demonstrated on average 22.4 seconds out of all our samples. We hope to make some design choices that involve synchronizing when the first and second camera takes image captures to improve the latency. Likewise, we can reduce the polling time it takes to stabilize the image to speed up this process because the quickest speed from one training sample is 16.5 seconds. We found that it does not take too long for the product and weight to stabilize, so a slight alteration in the Python script will improve latency tremendously.

Regarding progress, Steven was able to debug and fix our ML classification model with ResNet-18. There were many bugs with integrating it into the front-end, so Steven was able to tweak it to run properly and output the classified results. Before it was not outputting any result, so Steven was able to modify it to be able to process images correctly. Likewise, he was able to improve our background reduction algorithm to make the RPi image clearer and easier to classify due to less external features in the database entry. This ultimately improved accuracy to around 89% which is roughly similar to our desired goals. He will work to fine-tune it more to get the accuracy up to 95%.

Grace and Steven also conducted more testing and validation regarding the ML components which included a lot of unit test with our compiled data that we had from previous weeks. These tests will be talked about more in the team status report, but we did speed and accuracy tests on the ML algorithms for image classification and text extraction. Most of these tests were done locally first with stock images to ensure quick and efficient testing. The next step is live testing with live pictures from the RPi on the website.

Surya was able to get a high accuracy on the Seven Segment Optical Character Recognition that effectively establishes communication between the scale and the MySQL database. The extracted weight would be forwarded to the database to be written and also used for backend calculations. This was after failed testing with the Tesseract OCR since it has trouble extracting digital text fonts. He plans to work on fine-tuning this algorithm with the remaining time to work across broad testing inputs.

Grace spent most of her time improving frontend components involving user experience, performance, and functionality. Since most efforts were shifted towards creating the database retrieval system in MySQL, improvements on the design and interactive sides of the website as it could initially be overwhelming to navigate all the different pages. I am thinking instructions could be included somewhere to help users with the functionalities. Another issue that emerged after integrating the subsystems together was providing enough feedback and error states/messages to users after their actions such as scanning the product with the camera. It is crucial to provide helpful error messages to immediately help users identify their incorrect behavior and receive necessary feedback as soon as possible to improve upon.

Likewise, we all worked closely to deploy the web application on Nginx to create a MySQL database to be connected with the RPi using a MySQLConnector.  This involved work with coordinating between two USB cameras and testing with one vs two RPis. Likewise, we coded python scripts to take images of both the object and the digital scale output.

This week we had a wide variety of tasks that we needed to complete. The first involved the final presentation. We all worked to combine aspects of our previous presentations in addition to our new testing results and design tradeoffs. This involved collection of data and recording videos for the presentation inclusion to capture our team’s current progress thus far.

We also focused our attention on forwarding data from the rpi cameras to the database. We will need to record both the image of the product as well as the scale reading. These recordings will be stored in a MySQL database which we access through the cloud using Nginx cloud deployment. These tasks involved a lot of python code on the rpi and work configuring the rpi to properly write to the database. Since we have only recently focused our efforts on the scale integration, we came up with a comprehensive strategy in regards to its testing. Some calibration was done to ensure the scale provides accurate readings. Another question emerging was whether or not it could easily detect small changes in weight or not. Sensitivity is a big component to consider since we want to minimize latency between weighing and transferring the data to our user interface.

Another area where we concentrated significant effort was in the ML portion of our project. We tested ML classification in both the website end as well as the rpi end. Furthermore, we integrated ResNet-18 instead of GoogLeNet to improve performance. This involved a lot of configuration details that we had a lot of errors in. Likewise, we needed to fine-tune this classification algorithm to make it classify into the four primary groups. In addition to that, we coded up some functions to alter images to reduce the effect of the background. This involved some background subtraction code and experimentation using Mask R-CNN to extract instances of images in the picture. In our case, there is only one image in the photo, so Mask R-CNN will solely extract the pixels for that while ignoring the background. There are still some bugs in the implementation which we are currently debugging, but we hope to finish a working algorithm which would greatly improve the accuracy.

In addition to classifying canned foods and fresh produce, we also implemented 7-segment display OCR to streamline the process of digital capture of the digital scale reading. The camera inside the box had to be displayed in a manner where it could get a clear, unobstructed view of the reading. In a similar process to the previous ML classification, image preprocessing was done including conversion to black and white and cropping the image to focus on the weight reading only. This ensures the images to all be similar besides the actual reading. Currently, we do not have the best accuracy results with using Tesseract but we will conduct further testing with various light conditions to verify that this subsystem correctly interfaces with the web application. A possible solution is to experiment with the different page segmentation modes to treat the images as a single line of text (i.e. ‘–psm 11’). Of course, good lighting will also play an important role in ensuring the text to be as legible as possible without any shadowing and reflections.

Last but not least, we worked a little on the front-end to better display the data. We implemented the infrastructure of the code to access values in the MySQL database and query based of username. We were able to finish the code to create tables, lists, and graphs as needed in the web application. We hope to have all this connected and working by end of next week with the rpi and hardware. We shoot to have a working, fully-integrated product that resembles our MVP by end of next week.

To start the week off, we were able to make good progress in showcasing our project’s individual components (Rpi integration, Arducam, scale, physical box design, website ML functionality, SQL database access, and retrieval) for the interim demo. These were all separately shown during the demo.

In the context of validation, this is an evaluation that is more system-centric and requires us to analyze how well the final product matches our problem statement and proposed approach. Some of the system-wide approaches we plan to consider include the following:

1. 1. Image forwarding: How well does the RPI forward information to and from the mySQL database? With a 30-second overall latency requirement, image capturing and forwarding should be done in a timely manner (which also means the first-level classification performed on the RPI should be rather light and efficient given that an exact match is not required for that stage of the design)
   2. Concurrency: Principles of ACID and two-phase locking implemented in the views of the web application are important to ensure safe database access and retrieval. From our coursework, we have learned that the database is the most vulnerable component regarding system failure. Other than data sanitation and validation (which are separate concerns), we have to regulate access to elements in the database that are constantly being deleted and re-inserted. Particular edge cases include handling duplicates (eating the same thing multiple times in the day) and ensuring the DB generates new entries for each object.
   3. Tesseract OCR: There are still accuracy issues persisting with text recognition in relation to the image quality, so we have to perform more testing under different conditions for real-time processing scenarios. More diverse set of images can be used and calibrated to see how the text is best extracted in the most accurate way possible such as grayscale conversion, image cropping, font sizes, and background. By systematically testing Tesseract performance on these various images, we are aiming to find optimal settings to achieve the most accurate text extraction for reading food labels. In doing so, we aim to enhance the precision of our text recognition for this real-time application.
   4. Web App User Interaction: How can we ensure all of the user features will work as expected? Performance issues will be tested to access the web application’s responsiveness and scalability, where bottlenecks like slower queries and inadequate indexing are even more important to identify with the inclusion of data forwarding from the camera and scale readings. Upon these inclusions, usability regarding navigation and the transition between scanning and using the application is important to note for user’s ease of use and navigation between different functionality tabs.
   5. ML and Camera Integration: How accurate and quick will ML classification using the Rpi with RasNet-18 model be? We will test this using a testing dataset as well as the time function in the Rpi. Likewise, we will compare this with the results of classification from the website backend to determine which method is better and more efficient. Furthermore, we need to test the image quality using either a USB camera or an Arducam to determine the best fps to achieve maximum accuracy in terms of classification.
   6. Proper Database Retrieval and Storage: How will data (usernames, caloric intake, inventory items) be stored properly in the database once scanned and how can we properly display it in the front-end? We have pages that will graph caloric intake as well as display caloric intake at a daily granularity in the form of a list. We need to make sure that the items and information is properly stored based off the logged-in user to prevent security issues. Likewise, we need to make sure that the users can only modify the database by scanning and/or removing objects directly from the inventory list. We will do extensive testing to prevent SQL injections, javascript attacks, and X-site scripting attacks. Likewise, we will use the browser console to detect irregularities and potential security concerns using error and DB retrieval statements.

Summary of individual work
Surya continued his work on RPI and camera integration. Despite challenges with operating the RPI in a headless setup, including slow SSH connections and lag in camera performance, he successfully implemented a rudimentary algorithm to capture images with 80 percent confidence for classification. The next steps for him involve improving camera performance with a headed monitor setup and integrating image capturing into the database, along with exploring options for classification performance. He thanks the 18500 teaching staff for their feedback on the hardware behind this project during the interim demo with specific regards to camera performance and options.

Grace included additional functional pages on the web application in accordance to feedback and improving the general user experience. These pages include but are not limited to a chat function between users to discuss fitness related goals, a calorie consumption progress graph demonstrating user’s daily improvement, adding caloric entries by date and displaying by day (eventually connected to the hardware components of classification), and a page allowing users to set a daily calorie limit to measure necessary improvements.  She plans to continue working on the frontend and ensuring they match how the user can use the application in an efficient and beneficial way. Also testing for each page that includes user modifications or uploads that could be vulnerable to security attacks.

Steven continued working on his ML features and working with Grace to integrate them into the web application. Tesseract OCR components were added into views.py to extract food label texts from the user’s uploaded images. Additional work will need to be done to improve accuracy, but images were turned into gray-scale and enhanced through enlargement of the original images themselves. During the individual testing phase, these improvements were able to enhance text extracting results. The next step would involve incorporating Surya’s camera subcomponent into capturing these OCR readings and fully integrating it onto a web application page. Steven also included a new model to store entries in a database where users are inputting food items and their respective caloric values. More improvements will be done to include individual food inventories for the user that is currently logged into the application.

As a team, we enjoyed the focus on ethics this week. We enjoyed the pod discussions and began to consider things that we would have never thought of. A big issue we identified with our product involved both body image concerns as well as privacy issues. We want our product to promote a healthy lifestyle, but we do not want our users to develop eating disorders and other personal issues. Likewise, we do not want private data on the consumption habits of our users to be leaked to other users as well as companies who participate in targeted advertising. As a team, we were able to discuss fixes to these issues including database security design and user-friendly/encouraging dialogue on the front end. Despite being done with discussions on ethics, we plan to keep all these principles in mind as we further our design.

We found it interesting in the pod discussion that two other groups had a product of similar functionality to ours. While one group anticipated projecting ingredients and recipes onto a tabletop using a projector at a calculated angle, the other is using AI to generate recipes on a phone app from ingredients added to their food inventory. We were able to clearly address that the MVP for our system would only be able to classify canned foods and certain types of fruits as opposed to cooked foods or ingredients in a bowl. The biggest difference between our product and these is that ours is used as a calorie tracking device and more focused on physical wellness, so more ethical concerns arise with this focal point. This will definitely be a greater consideration of ours while working on user interface integration and user experience with our web application.

The parts for scale integration have steadily come in through the ECE delivery center, so Surya began work on understanding the hardware layout of the scale and assessing the 2 main approaches to writing scale measurements to the database. He is waiting on the RPIs to assess the camera approach to read scale values. Another option could be to start on this with the Arduino chip that came in the mail already, but typically, Arduinos are not great choices for image processing because of limited on-board RAM and limited functionality with other cameras (RPIs are fantastic for such applications and several resources exist online for subsequent support). Additionally, he plans on working with Steven and Grace in sculpting a presentation strategy for the rapidly approaching interim demo.

 

In the meantime, he has also learned more about how the load cell amplifier works and the wiring topology. An important thing to do when working with a functional scale is to ensure that the correct wires are snipped and soldered; a wiring schematic can be found below for the reader’s convenience:

 

Steven did a lot of work patching up the ML infrastructure of the project. He optimized the accuracy of the various components. The first optimization was done to the classification algorithm for canned foods and fruit using the AdaBoost algorithm to combine multiple decision boundaries. The second one involved classification within the groups of fruit using k-nearest neighbors. This was combined with GoogLeNet and OpenCV to produce better results more specific to our project. Lastly, the ChatGPT API does not need optimization, but Steven worked with integration into the front-end. He plans to work alongside Grace in the upcoming weeks to test the basic functionality and syntax of the API to perform label reading and classification, if needed.