Category: Team Status Reports

Below are some of the tests that our team conducted:

1. Image Classification Accuracy Tests: We had a dataset of 500 compiled images of all forms (white background, normal background, and blurred background) and tested the classification accuracy of it. The results fell into one of four categories: canned food, bananas, oranges, and apples. However, our system will be able to classify into other groups if needed. The purpose of the four categories was to simplify the confusion matrix into a smaller form. The accuracy was around 89%; however, we hope to aim for around 95%. The design changes that we will make to improve the accuracy involve background reduction to make the image consist of only the primary object.
2. Image Classification Latency Tests: We had a dataset of 500 compiled images of all forms (white background, normal background, and blurred background) and tested the classification speed of it. Our goal is for it to take around 2-3 seconds. The tests took 4-5 seconds on average which is not too far from the final goal. Our primary focus is on accuracy, but once we get accuracy to the ideal threshold, we will work on optimizing the classification algorithm (potentially removing layers).
3. Text Extraction Accuracy Tests: We used our sample of 125 canned food images as well as an additional 75 pictures that we took to create a testing set of 200 samples. We had two focuses: calorie extraction and product label extraction. The result indicated a 92% accuracy in obtaining the right calories which we ideally want to be above 96%. The primary issue is that sometimes the text extraction algorithm formats it into a weird way in which the calorie amount does not follow the word “calorie.” As a result, we plan to tweak our text extraction algorithm to take care of edge cases like that. In addition to that, we were able to extract the product label 98% of the time. The issue is the outputted text includes too many extraneous characters. We will work on a parsing algorithm to make the label clear and concise for inventory documentation purposes.
4. Text Extraction LatencyTests: Using the 200 samples, we reproduced a score of 3-4 seconds for the whole calorie and product label extraction process. The results were skewed a little by the fact that some tests took exceptionally long trying to find the calorie amount. We expect to see this value to decrease to 1-2 seconds after tweaking the calorie extraction algorithm. Our goal is 2-3 seconds, so this gives us some buffer in adding a product label sanitization algorithm to make the product label clear and concise rather than a blob of all the text.
5. RPi and MySQL Connector Latency Tests: We used a set of 100 image captures from the RPi (200 in total: 100 for product images and 100 for scale readings) and timed the time it takes from sending to being accessed via the web application. Our ideal time range includes 10-15 seconds. However, the results demonstrated on average 22.4 seconds out of all our samples. We hope to make some design choices that involve synchronizing when the first and second camera takes image captures to improve the latency. Likewise, we can reduce the polling time it takes to stabilize the image to speed up this process because the quickest speed from one training sample is 16.5 seconds. We found that it does not take too long for the product and weight to stabilize, so a slight alteration in the Python script will improve latency tremendously.

Regarding progress, Steven was able to debug and fix our ML classification model with ResNet-18. There were many bugs with integrating it into the front-end, so Steven was able to tweak it to run properly and output the classified results. Before it was not outputting any result, so Steven was able to modify it to be able to process images correctly. Likewise, he was able to improve our background reduction algorithm to make the RPi image clearer and easier to classify due to less external features in the database entry. This ultimately improved accuracy to around 89% which is roughly similar to our desired goals. He will work to fine-tune it more to get the accuracy up to 95%.

Grace and Steven also conducted more testing and validation regarding the ML components which included a lot of unit test with our compiled data that we had from previous weeks. These tests will be talked about more in the team status report, but we did speed and accuracy tests on the ML algorithms for image classification and text extraction. Most of these tests were done locally first with stock images to ensure quick and efficient testing. The next step is live testing with live pictures from the RPi on the website.

Surya was able to get a high accuracy on the Seven Segment Optical Character Recognition that effectively establishes communication between the scale and the MySQL database. The extracted weight would be forwarded to the database to be written and also used for backend calculations. This was after failed testing with the Tesseract OCR since it has trouble extracting digital text fonts. He plans to work on fine-tuning this algorithm with the remaining time to work across broad testing inputs.

Grace spent most of her time improving frontend components involving user experience, performance, and functionality. Since most efforts were shifted towards creating the database retrieval system in MySQL, improvements on the design and interactive sides of the website as it could initially be overwhelming to navigate all the different pages. I am thinking instructions could be included somewhere to help users with the functionalities. Another issue that emerged after integrating the subsystems together was providing enough feedback and error states/messages to users after their actions such as scanning the product with the camera. It is crucial to provide helpful error messages to immediately help users identify their incorrect behavior and receive necessary feedback as soon as possible to improve upon.

Likewise, we all worked closely to deploy the web application on Nginx to create a MySQL database to be connected with the RPi using a MySQLConnector.  This involved work with coordinating between two USB cameras and testing with one vs two RPis. Likewise, we coded python scripts to take images of both the object and the digital scale output.

This week we had a wide variety of tasks that we needed to complete. The first involved the final presentation. We all worked to combine aspects of our previous presentations in addition to our new testing results and design tradeoffs. This involved collection of data and recording videos for the presentation inclusion to capture our team’s current progress thus far.

We also focused our attention on forwarding data from the rpi cameras to the database. We will need to record both the image of the product as well as the scale reading. These recordings will be stored in a MySQL database which we access through the cloud using Nginx cloud deployment. These tasks involved a lot of python code on the rpi and work configuring the rpi to properly write to the database. Since we have only recently focused our efforts on the scale integration, we came up with a comprehensive strategy in regards to its testing. Some calibration was done to ensure the scale provides accurate readings. Another question emerging was whether or not it could easily detect small changes in weight or not. Sensitivity is a big component to consider since we want to minimize latency between weighing and transferring the data to our user interface.

Another area where we concentrated significant effort was in the ML portion of our project. We tested ML classification in both the website end as well as the rpi end. Furthermore, we integrated ResNet-18 instead of GoogLeNet to improve performance. This involved a lot of configuration details that we had a lot of errors in. Likewise, we needed to fine-tune this classification algorithm to make it classify into the four primary groups. In addition to that, we coded up some functions to alter images to reduce the effect of the background. This involved some background subtraction code and experimentation using Mask R-CNN to extract instances of images in the picture. In our case, there is only one image in the photo, so Mask R-CNN will solely extract the pixels for that while ignoring the background. There are still some bugs in the implementation which we are currently debugging, but we hope to finish a working algorithm which would greatly improve the accuracy.

In addition to classifying canned foods and fresh produce, we also implemented 7-segment display OCR to streamline the process of digital capture of the digital scale reading. The camera inside the box had to be displayed in a manner where it could get a clear, unobstructed view of the reading. In a similar process to the previous ML classification, image preprocessing was done including conversion to black and white and cropping the image to focus on the weight reading only. This ensures the images to all be similar besides the actual reading. Currently, we do not have the best accuracy results with using Tesseract but we will conduct further testing with various light conditions to verify that this subsystem correctly interfaces with the web application. A possible solution is to experiment with the different page segmentation modes to treat the images as a single line of text (i.e. ‘–psm 11’). Of course, good lighting will also play an important role in ensuring the text to be as legible as possible without any shadowing and reflections.

Last but not least, we worked a little on the front-end to better display the data. We implemented the infrastructure of the code to access values in the MySQL database and query based of username. We were able to finish the code to create tables, lists, and graphs as needed in the web application. We hope to have all this connected and working by end of next week with the rpi and hardware. We shoot to have a working, fully-integrated product that resembles our MVP by end of next week.

To start the week off, we were able to make good progress in showcasing our project’s individual components (Rpi integration, Arducam, scale, physical box design, website ML functionality, SQL database access, and retrieval) for the interim demo. These were all separately shown during the demo.

In the context of validation, this is an evaluation that is more system-centric and requires us to analyze how well the final product matches our problem statement and proposed approach. Some of the system-wide approaches we plan to consider include the following:

1. 1. Image forwarding: How well does the RPI forward information to and from the mySQL database? With a 30-second overall latency requirement, image capturing and forwarding should be done in a timely manner (which also means the first-level classification performed on the RPI should be rather light and efficient given that an exact match is not required for that stage of the design)
   2. Concurrency: Principles of ACID and two-phase locking implemented in the views of the web application are important to ensure safe database access and retrieval. From our coursework, we have learned that the database is the most vulnerable component regarding system failure. Other than data sanitation and validation (which are separate concerns), we have to regulate access to elements in the database that are constantly being deleted and re-inserted. Particular edge cases include handling duplicates (eating the same thing multiple times in the day) and ensuring the DB generates new entries for each object.
   3. Tesseract OCR: There are still accuracy issues persisting with text recognition in relation to the image quality, so we have to perform more testing under different conditions for real-time processing scenarios. More diverse set of images can be used and calibrated to see how the text is best extracted in the most accurate way possible such as grayscale conversion, image cropping, font sizes, and background. By systematically testing Tesseract performance on these various images, we are aiming to find optimal settings to achieve the most accurate text extraction for reading food labels. In doing so, we aim to enhance the precision of our text recognition for this real-time application.
   4. Web App User Interaction: How can we ensure all of the user features will work as expected? Performance issues will be tested to access the web application’s responsiveness and scalability, where bottlenecks like slower queries and inadequate indexing are even more important to identify with the inclusion of data forwarding from the camera and scale readings. Upon these inclusions, usability regarding navigation and the transition between scanning and using the application is important to note for user’s ease of use and navigation between different functionality tabs.
   5. ML and Camera Integration: How accurate and quick will ML classification using the Rpi with RasNet-18 model be? We will test this using a testing dataset as well as the time function in the Rpi. Likewise, we will compare this with the results of classification from the website backend to determine which method is better and more efficient. Furthermore, we need to test the image quality using either a USB camera or an Arducam to determine the best fps to achieve maximum accuracy in terms of classification.
   6. Proper Database Retrieval and Storage: How will data (usernames, caloric intake, inventory items) be stored properly in the database once scanned and how can we properly display it in the front-end? We have pages that will graph caloric intake as well as display caloric intake at a daily granularity in the form of a list. We need to make sure that the items and information is properly stored based off the logged-in user to prevent security issues. Likewise, we need to make sure that the users can only modify the database by scanning and/or removing objects directly from the inventory list. We will do extensive testing to prevent SQL injections, javascript attacks, and X-site scripting attacks. Likewise, we will use the browser console to detect irregularities and potential security concerns using error and DB retrieval statements.

Summary of individual work
Surya continued his work on RPI and camera integration. Despite challenges with operating the RPI in a headless setup, including slow SSH connections and lag in camera performance, he successfully implemented a rudimentary algorithm to capture images with 80 percent confidence for classification. The next steps for him involve improving camera performance with a headed monitor setup and integrating image capturing into the database, along with exploring options for classification performance. He thanks the 18500 teaching staff for their feedback on the hardware behind this project during the interim demo with specific regards to camera performance and options.

Grace included additional functional pages on the web application in accordance to feedback and improving the general user experience. These pages include but are not limited to a chat function between users to discuss fitness related goals, a calorie consumption progress graph demonstrating user’s daily improvement, adding caloric entries by date and displaying by day (eventually connected to the hardware components of classification), and a page allowing users to set a daily calorie limit to measure necessary improvements.  She plans to continue working on the frontend and ensuring they match how the user can use the application in an efficient and beneficial way. Also testing for each page that includes user modifications or uploads that could be vulnerable to security attacks.

Steven continued working on his ML features and working with Grace to integrate them into the web application. Tesseract OCR components were added into views.py to extract food label texts from the user’s uploaded images. Additional work will need to be done to improve accuracy, but images were turned into gray-scale and enhanced through enlargement of the original images themselves. During the individual testing phase, these improvements were able to enhance text extracting results. The next step would involve incorporating Surya’s camera subcomponent into capturing these OCR readings and fully integrating it onto a web application page. Steven also included a new model to store entries in a database where users are inputting food items and their respective caloric values. More improvements will be done to include individual food inventories for the user that is currently logged into the application.

We shifted our whole focus into our demo and getting our individual components to work. A big thing that we discovered in all our components were a lot of underlying issues that came when integrating into our project. All the testing was local and independent, so we ran into a lot of complications.

A few complications arose from our ML design. The first being the difficulty of using the ChatGPT API to take in images and process them. It was slow and hard to input an image efficiently. Furthermore, our local tests for some of the ML classification schemes were difficult to integrate into the Django web app. As a result, we had to shift base and adjust on the fly. This included using some backup plans we had such as using pre-trained data sets and classification APIs. The big issue with these were the configuration issues that we spent hours dealing with and making sure we had everything installed to the right versions. Lastly, we decided to change our fruit testing data to oranges, bananas, and strawberries instead of apples. We hope this change allows us to move on from classification at the moment and shift our focus towards label reading and OCR design as well as hardware integration.

Surya made major progress in integrating the Raspberry Pi and configuring it with our internet. The major issue is configuring it with CMU wifi which is the primary issue right now. However, he was able to set it up with the ssh server and download required packages for the video stream. We shifted focus to doing a lot of computation on the Raspberry Pi itself to add into our designed trade studies. We hope to show all of these in the demo to showcase our experimentation process. Surya lastly did a lot of work configuring MacBook settings to run our website with all the required packages. There were many hardware issues that he had to resolve and fix to even get the website to run. Ultimately, the website was able to run successfully on his computer and classify images to an acceptable accuracy.

Lastly, Grace was able to create a new Gantt chart schedule that reflected changes in our schedule that had to be made since several technical challenges were encountered during our testing process. While there were some unexpected delays with the OCR libraries and hardware configurations, we remained on track in terms of our original project schedule with the appropriate allocated slack time and abilities to adjust to such changes. Since we added additional features and ideas to our project throughout the capstone process including the in-class ethics discussion, some slack was allocated towards those features and some extra time was spent handling unanticipated technical issues. Ultimately while schedule changes to our project were necessary, they ended up contributed a lot towards our ability to work together as teammates and also learn to adapt to necessary changes to adapt to our project framework.

We hope to have a productive demo next week and take in the feedback of everyone to get closer to completing our final project. Likewise, we will start drafting the final report.

As a team, we enjoyed the focus on ethics this week. We enjoyed the pod discussions and began to consider things that we would have never thought of. A big issue we identified with our product involved both body image concerns as well as privacy issues. We want our product to promote a healthy lifestyle, but we do not want our users to develop eating disorders and other personal issues. Likewise, we do not want private data on the consumption habits of our users to be leaked to other users as well as companies who participate in targeted advertising. As a team, we were able to discuss fixes to these issues including database security design and user-friendly/encouraging dialogue on the front end. Despite being done with discussions on ethics, we plan to keep all these principles in mind as we further our design.

We found it interesting in the pod discussion that two other groups had a product of similar functionality to ours. While one group anticipated projecting ingredients and recipes onto a tabletop using a projector at a calculated angle, the other is using AI to generate recipes on a phone app from ingredients added to their food inventory. We were able to clearly address that the MVP for our system would only be able to classify canned foods and certain types of fruits as opposed to cooked foods or ingredients in a bowl. The biggest difference between our product and these is that ours is used as a calorie tracking device and more focused on physical wellness, so more ethical concerns arise with this focal point. This will definitely be a greater consideration of ours while working on user interface integration and user experience with our web application.

The parts for scale integration have steadily come in through the ECE delivery center, so Surya began work on understanding the hardware layout of the scale and assessing the 2 main approaches to writing scale measurements to the database. He is waiting on the RPIs to assess the camera approach to read scale values. Another option could be to start on this with the Arduino chip that came in the mail already, but typically, Arduinos are not great choices for image processing because of limited on-board RAM and limited functionality with other cameras (RPIs are fantastic for such applications and several resources exist online for subsequent support). Additionally, he plans on working with Steven and Grace in sculpting a presentation strategy for the rapidly approaching interim demo.

 

In the meantime, he has also learned more about how the load cell amplifier works and the wiring topology. An important thing to do when working with a functional scale is to ensure that the correct wires are snipped and soldered; a wiring schematic can be found below for the reader’s convenience:

 

Steven did a lot of work patching up the ML infrastructure of the project. He optimized the accuracy of the various components. The first optimization was done to the classification algorithm for canned foods and fruit using the AdaBoost algorithm to combine multiple decision boundaries. The second one involved classification within the groups of fruit using k-nearest neighbors. This was combined with GoogLeNet and OpenCV to produce better results more specific to our project. Lastly, the ChatGPT API does not need optimization, but Steven worked with integration into the front-end. He plans to work alongside Grace in the upcoming weeks to test the basic functionality and syntax of the API to perform label reading and classification, if needed.

With the interim demo coming up on April 1st, coming out of Spring break with a clear plan of attack was important for our group. We spent this week planning out some of the main features we would like to highlight for the interim demo; furthermore, with a realization that the interim demo is a presentation of an intentionally incomplete product, we spent some time deciding how the features we decide to demonstrate will interact with one another.

Surya continued work on scale integration and helped Grace tidy up the web application. In addition, he developed potential approaches to scale integration through a block diagram as seen below:

Scale integration can be done either by hacking into a scale and forwarding values through a microcontroller, or through an OCR on the digital scale reading panel. In evaluating the first option, a strong understanding of scale hardware is required before it can be opened up. The main aspects he focused on were the difference between strain gauges and load cells, how and why they are configured in a Wheatstone arrangement, and the rewiring necessary to forward the load cell measurement through an amplifier and into a microcontroller such as an Arduino.

Given the delicacy of this process and the surprising nuance of hardware behind camera scales, he discerned that an OCR model reading the segmented display panel of the scale may be cleaner to implement for an MVP. This, however, presents its own challenges. Images require more memory to deal with than a stream of bits representing scale readings, so latency becomes a pronounced issue.  Furthermore, the accuracy of digit classification is an additional factor that is simply not a problem when one is simply forwarding bits to and from components. The counterargument to this approach is the reduced potential of damages, and for the sake of the MVP and wanting an operational scale as first priority, this is a significant consideration to keep in mind.

In either case, because both options are largely cost-effective, Surya began ordering parts common between both approaches and plans to get his hands dirty with both approaches to see which method works better for the application over the next 2 weeks. He encourages readers to view his weekly status report for more details on the above design plan for scale integration.

Steven made significant progress in testing various ML methodologies. He was able to determine that soft-margin SVMs was not effective enough to include in our implementation. However, the SVMs have provided nice decision boundaries that we plan to use for our backup option: AdaBoost Algorithm. This algorithm establishes weights for varying decision boundaries, so it takes into account multiple boundaries. He was able to research the math and coded up the preliminary functions to compute the weights and train the model.

Steven also shifted a lot of focus into working with GoogLeNet and working with a k-nearest neighbors approach to boost the accuracy of the algorithm in the classification of different fruits. He plans to work on testing next week as well as validation. We hope to have all this tested, and the results analyzed in no more than two weeks. However, another goal next week is to integrate the GoogLeNet classification algorithm directly into the website without any modifications to test the compatibility as well as the preliminary accuracies and latencies.

Regarding progress in integration, Steven did research on ChatGPT4. We are currently hesitant on purchasing the API to save some money. However, Steven wrote the pseudocode for integrating it into the front-end. Likewise, he looked into creating formatted responses given images and formatted inputs. Steven will also begin shifting focus away from local testing on his MacBook Camera and work closely with Surya and Grace to take Arduino or Rpi images and/or images from the website.

Grace was able to take this week’s ethics assignment and apply it towards public safety and public welfare considerations. We realized food tracking applications could induce negative side effects for users with high self-discipline, so an additional globals page could help promote a body positive environment that shows users what others are up to as well. A caption can also be added so our web application is used more as a social media, but of course this is optional for users and they can always choose to opt-out. With this potentially being more on a global scale, I want to consider larger options for file uploads to be able to accommodate this. One option is instead of using the web application’s file system, a content delivery network could be used since their servers are scattered around the world. This would definitely help improve speed and performance of our web application in the long-run.

In addition to addressing how our product meets market needs in terms of considerations to public health, safety, and welfare, as social, and economic factors related to the system of production, distribution, and consumption of goods and services, our group would like to explore public concerns from a different perspective. Hence, A was written by Steven, B was written by Surya, and C was written by Grace.

Part A: With consideration of global factors, NutrientMatch can be used by just about anyone across the globe, whether or not their wellness goal consists of consuming more calories or better keeping track of the types of foods they are consuming daily. Regarding younger adults, as fitness is becoming a bigger trend with the emergence of technology and the expansion of gyms, more of these people are aware of their nutrient intake. Our product is heavily targeted towards this group since they tend to live with roommates as opposed to other family members. With this in mind, it is easier to get foods mixed up when groceries are purchased separately. NutrientMatch is able to track each user’s food inventory as soon as they are logged in to avoid these confusions. On the other hand, family members may also want to use this product to track their own intake. Our product can also be used by those who are not as tech-savvy as others. The usage process is easy: after each food item is either scanned or classified and weighed, it is automatically forwarded to the cloud database which will make backend calculations to be displayed on the webpage for users to see. Hence, while NutrientMatch is not a necessity in people’s lives, the increasing trend of physical wellness makes it a more desirable product overall.

Part B: Cultural considerations in the design process are crucial to making NutrientMatch an accessible and adaptable product for those around the globe. The most obvious example that comes to mind is the acknowledgment of dietary preferences in cultural and religious contexts, including but not limited to plant-based diets and veganism, Halal foods in Arabic Culture, and Kosher foods as delineated by the Jewish faith. To that end, we will dedicate significant resources to training a recognition model that can broadly recognize and perform similarly across a variety of testing inputs sampled across these cultural lines, so as to not perpetrate any implicit biases in the recognition process.

Another vital cultural aspect to consider is the idea of nutritional makeup and body image. Various cultures around the world prioritize specific eating lifestyles and different preferences for caloric makeup and subsequent ideas for how the ideal physique should be maintained. While some groups may prefer thinness, with an increased focus on portion control and calorie counting, in other cultures food is synonymous with hospitality and status. Despite these seemingly disparate perspectives on the notion of “ideal” nutrition, NutrientMatch should perform similarly and without bias in all scenarios. Our product will be carefully designed to avoid reinforcing specific nutritional ideals; similar to the idea above, our recognition model and recommendation algorithms will perform as intended irrespective of the actual nutritional habits of a certain user.

Finally, the notion of privacy and communication/integrity of data is an issue that crosses cultural lines as well.  Some cultures view technology and its role in human life differently than others. In developing NutrientMatch, we seek to build trust with our customer base by demonstrating a commitment to the privacy and confidentiality of data; this will help us garner popularity and positive standing even amongst cultures that may demonstrate initial skepticism towards such technology. MySQL automatically hashes and encrypts the stored data in the database, and the HTTPS protocol that will be used to interact with such a backend will prevent hackers from intercepting data based on an extensive SSL certificate authorization process that checks the users and the domain name against what is listed on the SSL certificate.

Part C: Lastly, there is a strong connection between previously analyzed economic factors to environmental considerations. One of our greatest concerns when doing research for the product’s use case requirements was the high amount of foods that spoiled in the United States. This does not only directly impact the economy as people are wasting their groceries, but more foods are added to landfills. Obviously, landfills take up a lot of space and need to be managed periodically. In addition to this, landfills produce odors and methane gas that directly contribute to global warming and the air we breathe. It is important to effectively utilize all the resources we have as the global population continues to increase, and NutrientMatch can help users maximize this use by tracking every item entering their food inventory system. This reduces user stress in remembering the groceries they bought but can also help them manage what to buy next or what types of foods they are currently lacking. Besides personal health, the environment remains one of our greatest concerns and we hope that NutrientMatch encourages users to consume and productively make use of all the ingredients in their home before directly disposing of them.

Shifting towards a discussion on our team’s progress over these past two weeks, we were successful in a lot of the software design from ML to web app development. First, Steven was able to progress substantially in the ML design portion. He was able to finalize tests and validation for the soft-margin SVM formulation to distinguish between fruit and canned foods; however, there are still issues with accuracy which will need to be fixed in the upcoming weeks. Likewise, Steven made substantial progress in researching ChatGPT4 capabilities and the API to integrate its label-reading algorithms into our web app design. Everything is on if not ahead of schedule regarding the ML aspect of our project which will give us extra flexibility in fine-tuning and maximizing the results. We hope to gather statistics to support our design choices as well shortly.

Additionally, Grace was able to help Steven gather data for the ML testing portion, including feature data that would help distinguish between the packaging between fruits and canned foods, label data for training the SVM classifier, and some other relevant information that was used during the labeling of each food item. With the given issues, some fine-tuning may still need to be done further. Regarding individual progress, Grace has completed mockups for the design report and used this towards calibrating the front-end designs. Rendering functionalities between user login, user registration, user home page, and logout pages have been completed. More progress is being made on the web application database retrieval with MySQL and setting up the tables regarding what information should be displayed to users. More testing will need to be done to handle database requests and deployment to the server environment.

Surya focused on advancing the web application in advance of a functional demonstration of the basic web app features at the next staff meeting on Monday, March 11. To reiterate, the demo will focus on site navigation, front-end development, user interaction, and view routing. Some of the particular accomplishments include a refinement of the OAuth feature for registering new users through their email addresses, as well as weight-tracking visuals using the chart.js framework. Looking ahead, the team plans to develop more user interaction features; these include progress posting, friend networks, and a Websocket-based chat service.

Surya also continued research on the scale integration process and will order parts this week. Despite initial hesitancy due to the unfamiliarity with market options, he compiled a list of microcontrollers, with a consideration of factors such as power consumption, pin layouts, and opportunities for streamlined design through custom SoC options. This constitutes a fascinating stretch goal, and one we look to tackle in early April, but for now, we would like to proceed as planned with our simple yet effective MVP approach with ESP protocols.

Despite the progress schedule remaining largely unchanged, we understand this is the point of the semester where we expect to make significant progress toward our product with the interim demonstration arriving soon.

This week, our team focused on preparing our design presentation and report. We collaborated on designing slides, creating speaker notes, and beginning the writing process for the Introduction and Machine Learning documentation in our design report.

In terms of progress on our project’s design component, we made significant strides. We experimented with Support Vector Machines (SVMs), GoogLeNet, and ChatGPT’s API to optimize our classification results. We conducted tests on SVM formulations, continued building our dataset, and explored using GoogLeNet for image classification. Additionally, we tested ChatGPT4’s ability to read labels, which proved to be highly accurate.

Throughout the week, we didn’t encounter any major complications. However, I acknowledge that I could have contributed more to designing the physical product and creating the box. Moving forward, I aim to gain more hands-on experience with the physical design process while finalizing the Machine Learning algorithms (Steven).

Regarding the design presentation, we addressed key considerations such as project implementation, load cell amplifiers, food item differentiation, validation metrics, and backend complexity. We’re now focusing on completing the design report, which will require a deeper understanding of our project’s development procedures. For instance, we will use our block diagram and implementation plan to devise a more descriptive architecture plan.  We will also put more efforts towards connecting initial public health/safety/welfare, social, and economic factors we want our product to solve to the formulation of this solution. Of course, many tradeoffs emerge with designing an engineering solution, so we will dive into these considerations through our testing experience so far and feedback we have received from peers during our proposal and design presentations.

In terms of hardware-related risk mitigation strategies, we’re addressing concerns such as buffer overflow, data loss, recovery, and mismatched baud rates. We plan to implement control mechanisms for buffer overflow, backup data on nonvolatile memory for data loss recovery, and ensure proper synchronization of baud rates.

Overall, our progress is on track. We’re aiming to complete the imaging component of the project by Spring break, with each team member making significant contributions to their respective tasks.

Ensuring that our product addresses a desired need in the market is critical to its reception and success amongst the public. This week, the team would like to explore this issue through 3 lenses: Steven will explore part A, Surya will explore Part B, and Grace will explore part C.

Steven:  With respect to considerations of public health, safety and welfare, our solution is designed to improve physical well-being for users by monitoring caloric intake. Likewise, our product aims to reduce the psychological burden of tracking what and how much you eat as our product does all the tracking for you. Furthermore, keeping track and inventory of all the food you buy at the store can be a burden, and people are quite forgetful sometimes. Our product reduces the stress in remembering what is in your fridge or pantry by keeping count of every item. We have ambitious goals for our product to reduce over and under eating which aims to better the physical health and well-being of the public.  With respect to safety, our product will have limited hazards as it will consist of a sturdy physical structure with a scale, camera, and lights. The components will be as minimalistic as possible, and we will conduct rigorous testing to make sure there are no malfunctions. The lights will require as little power as possible and will be properly embedded into the walls. Likewise, the camera will be off whenever the scale reading is 0 as a result the product will only turn on when an item is put on the scale.

Surya: Consideration of social factors, which relate to extended social groups with distinct sociocultural, political, and/or economic affiliations, is crucial in our increasingly interconnected environment. With this in mind, there are several key themes that we seek to account for in our design. First is the notion of accommodating varying dietary preferences: whether this be due to medical reasons, religious restrictions, or observance of regional cuisine, our algorithm looks to maintain as unbiased a performance as possible in its classification approach.  We look to achieve this accessibility in several ways. This involves details in the image recognition training process and enhanced user-user interaction on the application side that allows for features such as transparent sourcing of nutritional information and customizable dietary profiles (vegan, gluten-free, keto, etc.). Embracing diversity through building a prototype that lacks systematic bias is key to our consideration of social factors we look to meet in our design.

Grace:  With consideration of economic factors related to the system of production, distribution, and consumption of goods and services, a big concern that arose during our research is the amount of food waste generated by people on a daily basis in the United States. Not only does this pose sustainability challenges in the long run from an economic point of view, but also contributes to a substantial amount of landfill waste. Increased amounts of waste also require more management and environmental cleanup, leading to costly endeavors and potentially higher taxes for residents. Our solution, rather than focusing solely on mitigating the system of production and distribution of food goods, aims to assist users minimize waste during their food purchasing and consumption processes. While it does not directly address issues that may arise during production and distribution, helping users automate their daily calorie tracking can at least help them make more informed purchases and reduce waste as a whole. By providing accurate nutritional information, from label reading and food classification, along with better control over food inventory, our solution enables users to make more efficient use of the food at hand and consequently decrease food spoilage rates in general. We anticipate our product to not only  peoples’ wellness habits, but also make a positive economic impact on our society as a whole.

For our team’s status, we identified various risks that might prove to be an issue for our project. Surya was able to write down various risks regarding scale integration and using the Arduino as a middle component in our product. Likewise, Steven discovered risks with the image classification design and the various trade offs. As a result, we need to analyze our design decisions more next week and finalize it. Our contingency plans involve the various backup options we have which include the other design choices (i.e. buying multiple scales and Arduinos).

Likewise, we specified our schedule more as we got a better understanding of the necessary work and steps of our project. Here is the new Gantt chart that will be added to our design presentation:

 

Next week, we will make extensive efforts to produce results in terms of website design, image classification design, and physical product creation and integration.

As a team, we had a productive week in which we completed our presentation and all were on schedule in completing the planned tasks. Here is an image of a nutritional label algorithm Steven tested out:

It reads macronutrient values from a .png image that we found online. We are aware of some issues with this algorithm not calculating the calories, and the model is relatively outdated. We are expecting challenges in fine-tuning the code and/or making our own nutritional label reading algorithm. 

Furthermore, Surya was able to work with a potential camera that we will integrate into our design. He also worked alongside Steven to finalize the list of parts from inventory and from the purchase list, including Arduinos that are optimized to work with scale models for the next stage of the design (RS-232). While the group doesn’t plan to work on the stage in the next 2-3 weeks, prioritizing energy and resources to a robust recognition system in the meantime, given all the parts are interconnected/have to work with one another, it is an important design feature to design functionally-independent yet cohesive components that work well with one another in a complex project.

 

In addition to that, Grace was able to build the Django framework for the front-end by initializing all the files. She plans to have a front-end implementation next week to represent the user interface that we plan to connect the database to. The website portion of our project is up to speed with our anticipated schedule.

 

Through our work this week, we explored our options more in-depth and did research on previous projects. We are expecting to see risks in integrating the Arduino to do all the OCR and computational work. Our backup plan involves defaulting to using our computer to do the primary workload. Likewise, the equipment we have and plan to have are compatible with both a computer and/or an Arduino. 

 

We hope to make significant progress in the creation of the physical product (a wooden box with LED lights to illuminate an object). Likewise, we seek to make progress in our individual assignments which involve website design, algorithm development and integration, and interfacing the hardware with the software. 

 

Lastly, thank you to all the students, TAs, and professors for providing commentary on our presentation and overall project. We received great feedback to better guide us in our design. We are now more aware about potential problems and have thought about alternatives in the event of issues. We recognize the value of this course towards our personal and professional development towards becoming engineers who can both solve problems and communicate their solutions effectively.