Posted on

Team Status Report 03/02/24 and 03/09/24

In addition to addressing how our product meets market needs in terms of considerations to public health, safety, and welfare, as social, and economic factors related to the system of production, distribution, and consumption of goods and services, our group would like to explore public concerns from a different perspective. Hence, A was written by Steven, B was written by Surya, and C was written by Grace.

Part A: With consideration of global factors, NutrientMatch can be used by just about anyone across the globe, whether or not their wellness goal consists of consuming more calories or better keeping track of the types of foods they are consuming daily. Regarding younger adults, as fitness is becoming a bigger trend with the emergence of technology and the expansion of gyms, more of these people are aware of their nutrient intake. Our product is heavily targeted towards this group since they tend to live with roommates as opposed to other family members. With this in mind, it is easier to get foods mixed up when groceries are purchased separately. NutrientMatch is able to track each user’s food inventory as soon as they are logged in to avoid these confusions. On the other hand, family members may also want to use this product to track their own intake. Our product can also be used by those who are not as tech-savvy as others. The usage process is easy: after each food item is either scanned or classified and weighed, it is automatically forwarded to the cloud database which will make backend calculations to be displayed on the webpage for users to see. Hence, while NutrientMatch is not a necessity in people’s lives, the increasing trend of physical wellness makes it a more desirable product overall.

Part B: Cultural considerations in the design process are crucial to making NutrientMatch an accessible and adaptable product for those around the globe. The most obvious example that comes to mind is the acknowledgment of dietary preferences in cultural and religious contexts, including but not limited to plant-based diets and veganism, Halal foods in Arabic Culture, and Kosher foods as delineated by the Jewish faith. To that end, we will dedicate significant resources to training a recognition model that can broadly recognize and perform similarly across a variety of testing inputs sampled across these cultural lines, so as to not perpetrate any implicit biases in the recognition process.

Another vital cultural aspect to consider is the idea of nutritional makeup and body image. Various cultures around the world prioritize specific eating lifestyles and different preferences for caloric makeup and subsequent ideas for how the ideal physique should be maintained. While some groups may prefer thinness, with an increased focus on portion control and calorie counting, in other cultures food is synonymous with hospitality and status. Despite these seemingly disparate perspectives on the notion of “ideal” nutrition, NutrientMatch should perform similarly and without bias in all scenarios. Our product will be carefully designed to avoid reinforcing specific nutritional ideals; similar to the idea above, our recognition model and recommendation algorithms will perform as intended irrespective of the actual nutritional habits of a certain user.

Finally, the notion of privacy and communication/integrity of data is an issue that crosses cultural lines as well.  Some cultures view technology and its role in human life differently than others. In developing NutrientMatch, we seek to build trust with our customer base by demonstrating a commitment to the privacy and confidentiality of data; this will help us garner popularity and positive standing even amongst cultures that may demonstrate initial skepticism towards such technology. MySQL automatically hashes and encrypts the stored data in the database, and the HTTPS protocol that will be used to interact with such a backend will prevent hackers from intercepting data based on an extensive SSL certificate authorization process that checks the users and the domain name against what is listed on the SSL certificate.

Part C: Lastly, there is a strong connection between previously analyzed economic factors to environmental considerations. One of our greatest concerns when doing research for the product’s use case requirements was the high amount of foods that spoiled in the United States. This does not only directly impact the economy as people are wasting their groceries, but more foods are added to landfills. Obviously, landfills take up a lot of space and need to be managed periodically. In addition to this, landfills produce odors and methane gas that directly contribute to global warming and the air we breathe. It is important to effectively utilize all the resources we have as the global population continues to increase, and NutrientMatch can help users maximize this use by tracking every item entering their food inventory system. This reduces user stress in remembering the groceries they bought but can also help them manage what to buy next or what types of foods they are currently lacking. Besides personal health, the environment remains one of our greatest concerns and we hope that NutrientMatch encourages users to consume and productively make use of all the ingredients in their home before directly disposing of them.

Shifting towards a discussion on our team’s progress over these past two weeks, we were successful in a lot of the software design from ML to web app development. First, Steven was able to progress substantially in the ML design portion. He was able to finalize tests and validation for the soft-margin SVM formulation to distinguish between fruit and canned foods; however, there are still issues with accuracy which will need to be fixed in the upcoming weeks. Likewise, Steven made substantial progress in researching ChatGPT4 capabilities and the API to integrate its label-reading algorithms into our web app design. Everything is on if not ahead of schedule regarding the ML aspect of our project which will give us extra flexibility in fine-tuning and maximizing the results. We hope to gather statistics to support our design choices as well shortly.

Additionally, Grace was able to help Steven gather data for the ML testing portion, including feature data that would help distinguish between the packaging between fruits and canned foods, label data for training the SVM classifier, and some other relevant information that was used during the labeling of each food item. With the given issues, some fine-tuning may still need to be done further. Regarding individual progress, Grace has completed mockups for the design report and used this towards calibrating the front-end designs. Rendering functionalities between user login, user registration, user home page, and logout pages have been completed. More progress is being made on the web application database retrieval with MySQL and setting up the tables regarding what information should be displayed to users. More testing will need to be done to handle database requests and deployment to the server environment.

Surya focused on advancing the web application in advance of a functional demonstration of the basic web app features at the next staff meeting on Monday, March 11. To reiterate, the demo will focus on site navigation, front-end development, user interaction, and view routing. Some of the particular accomplishments include a refinement of the OAuth feature for registering new users through their email addresses, as well as weight-tracking visuals using the chart.js framework. Looking ahead, the team plans to develop more user interaction features; these include progress posting, friend networks, and a Websocket-based chat service.

Surya also continued research on the scale integration process and will order parts this week. Despite initial hesitancy due to the unfamiliarity with market options, he compiled a list of microcontrollers, with a consideration of factors such as power consumption, pin layouts, and opportunities for streamlined design through custom SoC options. This constitutes a fascinating stretch goal, and one we look to tackle in early April, but for now, we would like to proceed as planned with our simple yet effective MVP approach with ESP protocols.

Despite the progress schedule remaining largely unchanged, we understand this is the point of the semester where we expect to make significant progress toward our product with the interim demonstration arriving soon.

Posted on

Steven Zeng’s Status Report for 03/02/24 and 03/09/24

I wrote about section A for the team report; below is my answer.

With consideration of global factors, NutrientMatch can be used by just about anyone across the globe, whether or not their wellness goal consist of consuming more calories or to better keep track of the types of foods they are consuming daily. Regarding younger adults, as fitness is becoming a bigger trend with the emergence of its technology and the expansion of gyms, more of these people are aware of their nutrient intake. Our product is heavily targeted towards this group since they tend to live with roommates as opposed to other family members. With this in mind, it is easier to get foods mixed up when groceries are purchased separately. NutrientMatch is able to track each user’s food inventory as soon as they are logged to avoid these confusions. On the other hand, family members may also want to use this product to track their own intake. Our product can also be used by those who are not as tech savvy as others. The usage process is easy: after each food item is either scanned or classified and weighed, it is automatically forwarded to the cloud database that will make backend calculations to be displayed on the webpage for users to see. Hence, while NutrientMatch is not a necessity in peoples’ lives, the increasing trend of physical wellness makes it a more desirable product overall.

Regarding my progress, I primarily did work before Spring break, and used the designated slack time to relax and clear my head. I focused most of my effort on the design report including analysis of design trade-offs for the ML algorithms. Recently, I have tested soft-margin SVMs to distinguish between fruits and canned foods. I have computed optimal parameters and hyper-parameters using 5-fold cross validation due to a limited data set. It correctly classifies objects only around 74 percent of the time. However, I expect this number to go up as I tune the algorithm to only address the three types of fruits (banana, orange, and apple) and canned foods.

Furthermore, I did more research and experiments on ChatGPT4 and its API to read labels. The results were quite promising, and our focus will be on making sure the image is clear enough for ChatGPT to process. Likewise, I am experimenting with database storage and narrowing down what exactly our group wants to store in the database logs. The current plan is to take a picture of the image then pass it into the classification algorithm. If it is a fruit, our product will directly store the name of the fruit using the GoogLeNet classification algorithm along with caloric information calculated using the weight and online data. If it is canned food, our algorithm will read the label from the front and store the name into the database alongside an image of the back with the nutrition label. This will reduce the latency of storing an object into our database and offload the extra computation to database retrieval in which the image will need to be processed. The other alternative is to do all the computation and calculations when the object is being scanned. This will lead to better memory utilization but worse latency which will be an issue in terms of user experience. We plan to test both these approaches out and determine the differences in latency and memory utilization.

Lastly, I read up on a lot of the work Surya has done with the Arduino as well as cloud deployment. The ML portion is highly dependent on retrieving information from the Arduino (weight and image retrieval) and the ability of an Amazon EC2 server to efficiently run all the computations to populate the database. This involved reading a lot of documentation online to prepare myself for future stages. A lot of the detail and info came from https://aws.amazon.com/blogs/machine-learning/category/compute/amazon-ec2/ which I highly recommend. It gives a lot of fascinating stories and helped me understand the space better.

Overall, I had a relatively productive two weeks and got a good break to clear my head.

Posted on

Steven Zeng’s Weekly Status Report 02/24/24

This week I concentrated my efforts into the design presentation and report. I helped my team in designing slides and creating speaker notes. Likewise, I began writing the Introduction and ML documentation in our design report.

 

Regarding the progress in forwarding the design component of our project, I experimented with SVMs, GoogLeNet, and ChatGPT’s API. I was able to conduct tests on the weights and constants of the SVM formulation to best optimize the classification results. Likewise, I am still adding to a data set for training, testing, and validation data distinguishing canned foods and fruit. I plan to use 5-fold cross-validation to properly tune the hyper parameters (i.e. C and gamma). I also began looking at the GoogLeNet library and watching videos on how to use it. I seek to use function calls to distinguish between apples, bananas, and oranges. Finally, I tested ChatGPT4s ability to read labels which was surprisingly high and accurate. I plan to learn how to use the API to accept input and direct prompt responses in a formatted way to forward to the database.

 

There were no major complications or challenges this week in my progress. I wish I could have helped my teammates more in terms of designing the physical product and creating the box. I hope to get a more hands-on experience next week with the physical design process as I finalize the ML algorithms. Likewise, I hope to shift a lot of focus and effort into completing the design document for next week.

Posted on

Steven Zeng Status Report for February 17, 2024

 

This week, I focused primarily on image classification rather than label reading. I looked through the Tesseract library and watched several videos on using it to classify images. Likewise, I explored using support vector machines to hand-make our own classification algorithm. This included a compilation of training data from Kaggle’s 2022 Food Recognition set (https://www.kaggle.com/datasets/sainikhileshreddy/food-recognition-2022). This training data can help for whatever method we chose to apply. Finally, I tested ChatGPT and its API more to potentially use it for classification and label reading. As a result, we have these three potential design paths that we could take. Each of these paths have their own tradeoffs that I have explored and written down below, and I hope to make a decision next week after doing more research. 

1.  Online Label Reading and Classification Library: The benefits are that all the hard work and algorithms are created. Likewise, the method has been tested and published online. All I need to do is import the libraries and write functions that call these libraries. This will reduce the design process tremendously. On the cons side, it will take a lot of preliminary research and understanding of other people’s code. Likewise, some of the code may be outdated and lack the accuracy we need for our project.

2. Incorporate Tesseract and Create Our Own Algorithm: The primary benefit of this is better top-down understanding of everything. Likewise, I can fine-tune it to fit our product specifically. Maintenance and product updates would be quicker since I will have full understanding of a majority of the code. The con is that this will take a lot of time upfront with learning everything and small design choices along the way. For example, I have looked into the tradeoffs between using Support Vector Machines vs Naive Bayes vs Trees. All of them have their unique choices, but I was able to study up on the math and principals behind each of these algorithms.

3. Chat GPT API: The benefit of this is reliability, precision, and speed. I queried various inputs and formatted the replies of Chat GPT to be compatible with key-word extraction to store values in the database. The accuracy was quite remarkable, and it could classify images as well as read labels. The main con is understanding the API and the potential to have a high cost. However, we are definitely interested in somehow incorporating Chat GPT into our product.

In addition to that, I helped with creating the design presentation. I focused primarily on the implementation plan, testing, and finalizing our schedule via the Gantt chart. I worked with my team members to create the slides and write the script for the presentation. I went through feedback from our first presentation to better design our next presentation and our product in general. Thanks again to all the students, TAs, and professors who provided commentary on it.

Regarding the special prompt we are supposed to answer for this report, I have decided to write part A. With respect to considerations of public health, safety and welfare, our solution is designed to improve physical well-being for users by monitoring caloric intake. Likewise, our product aims to reduce the psychological burden of tracking what and how much you eat as our product does all the tracking for you. Furthermore, keeping track and inventory of all the food you buy at the store can be a burden, and people are quite forgetful sometimes. Our product reduces the stress in remembering what is in your fridge or pantry by keeping count of every item. We have ambitious goals for our product to reduce over and under eating which aims to better the physical health and well-being of the public. 

With respect to safety, our product will have limited hazards as it will consist of a sturdy physical structure with a scale, camera, and lights. The components will be as minimalistic as possible, and we will conduct rigorous testing to make sure there are no malfunctions. The lights will require as little power as possible and will be properly embedded into the walls. Likewise, the camera will be off whenever the scale reading is 0 as a result the product will only turn on when an item is put on the scale.



Posted on

Team Status Report for February 10th, 2024

As a team, we had a productive week in which we completed our presentation and all were on schedule in completing the planned tasks. Here is an image of a nutritional label algorithm Steven tested out:

It reads macronutrient values from a .png image that we found online. We are aware of some issues with this algorithm not calculating the calories, and the model is relatively outdated. We are expecting challenges in fine-tuning the code and/or making our own nutritional label reading algorithm. 

Furthermore, Surya was able to work with a potential camera that we will integrate into our design. He also worked alongside Steven to finalize the list of parts from inventory and from the purchase list, including Arduinos that are optimized to work with scale models for the next stage of the design (RS-232). While the group doesn’t plan to work on the stage in the next 2-3 weeks, prioritizing energy and resources to a robust recognition system in the meantime, given all the parts are interconnected/have to work with one another, it is an important design feature to design functionally-independent yet cohesive components that work well with one another in a complex project.

 

In addition to that, Grace was able to build the Django framework for the front-end by initializing all the files. She plans to have a front-end implementation next week to represent the user interface that we plan to connect the database to. The website portion of our project is up to speed with our anticipated schedule.

 

Through our work this week, we explored our options more in-depth and did research on previous projects. We are expecting to see risks in integrating the Arduino to do all the OCR and computational work. Our backup plan involves defaulting to using our computer to do the primary workload. Likewise, the equipment we have and plan to have are compatible with both a computer and/or an Arduino. 

 

We hope to make significant progress in the creation of the physical product (a wooden box with LED lights to illuminate an object). Likewise, we seek to make progress in our individual assignments which involve website design, algorithm development and integration, and interfacing the hardware with the software. 

 

Lastly, thank you to all the students, TAs, and professors for providing commentary on our presentation and overall project. We received great feedback to better guide us in our design. We are now more aware about potential problems and have thought about alternatives in the event of issues. We recognize the value of this course towards our personal and professional development towards becoming engineers who can both solve problems and communicate their solutions effectively.



Posted on

Steven Zeng’s Status Report for February 10th, 2024

Regarding my progress, I looked through online libraries for OCR (Tesseract) and nutritional label reading algorithms. I looked through them for important functions and how to integrate them into our project. The githubs that I have viewed are the following: https://github.com/openfoodfacts/off-nutrition-table-extractor/blob/master/nutrition_extractor/symspell.py and https://github.com/tesseract-ocr/tesseract. The purpose of doing this is to provide myself with a better understanding of how these libraries work and areas in which I can fine-tune them for our specific project. I found that the nutrition label extractor seems to only take.png images, and I changed the code to support newer versions of numpy and Tesseract. This algorithm does not do as well as we expected, so we might need to modify the algorithm or completely scratch it out.

 

In addition to this, I did research online to create a purchase list. I first found a convenient scale to use. We are looking for a scale that has the ability to connect to a laptop and send weight data to the computer. I will work on this part of the project later when we receive the scale; the scale seems to work with excel sheets, so I would need to understand how to convert an excel sheet into database entries. I also looked for LED strips to illuminate the item. All of this work has been added to a parts list that is shared amongst my team.

 

Currently, my progress is on schedule; however, looking through the library, I expect some issues with integrating various functionalities which might exceed the estimated time. However, I plan to get a better idea next week after I do basic small tests using my macbook camera. 

 

Overall, this week I focused primarily on presentations and organizing work for the following weeks. Next week, I hope to get physical proof that the OCR and nutritional label algorithms are working more effectively with example items and classifications displayed on a console. Next, I will look into image classification to classify fruits and vegetables. Finally, I plan to work with Surya to get access to a makerspace at TechSpark to do wood work.