March 16th 2024 Status Report — Surya Chandramouleeswaran

Coming off break and getting back into the swing of things, this week offered an opportunity for us as a team and individually to reflect on our progress to meet our NutrientMatch MVP. Entering the project work period where we had an ambitious idea but were largely uninformed of implementation strategies, we are in a much better space now as we have taken a deep dive into building the stages of the design. For me, this was a week where my preconceived notions and ideas of integrating the scale into the project were challenged, and I considered alternative options in the form of risk mitigation and to complete our MVP. I’d like to dedicate this week’s blog posts to explaining what I learned and subsequent approaches.

 

Our first idea to integrate the scale with a remote database (and subsequent display to a custom website) involved a physical scale, signal amplifiers, microcontrollers and logic, and a WiFi chip which would make requests to a backend engine for website rendering. The first design consideration was whether to build a custom scale or hack into an existing one. We favored the latter, as scales built in the market today are more complex than one would think, and have several nuanced electronic components that ensure the measurements are accurate. The downside to this approach, however, is the lack of customizability given the scale is prebuilt, and the fact that we would need to exercise extreme caution so as to not cut the wrong wires or the solder components to the wrong sections of the scale.

 

The next step involved rewiring the load cells to run through an amplifier. A load cell is just a component that translates experienced mechanical force into a signal to be interpreted digitally. When running these signals to an Arduino for display, the signals themselves are typically low in amplitude compared to the sensitivity of the ADC input of the microcontroller; hence the need for a load cell amplifier.

 

For testing, a nice thing to do would be to wire the load cell amplifier outputs to a 7-segment display, just to get an idea of the numbers we are dealing with. In implementation, the next step would be to send these signals to the ESP32 chip. This is where we will develop a significant component of the logic to control when the load cells are sampled. Furthermore, the ESP32 script needs to generate an HTTP request wirelessly to our cloud-hosted web application so it can write in the measured weight when certain conditions are met. These include if the weight is stable, if the website hasn’t been updated yet, and so on. Find below some of the logic I wrote for the ESP implementing the above concepts in detail. Some of it is in pseudocode, for simplicity:

 

void loop() 
{
  static float history[4];
  static float ave = 0;
  static bool stable = false;
  static bool webUpdated = false;
  static float weightData = 0.0;

  // When should we send a request to the server? this controls that 
  if (stable && !webUpdated)
  {
    // Only try if the wireless network is connected
// WL_CONNECTED is an imported state from the earlier logic in the ESP
    if((wifiMulti.run() == WL_CONNECTED)) 
    {
      HTTPClient http;

      Serial.print("[HTTP] begin...\n");
      Serial.println(weightData);

      // Write string address of where website is hosted here 

      String weight = String(weightData, 1);
      String fullAddress = String(address + weight);
      http.begin(fullAddress);

      Serial.print("[HTTP] GET...\n");

      // start connection and send HTTP header
      int httpCode = http.GET();
     
      if(httpCode > 0) 
      {
        // HTTP header has been send and Server response header has been handled
        Serial.printf("[HTTP] GET... code: %d\n", httpCode);

        // file found at server, response 200
        if(httpCode == HTTP_CODE_OK) 
        {
          // clear the stable flag as the data is no longer valid
          stable = false;
          webUpdated = true;
          Serial.println("Web updated");
        }
      } 
      else 
      {
// debugging case 
        Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
      }

      http.end();
    }
  }

  // Check the weight measurement every 250ms (THIS IS ARBRITRARY)
  if (millis() - scaleTick > 250)
  {
    scaleTick = millis();
    // Read from the HX711. have 2 do this before using data
    LoadCell.update();
   
    weightData = abs(LoadCell.getData());
    //running average
    history[3] = history[2];
    history[2] = history[1];
    history[1] = history[0];
    history[0] = weightData;
    ave = (history[0] + history[1] + history[2] + history[3])/4;

    // Logic to control when to write in the weight measurement 
    if ((abs(ave - weightData) < 0.1) && 
        (ave > 30) && 
        !webUpdated)
    {
      stable = true;
    }

    // IF we've updated the website AND
    //  the average weight is close to zero, clear the website updated flag
    //  so we are ready for the next weight reading
    if (webUpdated && ave < 1)
    {
      webUpdated = false;
    }

    Serial.print("Load_cell output val: ");
    Serial.println(weightData);

    // Write in the scale reading here!
  }
}

The web application handles incoming HTTP requests from here on out, so there is no more to implement on the scale side of things.

Now, this plan that our team developed in theory works very well, but some of the things that concern us mainly have to do with uncertainties on the scale end. Hacking a scale requires a strong understanding of how the scale is built to begin with, so we can avoid damaging existing wiring and components. So I began to think about alternative options for scale integration that wouldn’t involve the internals of the scale electronics. A less risky, but equally stimulating implementation idea would be to hook up an RPI camera and physically read the digital scale reading; this would also require talking to an external sensor which would tell the camera to only “sample” and take pictures when the door of our apparatus is closed. The module would forward the image result to an ML algorithm which would detect the weight measured and write it to the database.

The drawbacks to this approach are the increased probability of measurement inaccuracy, as well as the fact that it would be harder to control the logic for when to capture the images. Also, images are more memory intensive than streams of data, so the RPI would need to have a decent amount of memory onboard to handle the image capturing. In either case, we have plenty of money under the budget to implement both (highly cost effective) ideas, so we will be trying both in parallel and seeing which approach makes more sense.

I envision the next 2 weeks in advance of our demo to be an important time for our project development; I’m excited to get to work on the scale once our parts come in!

Steven Zeng Status Report 03/16/24

This week I continued on my progress in developing the ML backbone to this project. The first set of tasks I completed involved the soft-margin SVMs. I was able to fine-tune it to the best of my ability, but the accuracy values are not up to the value we like. As a result, I will experiment with more decision boundaries using the AdaBoost algorithm. This will assign different weights to a set of unique decision boundaries to improve the classification accuracy from the SVM formulation. The AdaBoost algorithm allows the classifier to learn from misclassifications by adjusting the weights of various decision boundaries. I did a lot of research on the mathematics and rationale behind this algorithm, so next week, I hope to implement a rough version of it and analyze its effect on improving classification between canned foods and fruits.

Next, I looked more into the GoogLeNet model.  Thanks to feedback from the Professor Marios and Neha, I decided to steer away from fine-tuning the last layer. Instead, the plan is to experiment with k-nearest neighbors to classify the real data using the training data. I created a design tradeoff report on the usage of k-nearest neighbors. I began coding up the algorithm behind it using google Colab, and I will compile a testing dataset of bananas, oranges, and apples at the same time. The plan right now is to start with k = 5; however, if time permits, I plan to use 5-fold validation to experiment with k= 3, 5, 10, 20, 40.

The last thing I did this week involved familiarizing myself with the ChatGPT4 API. I tested it locally on our website using localhost, and it worked quite well. The next step is to format the inputs to the API and get formatted outputs back to store into the database. The computational power and accuracy is quite remarkable, and I plan to experiment more with all the features it has. The goal is to have sufficient knowledge in using this API in the event that the other ML models and algorithms we plan to employ end up missing the accuracy benchmarks.

Next week, I plan to work closely with the website design to integrate the various features I have been testing. Currently, all the testing has been done using my MacBook camera and locally on my computer in a contained environment. As a result, there was not many errors or difficulties in running tests and experiments. Likewise, I hope to conduct latency tests on the website using dummy data and values. The goal is to be able to retrieve values from the database and feed them in as input to our ML models to output responses. Likewise, I plan to work with Surya to figure out the camera design of our system and how to accept images from the Arduino.

Grace Liu’s Status Report for March 9th, 2024

The week prior to spring break, our group spent most of efforts in producing our design report since this is a significant part of our capstone. Since most of my work involves knowledge from the class Web Application Development, I spent a lot of time combing through old lectures for React and JavaScript help since it had been a while since I’ve taken the class. The mockups added into the report for a registration page, a login page, and an inventory/main page can be seen below and have been used as inspiration for the actual frontend designs:

The rendering between these pages have been complete in view.py along with the logout button functionality. Users have the option of logging in using Google OAuth as opposed to creating a separate account. Many benefits come with this service, including the elimination of users sharing their passwords to third-party applications by using a token-based authentication system and providing a more seamless user experience that is one click away. While our product is pretty self explanatory to use, we still want our users to have a most simplified user experience as possible. This was set up on the Google Developer Console to configure the OAuth consent screen for how it will be displayed to users. During further testing processes, we will test that the authentication flow works as smoothly as possible with the best security measures.

Since most of the frontend application has been completed, I was able to take some time and start on setting up the database retrieval process on our website using Django. Upon evaluation and comparison of different databases in our design report, we ended up settling on MySQL for many reasons. It all ended up boiling down to its reliability in handling large datasets and high compatibility with web application and Arduino programming. The structure of our database will look similar to the personal inventory page as pictured above, but may be incline to slight changes depending on how we want the item status to be particularly displayed.

I also assisted Steven in gathering data for his ML testing. Specifically, this included feature data that would help distinguish between the packaging between fruits and canned foods, label data for training the SVM classifier, and some other relevant information that was used during the labeling of each food item. For the feature data, various attributes such as visual appearance and material composition were collected to help capture the differences in packaging appearances related to fruits and canned foods. Additionally, the labeling process involved the careful examination of the packaging and verification against established criteria to ensure enough accuracy. This comprehensive dataset served as a strong foundation for Steven’s ML testing and will enable him to develop an effective classification system.

Team Status Report 03/02/24 and 03/09/24

In addition to addressing how our product meets market needs in terms of considerations to public health, safety, and welfare, as social, and economic factors related to the system of production, distribution, and consumption of goods and services, our group would like to explore public concerns from a different perspective. Hence, A was written by Steven, B was written by Surya, and C was written by Grace.

Part A: With consideration of global factors, NutrientMatch can be used by just about anyone across the globe, whether or not their wellness goal consists of consuming more calories or better keeping track of the types of foods they are consuming daily. Regarding younger adults, as fitness is becoming a bigger trend with the emergence of technology and the expansion of gyms, more of these people are aware of their nutrient intake. Our product is heavily targeted towards this group since they tend to live with roommates as opposed to other family members. With this in mind, it is easier to get foods mixed up when groceries are purchased separately. NutrientMatch is able to track each user’s food inventory as soon as they are logged in to avoid these confusions. On the other hand, family members may also want to use this product to track their own intake. Our product can also be used by those who are not as tech-savvy as others. The usage process is easy: after each food item is either scanned or classified and weighed, it is automatically forwarded to the cloud database which will make backend calculations to be displayed on the webpage for users to see. Hence, while NutrientMatch is not a necessity in people’s lives, the increasing trend of physical wellness makes it a more desirable product overall.

Part B: Cultural considerations in the design process are crucial to making NutrientMatch an accessible and adaptable product for those around the globe. The most obvious example that comes to mind is the acknowledgment of dietary preferences in cultural and religious contexts, including but not limited to plant-based diets and veganism, Halal foods in Arabic Culture, and Kosher foods as delineated by the Jewish faith. To that end, we will dedicate significant resources to training a recognition model that can broadly recognize and perform similarly across a variety of testing inputs sampled across these cultural lines, so as to not perpetrate any implicit biases in the recognition process.

Another vital cultural aspect to consider is the idea of nutritional makeup and body image. Various cultures around the world prioritize specific eating lifestyles and different preferences for caloric makeup and subsequent ideas for how the ideal physique should be maintained. While some groups may prefer thinness, with an increased focus on portion control and calorie counting, in other cultures food is synonymous with hospitality and status. Despite these seemingly disparate perspectives on the notion of “ideal” nutrition, NutrientMatch should perform similarly and without bias in all scenarios. Our product will be carefully designed to avoid reinforcing specific nutritional ideals; similar to the idea above, our recognition model and recommendation algorithms will perform as intended irrespective of the actual nutritional habits of a certain user.

Finally, the notion of privacy and communication/integrity of data is an issue that crosses cultural lines as well.  Some cultures view technology and its role in human life differently than others. In developing NutrientMatch, we seek to build trust with our customer base by demonstrating a commitment to the privacy and confidentiality of data; this will help us garner popularity and positive standing even amongst cultures that may demonstrate initial skepticism towards such technology. MySQL automatically hashes and encrypts the stored data in the database, and the HTTPS protocol that will be used to interact with such a backend will prevent hackers from intercepting data based on an extensive SSL certificate authorization process that checks the users and the domain name against what is listed on the SSL certificate.

Part C: Lastly, there is a strong connection between previously analyzed economic factors to environmental considerations. One of our greatest concerns when doing research for the product’s use case requirements was the high amount of foods that spoiled in the United States. This does not only directly impact the economy as people are wasting their groceries, but more foods are added to landfills. Obviously, landfills take up a lot of space and need to be managed periodically. In addition to this, landfills produce odors and methane gas that directly contribute to global warming and the air we breathe. It is important to effectively utilize all the resources we have as the global population continues to increase, and NutrientMatch can help users maximize this use by tracking every item entering their food inventory system. This reduces user stress in remembering the groceries they bought but can also help them manage what to buy next or what types of foods they are currently lacking. Besides personal health, the environment remains one of our greatest concerns and we hope that NutrientMatch encourages users to consume and productively make use of all the ingredients in their home before directly disposing of them.

Shifting towards a discussion on our team’s progress over these past two weeks, we were successful in a lot of the software design from ML to web app development. First, Steven was able to progress substantially in the ML design portion. He was able to finalize tests and validation for the soft-margin SVM formulation to distinguish between fruit and canned foods; however, there are still issues with accuracy which will need to be fixed in the upcoming weeks. Likewise, Steven made substantial progress in researching ChatGPT4 capabilities and the API to integrate its label-reading algorithms into our web app design. Everything is on if not ahead of schedule regarding the ML aspect of our project which will give us extra flexibility in fine-tuning and maximizing the results. We hope to gather statistics to support our design choices as well shortly.

Additionally, Grace was able to help Steven gather data for the ML testing portion, including feature data that would help distinguish between the packaging between fruits and canned foods, label data for training the SVM classifier, and some other relevant information that was used during the labeling of each food item. With the given issues, some fine-tuning may still need to be done further. Regarding individual progress, Grace has completed mockups for the design report and used this towards calibrating the front-end designs. Rendering functionalities between user login, user registration, user home page, and logout pages have been completed. More progress is being made on the web application database retrieval with MySQL and setting up the tables regarding what information should be displayed to users. More testing will need to be done to handle database requests and deployment to the server environment.

Surya focused on advancing the web application in advance of a functional demonstration of the basic web app features at the next staff meeting on Monday, March 11. To reiterate, the demo will focus on site navigation, front-end development, user interaction, and view routing. Some of the particular accomplishments include a refinement of the OAuth feature for registering new users through their email addresses, as well as weight-tracking visuals using the chart.js framework. Looking ahead, the team plans to develop more user interaction features; these include progress posting, friend networks, and a Websocket-based chat service.

Surya also continued research on the scale integration process and will order parts this week. Despite initial hesitancy due to the unfamiliarity with market options, he compiled a list of microcontrollers, with a consideration of factors such as power consumption, pin layouts, and opportunities for streamlined design through custom SoC options. This constitutes a fascinating stretch goal, and one we look to tackle in early April, but for now, we would like to proceed as planned with our simple yet effective MVP approach with ESP protocols.

Despite the progress schedule remaining largely unchanged, we understand this is the point of the semester where we expect to make significant progress toward our product with the interim demonstration arriving soon.

March 9th 2024 Status Report — Surya Chandramouleeswaran

Leading into the Spring Break, my main goals were to continue work on our web application while continuing to develop ideas for implementation of the scale integration in our project.

Regarding the web application, we thought it would be appropriate to have a working demonstration of some of its basic features ready by our next weekly meeting with TAs and instructors – planned for Monday, March 11th. Because hosting it on the Amazon Elastic Compute (EC) instance is not a must for this meeting, we will perform this demonstration on a localhost and will focus on site navigation, frontend development, user-user interaction, and view routing in the presentation of our progress to the course staff. This has been a shared effort amongst the entire group; I would now like to share some of my contributions to the web application:

Google Oauth Feature that registers and authenticates users with their Gmail addresses:

Weight-tracking visuals implemented with chart.js (a JavaScript framework for data management and plotting):

 

This, along with general routing of the pages, constituted some of the things I worked on the past 2 weeks on the web application front. We hope to flush out some of the main features under the context of user-user interaction, such as posting progress, friend networks, following/unfollowing, and chatting with Websockets, in time for the demonstration on Monday.

 

With the scale integration component of the project arriving soon, I took some time to compare and contrast the numerous market options. I had been hesitant to start ordering parts due to my unfamiliarity with the market options; however, in preparation for the design report, I performed a thorough evaluation of the possible microcontrollers that we could use for this stage of the project:

The market evaluation and design trade studies sections of the report required a thorough evaluation of factors I hadn’t previously considered, such as power consumption, pin layouts, and opportunities to streamline the entire design pipeline through custom SoC options. Favoring simplicity for the MVP, we will proceed with ESP-related options for scale communication, while researching more robust and impressive options in parallel. The benefit of these SoC designs is the potential to integrate camera and scale communication all in a single chip. This would greatly reduce the complexity of the project and alleviate concerns with separate stages of the project working with one another. Still, we consider this a potential stretch goal in the context of a simple yet effective MVP.

Steven Zeng’s Status Report for 03/02/24 and 03/09/24

I wrote about section A for the team report; below is my answer.

With consideration of global factors, NutrientMatch can be used by just about anyone across the globe, whether or not their wellness goal consist of consuming more calories or to better keep track of the types of foods they are consuming daily. Regarding younger adults, as fitness is becoming a bigger trend with the emergence of its technology and the expansion of gyms, more of these people are aware of their nutrient intake. Our product is heavily targeted towards this group since they tend to live with roommates as opposed to other family members. With this in mind, it is easier to get foods mixed up when groceries are purchased separately. NutrientMatch is able to track each user’s food inventory as soon as they are logged to avoid these confusions. On the other hand, family members may also want to use this product to track their own intake. Our product can also be used by those who are not as tech savvy as others. The usage process is easy: after each food item is either scanned or classified and weighed, it is automatically forwarded to the cloud database that will make backend calculations to be displayed on the webpage for users to see. Hence, while NutrientMatch is not a necessity in peoples’ lives, the increasing trend of physical wellness makes it a more desirable product overall.

Regarding my progress, I primarily did work before Spring break, and used the designated slack time to relax and clear my head. I focused most of my effort on the design report including analysis of design trade-offs for the ML algorithms. Recently, I have tested soft-margin SVMs to distinguish between fruits and canned foods. I have computed optimal parameters and hyper-parameters using 5-fold cross validation due to a limited data set. It correctly classifies objects only around 74 percent of the time. However, I expect this number to go up as I tune the algorithm to only address the three types of fruits (banana, orange, and apple) and canned foods.

Furthermore, I did more research and experiments on ChatGPT4 and its API to read labels. The results were quite promising, and our focus will be on making sure the image is clear enough for ChatGPT to process. Likewise, I am experimenting with database storage and narrowing down what exactly our group wants to store in the database logs. The current plan is to take a picture of the image then pass it into the classification algorithm. If it is a fruit, our product will directly store the name of the fruit using the GoogLeNet classification algorithm along with caloric information calculated using the weight and online data. If it is canned food, our algorithm will read the label from the front and store the name into the database alongside an image of the back with the nutrition label. This will reduce the latency of storing an object into our database and offload the extra computation to database retrieval in which the image will need to be processed. The other alternative is to do all the computation and calculations when the object is being scanned. This will lead to better memory utilization but worse latency which will be an issue in terms of user experience. We plan to test both these approaches out and determine the differences in latency and memory utilization.

Lastly, I read up on a lot of the work Surya has done with the Arduino as well as cloud deployment. The ML portion is highly dependent on retrieving information from the Arduino (weight and image retrieval) and the ability of an Amazon EC2 server to efficiently run all the computations to populate the database. This involved reading a lot of documentation online to prepare myself for future stages. A lot of the detail and info came from https://aws.amazon.com/blogs/machine-learning/category/compute/amazon-ec2/ which I highly recommend. It gives a lot of fascinating stories and helped me understand the space better.

Overall, I had a relatively productive two weeks and got a good break to clear my head.