Posted on

Feb 17th 2024 Status Report — Surya Chandramouleeswaran

My roles this week involved a dive into the design process in preparation for the upcoming design presentations. In particular, I decided to take the opportunity to present our team’s design plans next week. Preparing for this presentation requires strongly re-evaluating current project parameters and scope. Additionally, we found the feedback from our peers and faculty quite valuable from the proposal presentation, and we are eager to reflect on their suggestions in our design process. Building these slides, in concert with feedback and an improved understanding of our project, forced our group to critically examine every component that we plan to develop, in the hopes of convincing our peers of the value of our product and the merits of our implementation approach.

For my specific role in the project, I started looking ahead into scale integration. Learning the requisite communication methods was a focus of mine this week, especially in the context of seamless integration with mySQL. I came up with 3 proposals, weighted carefully for their relative merits and drawbacks:

  1. Arduino RS-232: There is a lot of support for this chip, which serially communicates the weight reading to a computer through USB. This approach would involve a fair amount of scripting to implement a “polling feature” that Prof. Savvides and our group discussed this week in our meetings. In particular, the code would need to sample the weight reading at a high enough frequency (possibly every second) to detect significant weight changes, to then write to the database. Although this is a good starting point for the project and is largely scale-choice invariant, I am not too fond of this approach given that RPIs are a little easier to work with in this context than Arduinos, and the fact that a USB connection may be rather unwieldy.
  2. ESP-8266: This option was one that I had to do a bit of searching for, but this chip works quite seamlessly with mySQL. Along with the scale and the chip, the final design would presumably involve a load cell amplifier, 5-volt power supply, and jumper cables to hook everything together. The chip would have to be controlled through PHP code, which is something that I would need to familiarize myself with, and the documentation/support is not nearly as good as it is for a product like Arduino or RPI. Much like the previous example, it functions as a good proof-of-concept approach.
  3. Xiaomi Smart Scale with Bluetooth RPI communication: Also works seamlessly with mySQL, and has the benefit of being a Bluetooth communication in terms of elegance, but requires most implementation and background knowledge to successfully build. Some of the challenges include proper recognition of the MAC address of the Bluetooth dongle (for the program to recognize the RPI), and some additional code for the busy-polling feature we would like to implement.

 

My preference is to leverage one of the first 2 options towards a working idea, and then, maybe as a stretch goal, work with my teammates to getting a Bluetooth communication setup working. It would greatly improve the aesthetic of the project and would automate several of the features we are looking to incorporate.

 

As a group, we started laying the groundwork for our project’s essentials, and I helped across the scope for such collaborative ventures: exploring image recognition and classification architectures with Steven, to working with Grace on backend deployment. Grace’s expertise in local deployment (learned from her web applications course) gave us a proof-of-concept idea of deploying a scalable application through DB.sqlite3. We are debugging some of the view actions as part of the MVC architecture, and remain committed to our goal of transitioning to a flexible mySQL database. One of the main things we learned about is the importance of the database in achieving our latency goals. Updating and reading from this database are operations that should fit within our overall latency requirements of 30 seconds or less. Steven has expertise in efficient database manipulation and traversal, and I look forward to working with him to ensure the complexity of database traversals can be done in logarithmic time as opposed to linear. We plan on leveraging the fact that mySQL organizes workbench data as an object hierarchy (GRT Tree), which should go a long way toward efficient traversal and manipulation.

 

We look forward to design presentations this week as the semester begins to unfold.

Posted on

Grace Liu’s Status Report for February 10th, 2024

During this week, I presented for my group’s proposal presentation on Monday. My group and I initially worked together to expand on some of the ideas from our abstract and accurately reflected them onto the required themes of the presentation. I was then able to elaborate on the slide requirements to help the audience better gauge our product use. Since we had already established our problem and solution requirements, reasonable quantitative metrics were defined that state facts about our system.

Using the hardware components my team members scoped out, I was able to sketch a mockup of our envisioned product in CAD with the corresponding dimensions to be included in our solution approach. I believe this was an important step to visually demonstrate to our audience how users would utilize our food tracking product in realtime. Afterwards, the steps to the project were split into three different components that could be worked on asynchronously: image classification, storing scanned information to a cloud database, and the website interface for users to interact on. 

Moving onto technical challenges and testing verification, communication between each component would be hard to achieve in an efficient manner since image classification, scale weighing, and forwarding backend calculations to the database would all need to be processed and displayed in a relatively short amount of time. There also exists the decision of choosing between a wireless implementation or connecting our hardware to be offloaded to a computer for debugging purposes. We will decide between the two upon further testing. In parallel to the steps of our solution approach, verification and metrics were defined to guide our initialization process.

In terms of progress, I am on schedule with what I wanted to accomplish this week. However, upon receiving feedback during our Q&A session and overall notes from the faculty, there are many additional factors that need to be considered. I will need to review more on the technicalities of the OCR algorithms we will be testing with. Additionally, we have established that cooked meals will not be part of our testing plan and our product serves more as an inventory tracker for food.

A mockup of the website framework has been created that allows users to login through either a new account or Google OAuth. At the moment upon logging in, there is dummy data that will eventually be the food items users scan in. I plan to add functionality that lets users delete food items as they desire to be added to the total macronutrients count. Also, my group and I may discuss more on how interactive we would like to make this user interface and ask Professor Savvides for additional feedback.

Posted on

Team Status Report for February 10th, 2024

As a team, we had a productive week in which we completed our presentation and all were on schedule in completing the planned tasks. Here is an image of a nutritional label algorithm Steven tested out:

It reads macronutrient values from a .png image that we found online. We are aware of some issues with this algorithm not calculating the calories, and the model is relatively outdated. We are expecting challenges in fine-tuning the code and/or making our own nutritional label reading algorithm. 

Furthermore, Surya was able to work with a potential camera that we will integrate into our design. He also worked alongside Steven to finalize the list of parts from inventory and from the purchase list, including Arduinos that are optimized to work with scale models for the next stage of the design (RS-232). While the group doesn’t plan to work on the stage in the next 2-3 weeks, prioritizing energy and resources to a robust recognition system in the meantime, given all the parts are interconnected/have to work with one another, it is an important design feature to design functionally-independent yet cohesive components that work well with one another in a complex project.

 

In addition to that, Grace was able to build the Django framework for the front-end by initializing all the files. She plans to have a front-end implementation next week to represent the user interface that we plan to connect the database to. The website portion of our project is up to speed with our anticipated schedule.

 

Through our work this week, we explored our options more in-depth and did research on previous projects. We are expecting to see risks in integrating the Arduino to do all the OCR and computational work. Our backup plan involves defaulting to using our computer to do the primary workload. Likewise, the equipment we have and plan to have are compatible with both a computer and/or an Arduino. 

 

We hope to make significant progress in the creation of the physical product (a wooden box with LED lights to illuminate an object). Likewise, we seek to make progress in our individual assignments which involve website design, algorithm development and integration, and interfacing the hardware with the software. 

 

Lastly, thank you to all the students, TAs, and professors for providing commentary on our presentation and overall project. We received great feedback to better guide us in our design. We are now more aware about potential problems and have thought about alternatives in the event of issues. We recognize the value of this course towards our personal and professional development towards becoming engineers who can both solve problems and communicate their solutions effectively.



Posted on

Steven Zeng’s Status Report for February 10th, 2024

Regarding my progress, I looked through online libraries for OCR (Tesseract) and nutritional label reading algorithms. I looked through them for important functions and how to integrate them into our project. The githubs that I have viewed are the following: https://github.com/openfoodfacts/off-nutrition-table-extractor/blob/master/nutrition_extractor/symspell.py and https://github.com/tesseract-ocr/tesseract. The purpose of doing this is to provide myself with a better understanding of how these libraries work and areas in which I can fine-tune them for our specific project. I found that the nutrition label extractor seems to only take.png images, and I changed the code to support newer versions of numpy and Tesseract. This algorithm does not do as well as we expected, so we might need to modify the algorithm or completely scratch it out.

 

In addition to this, I did research online to create a purchase list. I first found a convenient scale to use. We are looking for a scale that has the ability to connect to a laptop and send weight data to the computer. I will work on this part of the project later when we receive the scale; the scale seems to work with excel sheets, so I would need to understand how to convert an excel sheet into database entries. I also looked for LED strips to illuminate the item. All of this work has been added to a parts list that is shared amongst my team.

 

Currently, my progress is on schedule; however, looking through the library, I expect some issues with integrating various functionalities which might exceed the estimated time. However, I plan to get a better idea next week after I do basic small tests using my macbook camera. 

 

Overall, this week I focused primarily on presentations and organizing work for the following weeks. Next week, I hope to get physical proof that the OCR and nutritional label algorithms are working more effectively with example items and classifications displayed on a console. Next, I will look into image classification to classify fruits and vegetables. Finally, I plan to work with Surya to get access to a makerspace at TechSpark to do wood work.



Posted on

Feb 10th 2024 Status Report — Surya Chandramouleeswaran

This week, I helped scope out some of the hardware tools needed to interface with our imaging system, which is the first component of interest for our product. I helped develop a plan for our team to have this component of our product finished and fully tested in about 6 weeks.

The first aspect of this involved researching existing cameras that met the frame rate and pixel resolution benchmarks that we needed for our OCR technology. Last week, during our weekly meetings, Prof. Savvides was kind enough to let me borrow a camera for the image recognition pipeline, suggesting that simple desk cameras would suffice for our applications, given the product would be held in a stationary manner. Additionally, Prof. Savvides offered to extend his expertise for making the detection algorithm as efficient as possible; I look to work with him and my teammates to build this processing pipeline.

Additionally, the layout of our apparatus, as well as details of the interconnect, were finalized; as the primary person in charge of hardware in this group, I took the initiative on this front. For computing reasons, and after prolonged discussion with the software-inclined members in our team, I decided that running the detection algorithms on an Arduino would not be feasible. While we certainly will order Ardunios (RS232 is rather cheap given the budget constraints of the class), I envision that offloading the computational processes to an external computer may be the best course of action in the beginning. After speaking with Prof. Savvides and our TA Neha last week, we agreed that it is best to test the performance of such tools before dismissing them.

 

We are on schedule progress-wise. We need to start building the recognition system and integrating that with the camera. I look forward to working with Steven on building some helper functions for our algorithm through Tesseract. Right now, we would like to test classification on our MacBook Laptop Camera, then integrate with the web camera that Professor Savvides provided us.

 

One of the design questions that has arisen is if we should implement an auto-capture feature on the camera (where the photo is captured when the camera deems it in focus). For training purposes, it is best if the image set is rather standardized in nature. This weekend I will brainstorm approaches to ensuring that the image classification algorithm gets quality images with ambient lighting and clarity in reading.

 

In the next week, I look to work with Steven in scaffolding the image recognition system and understanding what is expected from the camera which the system should speak to. I plan to begin testing on our built-in computer cameras, as I expect to work on some form of an autofocus-autocapture feature on the camera to ensure the training process of our classification neural network goes smoothly.

 

Attached please find a copy of our proposed design sketch:

Design Sketch