Author: gmliu

As Steven prepared for the final presentation, Surya and I helped provide any necessary feedback to ensure the presentation effectively portrayed our current work in progress along with incorporating previous presentation suggestions. While most of the hardware components remained static, the design tradeoffs involved weighing the pros and cons of different image classification libraries, showing greater overall benefits in Tesseract OCR and ResNet-18. With that being the main focus of the presentation, we tried allocating less time towards the video demos but rather verbally explain more.

Since we are approaching all of the final components to our project, I noticed there were still some frontend components on our web application that could be improved involving user experience, performance, and functionality. First, focusing on the UI, the previous designs had too much clutter on one side of the screen so it is important to ensure appropriate whitespace is utilized to not overwhelm users with too much information. I had some trouble with maintaining a consistent branding and typography on the pages since they do range from creating new posts for the global page to displaying parsed caloric information from uploaded images. I will receive further feedback from user testing to see which pages are easier to navigate and proceed from there.

Additionally, since we have made substantial progress in integrating the subsystems together, there were improvements to be made on the interactive elements of the web application upon its setup on Nginx. Since all the necessary buttons and forms have been implemented onto the UI, I worked on making certain ones stood out from others to ensure users visually notice them first. Another issue that emerged was providing feedback and error states/messages to users after their actions such as scanning the product with the camera. Providing helpful error messages can immediately help users identify their incorrect behavior and receive necessary feedback as soon as possible to improve.

Since I previously helped Steven in gathering data for the ML algorithms, we also worked together in its testing and validation. Specifically regarding testing, because we did not yet reach our initial goals in accuracy determination, we conducted various tests on our label reading and image classification algorithms to improve their validity. The crucial part of these tests is to ensure they perform well under controlled conditions before letting users use the product in real time. We previously spent a lot of time with clear images of these nutritional labels and fresh produce, so the necessary action now is to conduct live testing with the RPi cameras, allowing us to assess how the ML algorithms adapt and perform in real-time. This transition in testing is intended to close the gap between theoretical accuracy and real life applicability of the various trained algorithms.

With the remaining time, our group will shift efforts towards maximizing efficiency of our algorithms and make necessary fixes before the final demo on Friday. As I have previously worked on research posters prior to this class, I will be mainly working towards finishing that before the deadline. My group and I look forward to presenting our semesters’ worth of work to the faculty along with others who may be interested in learning more about our product and how it could be expanded to a wider application in the future.

With the final presentation and many other deadlines coming up, I am anticipating the final steps we will make and progress towards having a completed MVP for the final demo. A portion left on the user interface was integrating the MySQL database into the environment and setting up user roles for what they are allowed to and not allowed to manipulate. I was able to learn most of this information from online tutorials as well as previous knowledge from my Web Application Development and Security classes to prevent injection attacks.

Different data input/output operations were defined including the upload of new entries after successful scans from the hardware, deletion by user after consumption, and updating daily intake after this deletion. I also did individual database testing to ensure each of them operates correctly. In the remaining time, I will work with Steven and Surya to further check the integration between the hardware and web application display.

Since another presentation is coming up next week, a lot of the group’s efforts were concentrated on revising and completing the final presentation. I helped Steven with a video demo to be included in this. Most of the changes from the design presentation include highlighting changes made between the original design and our current design now, solution approach evolutions such and social and global considerations, and most of the completed testing to balance between design trade-offs. Most of the designated software/hardware design changes have been highlighted in our previous presentation, but we’ve decided on additional pages to be included on our web application for more versatile usage. For instance, in consideration of social concerns like higher interaction between different users and prevention of misusage, we’ve included a more social aspect of the application that allows users using the same device to optionally see each other’s food activity. Additionally, there is a page that takes in user inputted calorie threshold for them to view whether or not their calorie intake has exceeded their personal goal or not. 

In regards to testing, since most of the web application functionalities have been completed for the interim demo, I used the extra time to help Steven with his ML algorithms and Surya with scale calibrations. Since our project diverged towards ResNet-18 for CNN, I helped Steven with image preprocessing to ensure the system would receive the most readable images and achieve higher accuracy reading. We conducted batch testing to increase efficiency and make full use of our computational resources.

In regards to the scale, range and sensitivity testing was done to see if small weight changes could be detected. That is most of the testing that needed to be done, and if there is additional time, we can also conduct durability testing to verify the scale’s performance under different temperature conditions. While improving the performance of each subsystem, we are shifting our efforts towards system integration testing to monitor the system’s abilities to handling errors and how they could possibly be reported to the user.

This week was decently busy as my group members and I prepared for the interim demo. While not all of the components are properly connected together at this moment, we made good process on each of the subcomponents. I made some additional progress based on feedback and takeaways from the ethics discussion class since the last time we presented during our team meeting.

During this stage where we were trying to merge components of the web application including Surya’s scale readings, Steven’s ML classification, and my frontend additions for the user, there were conflicting changes that prevented pulls from our respective branches. Additionally, there were authentication issues that came up with certain additions, so I had to trace back to the most recent git commit that did not have this problem. This caused a lot of lost time to work on the functionalities, so the additional time to figure out the authentication problems we were kindly granted allowed me to finish most components we planned on having on our website.

To elaborate on the additions, I will include screenshots and explanations of pages included in our demo below:

Our WebSocket library and server have already been set up, but latency issues still persist so during the validation phase, I will look more into content delivery networks or choosing a different message format to minimize additional network traffic. For now, the page is able to display chat messages that are updated in the UI but more testing during the deployment stage will have to be done to ensure users are receiving their respective messages.

The calorie consumption progress page below is something we considered when brainstorming how to improve the user experience and provide more fitness encouragement. By displaying such a line graph of all of the user’s daily tracking, they are able to see their overall improvement after using our application. I ran into issues with rendering the inputted data and customizing the graph to update with each additional update. Enough testing was implemented to ensure efficient data synchronization between the client as well as the server.

This page is more to show proof of concept since there is still more work to be done with connecting the classification and scale readings to the cloud database. But essentially, inputting values for a certain day will be retrieved and displayed in the rendered chart below. I was having trouble saving values from previous days, but that is something that would be included in the end product to show progress. Perhaps a diagram that can directly compare improvements and vice versa.

Lastly, also for proof of concept and connected to the data retrieved from the previous calorie progress tracking page, users can set a calorie limit to see whether or not they have exceeded this limit on a daily basis. This is achieved through a simple if-else statement on the HTML page where red means they have exceeded their set calorie limit, and vice versa for green entries.

To answer this week’s additional prompt, more testing has to be done regarding individual components I am working on as well as API integrations. For instance, unit testing will be done to ensure the backend is able to reject invalid data like negative calorie entries or if a certain limit is exceeded. CRUD operations will need to be tested to ensure they work correctly and this changed data would be stored correctly. This would include deleting chats between users, deleting posts or comments under posts, or deleting caloric entries under the different pages that include these functionalities. In addition to proof of concept, we will also include deleting entries that will be added to total daily consumption.

Some basic testing for user security would be compatibility between different browsers to ensure consistent behavior across them, usability testing to gather navigation feedback and general user interface feedback, and more security tests. Previously, I conducted tests on SQL injection and cross-site scripting attacks. Since OAuth is an inclusion, proper authentication as in only CMU users are able to use this application upon signing in. Another thing that will be considered is file upload testing. Users are uploading profile pictures or (for now) pictures for posts that will eventually be pictures from the camera scanning, vulnerabilities will have to be tested to ensure malicious files would not compromise the server.

Considering most of our subsystems have not been connected together yet, I plan to finish all my testing to prevent future complications with my specific subsystem. Specific requirements that must be met include proper updating and including as many vulnerability prevention measures as possible for most common security vulnerabilities. I look forward to further discussion with my group members and identifying important validation points for the merging of subcomponents.

This week, our efforts were focused towards completely integrating one main subsystem for the interim demo that will be happening next week. Since our main goal is to integrate correct food classifications using ML into the website, I focused on our shift towards TensorFlow for OCR that seems to have a differing approach from traditional OCR libraries in which we were previously using the ChatGPT API. This approach had a lot more configuration issues that Steven and Surya encountered but would ultimately be more beneficial for classification and detection for our MVP in the end. The connection between their work and my role is to experiment with how the TensorFlow model could be integrated into our web application for proof of concept.

After the model selection, I set up a Python environment along with some necessary libraries for our web server for something that best suits our needs. I created the user interface where files are uploaded to the server from the camera, but some more work needs to be done involving image compatibility and resizing and resolution considerations to maximize user experience on the globals page. The good thing about the TensorFlow Python model is that it is able to extract text from the user scanned image that can be converted into a readable format. This text is saved into the format we want that would be displayed under posts once that functionality can work. This should also be saved into the database for the food inventory. Further testing with the OCR process will need to be done to improve performance and scalability and ensure it works as expected under various different conditions such as errors and edge cases.

In addition to TensorFlow integration into our web application, I was able to tackle some challenges in relation to GitHub version control. We encountered some merge conflicts involving branch pulling and pushing that temporarily hindered us from making quick progress. After going through Git’s version control features and going through each of our branch changes, these issues were resolved efficiently to address merge conflicts.

Looking ahead despite encountering various challenges, I will focus on refining our TensorFlow integration into the web application and tinkering the other features we were planning to focus more towards in consideration to the previous ethics discussion that happened in class. The integration of TensorFlow will help enhance our functionality and perform more advanced and high performance image analysis.

This week, I made significant progress in frontend design and usability. I used bootstrapping to make formatted input and output design which is particularly ideal for responsive designs where CSS is more ideal for layouts that require more flexibility (also used in our web application). I also worked towards building secure database access. In order to do so, I sanitized user inputs and outputs to prevent SQL injections and executable JavaScript attacks. It is important to validate and catch any potential malicious user input as well as encoding the output to stop malicious inputted data from users that would trigger any questionable behavior from the web browser. This was motivated primarily by the ethics discussions in which privacy was of utmost importance for our team and we now know how big of a responsibility it is in regards to out capstone project.

Another thing I looked into was web sockets and improving its speed, efficiency, and syncing functionalities. There was a big issue with a loop that caused extremely slow access time to the database. Going back to the previous discussion, security concerns that arise with this API include cross-site scripting, cross-site request forgery, and injection attacks. Input validation is also a huge part of preventing this types of attacks. I defined data types for expected input structure so there are constraints on user input messages for instance on our globals page where users can make posts. I added a lot of data to the database to simulate a real-world use case which includes images of all types that I gathered and gave Steven for his ML training dataset from the previous week. This included around a couple thousand data points and values for the modeling.

Likewise, I worked with Steven in figuring out how to integrate ChatGPT API into the web design framework. We made a dummy prototype page to test if we could easily interface it with basic input and outputs using the API and also considering the user experience and interface design. Now there is a somewhat smooth communication on our framework between users and ChatGPT since we are considering more engaging interactions. The next step we look forward to working on more is to allow ChatGPT to recognize and analyze images which we will experiment with next week. With our sights set on this experimentation, we believe this propels our project towards even more functionality and are excited how this will pan out during our interim demo.

This week, our group really focused on understanding each component of the ethics assignment and how our project is applicable in real world scenarios. Out of the two case studies, I thought “Do Artifacts Have Politics?” by Langdon Winner was especially interesting since this is a perspective that most people wouldn’t consider merely looking at a piece of technology. It is very eyeopening to see how things such as a bridge design in Long Island had such symbolic meaning behind it, truly reflecting the creator’s political viewpoints and opinions. While this is applicable to inventions of things to resolve societal affairs (keeping the lower class from using recreational resources), another category of these inventions involves those that are part of political relationships. I liked taking a current technology that is hot right now and applying these types of concepts to it since this really helped me realize the design and ethics components behind each step of the design process. In terms of considerations for public health, public safety, and public welfare, the added perspectives from the case studies will definitely inspire us to pay more attention to these details to ensure a safe and friendly product for our users.

In terms of the web application, changes were added to include public posts for users to interact with each other. This was inspired by the realization of public welfare and the negative effects they may cause for users in terms of mental health and body image issues. The globals page is in attempts to promote an environment that is inclusive and encourages positive self-image where users can voluntarily choose to post their food consumptions and add a caption to it as well. While for MVP, this feature would only be useable for those who share the product in the same household where all the food inventory is gathered, we envision there is a lot more potential on a global scale for it to become something more impactful and makes our product something beyond a food tracking tool.

From the previous week, while OAuth was set up to work properly, some debugging on the registration page had to be done to ensure all the information was rendered properly on the profile page. An issue that emerged was displaying the uploaded profile picture is that we have to take into consideration the file size and file format compatibility of the image users choose to upload. A large image could potentially consume too much bandwidth, so either limiting file sizes would have to be implemented or we could use a content delivery network to improve our website’s performance and speed. This approach can be particularly more beneficial since files can quickly be uploaded from any part of the world, and this is something important to consider since we want our application to be used on a global scale for more user interaction and positivity promotion. I would still like to do a bit more research on this approach since it can be more costly as opposed to other methods such as using the web server’s file system or cloud storage that would also come at an additional cost.

I look forward to seeing more of our project flesh out as Surya gathers the physical components together and more progress is being made on the ML algorithm with the data I collected with Steven last week. I envision a user-friendly product that will really be an application beyond merely calorie tracking and a food inventory.

The week prior to spring break, our group spent most of efforts in producing our design report since this is a significant part of our capstone. Since most of my work involves knowledge from the class Web Application Development, I spent a lot of time combing through old lectures for React and JavaScript help since it had been a while since I’ve taken the class. The mockups added into the report for a registration page, a login page, and an inventory/main page can be seen below and have been used as inspiration for the actual frontend designs:

The rendering between these pages have been complete in view.py along with the logout button functionality. Users have the option of logging in using Google OAuth as opposed to creating a separate account. Many benefits come with this service, including the elimination of users sharing their passwords to third-party applications by using a token-based authentication system and providing a more seamless user experience that is one click away. While our product is pretty self explanatory to use, we still want our users to have a most simplified user experience as possible. This was set up on the Google Developer Console to configure the OAuth consent screen for how it will be displayed to users. During further testing processes, we will test that the authentication flow works as smoothly as possible with the best security measures.

Since most of the frontend application has been completed, I was able to take some time and start on setting up the database retrieval process on our website using Django. Upon evaluation and comparison of different databases in our design report, we ended up settling on MySQL for many reasons. It all ended up boiling down to its reliability in handling large datasets and high compatibility with web application and Arduino programming. The structure of our database will look similar to the personal inventory page as pictured above, but may be incline to slight changes depending on how we want the item status to be particularly displayed.

I also assisted Steven in gathering data for his ML testing. Specifically, this included feature data that would help distinguish between the packaging between fruits and canned foods, label data for training the SVM classifier, and some other relevant information that was used during the labeling of each food item. For the feature data, various attributes such as visual appearance and material composition were collected to help capture the differences in packaging appearances related to fruits and canned foods. Additionally, the labeling process involved the careful examination of the packaging and verification against established criteria to ensure enough accuracy. This comprehensive dataset served as a strong foundation for Steven’s ML testing and will enable him to develop an effective classification system.