Accomplishments
This week, my team and I worked on integrating our parts together. Since we had some issues with the mobile app and using CoreML, we decided to switch to a web app after evaluating the trade offs. To do this, Ran and I developed the frontend and backend functionalities. MediaPipe provides examples and support on JavaScript, so we extracted MediaPipe landmarks in the frontend, and sent these landmarks to the backend. I structured these landmarks in a way that was readable by the Python code, and used the existing model to output a prediction. I then sent this prediction back to the frontend, as depicted in the image below.
My progress is on schedule as we are working on integrating our parts, but we will have to do some more testing as a team tomorrow before the final presentation, ensuring our validation and verification metrics are satisfied by our working solution.
Next week, I will complete our integration, add to the UI to ensure a seamless user experience, and perform more testing as my team prepares for the final presentation, final report and final demo.
Additional question for this week:
To accomplish our tasks, I needed to refer to documentation regarding the OpenAI API to integrate the LLM processing to our sentence prediction functionality. After switching to a web app, we needed tools regarding the best way to send information between the frontend and backend using Django, the framework we are using to accomplish this. We also found it necessary to use demos and existing implementations regarding incorporating MediaPipe with Javascript. Some learning strategies learned were how to use existing video tutorials or demos and iterate upon them in order to accomplish our unique task. I also utilized github and github issue forums to help debug, if others had any similar errors as I did.