Main Accomplishments for This Week
- 3D printing of phone attachment
- CAD model completed
- Printing in progress
- Integration of ML and MediaPipe functionality into web application
- Progress on displaying text via bluetooth
Risks & Risk Management
- As we are completing our integration, the biggest risk would be if the prediction translation seems to be significantly less accurate than the previous implementation using just a laptop webcam. If this is the case, we will use the next week to improve our integration before the final demo, or revert back to a laptop based implementation.
Design Changes- We decided to switch from a mobile app to a web app due to lack of MediaPipe support for IOS. In order to receive an accurate prediction, it is necessary to incorporate pose landmarks, which there is limited support for in IOS. By using a web app, we can incorporate necessary landmarks in a way we are already familiar with.
Schedule Changes
- No schedule changes
Additional question for this week
To accomplish our gesture and pose detection feature, we utilized the mediapipe module a lot. While this is a powerful library for generating landmarks on input image, video, or livestream video, it took us a couple weeks to study its official google developer website, read its documentation, and follow its tutorials to build the dependencies and experiment with the example code. In addition, we also watched YouTube videos to learn the overall pipeline to implement the CV module for gesture and pose recognition.
To accomplish our tasks, we needed to refer to documentation regarding the OpenAI API to integrate the LLM processing to our sentence prediction functionality. Although we are no longer focusing on mobile app development, while developing the mobile app, we needed to refer to IOS app development documentation and demos to gain familiarity with Swift. After switching to a web app, we needed tools regarding the best way to send information between the frontend and backend using Django, the framework we are using to accomplish this. We also found it necessary to use demos and existing implementations regarding incorporating MediaPipe with Javascript. Some learning strategies learned were how to use existing tutorials or demos and iterate upon them in order to accomplish our unique task.