Main Accomplishments for This Week
- The issues we encountered last week carried on to this week as well, but we have made progress in resolving them and continue to work on their solutions:
- The dynamic machine learning models were not performing as expected – regardless of gestures made, the same word(s) are being predicted. We narrowed the vulnerability to be from integration and received feedback to focus on how we are extracting our coordinates.
- We were advised to identify a center of mass or main focal point such as the wrist to subtract from, rather than use raw xyz coordinates for landmarking.
- Hence, we updated our dynamic processing model code and now have been getting improved predictions.
- The transmission from video to database is currently questionable. We desired real-time streaming from the phone camera to the cloud environment so that gestures can be processed and interpreted immediately as they are happening.
- We received lots of articles on this concept to study further. With the diversity of solutions to solving this problem, it’s a little difficult to identify which is best suitable for our situation.
- Hence, we are considering just having the iOS app and Xcode environment directly handle the machine learning and computer vision rather than outsource the operations to a cloud database storage. Research was done on whether in Xcode, python scripts and related code that utilize OpenCV, Mediapipe, Tensorflow, and Keras can be packaged with the app and retrieved within that package. So far, there is promise shown that this can be achieved, but for safety, we will maintain our database.
- The dynamic machine learning models were not performing as expected – regardless of gestures made, the same word(s) are being predicted. We narrowed the vulnerability to be from integration and received feedback to focus on how we are extracting our coordinates.
- Progress on Amplify setup
- Arduino and Mobile App Bluetooth connection
Risks & Risk Management
- With the interim demo approaching, we hope to have definitive outcomes in all our parts.
- We are working on further accuracy and expansion of training data for our machine learning models. Our basic risk mitigation tactic for this in case of setbacks is to remain with static model implementation.
- Regarding hardware, there is a safety concern with operating the LiPo battery, but that has been minimized by extremely careful and proper handling in addition to budget available in case of part replacement needed.
- As mentioned in the Main Accomplishment section, there is a challenge with our plans to integrate ML and CV with the mobile app. At first, we thought of the database, but because of streaming issues, we shifted to having the mobile app have local script access to ML and CV. We will be steadily trying to achieve this, but we will have a backup of the database and even delegate operations into a web app to be converted into a mobile app if the database-to-app transmission continues to be a risk.
Design Changes
- No design changes
Schedule Changes
- We are currently approaching our milestone to launch the mobile app. We will be working together to integrate it, but if we do not achieve everything we want to accomplish by then in regards to the iOS app, then we will change the date.
