Author: ranf

• What did you personally accomplish this week on the project? 

I was the speaker of the design review presentation, so I spent some time reviewing and practicing for it in the first couple days of last week. Then I started on setting up the backend codebase in Swift. Not familiar with Xcode or Swift language before, I watched a few online tutorials to help me get started. I encountered various issues when implementing the real app code, including basic syntax errors, program termination from the simulator’s failure to detect a camera, and permission inconsistency when I tried to export the app to my phone for testing. Those issues took me a while to fix, and thanks to the help from my teammates, I was able to create a camera app that features both photo and video taking. Although the current app is technically the basic version of the iPhone’s default camera app, it is integrable with our own app.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress is mostly on schedule. However, I have some work that needs to be integrated with my teammates on Monday (mainly CV and ML phases). If the integration outcome fails to meet our expectations, we might devote some time over the spring break into it. Moreover, although I completed my assigned task of setting up the mobile app backend, I anticipated the upcoming workload for the mobile app might turn out to be heavier than we thought. Subsequently, I will spend more time in the spring break working on it.

• What deliverables do you hope to complete in the next week?

Continue with mobile app backend programming

CV and ML integration

Recognition accuracy testing

Design review report writing

• What did you personally accomplish this week on the project?

After obtaining MediaPipe hands recognition on loaded video, I continued to work on enabling live video input and detecting dynamic pose in addition to hands. Notably, with instructions from Professor Savvides and Neha, I was able to answer the frame processing rate question from last week. The reason was that a loaded mp4 video might have played at a high frame rate incompatible with OpenCV. However, this issue turned out to be irrelevant to our implementation plan, since our use-case only involves live feed-in from webcam or phone camera. I added a live frame rate monitoring function to quantify our recognition frame rate to be 10s-15s under real-time dynamic capture conditions. Thanks to Sejal’s primitive working ML model, at this point, I completed my assigned task of static alphabetical gesture recognition. Our team then decided to include detection of the user’s upper body’s movement, basically including arm & shoulder poses, as well as the intricate gesture of both hands. Accordingly, I experimented with the MediaPipe holistic model to successfully implement this recognition feature (as shown in the screenshot below, code in GitHub). Lastly, as the presenter for the design review presentation, I prepared for the presentation slides and rehearsed for the speech.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My progress in on schedule.

• What deliverables do you hope to complete in the next week?

Boost CV processing

Detect webcam feed-in of static alphabetical gestures

Design presentation slides and speech

• What did you personally accomplish this week on the project? 

After helping finish up the proposal slides, I worked on getting OpenCV and MediaPipe set up on my laptop. I first experimented with sampling a mp4 video into frames that are stored as images in a local folder, and it worked. Later on, after solving some capability issues when initializing MediaPipe, I successfully utilized the MediaPipe hand detection feature to generate landmarks dynamically on the video input. These landmarks, as we expected, exclusively cover the person’s hands and mark the fingers and knuckles. This accomplishment marked the completion of my assigned task of video processing. However, I noticed that the incorporation of MediaPipe’s hand detection model slowed down the frame processing rate, so I will continue researching for solutions to speed up the process.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

My process is on schedule.

• What deliverables do you hope to complete in the next week?

Boost CV processing

Detect webcam feed-in of static alphabetical gestures

Design presentation slides and speech