Accomplishments:
This week I transitioned from using MediaPipe Hands in Python to testing its JavaScript version with my computer webcam for real-time detection. I integrated my part into Hanning’s in-browser pipeline and verified that fingertip landmarks display correctly in live video. During testing, I noticed that when the palm is viewed nearly edge-on (appearing more like a line than a triangle), the detection becomes unstable—positions shake significantly or the hand is not detected at all. To address this, we plan to tilt the phone or tablet so that the camera captures the palm from a more favorable angle.
After completing the initial hand landmark detection, I began work on fingertip detection. Since MediaPipe landmarks fall slightly behind the true fingertip tips, I researched three refinement methods:
- Axis-based local search: extend along the finger direction until leaving a hand mask to find the most distal pixel.
- Contour/convex hull: analyze the silhouette of the hand to locate fingertip extrema.
- CNN heatmap refinement: train a small model on fingertip patches to output sub-pixel tip locations.
I have started prototyping the first method using OpenCV.js and tested it live on my webcam to evaluate alignment between the refined points and the actual fingertips. This involved setting up OpenCV.js, building a convex hull mask from landmarks, and implementing an outward search routine.
Next Week’s Goals:
- Complete testing and evaluation of the axis-based local search method.
- Implement the contour/convex hull approach for fingertip refinement.
- Collect comparison results between the two methods, and decide whether implementing the CNN heatmap method is necessary.