This week I focused on getting the vision side of FocusField off the ground so we have something concrete, demo-able, and measurable—rather than just a block diagram and good intentions.
Work Completed
Repository & Project Presence (Scalability & Clarity)
- Set up and organized the GitHub repository to support both MVP and full system builds, with clear separation between contracts, configs, documentation, scripts, and core modules (vision, audio, fusion, UI, benchmarking).
- Created/updated the GitHub project page/site to serve as a single public-facing reference for staff and TAs.
- The goal was to make “what exists” and “what’s next” obvious in under 60 seconds.
Face & Lip Detection Bring-Up (Core Vision Primitive)
- Implemented and validated face detection and facial landmark / lip-region extraction on live video input.
- Confirmed stable face identification, frame-to-frame tracking, and reliable extraction of a mouth/lip signal.
- This signal will later drive active-speaker scoring (using mouth activity as a proxy).
- This work establishes the foundation for the video heatmap and a stable VISION_ONLY target-lock mode prior to audio fusion.
Early Active-Speaker Signal Exploration
- Began translating lip/landmark motion into a normalized “mouth activity” value.
- Designed the signal to be smoothed and thresholded to avoid flicker or instability.
- The approach is intentionally robust to wide-angle distortion and non-studio lighting, reflecting real deployment conditions.
Reference Material
Useful video reference used to align intuition and implementation choices:
https://www.youtube.com/watch?v=03SQxyOz7f8
Risks & Constraints Identified (Hardware–Software Reality Check)
- Wide-angle / fisheye cameras introduce geometric distortion at the edges, which can break naive pixel-to-bearing mappings and degrade landmark quality.
→ Bearing estimation should be treated as a calibration problem (e.g., LUT-based mapping), not assumed linear. - Raspberry Pi compute limits mean multi-camera face/landmark inference will require disciplined FPS and resolution choices, plus tracking between detections to reduce inference load.
Next Steps (Planned for Next Week)
- Build the video heatmap layer: convert per-face mouth activity and global bearing into a 0–360° polar heatmap suitable for clean UI rendering.
- Implement VISION_ONLY target-lock behavior (stable acquire, hold, and handoff) to reliably highlight a speaker without audio input.
- Add basic calibration scaffolding (camera yaw offsets and bearing mapping) so the heatmap corresponds to real angles rather than heuristic visuals.
- Package a short demo artifact (screen capture + logs) for team review and staff feedback.
Net Result
The vision pipeline is no longer hypothetical. We now have working face and lip detection, a concrete path toward a robust speaking proxy, and repository scaffolding that cleanly supports future lock logic and UI integration without rework.