Author: oghabaye

A major part of the week was hardening the production boot path and reducing dependence on local machine state. I generated Pi-local runtime config during service boot, resolved cached model paths against the service user, accepted cached runtime models in boot validation, and relaxed service shutdown timing so the service can stop more cleanly. I also tightened the production defaults so the main appliance path is more deterministic at bring-up.

In parallel, I separated the production and demo flows more cleanly. I expanded the demo-safe configuration, added explicit Pi demo profiles for live and observe modes, and tuned the UMA-8 capture path for stability while reverting one audio-tuning regression. I also updated the Pi runbook and demo launch workflow so the demo path is repeatable without weakening the production settings.

Another important thread was making the hardware probe path less brittle on Raspberry Pi. I fixed the camera probe to open concrete V4L2 nodes, allowed probing when sysfs capture metadata is absent, and tightened the Pi contract behavior around those edge cases. That lowers the chance of false negatives during hardware detection on the target platform.

I also expanded the test surface around the areas that changed most: boot validation, production defaults, and Pi hardware-probe behavior. The result is a better guardrail against regressions in the appliance path, especially around cached-model handling and config correctness.

Verification
Verification this week centers on boot correctness, config determinism, and probe reliability. The relevant checks are the updated boot-validation, contract, and production-default tests, plus live boot and shutdown validation on actual Raspberry Pi hardware.

Validation
Validation for this work is whether the system behaves like an appliance instead of a dev box. The main questions are whether a Pi can boot cleanly under the service user, load the correct runtime config, probe cameras consistently, and enter production or demo-safe mode without manual intervention.

Analysis of Results
The main result is a narrower failure surface. The boot path now depends less on ambient state, the demo path is clearly separated from production defaults, and the camera probe is less likely to fail on Pi-specific metadata gaps. That should translate into fewer field failures, less bring-up friction, and a safer operating model for demos.

Schedule
On track with the current Pi hardening and demo-stability push.

Next Steps
Next week, I’d run the updated boot and demo flows on real Raspberry Pi hardware, confirm the production versus demo-safe split in practice, and keep tightening camera and audio defaults until the appliance path is stable enough to treat as the default operating mode.

This week, I focused on moving FocusField’s Raspberry Pi meeting-peripheral path closer to a field-ready appliance by improving production hardening, demo readiness, runtime observability, and vision-stack stability on Raspberry Pi hardware. My work was centered on making the system more reliable on the actual deployment platform, while also improving our ability to measure, diagnose, and validate performance under real operating conditions.

A major part of this week’s effort was formalizing the production path instead of relying on ad hoc setup and bring-up. I added a top-level production release gate, boot validation helpers, guided provisioning and recovery workflows, a support-bundle generator, and expanded the Raspberry Pi runbook. As a result, cold boot, service installation, model-asset validation, and recovery now follow a documented and repeatable workflow rather than depending on tribal knowledge. I also introduced separate meeting_peripheral and meeting_peripheral_demo_safe configurations so that we now have a clearer distinction between the production appliance path and a safer, more traceable demo path. These changes make the system significantly easier to deploy, reprovision, and debug on-device.

In parallel, I improved the demo and benchmarking workflow so that readiness can be judged using measurable evidence rather than informal observation. I added demo rehearsal and host-meeting gates, demo A/B capture and panel reporting scripts, bench profiles, scene catalog and labeling improvements, and expanded FocusBench metrics and scoring. The benchmarking stack now evaluates not only core audio quality, but also latency percentiles, queue pressure, output underruns, lock stability, runtime behavior, and conversation-level signals. This gives us a much more defensible way to compare candidate builds and determine whether a configuration is truly ready for a live demo or production use.

I also strengthened runtime telemetry and observability across the live system. The new performance monitor captures stage timing, queue age, audio capture pressure, output health, face reacquisition timing, fallback behavior, and lock-state dwell time. I added a Pi telemetry watch script for longer soak monitoring and updated the UI telemetry and live views so the active detector backend and runtime state are easier to inspect. Together with the support and recovery improvements, these changes make the system much easier to operate and debug under realistic load.

Another important part of the week was stabilizing the face-detection path on Raspberry Pi builds. I fixed YuNet readiness checks across different OpenCV variants, corrected the Haar cascade path probe on Pi, and lowered wide-FOV face-area thresholds across relevant configurations so distant faces are less likely to be dropped on the 160-degree camera setup. These were relatively small implementation changes, but they address exactly the sort of environment-specific brittleness that can make a hardware demo appear unreliable even when the core pipeline is otherwise functioning correctly.

I also expanded automated test coverage in the areas that changed most, including release gates, demo workflows, support workflows, production defaults, telemetry correctness, output-path hardening, and FocusBench metrics and scene handling. As a result, this week’s work was not just about adding scripts and configuration files, but about building a tighter operational story around how FocusField is provisioned, evaluated, monitored, and recovered when failures occur.

Verification

Verification for my subsystem focuses on whether the Raspberry Pi meeting-peripheral path satisfies the engineering requirements for reliability, observability, and runtime performance. The key tests I have run or am planning to run include cold-boot and reboot validation, service-install and model-asset checks, production release-gate execution, failure-recovery testing, and automated tests for configuration correctness and telemetry integrity.

I am also using FocusBench to verify measurable subsystem behavior. For each candidate build, I will analyze metrics such as latency percentiles, queue pressure, output underruns, lock stability, and conversation-level behavior to determine whether the build meets the expected performance thresholds. In addition, I will use long-running soak tests with the new telemetry-watch tooling to detect issues such as queue buildup, runtime instability, or degraded detector behavior over time.

For the vision subsystem specifically, I will verify that the detector path behaves consistently across Raspberry Pi and OpenCV variants, and that the updated wide-FOV thresholds improve distant-face detection without introducing unacceptable false positives. The goal of these tests is to confirm that the appliance path behaves predictably on the real hardware platform we care about, not just in a development environment.

Validation

Validation for the overall project focuses on whether FocusField delivers the intended meeting experience in realistic use cases. To evaluate this, I am using demo rehearsal flows, host-meeting gates, A/B capture workflows, and panel-based reporting to compare candidate builds in realistic meeting scenarios.

The main validation questions are whether the system remains stable during live use, whether tracking and fallback behavior are acceptable in a real room, and whether the audio and interaction quality are good enough for an actual meeting setting. I will validate this by comparing production and demo-safe configurations, reviewing benchmark and telemetry evidence, and correlating those measurements with observed behavior during real meeting sessions.

Analysis of Results

To analyze the measured results, I will compare each run against explicit engineering and use-case requirements rather than relying on anecdotal impressions. For verification, that means checking whether subsystem-level metrics stay within acceptable thresholds for latency, underruns, queue health, detector stability, and recovery behavior. For validation, that means determining whether those measured improvements translate into a more stable and usable end-to-end meeting experience.

I will also use A/B comparisons between candidate configurations to identify whether a change actually improves system behavior or simply shifts problems elsewhere in the pipeline. In particular, I plan to compare production versus demo-safe runs, examine telemetry trends during longer sessions, and correlate runtime events such as queue spikes or fallback activation with user-visible issues such as lag, instability, or tracking loss. This analysis will help demonstrate that my contribution is not only technically functional, but also meaningfully improves the project’s ability to meet its design and use-case requirements.

Schedule

On track with the current production-hardening and demo-readiness push.

Next Steps

Next week, I plan to run the new release and demo gates using real Raspberry Pi evidence, with particular attention to cold boot, reconnect recovery, and meeting-app input selection on the appliance path. I also plan to use the telemetry-watch and demo-safe tracing workflows for longer soak tests, compare production and demo-safe configurations with A/B capture, and continue tuning detector and fallback thresholds until the wide-FOV rig behaves more consistently under live conditions.

This week I focused on improving the stability, calibration, and observability of the FocusField system on Raspberry Pi in order to make the platform more reliable for integrated testing and upcoming demonstration work. My primary effort went into hardening the runtime so that the system is easier to configure, easier to diagnose, and more dependable when running on the target hardware. Rather than adding isolated features, I concentrated on reducing the friction between configuration, hardware bring-up, runtime execution, and UI visibility.

A major portion of my work involved stabilizing the Pi runtime and strengthening the calibration workflow. I updated the runtime and related tooling so that hardware-specific configuration is handled more clearly and more consistently on-device. This included improving the Pi setup and validation flow, refining calibration support for the camera and microphone configuration, and strengthening the runtime path so that the system better reflects the actual hardware state of the device it is running on. These changes help reduce ambiguity during setup and make it easier to move from development configuration to a reliable deployed configuration.

In parallel, I improved the calibration and modeling of the microphone array, particularly around the UMA-8 geometry and channel handling. I refined the device profile so that the active microphone geometry more accurately reflects the true acoustic layout, and I updated the calibration tooling so it produces more actionable outputs for both device profiles and local Pi configuration. I also added support for runtime yaw offsets and tighter validation around audio channel configuration, which helps ensure that beamforming and direction-of-arrival calculations are grounded in the actual hardware arrangement rather than relying on looser assumptions. This work is important because it directly affects localization accuracy, beamforming quality, and overall runtime confidence.

Another important area of progress was improving runtime visibility and telemetry. I added more detailed runtime metadata so that the system now surfaces resolved audio bindings, camera bindings, applied calibration overlays, active configuration details, and other runtime characteristics more clearly. I also expanded the telemetry layer so the UI can expose richer debugging and operational information, including microphone health summaries, camera calibration state, and other runtime context that previously was harder to inspect. This makes it easier to understand not only whether the system is running, but how it is running and what configuration it is actually using at the moment.

I also strengthened the system’s health and performance reporting so that failures and degradation are surfaced much more clearly. Specifically, I added logic to expose runtime failures in the health summary, including queue pressure, capture overflow conditions, beamformer fallback states, and microphone channel degradation or dropout. I extended performance monitoring to track persistent fallback behavior and queue-related pressure signals, which gives the system a more useful view of when it is entering a degraded operating mode. This is valuable both for live troubleshooting and for demo preparation, since it allows us to identify and explain unstable behavior before it becomes a visible failure.

In addition to the core runtime and telemetry work, I updated supporting scripts and documentation so that the bring-up process is more operationally sound. I improved the runbook and Pi-oriented scripts for preflight checks, local config generation, smoke testing, calibration, and service installation, which helps connect the engineering work to actual repeatable usage on hardware. I also added test coverage around the runtime camera calibration path so these changes are backed by verification rather than only manual testing.

Overall, this week’s work significantly improved the maturity of the software stack behind FocusField. The system is now better calibrated to the target hardware, more transparent in how it reports its runtime state, and more robust in surfacing failure conditions that matter during testing and demos. These changes move the project beyond basic functionality and toward a more dependable MVP that is easier to operate, debug, and present confidently.

Schedule
Currently on schedule.

Next Steps
Next week I plan to focus on final integration and demonstration readiness. This will include verifying end-to-end behavior on the target hardware, continuing to refine camera and microphone bring-up on the Pi, validating that calibration and runtime bindings remain stable across runs, and using the improved telemetry and health reporting to resolve any remaining reliability issues before the demo. I also plan to continue strengthening the host-facing microphone workflow and benchmark readiness so that the system is not only functional, but also defensible from an engineering and presentation standpoint.

Accomplishments This Week

• Camera Mounting Coordination: Spent significant time coordinating with Cole to nail down the precise physical dimensions required for mounting the cameras onto the enclosure. This involved multiple rounds of back-and-forth to reconcile mechanical constraints with our existing hardware layout, ensuring that mounting positions are both structurally sound and optically aligned.

• Code Modifications: Based on the dimensional requirements gathered from coordination with Cole, made a series of code changes to accommodate the updated mounting configuration. This included adjusting calibration parameters, updating positional offsets, and validating that the software correctly reflects the physical setup. Ran several verification passes to confirm that all changes were consistent and correct across the codebase.

• Raspberry Pi Deployment: Successfully deployed the latest build to the Raspberry Pi. Conducted a post-deployment sanity check to ensure the system boots correctly, camera feeds initialize as expected, and no regressions were introduced from the recent code changes.

• System Validation: After deployment, performed manual testing to confirm that the camera modules are recognized by the system and that the configuration aligns with the physical mounting setup. Minor adjustments were made on the fly as needed.

---

In Progress / Next Steps

• Continue refining mounting parameters with Cole as the physical assembly progresses, additional code tweaks may be necessary once the cameras are fully secured.

• Planning to meet with the full team in the coming week to sync on overall project status, assign remaining tasks, and address any blockers ahead of the next milestone.

• Will continue monitoring the deployed Raspberry Pi build for any stability issues and address any bugs surfaced during extended runtime testing.

---

Blockers / Risks

• Finalizing exact camera dimensions is dependent on Cole completing the mechanical design review; minor delays here could push back calibration work.

• No major blockers at this time, team coordination has been smooth.

Overview

This week’s focus was entirely on developer experience and system reliability, specifically centered on repairing the toy quickstart workflow so that new contributors and evaluators can actually run the documented example end-to-end without hitting cryptic failures or following outdated instructions. The work may not be flashy — it lives in the infrastructure layer rather than the feature layer — but it was genuinely overdue and had a real impact on how usable the project is for anyone outside the core team.

---

What I Did This Week

Commit: 9952ce7 — Fix Toy Quickstart Workflow (March 13, 2026)

The primary deliverable this week was a single focused commit that addressed a cluster of interconnected issues in the examples/toy_course/ quickstart path. Here’s a breakdown of everything that changed and why:

Restored Missing Corpus Assets
The examples/toy_course/corpus directory had gone missing from the repo, meaning anyone following the quickstart docs would hit an immediate dead end before even getting to run anything. I tracked down and restored the necessary corpus assets so the documented flow is once again executable from a clean clone.

Removed the Misleading python -m nsmc.cli.main preload Warning
The CLI was emitting a preload warning that didn’t correspond to any meaningful action in the current setup, and it was confusing to anyone reading the terminal output. Removing it cleans up the UX and means the output you see when you run the quickstart actually reflects what the system is doing.

Fixed Assignment-Text Discovery for Proposer Runs
The proposer was failing to locate assignment text during its run, which caused the entire propose loop to silently misbehave. The fix ensures the proposer correctly resolves its input regardless of working directory or invocation path, making the component substantially more robust.

Preserved effective_policy.yaml Overrides During Propose-Loop
There was a subtle but serious bug where overrides specified in effective_policy.yaml were being dropped when the propose loop iterated past the first cycle. This meant that any policy customization a user had made would quietly disappear mid-run — the kind of bug that’s very hard to diagnose unless you know exactly what to look for. The fix ensures overrides persist correctly for the full duration of the loop.

Corrected CLI Witness Path Resolution
The CLI was resolving witness paths incorrectly under certain invocation conditions, which caused downstream failures that were hard to attribute to their root cause. With the corrected logic, witness references are now stable regardless of how the CLI is invoked.

Improved Docker/Gate Failure Messaging
When Docker wasn’t running or gate execution failed, the system was surfacing raw, low-level error output that gave users essentially no guidance on what to do next. I replaced those failure paths with actionable error messages that tell the operator what’s wrong and what to do about it. This is a small change in terms of lines of code but a meaningful one in terms of operational experience.

---

Testing & Quality

I didn’t just patch the bugs — I backed every fix with targeted regression tests. This week’s commit includes 7 new test files designed to catch regressions in exactly the scenarios that were broken. The goal was to make sure that if any of these paths break again in the future, we know about it immediately rather than discovering it when someone tries to run the quickstart and it doesn’t work.

---

Documentation Alignment

One of the quieter wins from this week is that user-facing documentation is now back in sync with actual system behavior. Before this commit, the docs described a flow that didn’t match what the code actually did. That kind of drift erodes trust in the project and makes onboarding harder. The docs now accurately reflect what happens when you follow them.

---

Open Constraints & Next Steps

The one remaining architectural dependency worth calling out is that hermetic gate execution and grading still require Docker to be running. This is an expected constraint for now, but it’s worth keeping in mind for anyone running in resource-constrained environments. The difference this week is that the failure mode is now clear and operator-friendly — you get a message that tells you exactly what’s missing — rather than a raw crash that sends you digging through stack traces.

Going into next week, the quickstart is in a stable and reproducible state. The foundation is solid enough that I can shift attention back toward feature work without worrying that new contributors are going to be blocked by environmental issues before they even get started.

 

_ Sorry for a late submission, I have been moving apartments and did not have internet

Executive summary

This week I took the AV stack from a prototype pipeline toward a deployable, Pi-ready meeting-room front end, and I personally built out the new live operator UI. The result is a production-style ML runtime on main plus a much stronger monitoring/dashboard surface waiting on a feature branch.

Shipped on origin/main – ML/Pi runtime and perception

• Implemented a multiprocessing runtime that splits audio and vision into separate processes, with OS tuning hooks for affinity, scheduler, memory locking, and thread caps to make the Pi profile realistic under load.

• Added mic health analysis and a reliability-aware active-speaker posterior that fuses audio speech probability, visual speaking, DOA agreement, temporal continuity, and mic health instead of relying on simple heuristics.

• Upgraded VAD, denoise, visual speaking, and face tracking paths to model-backed/model-aware implementations, and introduced a beamforming fallback ladder (MVDR → subset MVDR → delay-and-sum → best-mic) plus refreshed telemetry, config validation, recording, and tests.

In progress – UI / operator experience (my work, not yet on main)

• Overhauled the streaming UI into a WebSocket-based live operator dashboard on the rilical/debug-ml-pi-pipeline branch, replacing slow polling with a proper telemetry stream.

• Built out a new dashboard surface: camera tiles, DOA compass, mic health bars, audio meters, lock timeline, event log, and a calibration panel so an operator can actually monitor and tune the system live.

• Added a camera config API with sidecar JSON persistence and camera auto‑reconnect, publishing status on vision.camera_status.{cam_id}, so cameras can be configured and recovered without code edits.

Validation and test foundations

• Integrated and extended regression tests for interferer suppression and end-to-end latency, so the new runtime and MVDR path are backed by measurable timing and off‑axis rejection checks instead of anecdotal performance.

• Used these tests to validate that the new Pi profile and multiprocess architecture behave as intended before we rely on them in real meeting-room scenarios.

This week focused on hardening the Raspberry Pi deployment pipeline and enforcing strict hardware contracts to eliminate silent degradation paths. The system now enforces a fail-fast startup model requiring 3+ USB capture cameras and 8+ microphone channels before runtime initialization. This significantly improves deployment reliability and operational predictability.

Major Accomplishments

1. Strict Hardware Contract Enforcement

• Implemented hard-stop behavior when minimum hardware requirements are not met.

• Integrated strict validation into configuration generation, preflight checks, and runtime startup.

• Added regression tests (test_pi_contract.py) to lock behavior into CI.

• Updated runbook documentation to reflect enforced requirements and expected failure semantics.

2. Camera Topology and Discovery Reliability

• Filtered non-capture V4L2 nodes to eliminate false-positive detections.

• Tightened V4L2 probing logic for improved capture-capable device identification.

• Standardized use of stable numeric /dev/videoN indices to reduce symlink ambiguity.

• Improved explicit 3-camera index generation behavior for safer configuration.

• Reduced noisy probing and unnecessary backend fallbacks.

• Added CAP_ANY fallback selectively to improve backend resilience where appropriate.

3. Preflight and Diagnostics Improvements

• Elevated pi_preflight.py into a formalized deployment gate.

• Improved diagnostic clarity around camera open failures.

• Hardened audio device resolution and operator-facing guidance.

• Documented face cascade fallback behavior and detection validation steps.

• Reduced setup fragility by removing unavailable wheel dependencies.

4. Runtime and System Stability Enhancements

• Enforced fail-fast runtime startup when hardware contract is violated.

• Improved device probing determinism and reduced ambiguous behavior.

• Strengthened configuration generation workflows.

• Added structured fallback profiles where explicitly allowed.

Current System State

The system has transitioned from foundational architecture build-out into operational hardening. The camera discovery stack is now deterministic and filtered to capture-only devices. Startup validation ensures that incorrect hardware configurations fail immediately rather than running in degraded states.

This materially improves deployment reliability and reduces debugging ambiguity in field environments.

Next Focus Areas

• Continued validation under varied USB topologies.

• Long-duration stability testing on Pi hardware.

• Performance measurement under full 3-camera + 8-mic load.

• Additional telemetry improvements for early anomaly detection.

Overall, the system now enforces a strict hardware contract before pipeline execution, significantly increasing reliability and deployment confidence.

Omar’s Status Report – 02/14/2026

This week I focused on transitioning FocusField from a local prototype into a deployable, reproducible system on the Raspberry Pi. The primary emphasis was observability, hardware bring-up, and eliminating ambiguity during debugging.

---

Run Artifacts & Deterministic Debugging

Every run now produces a fully self-contained artifact bundle under:

artifacts/<run_id>/

Each run includes:

• run_meta.json (platform, versions, git commit, resolved devices)
• config_effective.yaml (exact merged config used)
• logs/events.jsonl (structured logs)
• logs/perf.jsonl (latency + performance snapshots)
• traces/*.jsonl (VAD, DOA peaks, beamformer debug, lock state)
• audio/enhanced.wav (+ raw multichannel when enabled)
• thumbs/*.jpg (1 FPS camera thumbnails)
• crash/crash.json (traceback + last-known state on failure)

This allows us to zip a run folder and fully reconstruct what happened without attaching a debugger or guessing.

---

Raspberry Pi Bring-Up

I wrote a complete Pi runbook (docs/30_pi_runbook.md) covering:

• OS package installation
• Python virtual environment setup
• Editable project installation
• Stable /dev/v4l/by-id camera mapping
• UMA-8 calibration workflow
• Smoke testing procedures
• Common failure modes (USB bandwidth, MJPEG, underruns)

Supporting automation scripts were added:

• pi_setup.sh – installs dependencies and environment
• list_cameras.py – verifies stable camera paths
• calibrate_uma8.py – channel order + yaw alignment calibration
• pi_smoke.py – full pipeline validation test
• install_systemd_service.sh – deploys FocusField as a boot service

The smoke test validates 8-channel audio capture, enhanced output generation, camera frame ingestion, and artifact creation.

---

UMA-8 Calibration

I implemented a structured calibration helper for the miniDSP UMA-8:

• Tap-based channel detection (clockwise ring)
• Automatic channel order reconstruction
• SRP-PHAT-based orientation test for yaw_offset_deg
• Auto-generated YAML snippet for device_profiles.yaml

This ensures DOA and beamforming geometry align with physical orientation and camera reference.

---

Audio Pipeline Hardening

Major upgrades to the audio stack:

• Dynamic device resolution (substring matching + required channels)
• SRP-PHAT DOA with configurable bins and smoothing
• MVDR beamformer with diagonal loading and condition number checks
• Delay-and-sum fallback logic
• Wiener frequency-domain denoiser
• Channel drift detection (GCC-PHAT offset estimation)
• Beamformer debug telemetry (target, fallback, gains, conditioning)

The UI now includes:

• Beamformer status panel
• Gain visualization bars
• Health monitoring display
• Runtime latency metrics

---

Crash Handling & Fail-Fast

Thread-level and system-level exception hooks were installed:

• Automatic crash reports in crash/crash.json
• Traceback + last known system state
• Configurable fail_fast behavior

This makes Pi debugging viable even without live introspection.

---

Observability & Monitoring

• Health monitor with topic staleness thresholds
• Bus drop counters
• Runtime performance monitor (latency estimation)
• Structured JSONL logging
• Live telemetry aggregation

The system now explicitly surfaces degradation rather than silently failing.

---

Current Pi Setup Status

• Stabilizing 3-camera bring-up via /dev/v4l/by-id
• Validating UMA-8 7+1 geometry alignment
• Testing USB bandwidth under full load
• Repeated smoke tests for deterministic startup
• Preparing systemd deployment for continuous runtime

The objective is a configuration that boots cleanly, auto-starts the pipeline, generates valid artifact bundles every run, and remains stable under sustained load.

---

Next Steps

• Validate MVDR stability in real room conditions
• Tune DOA bin count vs CPU load on Pi
• Run extended stress tests (30+ minutes)
• Confirm thermal stability
• Finalize systemd deployment
• Begin quantitative FocusBench replay evaluation

FocusField has now transitioned from prototype to deployable system. The Raspberry Pi environment is structured, reproducible, and debuggable. The remaining work is stability tuning and performance validation.

— Omar

This week I focused on getting the vision side of FocusField off the ground so we have something concrete, demo-able, and measurable—rather than just a block diagram and good intentions.

---

Work Completed

Repository & Project Presence (Scalability & Clarity)

• Set up and organized the GitHub repository to support both MVP and full system builds, with clear separation between contracts, configs, documentation, scripts, and core modules (vision, audio, fusion, UI, benchmarking).
• Created/updated the GitHub project page/site to serve as a single public-facing reference for staff and TAs.
• The goal was to make “what exists” and “what’s next” obvious in under 60 seconds.

---

Face & Lip Detection Bring-Up (Core Vision Primitive)

• Implemented and validated face detection and facial landmark / lip-region extraction on live video input.
  

  

• Confirmed stable face identification, frame-to-frame tracking, and reliable extraction of a mouth/lip signal.
• This signal will later drive active-speaker scoring (using mouth activity as a proxy).
• This work establishes the foundation for the video heatmap and a stable VISION_ONLY target-lock mode prior to audio fusion.

---

Early Active-Speaker Signal Exploration

• Began translating lip/landmark motion into a normalized “mouth activity” value.
• Designed the signal to be smoothed and thresholded to avoid flicker or instability.
• The approach is intentionally robust to wide-angle distortion and non-studio lighting, reflecting real deployment conditions.

---

Reference Material

Useful video reference used to align intuition and implementation choices:

https://www.youtube.com/watch?v=03SQxyOz7f8

 

---

Risks & Constraints Identified (Hardware–Software Reality Check)

• Wide-angle / fisheye cameras introduce geometric distortion at the edges, which can break naive pixel-to-bearing mappings and degrade landmark quality.
  → Bearing estimation should be treated as a calibration problem (e.g., LUT-based mapping), not assumed linear.
• Raspberry Pi compute limits mean multi-camera face/landmark inference will require disciplined FPS and resolution choices, plus tracking between detections to reduce inference load.

---

Next Steps (Planned for Next Week)

• Build the video heatmap layer: convert per-face mouth activity and global bearing into a 0–360° polar heatmap suitable for clean UI rendering.
• Implement VISION_ONLY target-lock behavior (stable acquire, hold, and handoff) to reliably highlight a speaker without audio input.
• Add basic calibration scaffolding (camera yaw offsets and bearing mapping) so the heatmap corresponds to real angles rather than heuristic visuals.
• Package a short demo artifact (screen capture + logs) for team review and staff feedback.

---

Net Result

The vision pipeline is no longer hypothetical. We now have working face and lip detection, a concrete path toward a robust speaking proxy, and repository scaffolding that cleanly supports future lock logic and UI integration without rework.