Executive summary
This week I took the AV stack from a prototype pipeline toward a deployable, Pi-ready meeting-room front end, and I personally built out the new live operator UI. The result is a production-style ML runtime on main plus a much stronger monitoring/dashboard surface waiting on a feature branch.
Shipped on origin/main – ML/Pi runtime and perception
-
Implemented a multiprocessing runtime that splits audio and vision into separate processes, with OS tuning hooks for affinity, scheduler, memory locking, and thread caps to make the Pi profile realistic under load.
-
Added mic health analysis and a reliability-aware active-speaker posterior that fuses audio speech probability, visual speaking, DOA agreement, temporal continuity, and mic health instead of relying on simple heuristics.
-
Upgraded VAD, denoise, visual speaking, and face tracking paths to model-backed/model-aware implementations, and introduced a beamforming fallback ladder (MVDR → subset MVDR → delay-and-sum → best-mic) plus refreshed telemetry, config validation, recording, and tests.
In progress – UI / operator experience (my work, not yet on main)
-
Overhauled the streaming UI into a WebSocket-based live operator dashboard on the
rilical/debug-ml-pi-pipelinebranch, replacing slow polling with a proper telemetry stream. -
Built out a new dashboard surface: camera tiles, DOA compass, mic health bars, audio meters, lock timeline, event log, and a calibration panel so an operator can actually monitor and tune the system live.
-
Added a camera config API with sidecar JSON persistence and camera auto‑reconnect, publishing status on
vision.camera_status.{cam_id}, so cameras can be configured and recovered without code edits.
Validation and test foundations
-
Integrated and extended regression tests for interferer suppression and end-to-end latency, so the new runtime and MVDR path are backed by measurable timing and off‑axis rejection checks instead of anecdotal performance.
-
Used these tests to validate that the new Pi profile and multiprocess architecture behave as intended before we rely on them in real meeting-room scenarios.