Skip to content

WIP/experimental: render an AEC reference into VPIO (does NOT cancel yet — see real-hardware test)#4

Open
louis030195 wants to merge 1 commit into
mainfrom
feat/vpio-aec-reference-render
Open

WIP/experimental: render an AEC reference into VPIO (does NOT cancel yet — see real-hardware test)#4
louis030195 wants to merge 1 commit into
mainfrom
feat/vpio-aec-reference-render

Conversation

@louis030195

Copy link
Copy Markdown
Collaborator

Gives VoiceProcessingIO the downlink reference its echo canceller needs. Today screenpipe's VPIO is input-only (output element disabled), so Apple's AEC has nothing to subtract and removes 0 dB (screenpipe/screenpipe#3938).

New optional MacosVoiceProcessingInputConfig.aec_reference (a shared mono-f32 ring). When set: enable + mute the output element, force it to mono f32, and install a realtime render callback that drains the ring into it as the AEC far-end reference. The engine feeds it the captured system-audio lane. None = unchanged input-only behavior.

Compiles clean on macOS. Not yet runtime-validated — needs a real Mac (speakers, no headphones, join a call, confirm the remote stops appearing under "me"). Engine-side wiring (feed system-audio into the ring + pass it to this config) is a paired screenpipe PR.

VPIO cancels echo by subtracting whatever the app renders through the unit's
output element. screenpipe doesn't play the far-end (other apps do), so its VPIO
was configured input-only — output element disabled, nothing rendered — and
Apple's AEC had no downlink to subtract, removing 0 dB (mediar/screenpipe#3938).

Add an optional MacosVoiceProcessingInputConfig.aec_reference (a shared mono-f32
ring). When set: enable the output element, force it to mono f32, set
kAUVoiceIOProperty_MuteOutput=1 (reference is for AEC only, never replayed), and
install a render callback that drains the ring into the output element each
cycle. The caller (engine) feeds it the captured system-audio lane. None keeps
the historical input-only behavior. Render callback is realtime-safe: try_lock
+ zero-fill on contention/underflow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@louis030195

Copy link
Copy Markdown
Collaborator Author

⚠️ Real-hardware test: this bridge does not work yet (makes echo worse)

Built it into screenpipe-audio (cpal re-pinned here) and tested on a real Mac: opened a VPIO mic stream with the reference ring, played a speech clip through the speakers, fed the same clip into the ring, captured the mic, and compared echo correlation (mic↔far, wide-search) with the reference ON vs OFF.

  • Native mic rate is 96 kHz; VPIO is a voice unit, so I forced the stream to 48 kHz.
  • WITHOUT reference (baseline echo): corr ≈ 4.1
  • WITH reference: corr ≈ 6.0 — echo went UP, not down.

Interpretation: the rendered reference is not being mutedkAUVoiceIOProperty_MuteOutput isn't taking effect, so the far-end I render into the output element is being played back through the speakers, adding more echo to the mic. So the bridge is currently counterproductive, and separately it's still unproven that VPIO's AEC consumes an app-injected reference at all (vs only audio it routes through its own playback path).

Status: experimental / does not cancel. Needs real CoreAudio debugging — verify MuteOutput scope/element actually mutes, confirm the reference reaches VPIO's AEC reference bus, and validate VPIO engages AEC at the chosen rate. Not mergeable as-is.

@louis030195 louis030195 changed the title feat(coreaudio): render an AEC far-end reference into VoiceProcessingIO (#3938) WIP/experimental: render an AEC reference into VPIO (does NOT cancel yet — see real-hardware test) Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant