Use AudioEngine ADM for PlatformAudio on Apple platforms#1215
Use AudioEngine ADM for PlatformAudio on Apple platforms#1215hiroshihorie wants to merge 5 commits into
Conversation
Select the new kAppleAudioEngine audio layer when creating the platform ADM on iOS and macOS. The AVAudioEngine based ADM supports runtime switchable voice processing and device change handling. Construction happens in the proxy constructor on the worker thread, which the AudioEngine device binds its sequence checker to. The proxy also forwards the ADM platform voice processing interface (topology, path availability and toggle, processing state). WebRTC's audio processing resolution calls these on the ADM when track audio options are applied, so the coupled Apple AEC+NS path is managed by WebRTC itself with no new Rust API surface. Docs updated from VPIO wording to the AudioEngine ADM. Requires a libwebrtc build that includes CreateAudioEngineDeviceModule.
The AudioSourceInterface hunk stopped applying after the fork added SetOptions right where the patch expected the class to end. The build script tolerates patch failures (git apply || true), so building a prebuilt from current m144 would have silently produced a libwebrtc without external audio source support. Content is unchanged, only the surrounding context is regenerated against m144.
hiroshihorie
force-pushed
the
hiroshi/audioengine-platformaudio
branch
from
675467c to
dddb8f2
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The active_*_type getters reported Hardware whenever the builtin effect was available. On the AudioEngine ADM availability means allowed by policy rather than currently active, so macOS would report hardware processing while running the default software configuration. The getters now reflect the device level options last applied through configure_audio_processing, including a None result when the component is disabled, which also corrects the same misreport on Android. Also fixes doc contradictions: macOS is no longer lumped into the "hardware not available" desktop wording, and the platform notes state that Apple voice processing on macOS is available but opt in.
| let options = *self.handle.processing_options.lock(); | ||
| if !options.echo_cancellation { | ||
| AudioProcessingType::None | ||
| } else if options.prefer_hardware_processing && self.is_hardware_aec_available() { | ||
| AudioProcessingType::Hardware | ||
| } else { | ||
| AudioProcessingType::Software |
There was a problem hiding this comment.
🟡 Audio processing status queries return stale results after using individual toggle methods
The processing status queries read from a cached options snapshot (*self.handle.processing_options.lock() at livekit/src/platform_audio/mod.rs:1054) that is only written by configure_audio_processing, so using the individual toggle methods leaves the snapshot unchanged and the queries report the wrong state.
Impact: After toggling echo cancellation off via the convenience method, the status query still reports it as active (Software or Hardware), misleading callers.
Stale cached state after convenience method calls
The PR adds a processing_options: Mutex<AudioProcessingOptions> field to PlatformAdmHandle (livekit/src/platform_audio/mod.rs:304) and makes the active_aec_type, active_agc_type, and active_ns_type getters read from it (livekit/src/platform_audio/mod.rs:1054-1085). The configure_audio_processing method correctly writes to this field at livekit/src/platform_audio/mod.rs:1150.
However, the three convenience methods — set_echo_cancellation (livekit/src/platform_audio/mod.rs:1172-1180), set_auto_gain_control (livekit/src/platform_audio/mod.rs:1188-1196), and set_noise_suppression (livekit/src/platform_audio/mod.rs:1204-1212) — modify the actual hardware/builtin state but never update processing_options. Their doc comments even claim they are "equivalent to calling configure_audio_processing with only the [field] changed," but they are not.
Example scenario:
PlatformAudio::new()setsprocessing_optionsto defaults (all enabled)audio.set_echo_cancellation(false, false)disables hardware AEC but leavesprocessing_options.echo_cancellation == trueaudio.active_aec_type()readsprocessing_options, seesecho_cancellation == true, and returnsSoftwareinstead ofNone
(Refers to lines 1054-1061)
Prompt for agents
The three convenience methods set_echo_cancellation (line 1172), set_auto_gain_control (line 1188), and set_noise_suppression (line 1204) each modify the hardware/builtin audio processing state but do not update self.handle.processing_options. Since the active_aec_type, active_agc_type, and active_ns_type getters now read from processing_options to determine their return value, these convenience methods must also update the corresponding field in processing_options after successfully applying the change. Each method should lock processing_options and update the relevant field (echo_cancellation, auto_gain_control, or noise_suppression respectively) as well as prefer_hardware_processing to match the prefer_hardware argument. Alternatively, the convenience methods could be refactored to delegate to configure_audio_processing with the appropriate options, which would naturally keep the state in sync.
Was this helpful? React with 👍 or 👎 to provide feedback.
The macOS default of prefer_hardware_processing=false predates this branch and existed because AudioDeviceMac had no hardware processing at all. With the AudioEngine ADM the capability exists, and Apple voice processing is the stronger default for the hardware macOS users actually have, built-in speaker to microphone echo in particular. Opting out remains a single option: prefer_hardware_processing: false.
| #[cfg(any(target_os = "ios", target_os = "macos"))] | ||
| prefer_hardware_processing: true, | ||
| #[cfg(not(target_os = "ios"))] | ||
| #[cfg(not(any(target_os = "ios", target_os = "macos")))] | ||
| prefer_hardware_processing: false, |
There was a problem hiding this comment.
🔍 macOS default changes from software to hardware audio processing
This PR changes the default prefer_hardware_processing from false to true on macOS (livekit/src/platform_audio/processing.rs:124-125). Combined with the ADM switch from kPlatformDefaultAudio to kAppleAudioEngine (webrtc-sys/src/adm_proxy.cpp:64-65), this means existing macOS applications that relied on AudioProcessingOptions::default() will silently switch from WebRTC's software AEC/AGC/NS to Apple's voice processing pipeline. The changeset acknowledges this and says users can pass prefer_hardware_processing: false to keep the old behavior, but this is a breaking behavioral change for any macOS app that doesn't explicitly set this field. Worth confirming this is acceptable for existing consumers.
Was this helpful? React with 👍 or 👎 to provide feedback.
| #if defined(WEBRTC_IOS) || defined(WEBRTC_MAC) | ||
| // Use the AVAudioEngine based ADM on Apple platforms. It supports runtime | ||
| // switchable voice processing and device change handling. | ||
| platform_adm_ = webrtc::CreateAudioDeviceModule( | ||
| env_, webrtc::AudioDeviceModule::kAppleAudioEngine); |
There was a problem hiding this comment.
🔍 ADM type switch from CoreAudio to AudioEngine on macOS
The C++ constructor now selects kAppleAudioEngine instead of kPlatformDefaultAudio for both iOS and macOS (webrtc-sys/src/adm_proxy.cpp:61-65). This is a significant platform-level change: the AudioEngine ADM uses AVAudioEngine internally rather than the traditional CoreAudio HAL path. While this enables runtime-switchable voice processing, it also changes the audio I/O stack entirely. If kAppleAudioEngine is not available in the WebRTC build (e.g., an older fork or a build without the AudioEngine ADM compiled in), CreateAudioDeviceModule would return nullptr and the platform ADM would be set to null at line 77, falling back to synthetic-only mode with no platform audio. The error path handles this gracefully, but it's worth verifying the WebRTC build includes this ADM type.
Was this helpful? React with 👍 or 👎 to provide feedback.
| runtime: Arc<LkRuntime>, | ||
| // Device level processing configuration last applied via | ||
| // configure_audio_processing, used by the active_*_type getters | ||
| processing_options: Mutex<AudioProcessingOptions>, |
There was a problem hiding this comment.
suggestion: A parking_lot::RwLock is probably a better fit over Mutex since reading is more common than updating settings.
3 participants
Note
Stacked on #1212 (worker-thread-affine AdmProxy) — review only the top 3 commits. Retarget to
mainafter #1212 merges.What
Switches the platform ADM on iOS and macOS to the AVAudioEngine based
AudioEngineDevice. Deliberately minimal — no new Rust API surface:AdmProxycreates the platform ADM with thekAppleAudioEngineaudio layer on Apple platforms (kPlatformDefaultAudioelsewhere, Android unchanged). Creation stays in the proxy constructor on the worker thread, which the AudioEngine device binds its sequence checker to — the contract Make AdmProxy worker-thread-affine #1212 established.AdmProxyforwards the ADM platform voice processing interface (topology, path availability/toggle, processing state). This is required plumbing, not new API: WebRTC's own audio processing resolution (AudioProcessingControllerin the voice engine) calls these ondependencies.admwhen track audio options are applied, and manages the coupled Apple AEC+NS path itself.PlatformAudio's public API is unchanged;configure_audio_processingkeeps its existing builtin-effects behavior. Platform-vs-software processing resolution is owned by WebRTC per track, where it lives since webrtc-sdk's runtime audio processing modes work.Why
The AudioEngine ADM replaces the AudioUnit/CoreAudio-HAL default on Apple platforms, bringing runtime switchable voice processing (no engine rebuild to toggle AEC), proper device change handling, and parity with the Swift SDK's audio stack.
Dependencies — why CI is red
This PR cannot build against the currently pinned webrtc prebuilt (
webrtc-51ef663):kAppleAudioEngineand the platform voice processing interface only exist in newer webrtc. The chain to green:Expose C++ factory for the AudioEngine audio device module webrtc-sdk/webrtc#260 merges✅ merged. Fix macOS input device selection without voice processing in AudioEngineDevice webrtc-sdk/webrtc#261 (macOS device selection fix, found by testing this PR) should land before the prebuilt is cut.WEBRTC_TAGbump commit is added here (webrtc-sys/build/src/lib.rs).Until then this is verified against a local webrtc build via
LK_CUSTOM_WEBRTCrather than by CI.Testing
Verified end to end on macOS arm64 against a local libwebrtc build (webrtc-sdk
m144_release+ #260 + #261):AudioEngineDeviceconfirmed active (not a fallback): with hardware processing preferred, builtin AEC/AGC/NS availability and active types flip to hardware — the coupled voice processing path enabling throughAdmProxyon the worker thread.cargo run -p livekit --example platform_audio(from Make AdmProxy worker-thread-affine #1212) passes all phases against the AudioEngine ADM: lifecycle, device enumeration/selection/hot-swap while recording, recording on real hardware, processing reconfiguration, 16-thread concurrent churn, full runtime teardown/reacquire cycles.AudioEngineDevicebug in the process (macOS device selection with voice processing disabled) — fixed in Fix macOS input device selection without voice processing in AudioEngineDevice webrtc-sdk/webrtc#261; with that fix in the local build, device switching works in both processing modes.