[audio] Fix Linux PC Audio capture stall after TCI TX#4251
Conversation
There was a problem hiding this comment.
Solid, well-scoped fix. The root-cause evidence (ring at 12000/12000 while the Qt source stays Active/NoError) is convincing, and the fix follows from it directly: keep draining the Linux pull device during TCI suppression and discard the samples so the readyRead edge is never lost, without ever routing that audio to the radio. The #ifdef Q_OS_LINUX gate keeps the blast radius tight, the thread marshaling in setRadioTransmitting() is handled carefully, TxCaptureHealthTracker is a clean hardware-free state machine with sensible dedupe/rate-limiting, and all 7 CI checks are green. I verified the WASAPI silent-open watchdog is #ifdef Q_OS_WIN, so the Linux drain not setting m_txReceivedAnyBytes is harmless.
Nothing blocking. One non-blocking note below.
Non-blocking notes
audioEndpointDiagnostics()reads the tracker snapshot cross-thread — pre-existing pattern in that method, flagged only because this PR widens the unsynchronized-read surface. See inline.
Nice work — thanks for the thorough root-cause writeup and the stacked-diff discipline. (Heads-up as noted in the description: this needs #4233 to land first so the diff reduces to just d028ead8.)
🤖 aethersdr-agent · cost: $6.5795 · model: claude-opus-4-8
|
Reviewed the full diff (including the stacked #4233 content, since that's what's currently up for review here). The core fix itself — the One finding I'd like to escalate past how the earlier bot review characterized it, plus two diagnostic-accuracy gaps — all three live in the stacked #4233 code, not this PR's own fix commit, so I don't think they should block 1. The cross-thread read is a real race, not just a "widened surface." The earlier review flagged 2. The Linux drain empties the buffer before the tracker's own fallback check runs. 3. Smaller, non-blocking:
Net: |
|
Built and exercised this on real Linux hardware to add to the CI compile coverage — including a live run on PipeWire, since that's the backend the fix targets. Build
Unit test
Live radio + audio path — tested on BOTH audio backends:
Scope note (so this isn't overstated): Code-wise the drain reads the right device ( Testing and this write-up were done by Claude (Anthropic's Claude Code), driving the build, the on-hardware runs, and the audio/RF verification on my linux-aether box, with me (G0JKN) at the radio keying TX into a dummy load. Posting on my account. |
) ## Summary Adds bounded TX microphone capture-health diagnostics for #4230 without changing audio behavior. The diagnostics are **off by default**. TX capture-health summaries are written only when Help → Support's **CAT/rigctld** logging toggle—the existing category used by TCI debug logging—is enabled. The lightweight in-memory tracker and `get audio` fields remain available for automation and troubleshooting snapshots, but they do not produce support-log records unless that TCI debug category is on. The #4230 field log confirmed the failure signature: while TCI audio keeps the 200 ms suppression window fresh, `AudioEngine::onTxAudioReady()` returns without consuming the local `QAudioSource`. The Linux/PipeWire capture ring reached `12000/12000` bytes while Qt still reported `Active/NoError`; the last successful microphone read was 14,173 ms old. This PR detects that full-while-Active condition directly, while retaining Active-to-Idle with unread bytes as a fallback when capacity is unavailable. ## Merge order and follow-up fix **Merge this diagnostic PR before #4251.** The Linux-only behavioral fix in #4251 is intentionally stacked after this work. Once this PR lands, #4251 reduces to its single behavior commit while these opt-in diagnostics remain available to validate the fix on affected PipeWire systems. ## What is recorded - Tracks suppressed microphone callbacks and peak unread bytes while TCI audio is fresh. - Records saturation when buffered bytes reach the reported source capacity, even if Qt continues reporting the source as Active. - Retains the Active-to-Idle-with-unread-bytes signature as a fallback when source capacity is unavailable. - Counts later local, non-DAX TX starts only while the capture source is currently saturated. - Clears current saturation after a successful microphone read, while retaining lifecycle history and bounded one-warning-per-class rate limiting. - Excludes initial Idle state, TX owned by another client on the same radio, DAX TX, and TX while TCI audio is still fresh. - Includes device, Qt state/error, buffer availability/capacity, source lifetime, last successful microphone-read age, and aggregate counters. AetherSDR's own local TX remains tracked normally when the radio is operating in Multi-Flex mode. Only TX owned by another client is excluded, because that remote transmission does not consume this AetherSDR process's microphone capture stream and must not be counted as a local capture failure. The existing `get audio` automation snapshot exposes the same TX endpoint evidence: - `buffer_bytes_available` - `buffer_capacity_bytes` - `source_was_active` - `saturation_observed` - `tci_suppressed_callbacks` - `idle_during_tci_transitions` - `full_buffer_during_tci_observations` - `post_tci_local_tx_while_saturated` - `last_mic_read_age_ms` ## Review findings addressed - `audioEndpointDiagnostics()` now self-marshals with a blocking queued invocation when called off the AudioEngine thread. This removes the troubleshooting-dialog race on `QAudioSource`, `QIODevice`, `QElapsedTimer`, and the non-atomic tracker snapshot while preserving direct execution for callers already on the owner thread. - Current saturation is now separate from lifecycle-level observation/rate-limiting. A successful microphone read clears the current condition, so later healthy local TX attempts do not inflate `post_tci_local_tx_while_saturated`; historical evidence remains available in the lifecycle summary. ## Related PipeWire / Linux audio tracking These references are the working set of PipeWire/Linux audio failure and lifecycle reports relevant to field correlation. They are **not** asserted to share one root cause; the diagnostics distinguish local `QAudioSource` capture saturation from DAX routing, stream ownership, clocking, and device-change failures. Current reports: - Refs #4230 — TCI TX leaves the subsequent PC Audio SSB path stuck. The new log confirms local capture saturation while the source remains Active. - Refs #4152 — Linux/PipeWire DAX TX falls to near-zero RF until the `dax_tx` stream is recreated. - Refs #3805 — PipeWire DAX TX reaches AetherSDR but produces no SSB RF while the PC Audio PipeWire input works. Prior PipeWire/Linux audio lifecycle and quality reports: - Refs #2286 — PipeWire DAX TX smearing and incomplete end-of-transmission teardown. - Refs #752 — PipeWire DAX TX audio dropped when an external client owns PTT. - Refs #537 — WSJT-X receive timing drift after repeated TX cycles on Linux. - Refs #1008 — DAX latency on Linux, leading to the native PipeWire stream work. - Refs #75 — Ubuntu PipeWire/PulseAudio compatibility produced no enumerated PC audio devices. - Refs #2864 — PipeWire device-change churn repeatedly reopened the audio-device warning. - Refs #2895 — headless Linux/PipeWire DAX channels remained Idle except for slice 0. - Refs #3363 — TCI RX and DAX audio both stopped during operation. - Refs #3669 — TCI/DAX audio required mode and DAX toggles to re-arm. - Refs #4009 — Linux/macOS DAX+TCI re-assert loop caused stream churn and audio jitter. Issues that only concern packaging, PipeWire node naming, or channel-pair selection are intentionally not listed as capture-stall candidates. ## Validation - Configured and built the complete application with Ninja and 8 build jobs. - Full CTest suite: 114/114 passed; the quarantined clean-room test was skipped by its existing policy. - Hardware-free TX capture-health state-machine test: 22/22 assertions passed, including Active/full-buffer saturation, the Idle fallback, recovery after a successful read, rate limiting, DAX exclusion, and exclusion of TX owned by another client. - `tools/check_engine_boundary.py --strict`: zero blocking findings. - `git diff --check`: clean. - Agent automation bridge: `ping` and `get audio` passed; the TCI/CAT logging category was toggled off and restored; Slice Troubleshooting opened and refreshed repeatedly through the formerly racy path with the app remaining responsive. No radio connection or TX was performed. ## Field-validation result The first affected PipeWire support log supplied the decisive evidence this PR was designed to collect: the ring was full while the Qt source remained Active. #4251 applies the narrow Linux-only drain/discard fix; affected users can enable the TCI debug logging category when validating that they no longer reach saturation. ## Screenshots Not applicable: this is opt-in support logging and diagnostics instrumentation with no visual UI change. 👨🏼💻 Generated with OpenAI Codex (GPT-5.6 Sol) and tested by @jensenpat
d028ead to
8bee875
Compare
|
Rebased onto |
ten9876
left a comment
There was a problem hiding this comment.
Approving — tight, well-evidenced Linux-only fix
Post-rebase this is a 14-line, #ifdef Q_OS_LINUX change: drain and discard the PipeWire pull device during TCI suppression so its ring never fills and the readyRead edge needed to resume PC Audio is never lost. The root-cause evidence (ring at 12000/12000 while the Qt source stays Active/NoError) is direct and the mechanism follows from it — draining on every callback keeps the edge alive, and the discarded samples never reach the radio, so TCI stays the sole TX source. Windows/macOS keep existing behavior (blast radius tight), and it now sits on #4233's hardened TxCaptureHealthTracker.
The one open item — audioEndpointDiagnostics() reading tracker/bytesAvailable() cross-thread — is diagnostic-only and non-blocking; worth tightening later but doesn't gate this. Thanks @jensenpat.
…thersdr#4233) ## Summary Adds bounded TX microphone capture-health diagnostics for aethersdr#4230 without changing audio behavior. The diagnostics are **off by default**. TX capture-health summaries are written only when Help → Support's **CAT/rigctld** logging toggle—the existing category used by TCI debug logging—is enabled. The lightweight in-memory tracker and `get audio` fields remain available for automation and troubleshooting snapshots, but they do not produce support-log records unless that TCI debug category is on. The aethersdr#4230 field log confirmed the failure signature: while TCI audio keeps the 200 ms suppression window fresh, `AudioEngine::onTxAudioReady()` returns without consuming the local `QAudioSource`. The Linux/PipeWire capture ring reached `12000/12000` bytes while Qt still reported `Active/NoError`; the last successful microphone read was 14,173 ms old. This PR detects that full-while-Active condition directly, while retaining Active-to-Idle with unread bytes as a fallback when capacity is unavailable. ## Merge order and follow-up fix **Merge this diagnostic PR before aethersdr#4251.** The Linux-only behavioral fix in aethersdr#4251 is intentionally stacked after this work. Once this PR lands, aethersdr#4251 reduces to its single behavior commit while these opt-in diagnostics remain available to validate the fix on affected PipeWire systems. ## What is recorded - Tracks suppressed microphone callbacks and peak unread bytes while TCI audio is fresh. - Records saturation when buffered bytes reach the reported source capacity, even if Qt continues reporting the source as Active. - Retains the Active-to-Idle-with-unread-bytes signature as a fallback when source capacity is unavailable. - Counts later local, non-DAX TX starts only while the capture source is currently saturated. - Clears current saturation after a successful microphone read, while retaining lifecycle history and bounded one-warning-per-class rate limiting. - Excludes initial Idle state, TX owned by another client on the same radio, DAX TX, and TX while TCI audio is still fresh. - Includes device, Qt state/error, buffer availability/capacity, source lifetime, last successful microphone-read age, and aggregate counters. AetherSDR's own local TX remains tracked normally when the radio is operating in Multi-Flex mode. Only TX owned by another client is excluded, because that remote transmission does not consume this AetherSDR process's microphone capture stream and must not be counted as a local capture failure. The existing `get audio` automation snapshot exposes the same TX endpoint evidence: - `buffer_bytes_available` - `buffer_capacity_bytes` - `source_was_active` - `saturation_observed` - `tci_suppressed_callbacks` - `idle_during_tci_transitions` - `full_buffer_during_tci_observations` - `post_tci_local_tx_while_saturated` - `last_mic_read_age_ms` ## Review findings addressed - `audioEndpointDiagnostics()` now self-marshals with a blocking queued invocation when called off the AudioEngine thread. This removes the troubleshooting-dialog race on `QAudioSource`, `QIODevice`, `QElapsedTimer`, and the non-atomic tracker snapshot while preserving direct execution for callers already on the owner thread. - Current saturation is now separate from lifecycle-level observation/rate-limiting. A successful microphone read clears the current condition, so later healthy local TX attempts do not inflate `post_tci_local_tx_while_saturated`; historical evidence remains available in the lifecycle summary. ## Related PipeWire / Linux audio tracking These references are the working set of PipeWire/Linux audio failure and lifecycle reports relevant to field correlation. They are **not** asserted to share one root cause; the diagnostics distinguish local `QAudioSource` capture saturation from DAX routing, stream ownership, clocking, and device-change failures. Current reports: - Refs aethersdr#4230 — TCI TX leaves the subsequent PC Audio SSB path stuck. The new log confirms local capture saturation while the source remains Active. - Refs aethersdr#4152 — Linux/PipeWire DAX TX falls to near-zero RF until the `dax_tx` stream is recreated. - Refs aethersdr#3805 — PipeWire DAX TX reaches AetherSDR but produces no SSB RF while the PC Audio PipeWire input works. Prior PipeWire/Linux audio lifecycle and quality reports: - Refs aethersdr#2286 — PipeWire DAX TX smearing and incomplete end-of-transmission teardown. - Refs aethersdr#752 — PipeWire DAX TX audio dropped when an external client owns PTT. - Refs aethersdr#537 — WSJT-X receive timing drift after repeated TX cycles on Linux. - Refs aethersdr#1008 — DAX latency on Linux, leading to the native PipeWire stream work. - Refs aethersdr#75 — Ubuntu PipeWire/PulseAudio compatibility produced no enumerated PC audio devices. - Refs aethersdr#2864 — PipeWire device-change churn repeatedly reopened the audio-device warning. - Refs aethersdr#2895 — headless Linux/PipeWire DAX channels remained Idle except for slice 0. - Refs aethersdr#3363 — TCI RX and DAX audio both stopped during operation. - Refs aethersdr#3669 — TCI/DAX audio required mode and DAX toggles to re-arm. - Refs aethersdr#4009 — Linux/macOS DAX+TCI re-assert loop caused stream churn and audio jitter. Issues that only concern packaging, PipeWire node naming, or channel-pair selection are intentionally not listed as capture-stall candidates. ## Validation - Configured and built the complete application with Ninja and 8 build jobs. - Full CTest suite: 114/114 passed; the quarantined clean-room test was skipped by its existing policy. - Hardware-free TX capture-health state-machine test: 22/22 assertions passed, including Active/full-buffer saturation, the Idle fallback, recovery after a successful read, rate limiting, DAX exclusion, and exclusion of TX owned by another client. - `tools/check_engine_boundary.py --strict`: zero blocking findings. - `git diff --check`: clean. - Agent automation bridge: `ping` and `get audio` passed; the TCI/CAT logging category was toggled off and restored; Slice Troubleshooting opened and refreshed repeatedly through the formerly racy path with the app remaining responsive. No radio connection or TX was performed. ## Field-validation result The first affected PipeWire support log supplied the decisive evidence this PR was designed to collect: the ring was full while the Qt source remained Active. aethersdr#4251 applies the narrow Linux-only drain/discard fix; affected users can enable the TCI debug logging category when validating that they no longer reach saturation. ## Screenshots Not applicable: this is opt-in support logging and diagnostics instrumentation with no visual UI change. 👨🏼💻 Generated with OpenAI Codex (GPT-5.6 Sol) and tested by @jensenpat
## Summary Fixes the Linux/PipeWire PC Audio capture stall seen after TCI transmit in aethersdr#4230. While TCI audio owns the TX path, AetherSDR now continues draining the local Linux `QAudioSource` and discards those suppressed samples. This prevents the PipeWire/Qt pull-device ring from filling and losing the `readyRead()` edge needed when PC Audio resumes. The behavior change is gated by `Q_OS_LINUX`. Windows and macOS keep their existing behavior, minimizing blast radius while the evidence is specific to PipeWire/Linux. ## Merge order **Merge aethersdr#4233 first.** This PR is intentionally stacked on the corrected diagnostic instrumentation in aethersdr#4233, so its current diff contains both that instrumentation and the Linux behavior fix. Once aethersdr#4233 lands, this PR reduces to the single Linux-only fix commit (`d028ead8`). The aethersdr#4233 diagnostic is part of the fix strategy: it remains default-on and bounded so support logs can show whether the source reached a full buffer and whether the drain prevents recurrence in the field. ## Root cause evidence The new aethersdr#4230 support log provides the missing direct evidence: - TCI audio became active at 11:28:44.769. - The local microphone source reached `12000/12000` buffered bytes. - `lastMicReadAge` reached 14,173 ms. - The Qt source still reported `Active/NoError`, so an Active-to-Idle-only diagnostic would miss the failure. - `dax=0` was confirmed at 11:28:47.667, before the failed local TX at 11:28:50.486. That isolates the failure to AetherSDR's local capture-consumption path: the TCI suppression return in `AudioEngine::onTxAudioReady()` stopped reading the pull device until its ring was full. On an edge-driven PipeWire source, freeing it only after the handoff is too late because no new readiness edge is guaranteed. ## Fix - On Linux only, read and discard local microphone bytes while fresh TCI audio suppresses PC Audio. - Continue recording successful microphone consumption in the TX capture-health tracker. - Do not send the discarded samples to the radio; TCI remains the sole TX audio source during the suppression window. - Do not change DAX routing, radio TX ownership, or Multi-Flex ownership rules. Local AetherSDR TX remains tracked in Multi-Flex mode; transmissions owned by another radio client remain excluded from local capture-failure accounting. ## Diagnostic correction in aethersdr#4233 The prerequisite diagnostic now detects the actual signature shown by the field log: a buffer at capacity while the source remains Active. It retains Active-to-Idle with unread bytes as a fallback when Qt cannot report a capacity. The bounded summary and `get audio` snapshot expose: - `full_buffer_during_tci_observations` - `post_tci_local_tx_while_saturated` - `buffer_bytes_available` - `buffer_capacity_bytes` - `source_was_active` - `saturation_observed` - `tci_suppressed_callbacks` - `last_mic_read_age_ms` ## Validation - Full RelWithDebInfo Ninja build with 8 jobs: passed. - Full CTest suite: 114/114 passed; one existing quarantined clean-room test skipped by policy. - TX capture-health state-machine test: 18/18 assertions passed, including Active/full-buffer detection and the Idle fallback. - `tools/check_engine_boundary.py --strict`: zero blocking findings. - `git diff --check`: clean. - Automation bridge: `ping` and `get audio` passed; the corrected saturation fields were present and initialized. The local automation host is macOS, where the Linux-only branch is deliberately excluded. Linux CI will provide compilation coverage; final PipeWire behavior validation should use the original reporter's reproduction sequence with the default Audio Summary diagnostics enabled. ## Related reports Refs aethersdr#4152, aethersdr#3805, aethersdr#2286, aethersdr#752, aethersdr#537, aethersdr#1008, aethersdr#75, aethersdr#2864, aethersdr#2895, aethersdr#3363, aethersdr#3669, and aethersdr#4009. These reports are related for Linux/PipeWire field correlation; this PR does not claim they all share aethersdr#4230's root cause. Fixes aethersdr#4230. ## Screenshots Not applicable: this changes core audio consumption and diagnostics with no visual UI change. 👨🏼💻 Generated with OpenAI Codex (GPT-5.6 Sol) and tested by @jensenpat
Summary
Fixes the Linux/PipeWire PC Audio capture stall seen after TCI transmit in #4230. While TCI audio owns the TX path, AetherSDR now continues draining the local Linux
QAudioSourceand discards those suppressed samples. This prevents the PipeWire/Qt pull-device ring from filling and losing thereadyRead()edge needed when PC Audio resumes.The behavior change is gated by
Q_OS_LINUX. Windows and macOS keep their existing behavior, minimizing blast radius while the evidence is specific to PipeWire/Linux.Merge order
Merge #4233 first. This PR is intentionally stacked on the corrected diagnostic instrumentation in #4233, so its current diff contains both that instrumentation and the Linux behavior fix. Once #4233 lands, this PR reduces to the single Linux-only fix commit (
d028ead8).The #4233 diagnostic is part of the fix strategy: it remains default-on and bounded so support logs can show whether the source reached a full buffer and whether the drain prevents recurrence in the field.
Root cause evidence
The new #4230 support log provides the missing direct evidence:
12000/12000buffered bytes.lastMicReadAgereached 14,173 ms.Active/NoError, so an Active-to-Idle-only diagnostic would miss the failure.dax=0was confirmed at 11:28:47.667, before the failed local TX at 11:28:50.486.That isolates the failure to AetherSDR's local capture-consumption path: the TCI suppression return in
AudioEngine::onTxAudioReady()stopped reading the pull device until its ring was full. On an edge-driven PipeWire source, freeing it only after the handoff is too late because no new readiness edge is guaranteed.Fix
Diagnostic correction in #4233
The prerequisite diagnostic now detects the actual signature shown by the field log: a buffer at capacity while the source remains Active. It retains Active-to-Idle with unread bytes as a fallback when Qt cannot report a capacity. The bounded summary and
get audiosnapshot expose:full_buffer_during_tci_observationspost_tci_local_tx_while_saturatedbuffer_bytes_availablebuffer_capacity_bytessource_was_activesaturation_observedtci_suppressed_callbackslast_mic_read_age_msValidation
tools/check_engine_boundary.py --strict: zero blocking findings.git diff --check: clean.pingandget audiopassed; the corrected saturation fields were present and initialized.The local automation host is macOS, where the Linux-only branch is deliberately excluded. Linux CI will provide compilation coverage; final PipeWire behavior validation should use the original reporter's reproduction sequence with the default Audio Summary diagnostics enabled.
Related reports
Refs #4152, #3805, #2286, #752, #537, #1008, #75, #2864, #2895, #3363, #3669, and #4009. These reports are related for Linux/PipeWire field correlation; this PR does not claim they all share #4230's root cause.
Fixes #4230.
Screenshots
Not applicable: this changes core audio consumption and diagnostics with no visual UI change.
👨🏼💻 Generated with OpenAI Codex (GPT-5.6 Sol) and tested by @jensenpat