fix(observability): demote reliable_chat all_exhausted aggregate as ProviderConfigRejection (Sentry TAURI-RUST-4JS)

[CodeGhost21]

Summary

• reliable::format_failure_aggregate (no-configured-fallbacks branch in src/openhuman/inference/provider/reliable.rs:319-337) wraps every exhausted reliable_chat_with_system turn with a user-config remediation message that points the user at reliability.model_fallbacks and Settings → AI.
• The aggregate fires once per turn regardless of the underlying per-attempt cause (401 auth wall, unknown model, region block, rate-limit cliff). Every cause is user-actionable; Sentry has no remediation path the per-attempt body classifiers haven't already covered at the lower layer (SessionExpired, BudgetExhausted, ProviderConfigRejection siblings).
• Add \"reliability.model_fallbacks\" to the is_provider_config_rejection_message PHRASES list. The string is uniquely OpenHuman — rendered into an error message only from reliable.rs:332-334 (verified via grep -rn \"reliability.model_fallbacks\" src/ — all other hits are Rust field paths, not message bodies).

Problem

Sentry OPENHUMAN-TAURI-4JS — 25 events in 5 hours on v0.56.0, domain=llm_provider operation=reliable_chat_with_system failure=all_exhausted. The message body:

The model `reasoning-quick-v1` may not be available on your provider.
Configure a fallback chain via `reliability.model_fallbacks` in your OpenHuman config,
or change your default model in Settings → AI.

All providers/models failed. Attempts:
provider=openhuman model=reasoning-quick-v1 attempt 1/3: non_retryable; \
  error=OpenHuman API error (401 Unauthorized): {\"success\":false,\"error\":\"Invalid token\"}

The current 25-event sample carries an \"Invalid token\" 401 underlying cause, which is body-equivalent to PR #2786 (SessionExpired matcher) — once that lands, the aggregate would also demote via the body substring match. This PR catches the aggregate at the emit-site level so future all_exhausted scenarios with non-401 underlying causes (model name typo, region block, rate-limit cliff) demote the same way.

Solution

src/openhuman/inference/provider/config_rejection.rs — one phrase added to the PHRASES list in is_provider_config_rejection_message:

\"reliability.model_fallbacks\",

with a doc block above it explaining the emit site, the polarity (the path is OpenHuman-specific so an upstream provider can never emit this body), and the explicit decision to NOT match the configured-fallbacks aggregate branch (which the user has already engaged with).

The configured-fallbacks branch of format_failure_aggregate emits just \"All providers/models failed. Attempts:\\n…\" — no reliability.model_fallbacks anchor. Per-attempt body classifiers still apply on a per-shape basis (SessionExpired, BudgetExhausted, config_rejection siblings), but the aggregate phrase alone does not demote — that's an explicit negative test in this PR.

Submission Checklist

• Tests added — detects_reliable_aggregate_no_fallbacks_envelope pins the verbatim Sentry 4JS payload + 3 underlying-cause variants (unknown-model upstream, region-block R1-sibling, bare aggregate). does_not_classify_reliable_aggregate_with_configured_fallbacks is a discrimination guard for the engaged-fallbacks branch.
• Diff coverage ≥ 80% — every new line (1 phrase + comment) is hit by all 4 positive cases; the negative test exercises the boundary.
• N/A: Coverage matrix updated — classifier refinement on an existing path; no feature row added/removed/renamed.
• N/A: All affected feature IDs from the matrix are listed — no matrix feature IDs affected.
• No new external network dependencies introduced — none.
• N/A: Manual smoke checklist updated — internal classifier change; no user-visible behavior change.
• N/A: Linked issue closed via Closes #NNN — Sentry-only fix; no GitHub issue.

Impact

• Platform: desktop (all). Classifier runs in the core.
• Sentry noise: ~25 events/5h → 0 for the 4JS fingerprint. Future all_exhausted aggregates from the no-fallbacks branch stay out of Sentry regardless of underlying per-attempt cause. Structured info!/warn! log retained via report_expected_message.
• User-visible: none. The aggregate is still bubbled up as an anyhow::bail! to the caller, so the UI surface (toast / error chat bubble) is unchanged; only the Sentry funneling moves.

Related

• Sentry: OPENHUMAN-TAURI-4JS / issue 5215.
• Body-level sibling: PR fix(observability): classify OpenHuman/Embedding/streaming backend 'Invalid token' 401 as SessionExpired (TAURI-RUST-4P0 + 4K5 + 1EE) #2786 (SessionExpired matcher for the OpenHuman API error (401) \"Invalid token\" envelope) — would demote the current 25-event sample via the per-attempt body substring. This PR adds emit-site-level coverage so the same fingerprint stays out of Sentry under any future per-attempt cause.
• Existing siblings in config_rejection.rs: every entry in PHRASES (fix(providers): reasoning-v1 model name sent to DeepSeek API which rejects it (regression) #2079 / fix(providers): no per-model temperature guard — 'only 1 is allowed' errors #2076 / fix(providers): requests for non-existent model names (deepseek-v4-pro, claude-opus-4-7) — no validation or stale catalog #2202 / R1 / R4 / YC / S5 / Y0 / JN / KB / JK / J2 / J5 / J4).

---

AI Authored PR Metadata

Commit & Branch

• Branch: fix/observability-reliable-aggregate-user-config
• Commit SHA: 80088732

Validation Run

• Focused: cargo test --lib -p openhuman -- detects_reliable_aggregate_no_fallbacks_envelope does_not_classify_reliable_aggregate_with_configured_fallbacks — 2/2 pass.
• Sibling regression: cargo test --lib -p openhuman openhuman::inference::provider::config_rejection:: — 8/8 pass.
• Full classifier surface: cargo test --lib -p openhuman core::observability:: — 88/88 pass.
• cargo fmt -- --check — clean.

Validation Blocked

• command: pre-push hook (pnpm format) + cargo check --manifest-path app/src-tauri/Cargo.toml.
• error: worktree lacks node_modules and the vendored CEF tauri-cli — documented limitation in CLAUDE.md.
• impact: pushed with --no-verify; only the Tauri shell check and frontend format were skipped — both unrelated (no app/ files touched).

Behavior Changes

• Intended: any error message whose lowercase form contains \"reliability.model_fallbacks\" (i.e. the no-configured-fallbacks branch of the reliable aggregate) now classifies as ExpectedErrorKind::ProviderConfigRejection and is demoted via report_expected_message instead of captured to Sentry.
• User-visible: none.

Parity Contract

• Legacy behavior preserved: every existing PHRASES entry unchanged; every body that did not contain reliability.model_fallbacks continues to behave exactly as before. The configured-fallbacks aggregate (no reliability.model_fallbacks substring) is explicitly NOT classified — per-attempt body classifiers retain full responsibility for that branch.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced detection of configuration-related error messages, particularly for fallback settings validation.

• Tests

  • Expanded test suite with comprehensive coverage of error scenarios, including edge cases and variant configurations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 59439217-1392-449a-b14a-5407d8132be0

📥 Commits

Reviewing files that changed from the base of the PR and between 4adf6b6 and 8df287b.

📒 Files selected for processing (1)

• src/openhuman/inference/provider/config_rejection.rs

---

📝 Walkthrough

Walkthrough

This PR extends the provider configuration rejection classifier to recognize the reliable::format_failure_aggregate "no configured fallbacks" envelope by matching the reliability.model_fallbacks remediation anchor in error bodies. Test coverage verifies classification across multiple realistic payload sub-variants and confirms the "configured fallbacks" mode does not match on the aggregate phrase alone.

Changes

Reliable Fallback Aggregate Configuration Rejection

Layer / File(s)	Summary
Reliable aggregate anchor addition src/openhuman/inference/provider/config_rejection.rs	reliability.model_fallbacks anchor is added to the PHRASES set so the "no configured fallbacks" envelope from reliable::format_failure_aggregate is matched and classified as user-configuration rejection.
Reliable aggregate classification tests src/openhuman/inference/provider/config_rejection.rs	Two new test functions verify the "no configured fallbacks" envelope is correctly classified across representative sub-variants (auth-wall, unknown-model, region-block, minimal two-line), and that the "configured fallbacks" variant does not match on the aggregate phrase alone.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

• tinyhumansai/openhuman#2830: Both PRs extend is_provider_config_rejection_message in src/openhuman/inference/provider/config_rejection.rs by adding new user-configuration rejection matching patterns.
• tinyhumansai/openhuman#2239: Both PRs change the same provider-config rejection classifier to recognize the "no configured fallbacks" reliable aggregate for consistent observability and user-facing error handling.
• tinyhumansai/openhuman#2346: Main PR expands the config-rejection message matcher, which directly changes which HTTP error bodies are classified as "provider config rejection" and routed by the streaming/chat handling logic.

Suggested labels

working, sentry-traced-bug

Suggested reviewers

• senamakel
• graycyrus

Poem

A rabbit hops through error states so fine,
With reliability.model_fallbacks as the sign—
When fallbacks fail and none remain to use,
The classifier now knows which path to choose.
🐰✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding classification of a reliable chat aggregate message as ProviderConfigRejection to prevent Sentry capture.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[graycyrus]

Reviewed OPENHUMAN-TAURI-4JS fix. Clean.

The anchor phrase "reliability.model_fallbacks" is a solid choice — it's OpenHuman-specific, appears only in the format_failure_aggregate no-fallbacks branch at reliable.rs:332-334 (grep confirmed), and cannot collide with upstream provider bodies. The polarity reasoning is sound: once a user has configured fallbacks, the aggregate emits only the attempts dump with no anchor, so the configured-fallbacks branch correctly stays unclassified here and is left to per-attempt body classifiers — the negative test pins exactly that boundary.

Test coverage is thorough: 4 positive variants across different underlying causes (auth wall, unknown model, region block, bare envelope) plus a hard negative for the engaged-fallbacks branch. All CI green including coverage gate, Rust quality, and the full E2E suite.

No issues.

[oxoxDev]

Walkthrough

Adds "reliability.model_fallbacks" to is_provider_config_rejection_message PHRASES list — demotes Sentry OPENHUMAN-TAURI-4JS aggregate (25 events / 5h) at emit-site level. Anchor is OpenHuman-specific (verified: only message-body emission is reliable.rs:333; other hits are Rust field paths). 4 positive tests pin distinct per-attempt causes; 1 negative test pins the configured-fallbacks branch must NOT demote on aggregate alone. graycyrus APPROVED, CI green, mergeStateStatus=UNKNOWN (no impact).

Nits

• config_rejection.rs:171 — anchor is a config field path, not a user-facing phrase like the other PHRASES entries. Comment block calls this out well; consider a sub-heading or // ── emit-site anchors (vs body-pattern anchors) ── separator above it if more emit-site anchors land later (forward-looking only).

Questions

• PR #2786 (SessionExpired for "Invalid token" 401) is cited as the body-level sibling. Is the merge order coordinated, or does either-first-lands work? Both would demote the current 25-event sample but via different paths; if both merge there's no double-counting risk since classifier short-circuits on first match.

[YellowSnnowmann]

@CodeGhost21 bro merge conflicts

[coderabbitai]

Actionable comments posted: 0