feat: route with Switchyard libsy in process

[bbednarski9]

Overview

Replaces the service-backed Switchyard plugin with an in-process, library-first
integration driven by libsy::Algorithm::run_stream. Relay remains responsible
for provider dispatch, authentication, retries, fallback, and observability.

The draft now supports both weighted random routing and libsy's judge-backed LLM
classifier. It remains a placeholder while the focused Switchyard dependencies
land and compatible Switchyard 0.2 crates are published.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

• Drives the complete run_stream lifecycle. Switchyard requests provider work,
  Relay dispatches every CallLlm, and Relay returns the real buffered response,
  stream, or provider error through CallLlmRequest::respond.
• Supports weighted, reproducibly seeded random routing.
• Adds kind = "llm_classifier" with classifier, weak, and strong semantic
  targets; solve/confidence thresholds; optional capability floor; session
  affinity; and message-hash fallback.
• Dispatches both classifier consultation and final routed calls through Relay's
  configured provider bindings. The plugin does not use
  switchyard-llm-client or switchyard-server.
• Supports OpenAI Chat and OpenAI Responses classifier targets. Weak and strong
  targets may use OpenAI Chat, OpenAI Responses, or Anthropic Messages.
• Uses switchyard-translation as the sole request, response, and stream
  translation layer. The plugin does not fall back to Relay provider codecs.
• Starts a fresh libsy run for each retry. A failed classifier consultation
  follows libsy's fail-closed strong-target behavior; only a failed final routed
  call participates in Relay retry classification.
• Emits requested, call, decision, retry, error, and fallback routing marks from
  genuine libsy lifecycle data. Classifier consultations are marked
  is_routed_call = false.
• Adds buffered, streaming, cross-protocol, retry, affinity, fail-closed,
  telemetry, process-boundary, and standalone no-service classifier coverage.
• Removes the Decision API client, service health checks, service startup
  examples, dedicated Switchyard ATOF receiver, and service-backed version-1
  configuration.
• Temporarily pins switchyard-libsy, switchyard-protocol, and
  switchyard-translation to
  773a451125231e69209cadb697666a1f03d8f909, the source behind
  feat(translation): preserve raw stream events NVIDIA-NeMo/Switchyard#192.

Breaking change: the Switchyard plugin no longer supports service-backed
version-1 configuration.

Current upstream dependency:

• LIBSY-GAP-005 records that the pinned classifier builds its governing prompt
  as a normal system-role message. The OpenAI encoders consequently emit it as
  user content or omit the native instructions field. Relay does not rewrite the
  request around this gap.
• The focused Relay contract probe is intentionally ignored against the current
  pin. The rest of the classifier lifecycle is active and passing.
• A focused one-file Switchyard fix has been implemented and validated locally,
  but is not committed or pushed as part of this Relay-only update.

Validation:

• cargo test -p nemo-relay-switchyard — 31 passed, 1 upstream contract probe ignored
• cargo test -p nemo-relay-cli --features switchyard --test switchyard_process_e2e — 2 passed
• examples/switchyard/run-classifier-e2e.sh
• cargo clippy --workspace --all-targets -- -D warnings
• cargo fmt --all --check
• targeted uv run pre-commit run --files ...
• uv run pre-commit run --all-files
• just docs — 0 errors; authenticated Fern redirect check unavailable locally

Known repository-wide validation issue: just test-rust is not green in this
checkout. Its first failure is in a core pricing/plugin-registry test because
the pii_redaction component is not registered; later tests inherit poisoned
registry locks. The first failure reproduces in isolation and is unrelated to
the Switchyard classifier paths.

Where should the reviewer start?

Start with crates/switchyard/src/component.rs for algorithm construction and
the generic run_stream driver. Then review
classifier_run_stream_offloads_the_judge_and_routed_calls_to_relay and
switchyard_plugin_runs_classifier_and_routed_calls_without_a_service for the
library/Relay ownership boundary.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to feat(translation): preserve raw stream events NVIDIA-NeMo/Switchyard#192.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: a02b0df7-9b82-413c-b76f-fb83a5a92683

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-586.docs.buildwithfern.com/nemo/relay/dev

[github-actions]

License Diff

Compared against origin/release/0.6.

Lockfile license changes

Lockfile License Changes

Rust

Added

• switchyard-libsy 0.1.0 (Apache-2.0)

Removed

• None

Updated/Changed

opentelemetry

Before:

• opentelemetry 0.31.0 (Apache-2.0)

After:

• opentelemetry 0.31.0 (Apache-2.0)
• opentelemetry 0.32.0 (Apache-2.0)

rand

Before:

• rand 0.10.1 (Apache-2.0)
• rand 0.9.3 (Apache-2.0)

After:

• rand 0.10.1 (Apache-2.0)
• rand 0.8.7 (Apache-2.0)
• rand 0.9.3 (Apache-2.0)

rand_chacha

Before:

• rand_chacha 0.9.0 (Apache-2.0)

After:

• rand_chacha 0.3.1 (Apache-2.0)
• rand_chacha 0.9.0 (Apache-2.0)

rand_core

Before:

• rand_core 0.10.1 (Apache-2.0)
• rand_core 0.9.5 (Apache-2.0)

After:

• rand_core 0.10.1 (Apache-2.0)
• rand_core 0.6.4 (Apache-2.0)
• rand_core 0.9.5 (Apache-2.0)

Node

Added

• None

Removed

• None

Updated/Changed

• None

Python

Added

• None

Removed

• None

Updated/Changed

• None

Status output

[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (431 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (367 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (105 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/release/0.6 into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (426 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (367 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (105 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done