[bug] Filter unsupported input modalities per target

[ryana]

Symptom

A coding-agent trajectory containing rich input is forwarded unchanged after
routing, even when the selected model cannot accept that modality. The
upstream rejects the request instead of Switchyard removing unsupported
content for that target.

Switchyard also has no target configuration field that declares accepted input
modalities. An image-only boolean would solve the immediate reproduction but
would repeat the problem for audio, video, and files.

Reproduction

At commit 0a03235b, export a valid NVIDIA Inference Hub credential:

export NVIDIA_API_KEY="..."

Run the same OpenAI Responses request through two OpenAI-compatible
Switchyard targets:

import asyncio
import base64
import os
from pathlib import Path

from switchyard import ChatRequest, ProxyContext
from switchyard.lib.backends.llm_target import (
    BackendFormat,
    LlmTarget,
    llm_target_with_runtime_defaults,
)
from switchyard_rust.components import OpenAiNativeBackend


async def main() -> None:
    image = base64.b64encode(Path("assets/logo.png").read_bytes()).decode()
    request = ChatRequest.openai_responses({
        "model": "switchyard",
        "input": [{
            "type": "message",
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image briefly."},
                {
                    "type": "input_image",
                    "image_url": f"data:image/png;base64,{image}",
                },
            ],
        }],
    })

    for target_id, model in [
        ("opus", "azure/anthropic/claude-opus-4-7"),
        ("nemotron", "nvidia/nvidia/nemotron-3-super-v3"),
    ]:
        target = llm_target_with_runtime_defaults(LlmTarget(
            id=target_id,
            model=model,
            format=BackendFormat.RESPONSES,
            base_url="https://inference-api.nvidia.com/v1",
            api_key=os.environ["NVIDIA_API_KEY"],
        ))
        backend = OpenAiNativeBackend(target)
        try:
            response = await backend.call(ProxyContext(), request)
            print(model, "OK", response.to_body())
        except Exception as error:
            print(model, "ERROR", error)


asyncio.run(main())

Observed on July 27, 2026:

• azure/anthropic/claude-opus-4-7 succeeded and described the Switchyard logo.
• nvidia/nvidia/nemotron-3-super-v3 returned HTTP 400 indicating that it was
  not a multimodal model.

A later negative control returned a more generic HTTP 400 validation response
when the image was preserved, but succeeded after Switchyard removed the image.

Expected vs. actual

• Expected: A target can declare accepted input modalities. Switchyard
  removes recognized unsupported content from its cloned outbound request,
  preserves supported content and message structure, and leaves the inbound
  trajectory untouched for retries or later targets.
• Actual: LlmTarget has no input-capability metadata. Rich content is
  preserved and forwarded to every selected target.

Proposed contract

Add an optional input_modalities allowlist to LlmTarget and supported
configuration surfaces:

strong:
  model: azure/anthropic/claude-opus-4-7
  input_modalities: [text, image]
weak:
  model: nvidia/nvidia/nemotron-3-super-v3
  input_modalities: [text]

Initial typed values:

• text
• image
• audio
• video
• file

Semantics:

• An explicit list is authoritative. Recognized content whose modality is
  absent is removed after target selection and format translation.
• Omission means capabilities are unknown and preserves existing pass-through
  behavior.
• Invalid literals fail configuration loudly.
• Filtering is target-local and does not mutate the original request.
• Unknown provider-extension blocks and unrelated JSON such as tool schemas
  are preserved by the filtering step.

This metadata does not need to make routing modality-aware in the first
change. Automatic capability discovery and reject/reroute policies can be
follow-up work.

Environment

• Switchyard commit SHA: 0a03235b
• Python version: 3.14.3
• OS / arch: macOS / arm64
• Install path: source build with uv
• Inbound and backend format: OpenAI Responses
• Backend: NVIDIA Inference Hub

Additional context

The translation crate already has internal target capabilities for images,
audio, video, and files, but shipping native backends construct the default
unknown policy. Same-format native requests also bypass translation, so a
translation-only fix would not cover all paths.

A validated spike implementing the proposed contract is available on Ryan
Angilly's fork in ryana/Switchyard#2.

[ayushag-nv]

@grahamking @nachiketb-nvidia We should handle this as part of new rust-server ?

[ryana]

@ayushag-nv agent took a stab at it here ryana#2

[ryana]

@ayushag-nv agent took a stab at it here ryana#2

supports_images may be too myopic. taking another swing at handling other media formats or at the very least designing a better interface that can be extended to support them.