[codex] Support raw image refs for multimodal rendering

[eligotts]

Design update — inline/offload image storage

This PR now supports both raw image transport modes used by prime-rl:

• offload: existing behavior, raw image bytes are written to run-scoped image assets and refs carry a file-backed image id.
• inline: data-image URIs remain inline and raw refs carry the inline source instead of requiring raw_image_id.

This repo adds inline-capable mmraw:v3 refs while preserving mmraw:v2 parsing, keeps Qwen image hashes aligned to the raw decoded bytes, and emits raw descriptor items with either raw_image_id or raw_uri.

Validation after latest push: uv run pytest tests/test_client.py -q passed (14 passed).

Design update — dropped the None/cache-only image path

This PR and its companions (prime-rl #2836 / verifiers #1746 / renderers #89) no longer use the "send None for already-cached images" mechanism. Every image carries its raw descriptor ref at every slot (current and prior turns); /inference/v1/generate rematerializes each ref from disk every request.

Why: the None path coupled correctness to deployment (LRU cache present, single replica / DP-affinity, no eviction) and surfaced a miss as a hard vLLM EngineDeadError (qwen3-vl mrope dereferences a None image_grid_thw) that the retry net couldn't catch across the engine→API IPC. Dropping it is deployment-agnostic (a miss is impossible) and non-hacky. vLLM's mm_hash encoder cache still skips the expensive GPU re-encode for free — we only forgo the cheap IPC/CPU-reprocess dedup.

Validated: color-codeword (Qwen3-VL-4B) under DP=2, no affinity / no cache reliance: 0 crashes, 0 data=None, multi-turn accumulation correct, reward ~0.84. Also confirmed under TP.

This repo: every image emits a raw descriptor ref at every slot. _descriptor_only_mm_data no longer strips the pointer (pixel_values were never present in v1, so the strip was both stale and the root cause of the descriptor-only/rebuild churn). Removed the materialize_all_image_refs flag and the now-orphaned materialize_image_refs / materialize_kimi_image_refs.

---

Original description

Summary

• adds generic mmraw:v2 raw multimodal refs in renderers.mm_store, parsed as RawMMRef objects with family, fingerprint, modality, hash, asset id, and adapter-owned payload
• emits strict prime_raw_mm_item envelopes instead of processed image payloads for Qwen-VL and Kimi K2.5 image rendering
• keeps adapter-specific layout details in renderer-owned payloads (image_grid_thw for Qwen, grid_thws/media token metadata for Kimi)
• supports materializing all raw image refs for retry paths after vLLM multimodal cache misses
• keeps run-scoped image asset refs file-backed so downstream Prime-RL trainer materializes images with its own processor

Companion PRs

• Prime-RL: Support v1 raw multimodal image offload prime-rl#2836
• Verifiers: [codex] Support raw image offload in v1 train client verifiers#1746

Notes

• Draft/WIP: stacked with the Verifiers and Prime-RL raw image offload PRs.
• Verifiers is expected to offload image content to file://.../assets/images/... refs before rendering.
• This intentionally treats raw image refs as the supported path, not processed multimodal feature sidecars.

Validation

• uvx ruff@0.15.18 check . passed.
• uvx ruff@0.15.18 format --check . passed.
• uvx 'ty<0.0.22' check . exited 0; remaining diagnostics are warning-level advisories under the repo config.
• PYTHONPATH=/home/ubuntu/renderers uv run --no-project --active pytest -q tests/test_client.py passed: 14 passed.
• End-to-end hosted-style smoke through Prime-RL with /home/ubuntu/renderers, /home/ubuntu/verifiers, and /home/ubuntu/prime-rl-v1-raw-mm-offload completed inference, env rollouts, train batch creation, trainer step 0, and decoded strict trainer-bound raw image refs.

[!NOTE]

Support raw image refs for multimodal rendering in Qwen3-VL, Qwen3.5, and Kimi-K2.5 renderers

• Adds a multimodal_output config field ('raw' or 'processed') to BaseRendererConfig; renderers default to 'raw', emitting file-URI image references and baked layout metadata instead of processed pixel tensors.
• Introduces renderers/mm_store.py with utilities to construct, serialize, offload, and validate raw multimodal image references and layout fingerprints, with no torch/vLLM dependency.
• Refactors image handling in renderers/qwen3_vl.py, renderers/qwen35.py, and renderers/kimi_k25.py to delegate image processing through shared helpers (qwen_image_item_for_render, kimi_image_item_for_render) and select between raw refs or processed payloads based on config.
• Updates renderers/client.py to serialize multimodal features as raw references via _build_vllm_mm_features, removing the renderer-class-specific torch/vLLM encoding path.
• Adds a vision optional dependency group in pyproject.toml for pillow, torch, and torchvision, required only for multimodal_output='processed'.
• Risk: renderer constructors no longer accept an injected processor argument; callers relying on direct processor injection will break.

^{Macroscope summarized a49e0fc.}

Update: review hardening (e3c12e9)

• Fixed render_completion_update mutating the caller's previous_multi_modal_data in place (shallow dict copy + setdefault().extend()); the merge now lives in a shared merge_multi_modal_data helper in base.py that copies inner lists, and the bridge test asserts the previous sidecar is unmutated.
• Qwen resize math is imported from transformers (torch-free PIL-backend module, with a fallback for older layouts) instead of maintaining a port.
• Raw layout describes resolve and read each image asset once; mm_store uses full sha256 content-addressed filenames and raises on undecodable base64.
• Added test_raw_layout_math_matches_image_processor: parity against the real Qwen3-VL-4B and Kimi-K2.5 (pinned revision) processors at rounding-boundary dimensions. Full suite: 2182 passed.

Update: merged main (e64cc58)

Reconciled with renderers main (12 commits: Hy3/PrimeQwen3/LagunaXS21 configs, kept-tokens parsing, SFT stop-token opt-in, a qwen thinking-wrapper fix, and bdb96b0 — main's own independent fix for the same bridge-mutation bug this PR's merge_multi_modal_data() already covers).

• kimi_k25.py/qwen35.py/qwen3_vl.py: kept this branch's shared merge_multi_modal_data() helper over main's three duplicated inline fixes for the identical bug — same behavior, one implementation instead of three.
• test_multimodal.py: re-injected main's _skip_for_disabled_thinking_deviation skip guard into this branch's parallel renderer_cases/processor_cases structure at all 4 call sites; took main's tuple-based prior_mm/prior_counts assertion over this branch's redundant duplicate check.
• test_client.py, __init__.py, configs.py: non-overlapping additions on both sides, no semantic conflict.
• client.py: no textual conflict (main's fix landed near, not on, this branch's rewritten _build_vllm_mm_features) — manually verified the merged generate() is consistent (prompt_attr/mm_data naming intact).

Full suite: 2578 passed, 153 skipped, 1 xfailed.

Update: unwrapped the ref payload (7c58fd6)

raw_mm_ref serializes its payload as compact JSON instead of base64-wrapping it. The ref travels as a string inside a JSON request body, so the wrapper's 33% inflation bought nothing. The saving is small in offload mode (the payload is a short file:// URI), but it keeps the ref format identical to the inline bundle, where the payload carries the image source and the wrapper cost ~32 KiB per image slot. Refs parse with a single partition(":") now that the payload contains colons.

---

Note

High Risk
Large multimodal contract change across inference client, bridge merging, and three VL renderers; constructor API removal and mandatory offloaded image assets can break downstream integrations.

Overview
Multimodal inference path switches from embedding processed pixel_values in the sidecar to JSON-safe raw image descriptors (prime_raw_mm_item) with layout metadata and file:// URIs. Qwen-VL, Qwen3.5, and Kimi K2.5 compute placeholder counts via baked layout math (or transformers smart_resize for Qwen) without running the image processor in the default mode.

Adds multimodal_output on renderer config (raw default, processed for SFT/training), renderers/mm_store (offload, fingerprints, mmraw refs), and merge_multi_modal_data so bridge turns concatenate prior media without mutating the caller’s sidecar. generate() now builds vLLM features from raw refs only—removing renderer-specific torch/vLLM encoding—and expects offloaded images at every slot.

Breaking / API: VL renderer constructors no longer take processor=; processors lazy-load only for multimodal_output="processed". image_cache_max is removed from VL configs. New optional vision extra for Pillow/torch when using processed mode. Tests require file:// image fixtures for raw render parity.

^{Reviewed by Cursor Bugbot for commit a49e0fc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new multimodal rendering mode that changes how images are transmitted to inference endpoints (raw refs vs processed payloads). The scope of runtime behavior changes across multiple renderers and client code, combined with an unresolved dependency issue in the default mode, warrants human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e64cc58. Configure here.}

[cursor]

Raw mode needs undeclared Pillow

Medium Severity

Default multimodal_output='raw' still calls _image_dimensions, which imports Pillow, but Pillow is only declared under the optional vision extra with torch/torchvision. A bare renderers install therefore fails on the default multimodal path before any processed tensors are involved.

Additional Locations (1)

• renderers/qwen3_vl.py#L210-L217

 

^{Reviewed by Cursor Bugbot for commit e64cc58. Configure here.}