Skip to content

[codex] collect per-generation parsing errors#190

Draft
Ki-Seki wants to merge 2 commits into
mainfrom
codex/collect-generation-errors
Draft

[codex] collect per-generation parsing errors#190
Ki-Seki wants to merge 2 commits into
mainfrom
codex/collect-generation-errors

Conversation

@Ki-Seki

@Ki-Seki Ki-Seki commented Jun 19, 2026

Copy link
Copy Markdown
Member

Summary

  • add error_mode="raise" | "collect" to synchronous, asynchronous, and vLLM offline batch model interfaces
  • add GenerationResult to preserve each raw response together with either its parsed Result or serializable error details
  • isolate parsing and infill failures per candidate while keeping model request, configuration, and response-container failures as whole-call errors
  • preserve existing default behavior and return types when error_mode is omitted
  • document the behavior for OpenAI, vLLM server, and vLLM offline backends

Root cause

Multi-candidate and batch responses were passed through list-level infill helpers. A parsing exception from one generated string aborted the conversion of the entire response list, discarding otherwise valid candidates and queries.

Behavior

  • error_mode="raise" remains the default and preserves fail-fast behavior.
  • error_mode="collect" parses every generated string independently.
  • single-query calls preserve the generator's scalar/list shape, replacing Result elements with GenerationResult elements.
  • vLLM offline batch always returns a two-dimensional query/candidate structure.
  • request failures, invalid arguments, invalid response containers, empty candidate groups, and batch shape mismatches still raise normally.
  • existing response repair and non-strict infill behavior is unchanged.

Validation

  • uv run pytest — 73 passed
  • uv run mypy src — passed
  • uv run ruff format src tests — passed
  • uv run ruff check src tests — passed
  • pre-commit hooks — passed

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://SculptAI.github.io/GIMKit/pr-preview/pr-190/

Built to branch gh-pages at 2026-06-19 10:06 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@codecov

codecov Bot commented Jun 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant