Skip to content

Commit 2ecb7b1

Browse files
feat(llm): structured output (proposal 0016) (#42)
* chore: bump spec to v0.15.0; add jsonschema; skip deferred fixtures Spec submodule moves from v0.10.0 to v0.15.0 — covers the full 5-proposal batch (0011, 0014, 0015, 0016, 0017) in one bump per the skip-ahead governance principle. spec_version in pyproject.toml bumped to match. Adds jsonschema>=4.0 as a runtime dependency (used by the forthcoming structured-output validation path on the dict-schema side; Pydantic-class path uses its own validator). Adds skip markers to the conformance test files for fixtures whose runtime support lands in a later PR of the batch: - llm-provider 009-020 → 0015 multimodal (PR-2) - llm-provider 021-028 → 0016 structured output (this PR, wired up in a later commit) - pipeline-utilities 032-038 → 0011 parallel branches (PR-5) - pipeline-utilities 039-046 → 0014 state migration (PR-4) - graph-engine 021-observer-branch-name → 0011 parallel branches (PR-5) Skip markers also apply to test_fixture_parsing.py for the same set — the typed harness models in tests/conformance/harness/ don't yet know about the new directive shapes (state_migration, parallel branches state-schema variation, NodeEvent.branch_name); each deferring PR drops its own skip rows when it lands the harness work. * feat(llm): add StructuredOutputInvalid error category Adds the structured_output_invalid canonical category. Raised when a complete() call requested a response_schema and the provider's returned content could not be parsed as JSON OR did not validate against the schema. The exception carries response_schema, raw_content, and failure_description attributes for caller introspection. Non-transient by default — NOT added to TRANSIENT_CATEGORIES. The default RetryMiddleware classifier will not retry this category; callers wanting retry-on-validation-failure can include the category in a custom classifier's transient set. * feat(llm): add Response.parsed field Adds the parsed field to the Response record. Default None, populated by structured-output calls (response_schema set on complete() and the model returned structured content). The runtime type is a discriminated union over dict (when the caller passed a JSON-Schema dict) and BaseModel instance (when the caller passed a Pydantic class). Pydantic Response config now allows arbitrary types so a BaseModel instance can sit in the parsed slot. No public surface change for free-form callers — parsed defaults to None and remains None when response_schema is not supplied. * feat(llm): validate_response_schema + strict_mode_supported helpers Adds two provider-agnostic helpers in openarmature.llm.provider used by structured-output Provider implementations: - validate_response_schema(schema) — pre-send structural check that the value is a dict and its top-level type is "object". Raises ProviderInvalidRequest on failure. - strict_mode_supported(schema) — whether the schema satisfies the strict-mode constraint set (additionalProperties not true, properties fully covered by required) across the full schema tree. Walks anyOf/oneOf/allOf branches and follows $ref targets with cycle protection. An unresolvable $ref or unknown shape returns False (conservative fail). Both are exported from openarmature.llm so OpenAI-compatible providers and any future Anthropic/Gemini provider share the same constraint heuristic. * feat(llm): Provider Protocol gains response_schema parameter Extends the Provider Protocol's complete() method signature to accept an optional response_schema parameter. Accepts either a JSON Schema dict or a Pydantic BaseModel subclass; the implementation converts the class form to a JSON Schema at the boundary. Free-form callers (response_schema=None or absent) see no behavior change — the parameter defaults to None and the v0.4.0 contract is preserved. OpenAIProvider's complete() still has the v0.4.0 signature; the next commit wires the response_schema parameter through it. * feat(llm/openai): native response_format wire path + Pydantic overload Threads response_schema through OpenAIProvider.complete() → _do_complete() → _parse_response(). Accepts either a JSON Schema dict OR a Pydantic BaseModel subclass; the latter is converted via model_json_schema() at the boundary. Native wire path: when response_schema is supplied, the request body includes response_format: { type: "json_schema", json_schema: { name, schema, strict } }. The name field comes from schema.title when non-empty, otherwise a deterministic sha256 hash of the schema. The strict flag is set per strict_mode_supported() — true only when the schema cleanly satisfies the constraints across the full tree. Post-receive: parses message.content as JSON, then validates against the schema. Dict-input path validates with jsonschema and returns a dict. BaseModel-class-input path validates with model.model_validate() and returns a BaseModel instance. Either way, JSON parse failure or schema validation failure raises StructuredOutputInvalid carrying the schema, raw content, and failure description. parsed is absent on tool-call responses regardless of whether response_schema was supplied (mutually exclusive paths). Free-form calls (response_schema=None) see no behavior change — body omits response_format, parsed stays None. The prompt-augmentation fallback path is the next commit. * feat(llm/openai): prompt-augmentation fallback + inspect property Adds the prompt-augmentation fallback for OpenAI-compatible servers that don't implement response_format (older vLLM, some LM Studio releases, llama.cpp variants). Constructor: force_prompt_augmentation_fallback: bool = False When True, structured-output calls build the wire body by augmenting the message list with a system directive that includes the serialized JSON Schema, and omit response_format entirely. Native path is the default (False). Inspect property: uses_prompt_augmentation_fallback -> bool Read-only; lets callers verify which wire path is active without poking private state. _augment_messages_with_schema_directive returns a fresh list. When the first message is system, its content is extended with the schema directive (preserving caller intent); otherwise a new system message is prepended. The caller's original messages list is NOT mutated — Message instances are reused unchanged (immutable Pydantic models). Response parsing is unchanged from the native path: parse + validate post-receive raise StructuredOutputInvalid on failure. parsed is populated identically whether the wire took the native or fallback route. * test(conformance): capability-agnostic harness helpers for wire + carries Adds tests/conformance/harness/wire.py with helpers used by structured- output and content-block fixtures (and any future capability fixtures that need the same shapes): - match_wire_body(actual, expected) — recursive deep-equal with "*" wildcard support for string slots. - assert_response_format_absent(body) — asserts the wire body has no response_format key. - assert_system_references_schema(body, schema) — asserts the first message in the body is a system message whose content contains the canonical-JSON form of the schema as a substring. - assert_error_carries(exc, carries) — introspects a raised exception's attributes against an expected_carries block; supports _present / _mentions / literal-equal forms; handles the raw_response_content → raw_content fixture-vs-impl naming alias. Extends test_llm_provider.py to drive these from the existing fixture loop: - response_schema is read from call_spec and threaded through provider.complete(). - expected_wire_request literal compare + expected_wire_request_checks sibling checks fire after each captured chat-completions request. - caller_messages_unmodified takes a model_dump snapshot pre-call and asserts byte-equality post-call. - expected.response.parsed is compared for equality. - expected.raises.carries is fed to assert_error_carries. - retry_middleware: block wraps the call in a default-classifier retry simulator (transient = TRANSIENT_CATEGORIES membership); the captured-request count provides provider_call_count. - mock_provider.capabilities.supports_native_response_format: false constructs the provider with force_prompt_augmentation_fallback=True. The 0016 structured-output fixtures (021–028) remain skipped at this commit. The next commit removes their skip markers. * test: drive 0016 fixtures 021-028 + add structured-output unit tests Removes the deferred-fixture skip markers for the 8 structured-output conformance fixtures (021–028). All pass against the OpenAIProvider + harness extensions landed in earlier commits. Adds tests/unit/test_structured_output.py covering bits the conformance fixtures don't exercise directly: - validate_response_schema edge cases: non-dict, non-object top-level, missing type. - strict_mode_supported: required-coverage rule, additionalProperties true, nested-object violation, anyOf branch violation, internal $ref resolution, unresolvable $ref, $ref cycle (self-referential schema). - _derive_schema_name: title-when-present, hash-fallback, determinism, empty-title behavior. - _augment_messages_with_schema_directive: prepend-when-no-system, extend-existing-system, caller-list-not-mutated, serialized-schema- substring. - Pydantic-class overload: class-in returns validated BaseModel instance; pydantic ValidationError wraps in StructuredOutputInvalid; wire body produced from class equals wire body produced from the equivalent .model_json_schema() dict. - uses_prompt_augmentation_fallback inspect property: False by default, True when constructor flag is set. * docs: changelog entry for proposal 0016 under [Unreleased] Documents the structured-output surface added in this PR: the response_schema parameter, Response.parsed field, StructuredOutputInvalid error category, OpenAIProvider native + fallback wire paths, the provider-agnostic schema helpers, the capability-agnostic conformance harness extensions, and the jsonschema runtime dependency. Also records: - Spec pin bump 0.10.0 → 0.15.0 (skip-ahead governance) with per-proposal deferred-skip in the conformance suite until each PR lands. - Release gate: do not tag the consolidated release until all five PRs of the batch (0011, 0014, 0015, 0016, 0017) are merged. * fix(llm): address CoPilot review on PR #42 Addresses the 8 CoPilot review threads on the structured-output PR: - strict_mode_supported now requires additionalProperties to be EXPLICITLY false (not just missing-or-false). Missing implies the JSON Schema default of permitting extras, which OpenAI's strict mode rejects. Pydantic's .model_json_schema() omits the key by default, so the class-input path would have 400ed against OpenAI even with conformance fixtures passing. - _normalize_response_schema now raises ProviderInvalidRequest when the class form is not a BaseModel subclass, instead of letting AttributeError leak from model_json_schema. - validate_response_schema now runs jsonschema.Draft202012Validator .check_schema() at the boundary, wrapping SchemaError as ProviderInvalidRequest. Malformed schemas now fail at the API boundary instead of escaping at decode time. - _derive_schema_name now regex-checks the title against OpenAI's name constraint (^[a-zA-Z0-9_-]{1,64}$) and falls back to the hashed name when the title doesn't match. Sanitizing-in-place would silently mutate user intent; the hash is a more honest fallback. - Two comments claiming Message instances are immutable Pydantic models were updated. The models are not configured with frozen=True; the safety actually comes from the helpers not modifying them in place. - match_wire_body now fails on extra keys in actual. The previous permissive default defeated the point of expected_wire_request being a literal compare; partial assertions continue to live in the sibling expected_wire_request_checks block. - _iter_calls now propagates expected_wire_request, expected_wire_request_checks, response_schema, and retry_middleware from sibling-of-call into the call dict. Only expected was being copied before. Cases-form fixtures with case-level wire expectations were silently running without those assertions. The _iter_calls fix surfaced two pre-existing gaps in the harness's handling of cases-shape fixtures, fixed inline: - The harness was never wiring config from the call spec into provider.complete(); fixture 005's runtime_config_passthrough case was effectively a no-op. - OpenAIProvider was using json.dumps default formatting for tool_call.function.arguments (with spaces after colons), which doesn't match the canonical compact form OpenAI emits or the spec's fixture 005 expectations. Switched to compact form. New unit tests cover the missing-additionalProperties strict-mode case, the non-BaseModel class rejection, the malformed JSON Schema rejection, and the title-falls-back hash cases. * docs: upgrade hello-world to demo structured output Replaces the no-LLM hello-world in README.md with a version that makes a real LLM call via OpenAIProvider and uses a Pydantic class as the response_schema. The resulting Response.parsed flows through state as a typed Classification instance and drives the conditional edge that routes between research and summarize. Defaults to OpenAI public API (gpt-4o-mini) with env-var config: LLM_BASE_URL, LLM_MODEL, LLM_API_KEY. A trailing line in the README calls out OpenRouter, vLLM, LM Studio, llama.cpp as drop-in swaps via base_url/model. The example also lands as a runnable file at examples/00-hello-world/main.py and is added to the smoke test suite. examples/README.md gets a corresponding entry. * fix(examples): hello-world base_url default and observer trace Two bugs surfaced during live validation against OpenAI: - The default LLM_BASE_URL was https://api.openai.com/v1, but our OpenAIProvider's wire path posts to /v1/chat/completions itself. httpx URL join produced https://api.openai.com/v1/v1/chat/completions → 404. Convention is base_url = host root; impl adds /v1. Default now matches; doc-string + README comment make it explicit. - The observer trace fired on the OpenAIProvider LLM-span event (sentinel namespace, post_state=None) and crashed accessing .sources. Added a post_state is not None guard. * docs(examples): wire research + summarize nodes to real LLM calls The hello-world's research and summarize nodes were returning hard-coded source lists. Replaces both with real provider.complete() calls that emit typed structured output, so the example demonstrates the value of a structured-output pipeline end-to-end instead of just the framework's plumbing. The example now exercises both response_schema forms in one demo: - classify and summarize use Pydantic classes (Classification, Summary); Response.parsed comes back as a validated instance. - research uses a raw JSON Schema dict; Response.parsed comes back as a plain dict. State gains two intermediate-artifact fields (research_plan, summary). Final output prints whichever branch fired, in addition to the existing sources/metadata. The reducer-policy story stays intact (last_write_wins on the LLM outputs, append on sources, merge on metadata). Live-validated against OpenAI gpt-4o-mini; both branches verified (structured class instance + structured dict on Response.parsed). * docs: structured output concepts page + model-providers updates Adds docs/concepts/llms.md covering how LLM calls fit into the graph model: LLM calls as async IO inside nodes, structured output (both response_schema forms + native/fallback wire paths + strict mode), routing on parsed fields, and errors at the LLM boundary. Nav entry added to mkdocs.yml's Concepts section; concepts/index.md TOC extended. Updates docs/model-providers/index.md: Protocol signature now shows the response_schema parameter; errors table adds StructuredOutputInvalid; new Structured output section walks through both response_schema forms, the native/fallback wire paths, and strict-mode constraints. Updates docs/model-providers/authoring.md: skeleton's complete() signature now matches the Protocol (response_schema parameter); a new "Structured output" entry in Beyond the skeleton points custom- provider authors at validate_response_schema and strict_mode_supported. mkdocs builds clean in strict mode; the runnable example in the new Structured output section is verified by tests/test_docs_examples.py. * docs: fix Provider.complete docstring Returns rendering The Returns block on Provider.complete started with "A :class:Response carrying ...", which mkdocstrings' Google-parser misread as a name-type pair: it pulled out "A" as the Name column entry and split the multi-line description across three table rows. Moving the return-value sentence into the prose summary at the top of the docstring (matching the pattern OpenAIProvider.complete already uses) renders cleanly: no spurious Name column entry, single description block. * fix(llm): second CoPilot review pass on PR #42 Addresses 19 review threads from the second CoPilot pass; about half were duplicates of the same underlying issue: - examples/00-hello-world/main.py + README hello-world: api_key now uses `os.environ.get("LLM_API_KEY") or None` so an exported-but- empty env var falls through to no-auth (matters for local servers that reject an empty bearer header). - Both examples now close the OpenAIProvider in the finally block alongside graph.drain(). Long-running consumers that copy the snippet had been leaking the underlying httpx.AsyncClient. - errors.py header dropped the hard-coded "seven canonical categories" count after StructuredOutputInvalid landed. - strict_mode_supported docstring and the surrounding spec-anchor comment block both updated to match the implementation: additionalProperties must be EXPLICITLY false (an omitted key counts as non-strict, since JSON Schema's default permits extras). - _resolve_ref now handles ref == "#" as the document root before rejecting external refs. Root-recursive schemas that use the bare JSON-Pointer-root form now resolve correctly. Unit test added. - _strict_mode_check tightened to return False on unrecognized shapes (empty {}, const-only, enum-only, unknown keywords) instead of falling through to True. Primitive types (string/integer/ number/boolean/null) classified as terminal-strict-compatible. Two unit tests added. - _build_request_body now explicitly strips response_format from the body when the provider is in fallback mode. RuntimeConfig is extra="allow", so a caller could have piped response_format through the extras loop past the include_response_format gate. - provider.py module docstring's summary signature line updated to match the Protocol's response_schema parameter. - validate_response_schema's spec-anchor comment updated to reflect that JSON Schema validity is now checked at the boundary via Draft202012Validator.check_schema(), not delegated to parse time. - test_pydantic_class_wire_body_matches_dict_form: widened the assertion from response_format-only to full body equality, so any regression in the class-input wire mapping (not just response_format) gets caught. - test_inspect_property_native_default and test_inspect_property_fallback_when_forced converted to async with try/finally + aclose() to match the rest of the file's provider-lifecycle pattern. * fix: third CoPilot review pass on PR #42 Addresses 5 remaining review threads (3 substantive, 2 stale on already-fixed code): - LlmProviderResponseAssertion (the typed assertion model in harness/expectations.py) now lists `parsed: Any | None`. The runtime assertion in test_llm_provider.py already handled it, but the typed parser had it under extra="forbid" and would have rejected any future case-shape LLM fixture using `parsed`. The 021-028 fixtures slip past today on `calls:` form's permissive `LlmCallSpec.expected: dict[str, Any]`; this lines the two paths up. - docs/model-providers/authoring.md skeleton comment tightened: removed the "ignore it and return free-form text" option from the response_schema guidance. A provider that silently drops the parameter violates the Protocol contract; callers expect either Response.parsed populated or StructuredOutputInvalid raised. Now only two valid options surfaced: raise ProviderInvalidRequest until implemented, or wire it through. - docs/concepts/llms.md softened the static-typing claim in the Pydantic-class form section. Response.parsed is `dict[str, Any] | BaseModel | None`, so a type checker won't narrow from `response_schema=Classification` alone. The page now separates the runtime guarantee (validated instance) from static access (requires cast/isinstance/typed assignment); generic Response[T] flagged as a follow-up. The two stale threads (examples/00-hello-world/main.py provider cleanup, test_structured_output.py provider cleanup) were already fixed in commit 8ed334c; replies sent + threads resolved without code changes. * fix: fourth CoPilot review pass on PR #42 Addresses 6 review threads, several of which surfaced second-order issues from previous rounds: - openai.py complete(): the fallback flag was driving include_response_format=False for every call, including free-form ones. That triggered the response_format strip on calls that weren't structured-output at all, clobbering caller-supplied RuntimeConfig extras. Gating the flag on schema_dict being set so free-form calls preserve extras. Unit test added. - src/openarmature/__init__.py + tests/test_smoke.py: bumped __spec_version__ from "0.10.0" to "0.15.0" to match the pyproject.toml [tool.openarmature].spec_version bump. AGENTS.md flags these three values as required to stay in sync; the submodule-bump commit missed the runtime sources. - _strict_mode_check array branch: {"type": "array"} without `items` no longer returns True. Unconstrained array content is the array analog of an object with no additionalProperties: false: the walker can't statically verify nested shapes, so strict mode rejects. Unit test added. - docs/model-providers/authoring.md: skeleton's complete() now actually enforces what its comment promised. Added `if response_schema is not None: raise ProviderInvalidRequest` to the body and surfaced the exception in the import list, so a provider copied from the skeleton can't silently violate the Protocol contract. - docs/concepts/llms.md Pydantic-class snippet: added `from typing import Literal` so the example is copy-paste- runnable (the snippet uses Literal in the class but only imported BaseModel). - tests/unit/test_structured_output.py nested-recursion tests: test_strict_mode_recurses_into_nested_object and test_strict_mode_anyof_branch_must_satisfy were short-circuiting at the root because the root schema itself failed strict rules. Tightened both root schemas so the recursive walk actually fires; the tests now guard the recursion they claim to. * docs: cumulative strict-mode constraint list + spec-version-drift test Captures two follow-ups surfaced by the four CoPilot review rounds: - docs/concepts/llms.md "Strict mode" section expanded into the full constraint list. After four rounds of tightening the strict_mode_supported heuristic, the rule set is stable and the user-facing surface should list it directly rather than make callers read provider.py. The page frames the list as the authoritative set: anything not on it trips to non-strict. - docs/model-providers/index.md "Strict mode" subsection trimmed and now links into the concepts page for the full list, following the established split (concepts/ owns the deep-dive, model-providers/ stays terse). - tests/test_smoke.py adds test_spec_version_matches_pyproject: reads pyproject.toml's [tool.openarmature].spec_version and asserts it equals openarmature.__spec_version__. AGENTS.md flags these as required to stay in sync; the previous smoke test only checked internal consistency between __spec_version__ and its asserted value, so the pyproject side could drift silently (and did, in the original submodule-bump commit). * fix: fifth CoPilot review pass on PR #42 Addresses 4 review threads: - examples/00-hello-world/main.py: provider construction moved from module level to a lazy _get_provider() helper backed by a module global. Avoids opening an httpx.AsyncClient when tooling imports the module without running main() — the smoke test now doesn't trigger construction across 6 example loads. main()'s finally only closes when the cached instance is set. - src/openarmature/llm/provider.py: validate_response_schema now walks all $ref values via _check_refs_resolvable and raises ProviderInvalidRequest for any non-internal-resolvable ref. Draft202012Validator.check_schema doesn't traverse refs, so previously an external ref slipped past the boundary and surfaced as a raw referencing-library exception at validate time. Pre-validation surfaces the clean category at the API boundary. - src/openarmature/llm/providers/openai.py: _parse_and_validate now also catches jsonschema.SchemaError and maps it to StructuredOutputInvalid. Safety net for any schema-side exception (including ref-resolution failures) that pre- validation might miss. - tests/unit/test_structured_output.py: - test_strict_mode_unresolvable_ref_fails: root tightened with additionalProperties: false so the walk reaches the $ref branch (was short-circuiting at the root). - Added test_validate_response_schema_rejects_external_ref covering the new pre-validation path. - tests/test_smoke.py: added test_spec_version_matches_submodule_pin shelling to `git -C openarmature-spec describe --tags --exact-match HEAD` and asserting it equals v{__spec_version__}. Skips cleanly when the submodule isn't a git checkout (installed-package CI lanes). Completes the three-place drift check from AGENTS.md (__spec_version__ ↔ pyproject ↔ submodule pin). * fix(test): replace git-describe submodule check with CHANGELOG parse The git-describe-based submodule check from the previous commit passed locally but failed in CI because actions/checkout pins the submodule to its recorded SHA without fetching the spec repo's tags. `git describe --tags --exact-match` then finds nothing and the test fails with "submodule HEAD is not at any tag." Switching to parsing openarmature-spec/CHANGELOG.md: the spec follows Keep a Changelog, so the first non-[Unreleased] `## [X.Y.Z]` heading is the version at the pinned commit. This works regardless of CI tag-fetch state and catches the same drift class (submodule moved to a different release). Skips cleanly when CHANGELOG.md isn't present (installed-package lanes that don't ship the submodule checkout). * fix(test): satisfy CodeQL on the changelog-parsing test CodeQL flagged the for/else: pytest.fail() pattern as a potentially-uninitialized-local-variable warning because it doesn't model pytest.fail as NoReturn — the analyzer sees a path where submodule_latest is referenced after the loop without ever being bound. Pulling the parse into _read_latest_spec_version_from_changelog that explicitly returns the version or raises AssertionError. Eliminates the unreachable-after-fail pattern and reads cleaner. * fix: seventh CoPilot review pass on PR #42 Six second-order correctness fixes surfaced by the round-7 review, mostly hardening _resolve_ref, _check_refs_resolvable, and the Pydantic-class validation path. - _resolve_ref now distinguishes "unresolvable" (path doesn't exist / external ref) from "resolved to non-dict" via a module-level _UNRESOLVABLE sentinel. Boolean schemas (true/false) are valid JSON Schema subschemas; a $ref to one was being incorrectly rejected as ProviderInvalidRequest. Now resolves cleanly and strict-mode still returns False on bool targets (the correct conservative answer). - validate_response_schema's metaschema check now uses jsonschema.validators.validator_for(schema) instead of the hard-coded Draft 2020-12. A valid draft-07 schema (e.g. tuple- form items, common in tooling) was being rejected at the boundary but accepted at runtime. Boundary and runtime now agree. - _resolve_ref percent-decodes JSON Pointer tokens before applying the ~1 / ~0 unescape pair. Per RFC 6901 §6, a JSON Pointer in a URI fragment is percent-encoded; refs like #/$defs/Name%20With%20Spaces now resolve correctly. - _check_refs_resolvable now walks only known subschema-bearing keywords (properties, patternProperties, additionalProperties, items, prefixItems, contains, if/then/else, allOf/anyOf/oneOf/not, $defs/definitions, dependentSchemas, propertyNames, unevaluatedItems, unevaluatedProperties). A "$ref" key under data positions (default, const, enum, $comment, x-* extensions) is data, not a schema reference, and is no longer incorrectly resolved. - docs/concepts/llms.md "LLM calls are async IO inside a node" section reframed: module-level provider construction leaks the httpx.AsyncClient in tooling/test/docs-build imports. The page now documents application-startup / lifecycle-managed construction (lazy on-first-use plus aclose in finally / shutdown hook), matching the pattern the hello-world example was made lazy for. - _parse_and_validate's Pydantic-class path now runs jsonschema.validate against the generated JSON Schema BEFORE calling model_validate. Pydantic's default model_validate is coercive (accepts "30" for an int field), which diverged from the strict dict-schema path. Both paths now apply the same jsonschema check first; model_validate then constructs the typed instance. - jsonschema.ValidationError's failure description now includes exc.json_path (e.g. "$.age: '30' is not of type 'integer'"). The bare exc.message lost the field name, breaking caller diagnostics for the missing-field / wrong-type-at-path cases. Five new unit tests cover the bool-ref, draft-07, percent-encoded ref, ref-under-data, and Pydantic-coercion-rejection cases.
1 parent 89df03d commit 2ecb7b1

28 files changed

Lines changed: 2872 additions & 108 deletions

CHANGELOG.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,24 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The
66

77
## [Unreleased]
88

9+
### Added
10+
11+
- **Structured output (proposal 0016, spec v0.14.0).** `Provider.complete()` now accepts an optional `response_schema` parameter — either a JSON Schema dict or a Pydantic `BaseModel` subclass. When supplied, the provider constrains the model's output to the schema and populates `Response.parsed` with the validated value (`dict` for dict-schema input, a `BaseModel` instance for class input). New `StructuredOutputInvalid` error category (non-transient by default) raises on JSON parse failure or schema validation failure; carries the requested schema, the raw response content, and a failure description.
12+
- **`OpenAIProvider` native response_format wire path.** When `response_schema` is supplied, the chat-completions request body carries `response_format: { type: "json_schema", json_schema: { name, schema, strict } }`. The `strict` flag is determined by a deep recursive walk over the schema (object-property required-coverage rule across `anyOf` / `oneOf` / `allOf` and `$ref` targets, with cycle protection); unresolvable refs fall through to `strict: false`. The `name` field uses `schema.title` when present, otherwise a deterministic sha256-prefix hash.
13+
- **`OpenAIProvider` prompt-augmentation fallback.** Constructor flag `force_prompt_augmentation_fallback: bool` (default `False`) and read-only inspect property `uses_prompt_augmentation_fallback: bool`. When the flag is on, structured-output calls build a fresh message list with a system directive containing the serialized schema, omit `response_format` from the wire, and validate the response post-receive. The caller's original `messages` list is never mutated. Use for OpenAI-compatible servers (older vLLM, some LM Studio releases, llama.cpp variants) that reject or silently ignore `response_format`.
14+
- **Provider-agnostic schema helpers.** `openarmature.llm.validate_response_schema(schema)` (raises `ProviderInvalidRequest` when the schema is not a dict with a top-level `type: "object"`) and `openarmature.llm.strict_mode_supported(schema)` (the deep-tree strict-mode constraint check) are exported for reuse by future Anthropic/Gemini providers.
15+
- **Capability-agnostic conformance harness helpers.** `tests/conformance/harness/wire.py` adds `match_wire_body` (recursive deep-equal with `"*"` wildcard support), `assert_response_format_absent`, `assert_system_references_schema`, and `assert_error_carries` for the `expected_wire_request[_checks]` and `expected.raises.carries.{...}` fixture shapes. Used by the 0016 fixtures; available for the upcoming 0014 / 0015 / 0017 fixture sets.
16+
- **Runtime dependency: `jsonschema>=4.0`.** Used by the dict-schema validation path. The Pydantic-class path uses Pydantic's native validator and does not need `jsonschema`.
17+
18+
### Changed
19+
20+
- **Pinned spec version: 0.10.0 → 0.15.0.** Adopts the skip-ahead governance principle: the submodule jumps across v0.11.0–v0.15.0 (proposals 0009, 0011, 0014, 0015, 0016, 0017) in one bump. Only the surface introduced by proposal 0016 is implemented in this changelog entry; fixtures from 0011 / 0014 / 0015 / 0017 are marked deferred-skip in the conformance suite and unmark as their respective PRs land.
21+
22+
### Notes
23+
24+
- **Release gate: do not tag until all of {0011, 0014, 0015, 0016, 0017} are merged.** This batch implements one proposal per PR and lands a consolidated release after the fifth PR. Cutting a release tag before the batch is complete would ship a partial spec implementation against the v0.15.0 pin.
25+
- **Pre-1.0 MINOR.** Existing free-form callers (no `response_schema`) see no behavior change — the new field defaults to `None`, the wire body omits `response_format`, and `Response.parsed` remains absent.
26+
927
## [0.5.0] — 2026-05-10
1028

1129
First release on real PyPI. Catches the implementation up from spec v0.5.x to v0.10.0 across six phases — the spec accepted eight proposals while the python lib was at v0.3.1, and v0.5.0 lands all of them in one curated drop.

README.md

Lines changed: 72 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -55,26 +55,34 @@ The OpenTelemetry mapping mandates a private `TracerProvider`. That prevents the
5555

5656
## Hello World
5757

58-
About fifty lines that show the engine in action. Three reducer policies declared on one state class. Routing as a pure function of state, not a hidden state machine. An observer attached at compile time that sees every node boundary the engine emits. No LLM, no API key, no boilerplate. Copy it, run it, watch the events fire. Requires Python 3.12 or later.
58+
About a hundred lines that show the engine in action. Three reducer policies declared on one state class. Three LLM calls each returning typed structured output (Pydantic class on two, raw JSON Schema dict on the third). Conditional routing as a pure function of state, not a hidden state machine. An observer attached at compile time that sees every node boundary the engine emits. Requires Python 3.12 or later and an OpenAI-compatible endpoint (defaults to OpenAI public API; works against any local server too).
5959

6060
```python
6161
import asyncio
62-
from typing import Annotated
63-
64-
from openarmature.graph import (
65-
END,
66-
GraphBuilder,
67-
NodeEvent,
68-
State,
69-
append,
70-
merge,
71-
)
72-
from pydantic import Field
62+
import os
63+
from collections.abc import Mapping
64+
from typing import Annotated, Any, Literal
65+
66+
from openarmature.graph import END, GraphBuilder, NodeEvent, State, append, merge
67+
from openarmature.llm import OpenAIProvider, UserMessage
68+
from pydantic import BaseModel, Field
69+
70+
71+
class Classification(BaseModel):
72+
intent: Literal["research", "summarize"]
73+
rationale: str
74+
75+
76+
class Summary(BaseModel):
77+
one_liner: str
78+
confidence: float
7379

7480

7581
class PipelineState(State):
7682
query: str # last_write_wins (default)
77-
classification: str = "" # last_write_wins
83+
classification: Classification | None = None # set by classify
84+
research_plan: dict[str, Any] | None = None # set by research (dict-schema form)
85+
summary: Summary | None = None # set by summarize
7886
sources: Annotated[list[str], append] = Field( # appends across writes
7987
default_factory=list
8088
)
@@ -83,34 +91,56 @@ class PipelineState(State):
8391
)
8492

8593

86-
async def classify(state: PipelineState) -> dict:
87-
decision = "research" if "?" in state.query else "summarize"
88-
return {
89-
"classification": decision,
90-
"metadata": {"classified_by": "rule"},
91-
}
94+
provider = OpenAIProvider(
95+
base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com"), # host root; impl adds /v1
96+
model=os.environ.get("LLM_MODEL", "gpt-4o-mini"),
97+
api_key=os.environ.get("LLM_API_KEY") or None, # empty → no-auth
98+
)
9299

93100

94-
async def research(state: PipelineState) -> dict:
101+
async def classify(state: PipelineState) -> Mapping[str, Any]:
102+
response = await provider.complete(
103+
[UserMessage(content=f"Route to 'research' or 'summarize': {state.query!r}")],
104+
response_schema=Classification, # class → instance
105+
)
106+
return {"classification": response.parsed, "metadata": {"classified_by": "llm"}}
107+
108+
109+
async def research(state: PipelineState) -> Mapping[str, Any]:
110+
response = await provider.complete(
111+
[UserMessage(content=f"Plan research for {state.query!r}: list topics + follow-ups.")],
112+
response_schema={ # dict → dict
113+
"type": "object",
114+
"properties": {
115+
"topics": {"type": "array", "items": {"type": "string"}},
116+
"follow_up_questions": {"type": "array", "items": {"type": "string"}},
117+
},
118+
"required": ["topics", "follow_up_questions"],
119+
"additionalProperties": False,
120+
},
121+
)
95122
return {
123+
"research_plan": response.parsed,
96124
"sources": ["wikipedia", "arxiv"],
97-
"metadata": {"tool": "search"},
125+
"metadata": {"tool": "research"},
98126
}
99127

100128

101-
async def summarize(state: PipelineState) -> dict:
102-
return {
103-
"sources": ["cache"],
104-
"metadata": {"tool": "summarizer"},
105-
}
129+
async def summarize(state: PipelineState) -> Mapping[str, Any]:
130+
response = await provider.complete(
131+
[UserMessage(content=f"Summarize {state.query!r} in one sentence with confidence 0-1.")],
132+
response_schema=Summary, # class → instance
133+
)
134+
return {"summary": response.parsed, "sources": ["cache"], "metadata": {"tool": "summarize"}}
106135

107136

108137
def route(state: PipelineState) -> str:
109-
return state.classification
138+
assert state.classification is not None
139+
return state.classification.intent
110140

111141

112142
async def trace(event: NodeEvent) -> None:
113-
if event.phase == "completed" and event.error is None:
143+
if event.phase == "completed" and event.error is None and event.post_state is not None:
114144
print(f"{event.node_name}: sources={event.post_state.sources}")
115145

116146

@@ -127,22 +157,30 @@ graph = (
127157
)
128158
graph.attach_observer(trace)
129159

160+
130161
async def main() -> None:
131162
try:
132-
await graph.invoke(PipelineState(query="what is RAG?"))
163+
final = await graph.invoke(PipelineState(query="what is RAG?"))
164+
print(f"\nclassification: {final.classification}")
165+
if final.research_plan is not None:
166+
print(f"research_plan: {final.research_plan}")
167+
if final.summary is not None:
168+
print(f"summary: {final.summary}")
133169
finally:
134170
await graph.drain()
171+
await provider.aclose()
135172

136173

137174
asyncio.run(main())
138-
# classify: sources=[]
139-
# research: sources=['wikipedia', 'arxiv']
140175
```
141176

142-
A few things to notice in this short example:
177+
Set `LLM_API_KEY=sk-...` and run. To swap providers, point `LLM_BASE_URL` and `LLM_MODEL` at OpenRouter, vLLM, LM Studio, llama.cpp, or anything else that speaks the OpenAI Chat Completions wire format. The example also lives at [`examples/00-hello-world/main.py`](./examples/00-hello-world/main.py); see [`examples/`](./examples/) for more runnable demos.
178+
179+
A few things to notice:
143180

144-
- **Three reducer policies on one state schema.** `query` and `classification` get the default `last_write_wins`. `sources` is `Annotated[list[str], append]`, so successive writes concatenate. `metadata` is `Annotated[dict[str, str], merge]`, so successive writes shallow-merge. The merge policy lives on the schema, once.
145-
- **Conditional routing as a state function.** `route` reads `state.classification` and returns a node name. The graph engine doesn't care that this happens to be deterministic; it would accept an LLM-driven router with the same shape.
181+
- **Three reducer policies on one state schema.** `query` / `classification` / `research_plan` / `summary` get the default `last_write_wins`. `sources` is `Annotated[list[str], append]`, so successive writes concatenate. `metadata` is `Annotated[dict[str, str], merge]`, so successive writes shallow-merge. The merge policy lives on the schema, once.
182+
- **Structured output, two forms.** `response_schema=Classification` (a Pydantic class) returns `Response.parsed` as a validated `Classification` instance, typed end-to-end. `response_schema={...}` (a raw JSON Schema dict) returns `Response.parsed` as a plain dict. Same wire shape underneath; pick the form that fits.
183+
- **Conditional routing on a parsed field.** `route` reads `state.classification.intent` and returns the next node's name. The graph engine doesn't care the discriminator came from an LLM; it would accept a deterministic rule with the same shape.
146184
- **Observer sees both phases.** `trace` filters to `completed` events for brevity; the engine also delivers `started` events.
147185
- **The graph either compiles or it doesn't.** Remove `.set_entry()` and `.compile()` raises `NoDeclaredEntry` before `invoke()` runs.
148186

docs/concepts/index.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ the framework, or jump to whichever concept you need.
1212
data seam.
1313
- [Fan-out](fan-out.md): running the same subgraph many times in
1414
parallel, results merged back deterministically.
15+
- [LLMs](llms.md): how LLM calls fit into nodes, structured output,
16+
routing on parsed fields, errors at the LLM boundary.
1517
- [Observability](observability.md): node-boundary hooks, OTel mapping,
1618
log correlation.
1719
- [Checkpointing](checkpointing.md): save state at each node boundary,

0 commit comments

Comments
 (0)