docs: schema-validated documentation examples

[igrigorik]

Validation harness that ensures every JSON example in the spec docs is validated against UCP schemas. Catches documentation drift — when schemas change, stale examples break CI and docs builds.

Contract

Every json block requires an HTML comment annotation on the preceding line:

<!-- ucp:example schema=shopping/checkout op=read -->
<!-- ucp:example schema=shopping/checkout path=$.totals op=read -->
<!-- ucp:example schema=shopping/cart op=create direction=request -->
<!-- ucp:example skip reason="JSON-RPC transport binding" -->

Unannotated blocks are hard failures.

Core rule: "if you open it, you own it." When an example opens an object's braces, every required field (per the resolved schema for the declared op + direction) must be shown or acknowledged with an ellipsis. This applies recursively.

Three-tier value contract for annotated (non-skip) examples:

Tier	Syntax	Meaning
Full value	Real JSON	Validated against schema (types, constraints, enums)
Required field	{ ... }, [ ... ], "..."	Field must exist; value not shown; scaffold fills it for validation
Absent	Not in example	Only permitted for optional schema fields

Pipeline

1. Extract — find annotated ```json blocks (handles MkDocs tab indentation)
2. Unwrap — strip HTTP headers from request/response examples
3. Expand — substitute {{ ucp_version }} with valid date
4. Preprocess — convert bare ... to valid JSON ({ ... } → {"...":"..."})
5. Coverage — walk resolved schema, verify required fields acknowledged
6. Strip — remove ellipsis markers
7. Merge — deep-merge into scaffold (example wins, scaffold fills gaps)
8. Validate — ucp-schema validate on merged payload
9. Report — map errors to source file:line

Bugs caught and fixed

• checkout.md: item missing required price field in disclosure example
• checkout.md: totals array missing required subtotal entry
• checkout-rest.md: business outcome used non-schema available_quantity field on line_item; replaced with message-based pattern
• discount.md (4 instances): quantity wrongly nested inside item object instead of on line_item — schema requires quantity at line_item level
• discount.md: price and title shown in create requests where schema omits them (request-only fields: only item.id is valid)

Current state

python3 scripts/validate_examples.py --schema-base source/schemas/ — 52 passed, 0 failed, 0 errors

268 blocks across 39 files:

• 52 validated — REST request/response pairs, core spec examples, error responses, extension examples
• 216 skipped — transport bindings, schema definitions, configs (future: payload_path= for JSON-RPC unwrap)

🟢 Pausing fixing the rest (skipped) to get feedback: 🟢

• Does this strategy make sense?
• Anything we want to change or improve?

---

Type of change

• Documentation update
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings

[drewolson-google]

Looks good, one small comment on empty request bodies.

[drewolson-google]

Can we make the validator / annotation handle empty request bodies? This is a valid request, so having to skip it feels off.

[wry-ry]

The scaffold approach would require that we write (and maintain) a scaffold for most types, right? An alternative approach that I was exploring was:

• Force every example to not omit any required root level properties (value can be ellipsis)
• Use ucp-schema to resolve the schema based on the annotation
• Use jsonschema library to validate the example against the schema
  • Use jsonschema's extend function to create a custom validator class that skips ellipses

May be more convoluted, but would save the extra work of maintaining additional scaffold files. Curious to hear your thoughts.

[raginpirate]

I have no issue with this PR, but I would love to also challenge the fundamental requirement to have all required top-level keys be provided in an unignored example.
We don't want to bloat every snippet with whole complete top level properties, so I feel like many docs will end up leveraging the ignore block 🤔

Can we solve this using a custom key like "...": true or "__rest__": true to ignore evaluating any other required but not present fields in that obj. And perhaps we could display that nicely in the final output docs? {"id": 1234, "...": true} => {"id": 1234, ...}

This can be viewed as non-blocking if we just want to get V0 out, but it could be nice to disallow the "skip" unless you are providing completely custom, non-ucp json 😄