Skip to content

fix(schemas): add missing 'executive_synthesis_skipped' to Event.type enum#1

Open
MerouaneBen wants to merge 1 commit into
ProofAgent-ai:mainfrom
MerouaneBen:fix-event-enum-exec-synthesis
Open

fix(schemas): add missing 'executive_synthesis_skipped' to Event.type enum#1
MerouaneBen wants to merge 1 commit into
ProofAgent-ai:mainfrom
MerouaneBen:fix-event-enum-exec-synthesis

Conversation

@MerouaneBen

Copy link
Copy Markdown

Summary

When the Reporter's executive-synthesis LLM call raises, the graceful-fallback path emits Event(type="executive_synthesis_skipped", …) — but that string isn't in the Event.type Literal enum (schemas.py). Pydantic then raises ValidationError, which crashes the entire run at the reporter stage, after the (expensive) jury + consensus work is already done. The safety net is itself the failure.

Where

proofagent_harness/agents/reporter.py_generate_executive_synthesis() except Exception: block calls _emit(state, Event(type="executive_synthesis_skipped", …)). The Event.type Literal in schemas.py does not list that value.

Repro

Any Harness().evaluate(...) where the harness LLM fails/refuses/times out on the exec-synthesis call. We hit it reliably with a --metrics subset + a harness model that occasionally returns malformed JSON. Instead of degrading to the deterministic summary, the process dies with:

pydantic_core.ValidationError: 1 validation error for Event
type
  Input should be 'setup_start', … , 'done' or 'error' [input_value='executive_synthesis_skipped']

Fix

Add "executive_synthesis_skipped" to the Event.type Literal. Verified locally — the run then completes on the deterministic fallback as intended.

Context

Found while running a serious hands-on evaluation of a fleet of internal agents. Genuinely useful harness — thanks for open-sourcing it.

… enum

The reporter's executive-synthesis fallback emits
Event(type='executive_synthesis_skipped', ...) when the exec-summary LLM call
raises, but that value is absent from the Event.type Literal in schemas.py. Pydantic
then raises ValidationError, crashing the entire run at the reporter stage — after
the (expensive) jury + consensus work is already done. The graceful fallback is
itself the failure path.

Adding the enum value lets the run degrade to the deterministic summary as intended.

Found while evaluating a fleet of internal agents; reproduced reliably when the
harness LLM returns malformed JSON / times out on the exec-synthesis call.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant