Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow)

[thegovind]

Status

Design/alignment; implementation branch pending the RFC 008 base decision.

PostTrain/BenchFlow is ready to test an early branch. The draft RFC 008 stack (#942–#958) now proposes the shared adapter/planner/executor/report foundation, so this issue should implement the missing one-shot task-package lane rather than a second validation core. Reviews are underway on the author-facing contracts in #947 and #952.

Problem

PostTrain Arena uses containerized task packages rather than served environments. Its starting-kit package contains a task.md, an environment/, a verifier, and an oracle/solve.sh. The agent runs to completion, then the verifier reads final artifacts and produces a reward.

OpenEnv environments use a different lifecycle: a long-lived server is driven through reset / step / state, while agents interact through MCP and reward is returned by the environment.

flowchart LR
    subgraph Served["OpenEnv served lifecycle"]
      O["Trainer / orchestrator"] -->|"reset · step · state"| E["Long-lived environment"]
      A["Agent"] -->|"MCP tools"| E
      E -->|"Observation + reward"| O
    end

    subgraph OneShot["PostTrain one-shot lifecycle"]
      P["Task package"] --> R["Agent runs to completion"]
      R --> V["Verifier reads artifacts"]
      V --> W["Reward / evidence"]
    end

    Served -. "shared planning/reporting, different runner" .-> OneShot

Loading

Neither lifecycle should pretend to be the other. The missing capability is an authoring and validation path for the one-shot package.

Current direction

RFC 008 proposes the shared, spec-neutral pieces:

flowchart LR
    S["Trusted spec adapter"] --> N["Normalized subject + requirements"]
    N --> P["Policy + planner"]
    P --> X["Execution-model-specific runner"]
    X --> R["Shared validation report"]

    X -->|"served"| SV["OpenEnv runtime checks"]
    X -->|"one_shot"| OS["This issue"]

Loading

If that architecture is accepted, this issue owns the PostTrain one-shot adapter, policy, runner, CLI surface, and conformance fixtures. It does not own the generic validation planner/executor/report core.

Proposed v1 scope

• Pin an explicit PostTrain starting-kit profile/version rather than claiming broad BenchFlow compatibility.
• Add a trusted task.md adapter and structural checks for frontmatter, prompt, environment, verifier, oracle, and required paths.
• Emit RFC 008 shared reports with stable diagnostics and certified: false for local author validation.
• Add a separate task-oriented CLI (openenv validate-task or openenv task validate; spelling still to be decided).
• Add representative valid, structurally invalid, and execution-invalid fixtures supplied with PostTrain/BenchFlow.
• Add a one-shot Docker runner only after the lifecycle, reward-boundary, and isolation policy is agreed.
• Validate oracle and empty-trial behavior with policy-defined thresholds/margins, not brittle exact reward == 1.0 assertions.

Suggested implementation sequence:

flowchart TD
    D["Decide base + one-shot policy ownership"] --> A["PR 1: task.md adapter + structural checks"]
    A --> F["PR 2: shared diagnostics + conformance fixtures"]
    F --> R["PR 3: isolated one-shot runner"]
    R --> C["PR 4: task CLI + docs"]

Loading

Explicit non-goals for v1

• Reimplementing or wrapping served OpenEnv validation; openenv validate remains the served-environment command.
• Public plugin discovery in the first slice.
• Official certification of external task packages.
• Claiming full generic BenchFlow support from the smaller PostTrain starting-kit checker.
• Running hosted-verifier, GPU/TPU-only, multi-agent, or simulated-user packages locally without a defined runner and policy.

Decisions still needed

1. Base: branch from CLI: validation profiles and shared reports #947 after its CLI semantics are unified, or from a restacked surface that includes Validation: add structured author diagnostics and remediation #952 diagnostics without pulling unrelated remote-provider work.
2. Policy ownership: add the one-shot lifecycle/reward/isolation contract to RFC 008, or create a focused follow-up RFC.
3. Profile/version: which PostTrain starting-kit commit and which generic BenchFlow release, if any, are initial conformance targets.
4. Dependency strategy: pin/import supported BenchFlow APIs versus maintaining a small profile-specific checker with provenance tests.
5. Fixtures: representative good, structurally bad, and runtime-bad packages from PostTrain Arena.
6. CLI spelling: validate-task versus task validate.

Acceptance criteria for the first branch

• A pinned valid PostTrain fixture passes structural validation.
• Invalid fixtures produce stable, repository-relative diagnostics and actionable remediation.
• The adapter performs safe structural inspection only; it does not import or execute submitted code.
• Reports use the shared schema and never claim certification.
• No task package is launched through the served Uvicorn path.
• The implementation and docs state the exact profile/version being validated.

Related work

• RFC 008: Spec-neutral local and remote validation architecture #942: draft RFC 008 validation architecture and stacked implementation.
• [feature] Import external envs (ors, verifiers) #726: importer registry precedent.
• envs/tbench2_env: nearby one-shot/pytest task shape, not a drop-in validator.
• Harbor/ATIF/ACP trajectory and agent interoperability is a separate follow-up boundary, not part of this v1.

[AmyTao]

Hi @thegovind! This is Amy from the BenchFlow community. I think a separate validate-env command would be better here, to avoid conflicting with what the existing validate command already does.

[thegovind]

Direction update after reviewing RFC 008 (#942) and its implementation stack:

• @AmyTao's suggestion is right in substance: task-package validation should have a separate command instead of extending the existing openenv validate flow.
• RFC 008 now provides the spec-neutral adapter, planner, executor, and report foundation, and explicitly reserves one-shot task packages for this issue.
• This issue should therefore narrow to the missing PostTrain/BenchFlow lane: the task.md adapter and structural checks, a one-shot Docker oracle/empty-trial runner, a separate task CLI, docs, and one end-to-end example.
• The task command should not wrap OpenEnv-native validation; served OpenEnv environments remain under openenv validate. The exact task command can be settled as openenv validate-task versus openenv task validate. I would avoid validate-env, because that is too close to the existing environment command.
• Public plugins can support local author validation, but official certification must use pinned trusted adapters.

I have proposed to @zkwentz that I own this focused follow-up rather than opening a competing validation-core PR. The likely implementation sequence is:

1. PostTrain/BenchFlow adapter plus structural policy.
2. One-shot local runner and shared report integration.
3. Task CLI, plugin/docs surface, and example package.

I am holding off on a branch from main. If we develop before #942–#947 land, the clean base is #947 (rfc008-06-cli); otherwise this should start from updated main after that foundation merges.

[AmyTao]

@thegovind Hi Govind! Could you point me to the branch you are working on for the implementation? I'd like to test on the posttrain arena side! thx

[thegovind]

Hi Amy - thanks, and sorry for the delay. There is not an implementation branch yet; I deliberately held off because #898 now depends on the draft RFC 008 stack, and I do not want to create a second validation core in parallel.

I have now reviewed the other two isses Ben prioritized (#947 and #952) and left contract issues for Zach. My proposed first WIP slice is still the PostTrain task.md adapter plus structural checks off the smallest stable core base. the Docker oracle/empty-trial runner would follow after we agree on the one-shot reward and isolation policy. I will link the clearly labeled WIP branch here once Zach confirms whether that base is #947 as-is or a restacked version that includes the diagnostics contract.

To make the first slice useful on your side, could you share 2–3 representative fixtures: one valid PostTrain package, one structurally invalid package, and one that passes structure but should fail oracle/empty-trial validation? It would also help to know the starting-kit commit/profile you want us to pin for the initial conformance target.