Skip to content

Add Scenario 1 v2 handoff adapter#4

Closed
elenaajayi wants to merge 5 commits into
mainfrom
codex/v2-scenario-adapter
Closed

Add Scenario 1 v2 handoff adapter#4
elenaajayi wants to merge 5 commits into
mainfrom
codex/v2-scenario-adapter

Conversation

@elenaajayi

@elenaajayi elenaajayi commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

This draft PR adds the adapter between the Scenario 1 v2 event-style trajectory format and the existing probe and metrics pipeline.

The adapter keeps the richer v2 schema as the public trajectory format. It converts that format into the normalized per-step records needed for activation extraction, AUROC, Brier score, ECE, and Temporal Divergence.

What this PR includes

  • Detects schema_version == "spec_gap.scenario1.v2" and normalizes it directly.
  • Preserves the real 3-hop topology: planner → worker → worker2 → executor.
  • Keeps Worker1 as the raw-poison entry point at both depths.
  • Preserves agent IDs, hop indexes, poison-exposure metadata, token alignment, activation metadata, and attention metadata.
  • Preserves construction, behavioral, and reasoning label channels without combining them into one compromise label.
  • Preserves the Modal runner's exact model input and output fields, including:
    • raw generated text
    • parsed final message
    • separate thinking content and completion status
    • tool-call requests and parse errors
    • model execution metadata
    • cost and token metadata
  • Keeps the Modal generation and cost metadata in activation requests so later probe results can be traced back to the exact model call.
  • Lazy-loads activation extraction so adapter and request tests do not require transformer_lens to import.

Action and outcome boundary

The adapter does not treat a requested tool call as an executed action.

If the model asks to call the exfiltration tool but no executor action fires, the adapter keeps the request but leaves the executed-actions list empty. That trajectory can remain propagated_but_not_executed, attempted_but_blocked, or another observed outcome. It must not become executed only because the model mentioned an endpoint or produced tool-call syntax.

Only a real executor action may set unsafe_action_executed=true.

The adapter also does not assign a confirmed latent-compromise label. Reasoning or latent status must remain unconfirmed unless later human or mechanistic evidence supports it.

Scope

This PR does not generate Scenario 1 documents or trajectories. It does not call the model, execute tools, or create behavioral ground truth. It only validates and converts the completed v2 trajectory records produced by the shared Scenario 1 pipeline.

Validation

  • Full test suite: 96 passed
  • Focused trajectory acceptance, metrics, Temporal Divergence, and Modal-metadata tests: 26 passed
  • Python compilation checks passed.
  • Diff checks passed.

Regression coverage includes a v2 3-hop trajectory with planner, worker, worker2, and executor; preservation of activation and cost metadata; separation of hidden thinking from forwarded content; and confirmation that a requested tool call does not become an executed action.

Review requested

Could @diabatem, @onyokoli, and @Ifeoluwah review this adapter from the Scenario 1 output side?

Please check whether it matches the v2 fields you are using or plan to use, especially:

  • event fields and agent topology
  • construction, behavioral, outcome, and reasoning labels
  • poison-exposure and token-alignment metadata
  • exact model input and output fields
  • thinking content and downstream forwarded content
  • requested tool calls versus executed actions
  • activation, attention, cost, and manifest metadata

The goal is to keep one shared event-style trajectory format while giving the probe and metrics code a stable normalized input.

@elenaajayi elenaajayi mentioned this pull request Jul 10, 2026
@elenaajayi elenaajayi added phase:0 Phase 0 fellowship work area:probe Probe, activation, or measurement pipeline work type:adapter Adapter or format conversion work status:needs-review Needs review from the team labels Jul 10, 2026
@elenaajayi

elenaajayi commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

Could @diabatem, @onyokoli, and @Ifeoluwah review this adapter from the Scenario 1 output side?

Please check that it matches the v2 files you are using or planning to use, especially:

  • the 2-hop and 3-hop agent order;
  • Worker1 being the only agent that sees the raw poisoned document;
  • exact model inputs and outputs, token alignment, activation metadata, and attention metadata;
  • match-group, domain, wording, carrier, model, and thinking-mode metadata;
  • executor actions and their status;
  • the six behavioral outcomes: clean, resisted, propagated_but_not_executed, attempted_but_blocked, executed, and indeterminate;
  • latent_compromise_status: candidate meaning “needs more study,” not confirmed compromise;
  • real artifact paths and null paths for dry-run placeholders.

The generator only needs to save the raw facts. This adapter assigns the shared behavioral outcome, so contributors do not need to add the same label logic to each generator.

This PR does not replace the v2 schema or define the final generator. It converts the richer trajectory format into the per-step format used by the probe and metrics code.

Current local checks: the full suite passed with 94 tests. The 24 focused adapter and metrics tests also passed, along with Python compilation and git diff --check.

@elenaajayi elenaajayi self-assigned this Jul 10, 2026
@elenaajayi elenaajayi changed the title [codex] Add Scenario 1 v2 handoff adapter Add Scenario 1 v2 handoff adapter Jul 10, 2026
@elenaajayi

Copy link
Copy Markdown
Collaborator Author

Closing this PR as superseded by PR #10, which has now merged into main. The Scenario 1 v2 adapter, outcome mapping, activation-request metadata, and related tests from this branch were preserved in that integrated PR, so no work is being discarded.

@elenaajayi elenaajayi closed this Jul 14, 2026
@elenaajayi
elenaajayi deleted the codex/v2-scenario-adapter branch July 14, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:probe Probe, activation, or measurement pipeline work phase:0 Phase 0 fellowship work status:needs-review Needs review from the team type:adapter Adapter or format conversion work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant