Add Scenario 1 v2 handoff adapter#4
Conversation
|
Could @diabatem, @onyokoli, and @Ifeoluwah review this adapter from the Scenario 1 output side? Please check that it matches the v2 files you are using or planning to use, especially:
The generator only needs to save the raw facts. This adapter assigns the shared behavioral outcome, so contributors do not need to add the same label logic to each generator. This PR does not replace the v2 schema or define the final generator. It converts the richer trajectory format into the per-step format used by the probe and metrics code. Current local checks: the full suite passed with 94 tests. The 24 focused adapter and metrics tests also passed, along with Python compilation and |
|
Closing this PR as superseded by PR #10, which has now merged into main. The Scenario 1 v2 adapter, outcome mapping, activation-request metadata, and related tests from this branch were preserved in that integrated PR, so no work is being discarded. |
Summary
This draft PR adds the adapter between the Scenario 1 v2 event-style trajectory format and the existing probe and metrics pipeline.
The adapter keeps the richer v2 schema as the public trajectory format. It converts that format into the normalized per-step records needed for activation extraction, AUROC, Brier score, ECE, and Temporal Divergence.
What this PR includes
schema_version == "spec_gap.scenario1.v2"and normalizes it directly.transformer_lensto import.Action and outcome boundary
The adapter does not treat a requested tool call as an executed action.
If the model asks to call the exfiltration tool but no executor action fires, the adapter keeps the request but leaves the executed-actions list empty. That trajectory can remain
propagated_but_not_executed,attempted_but_blocked, or another observed outcome. It must not becomeexecutedonly because the model mentioned an endpoint or produced tool-call syntax.Only a real executor action may set
unsafe_action_executed=true.The adapter also does not assign a confirmed latent-compromise label. Reasoning or latent status must remain unconfirmed unless later human or mechanistic evidence supports it.
Scope
This PR does not generate Scenario 1 documents or trajectories. It does not call the model, execute tools, or create behavioral ground truth. It only validates and converts the completed v2 trajectory records produced by the shared Scenario 1 pipeline.
Validation
96 passed26 passedRegression coverage includes a v2 3-hop trajectory with planner, worker, worker2, and executor; preservation of activation and cost metadata; separation of hidden thinking from forwarded content; and confirmation that a requested tool call does not become an executed action.
Review requested
Could @diabatem, @onyokoli, and @Ifeoluwah review this adapter from the Scenario 1 output side?
Please check whether it matches the v2 fields you are using or plan to use, especially:
The goal is to keep one shared event-style trajectory format while giving the probe and metrics code a stable normalized input.