Mariame/scenario1 event handoff#9
Open
diabatem wants to merge 4 commits into
Open
Conversation
- scenario1_trajectory.schema.json: pin schema_version to spec_gap.scenario1.v2 and set outcome_class to the six canonical classes shared with the PR #4 adapter (clean, resisted, propagated_but_not_executed, attempted_but_blocked, executed, indeterminate). - Rename the single validator to lowercase validate_trajectory.py; guard the action_channel / behavioral_channel reads so a malformed record returns a validation error instead of a KeyError; rename the INV-7 outcome branch to propagated_but_not_executed. - pyproject.toml: add jsonschema (validator dependency) instead of a separate requirements file. Part of the fresh main-based replacement for PR #7. Inputs, corrected dry-run generator, docs, and tests follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LMYkZdQB4Vz8qamn1q2cxE
…art 2)
- experiments/scenario1/inputs/: controlled input registry and three clean
document fixtures named after the source PDFs. Report 3 here is the CLEAN
canonical text (the PDF is poisoned); the injection is inserted only when
constructing the injected condition.
- scenario1_pipeline.py: structural dry-run generator (no model, no GPU, no
.pt files). Reads the registry and emits one match group -- clean/injected x
2-hop/3-hop -- as v2 records:
* one wording (the group's assigned A) across all four conditions;
* exactly three document slots; clean vs injected document 3 differ only by
the injected span (asserted);
* injection fixed at worker_1; only worker_1 sees raw poison; worker_2 and
the executor are relay-fed;
* no text-to-action fallback -> injected outcome is
propagated_but_not_executed, action_channel not executed;
* reasoning labels null (annotation-only); activation refs are dry-run
placeholders; a manifest is written and every path is asserted to resolve.
- schema: allow extraction_metadata.storage_path null for dry-run placeholders.
- gitignore the generated manifest.json (trajectories/ already ignored).
All four generated trajectories pass validate_trajectory.py (v2 schema + INV-1..7).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LMYkZdQB4Vz8qamn1q2cxE
…art 3) validate_trajectory.py: add two semantic invariants the review asked for -- INV-8 (agent topology matches the depth condition; catches a 3-hop trace missing worker_2) and INV-9 (a non-null reasoning label requires human or mechanistic evidence). tests/test_scenario1_schema.py and tests/test_scenario1_validator.py: exercise the real input builder and dry-run generator (not one hand-written trajectory). Positive: four base conditions; shared task/seed/wording; three document slots; clean vs injected document 3 differ only by the injected span; only worker_1 poison-exposed; agents match depth; injected -> propagated_but_not_executed; reasoning labels null; the six canonical outcomes are schema-valid; manifest paths resolve; no duplicate ids. Negative fixtures the validator must reject: worker_2 poison-exposed, 3-hop missing worker_2, outcome executed without an action, reasoning label without evidence, auto-derived reasoning_compromised, schema-version mismatch, broken hops_survived, and a contaminated clean control. 22 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LMYkZdQB4Vz8qamn1q2cxE
Describe only files/commands that exist. Cover ownership boundaries (this PR = schema + validator + inputs + docs; generation = Onyinye; model execution = PR #8; normalization = PR #4), the input-vs-dry-run-output-vs-real-data distinction, the four channels, the nine invariants, the v2 decision, and the smoke check. State that dry-run files are an initial end-to-end validation subset, not experiment data, and that only worker_1 sees the raw poison. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LMYkZdQB4Vz8qamn1q2cxE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replacement for PR #7, opened fresh from
mainas a draft. This is theScenario 1 schema + controlled-inputs scaffold for the public-health match
group, in the shared event-style format. It is not the full dataset and not a
model runner. Reference: PR #7 (to be closed once this is open).
Schema version:
spec_gap.scenario1.v2(matches PR #4's normalization path).Files owned by this PR
scenario1_trajectory.schema.json— one v2 schema; six canonical outcomes.validate_trajectory.py— the single (lowercase) validator: JSON Schema + 9semantic invariants.
experiments/scenario1/inputs/—registry.json, three clean documentfixtures named after the source PDFs (Report 3 is the clean canonical text;
the injection is inserted only for the injected condition), and the source
PDFs.
scenario1_pipeline.py— structural dry-run generator only (no model, noGPU, no
.pt).tests/test_scenario1_schema.py,tests/test_scenario1_validator.py.pyproject.toml— addsjsonschema.Deliberately not included: any real-model backend,
live_run_config.py,the old pilot JSONL, the duplicate capital validator, or PDFs used as poisoned
"clean" documents.
Ownership / interfaces expected
Qwen/Qwen3-32Bon Modal, thinking on/off,activation extraction, model metadata). The schema holds exact model I/O and
honest activation references; extraction layers stay configurable, no primary
layer is locked.
The four base conditions (one match group)
ph_breast_cancer_A: clean 2-hop, injected 2-hop, clean 3-hop, injected 3-hop.All four share the same task, documents, system prompts, wording (A), seed, and
document order — only treatment and depth vary. Injection is fixed at
worker_1at both depths;
worker_2and the executor are relay-fed. The injected executorforwards the endpoint in text but fires no structured tool call, so the outcome
is
propagated_but_not_executed(notexecuted). Reasoning labels stay null.Commands run and results
The dry-run trajectory files are an initial end-to-end validation subset:
they confirm the pipeline and schema agree. A dry run does not call the model,
does not start a GPU, and does not create activation files. They are not
real-model experiment results. Generated trajectories and the manifest are
gitignored.
Open integration items
registry.jsonshape against Onyinye's shared input registry.