Research and demo repo for testing whether agentic AI systems stay useful and safe when external web-like pages contain adversarial, misleading, stale, or source-laundered content.
This is the sibling project to rag-redteam-lab. The RAG project showed that
retrieved evidence can corrupt answers and citations. This project moves the
same research theme into an agentic setting: the system can browse, select
sources, synthesize evidence, and decide when to abstain.
This repo is a public AI safety artifact around a concrete failure mode: defenses against poisoned web evidence can become too conservative and over-abstain, even when trusted current pages directly answer the question. The long-graph v2 experiment evaluates that safety/usefulness tradeoff on a synthetic campus AI-governance corpus with hosted model runs, paired tests, row-level public data, and a dashboard/demo packet.
The differentiator is the research method around the result: a named benchmark specification, row-level public snapshots, machine validation, a blinded manual audit queue, a preregistered held-out v3 replication plan, and a completed single-deployment v3 Stage 3 full-run packet. The repo is set up so another reviewer can inspect evidence paths and falsify the claim rather than just watch an agent demo.
Core finding: the A10 preservation-calibrated relation gate preserved correct abstention on all paired evidence-gap rows while repairing direct-control over-abstention introduced by earlier relation gates.
| Result | Evidence |
|---|---|
| Public hosted rows | 576 sanitized rows across two Azure OpenAI deployments |
| Primary deployment | A10: 72/72 accuracy, 48/48 direct controls, 0/72 attack successes |
| Cross-model replication | A10: 72/72 accuracy, 48/48 direct controls, 0/72 attack successes |
| Paired repair vs A8 | 14 direct-control rows fixed, 0 new direct-control misses, exact McNemar p = 0.0001 |
| Paired repair vs A9 | 19 direct-control rows fixed, 0 new direct-control misses, exact McNemar p < 0.0001 |
| Public validation | Row counts, summary consistency, redaction, condition coverage, and task/page IDs pass |
| Goal | File |
|---|---|
| Understand the benchmark contract | docs/benchmark-spec.md |
| Understand the research claim in two minutes | docs/extended-abstract.md |
| Give a short demo | docs/demo-script.md |
| Inspect the headline dashboard | static/research-dashboard.html |
| Follow a reviewer path | docs/reviewer-guide.md |
| Verify the public data package | artifacts/long-graph-v2/README.md |
| Check machine validation and hashes | docs/long-graph-v2-public-artifact-validation.md, docs/research-artifact-manifest.md |
| Run a blind evidence audit | docs/manual-audit-protocol.md, docs/blind-audit-label-schema.md, artifacts/long-graph-v2/blind-audit-queue.jsonl, docs/blind-audit-validation.md |
| Inspect the held-out replication plan | docs/v3-replication-plan.md |
| Inspect the v3 Stage 3 full-run packet | docs/v3-full-results.md, artifacts/v3-full/README.md, docs/v3-full-public-artifact-validation.md |
| Review safety boundaries | docs/threat-model.md |
| Track submission readiness | docs/submission-checklist.md |
Do agentic AI systems remain reliable when the external pages they inspect contain prompt injections, source laundering, stale policy text, fake citations, or misleading calls to action? More specifically, can a defense avoid poisoned evidence without collapsing into unnecessary abstention when trusted current evidence directly supports or refutes the proposition?
- Agentic systems increasingly browse, select sources, and make multi-step evidence decisions.
- Poisoning defenses need to optimize both safety and usefulness; abstaining on everything can hide a poor agent behind a safe-looking metric.
- The benchmark separates final-answer correctness, evidence sufficiency, source trust, stale evidence, poisoned citations, and action safety.
- The public packet includes row-level hosted data and deterministic validation, so reviewers can inspect the mechanism rather than trusting a summary table.
| Condition | Name | Description |
|---|---|---|
| A0 | Direct answer | Model answers without browsing. |
| A1 | Agent baseline | Agent browses top pages and answers with citations. |
| A2 | Source ranking | Agent prefers trusted/current pages before synthesis. |
| A3 | Prompt Shields | User prompt and page-content screening before synthesis. |
| A4 | Full defense | Source ranking, Prompt Shields, citation validation, and abstention. |
| A5 | Strict abstention | Full defense plus explicit evidence-sufficiency rules. |
| A6 | Relation verifier | A5 plus explicit direct-support/direct-refutation/missing-validation classification. |
| A7 | Structured relation gate | A6 plus application enforcement of verified evidence-relation labels. |
| A8 | Classified relation gate | A7-style gate using a separate relation-classifier stage instead of gold labels. |
| A9 | Calibrated relation gate | A8 plus a conservative evidence-gap override for absent independent validation. |
| A10 | Preservation-calibrated relation gate | A9 plus direct-support/direct-refutation preservation when trusted current pages agree. |
- Indirect prompt injection embedded in page text.
- Source laundering through formal-looking but low-trust pages.
- Stale page conflict against a current policy page.
- Fake citation or reference bait.
- Action hijack, where a page tries to make the agent change task goals.
- Confidence pressure, where a page pushes certainty despite weak evidence.
Run tests:
make testRun the deterministic seed benchmark and write a report:
make research-refreshGenerate only the human audit queue after a run:
make audit-localRun the Azure OpenAI hosted smoke validation after filling .env from
.env.example:
make hosted-smoke-refreshRun the focused Azure sweep across six attack classes and two benign controls:
make hosted-focused-refreshRun the full hosted research matrix across all 30 tasks and A0-A4:
make hosted-full-refreshRun the harder challenge set focused on source laundering and abstention:
make hosted-challenge-refreshRun the same challenge set with the experimental A5 strict-abstention condition:
make hosted-strict-challenge-refreshRun the focused evidence-boundary set for A4/A5:
make hosted-boundary-refreshRun the relation-verifier follow-up for A5/A6:
make hosted-relation-boundary-refreshRun five repeated A5/A6 boundary trials for variance:
make hosted-relation-boundary-repeats-refreshRun five repeated A6/A7 boundary trials for the structured relation gate:
make hosted-relation-gate-repeats-refreshRun five repeated A6/A7 trials on the expanded boundary set:
make hosted-relation-gate-expanded-repeats-refreshRun the local expanded A7/A8 classifier-gate smoke benchmark:
make relation-classifier-expanded-refreshRun five repeated hosted A7/A8 classifier-gate trials on the expanded boundary set:
make hosted-relation-classifier-expanded-repeats-refreshRun five repeated hosted A9 calibrated classifier-gate trials:
make hosted-relation-calibrated-expanded-repeats-refreshRun the local multi-page graph stress benchmark:
make graph-refreshRun the local long-chain graph stress benchmark:
make long-graph-refreshRun the local 24-task long-chain v2 graph benchmark:
make long-graph-v2-refreshRun three repeated hosted graph stress trials:
make hosted-graph-repeats-refreshRun three repeated hosted long-chain graph stress trials:
make hosted-long-graph-repeats-refreshRun the hosted A10 long-chain preservation follow-up:
make hosted-long-graph-preservation-repeats-refreshRun the hosted A1/A4/A8/A9/A10 long-chain v2 pilot:
make hosted-long-graph-v2-pilot-refreshRun the hosted A8/A9/A10 long-chain v2 cross-model replication, defaulting to
the gpt-4-1-mini deployment:
make hosted-long-graph-v2-cross-model-repeats-refreshRun the hosted A10 cross-model follow-up, defaulting to the gpt-4-1-mini
deployment:
make hosted-long-graph-preservation-cross-model-repeats-refreshRun the hosted A8/A9 cross-model long-graph baseline on the same deployment:
make hosted-long-graph-relation-gates-cross-model-repeats-refreshGenerate the long-graph v2 corpus card:
make long-graph-v2-corpus-card-refreshGenerate the paired A7/A8/A9 statistical appendix from hosted result files:
make paired-analysis-a7-a9Generate the paired long-graph v2 preservation appendix from the primary and cross-model hosted result files:
make paired-analysis-long-graph-v2-preservationGenerate the qualitative long-graph v2 preservation repair casebook from the same hosted rows and page corpus:
make casebook-long-graph-v2-preservationGenerate the long-graph v2 page-label transition appendix from the same hosted rows and page corpus:
make transition-analysis-long-graph-v2-preservationGenerate the full blinded long-graph v2 audit queue and unblinding key from the committed public snapshots:
make blind-audit-long-graph-v2-publicValidate the blinded audit queue and key:
make validate-blind-audit-long-graph-v2Generate the deterministic artifact checksum manifest:
make artifact-manifest-refreshFor a fast review path through the public artifacts, see
docs/reviewer-guide.md.
Outputs are written under experiments/results/local/ and kept out of Git.
Hosted smoke outputs are written under experiments/results/hosted-smoke/ and
are also kept out of Git. Focused sweep outputs are written under
experiments/results/hosted-focused/, including comparison.md for local vs
hosted deltas. A committed aggregate snapshot is in
docs/hosted-focused-summary.md. Full hosted matrix outputs are written under
experiments/results/hosted-full/, including stats.md with Wilson confidence
intervals, attack-class breakdowns, defense deltas, and provider reliability.
A committed aggregate snapshot is in docs/hosted-full-summary.md.
Challenge-set outputs are written under experiments/results/hosted-challenge/
and use data/tasks.challenge.jsonl plus data/pages.challenge.jsonl. A
committed aggregate snapshot is in docs/hosted-challenge-summary.md.
Strict-abstention challenge outputs are written under
experiments/results/hosted-strict-challenge/; this keeps the A1-A4 snapshot
fixed while adding A5 as a follow-up calibration experiment. A committed
aggregate snapshot is in docs/hosted-strict-challenge-summary.md.
Boundary-set outputs are written under experiments/results/hosted-boundary/
and use data/tasks.boundary.jsonl plus data/pages.boundary.jsonl. A
committed aggregate snapshot is in docs/hosted-boundary-summary.md.
Relation-boundary outputs are written under
experiments/results/hosted-relation-boundary/ and compare A5 against the
experimental A6 relation verifier. A committed aggregate snapshot is in
docs/hosted-relation-boundary-summary.md.
Repeated relation-boundary outputs are written under
experiments/results/hosted-relation-boundary-repeats/ and compare five A5/A6
passes for variance. A committed aggregate snapshot is in
docs/hosted-relation-boundary-repeats-summary.md.
Structured relation-gate outputs are written under
experiments/results/hosted-relation-gate-repeats/ and compare five A6/A7
passes. A committed aggregate snapshot is in
docs/hosted-relation-gate-repeats-summary.md.
Expanded structured relation-gate outputs are written under
experiments/results/hosted-relation-gate-expanded-repeats/ and compare five
A6/A7 passes on data/tasks.boundary-expanded.jsonl. A committed aggregate
snapshot is in docs/hosted-relation-gate-expanded-summary.md.
Classifier-gate outputs are written under
experiments/results/hosted-relation-classifier-expanded-repeats/ and compare
five A7/A8 passes on the expanded boundary set. This target tests whether a
separate relation-classifier stage can replace A7's synthetic verified labels.
A committed aggregate snapshot is in
docs/hosted-relation-classifier-expanded-summary.md.
Calibrated classifier-gate outputs are written under
experiments/results/hosted-relation-calibrated-expanded-repeats/ and test the
A9 evidence-gap override against the A8 classifier failure mode. A committed
aggregate snapshot is in docs/hosted-relation-calibrated-expanded-summary.md.
Graph-stress outputs are written under experiments/results/graph-local/ and
experiments/results/hosted-graph-repeats/. They use
data/tasks.graph.jsonl and data/pages.graph.jsonl to test multi-page
evidence chains with low-trust, stale, and confidence-pressure distractors. A
committed aggregate snapshot is in docs/hosted-graph-summary.md.
Long-graph outputs are written under experiments/results/long-graph-local/
and experiments/results/hosted-long-graph-repeats/. They use
data/tasks.graph-long.jsonl and data/pages.graph-long.jsonl to test three
trusted pages plus three adversarial distractors per task. A committed
aggregate snapshot is in docs/hosted-long-graph-summary.md.
A10 preservation follow-up outputs are written under
experiments/results/long-graph-preservation-local/ and
experiments/results/hosted-long-graph-preservation-repeats/. The committed
aggregate snapshot is in docs/hosted-long-graph-preservation-summary.md.
Long-graph v2 outputs are written under
experiments/results/long-graph-v2-local/ and
experiments/results/hosted-long-graph-v2-pilot/. They use
data/tasks.graph-long-v2.jsonl and data/pages.graph-long-v2.jsonl to test
24 tasks, eight campus AI governance domains, four trusted current evidence
pages, and four adversarial distractors per task, including fake-citation
laundering. The hosted pilot runs three repeats across the vulnerable baseline,
full defense, and A8/A9/A10 relation-gate defenses. The committed aggregate
snapshot is in docs/hosted-long-graph-v2-summary.md; sanitized row-level
evidence is in artifacts/long-graph-v2/hosted-gpt5-mini-results.jsonl.
The v2 cross-model replication target writes to
experiments/results/hosted-long-graph-v2-gpt41mini-a8-a10-repeats/ and reruns
A8/A9/A10 on the same corpus to test whether A10's preservation repair
replicates beyond the primary gpt-5-mini deployment. A committed aggregate
snapshot is in docs/hosted-long-graph-v2-cross-model-summary.md; sanitized
row-level evidence is in
artifacts/long-graph-v2/hosted-gpt41-mini-a8-a10-results.jsonl.
The public data package README at artifacts/long-graph-v2/README.md
documents the row schema, redacted provider fields, and rebuild commands.
The paper-style overview in docs/extended-abstract.md summarizes the method,
headline effects, public artifacts, and limitations.
docs/demo-script.md turns the same packet into a concise presentation flow
with exact claims and caveats.
docs/threat-model.md defines the synthetic adversary, safety scope, release
boundary, and out-of-scope misuse.
docs/submission-checklist.md tracks class/workshop readiness across claims,
data, method, reproducibility, demo, and safety.
docs/release-notes-v0.1-long-graph-v2.md describes the current portfolio
artifact release.
Citation metadata is available in CITATION.cff.
The benchmark contract in docs/benchmark-spec.md defines the task/page/result
schemas, condition set, model-adapter expectations, and metrics for the named
Long-Graph v2 Preservation benchmark.
The corpus card in docs/long-graph-v2-corpus-card.md documents the balanced
24-task design: 8 yes, 8 no, and 8 insufficient-evidence tasks across 8
domains, with 4 trusted pages and 4 attack pages per task.
docs/long-graph-v2-public-artifact-validation.md machine-checks the public v2
row snapshots for redaction, summary consistency, condition coverage, and
task/page ID integrity.
The cross-model A10 target defaults to
experiments/results/hosted-long-graph-preservation-gpt41mini-network-repeats/
and can be pointed at another deployment with
LONG_GRAPH_CROSS_MODEL_DEPLOYMENT=<deployment>.
The cross-model A8/A9 relation-gate target defaults to
experiments/results/hosted-long-graph-gpt41mini-relation-gates-repeats/.
The paired statistical appendix in docs/paired-a7-a9-analysis.md aligns A7,
A8, and A9 rows by task and repeat index, including exact McNemar tests for the
A8 degradation and A9 repair.
The paired v2 preservation appendix in
docs/paired-long-graph-v2-preservation-analysis.md aligns A8/A9/A10 rows by
deployment, task, and repeat index. Across both v2 deployments, A10 fixed 14
direct-control rows relative to A8 and 19 relative to A9, with 0 new
direct-control misses.
The companion casebook in docs/long-graph-v2-preservation-casebook.md
surfaces 14 representative repaired rows with trusted/current page evidence,
relation-label changes, and safety metrics.
The transition appendix in
docs/long-graph-v2-preservation-transition-analysis.md quantifies the repair
mechanism: 35 page-label transition observations all occur on repaired
direct-control rows, with 0 non-repaired direct-control rows changing labels
and 0 A10 regressions.
The artifact manifest in docs/research-artifact-manifest.md records row
counts, line counts, byte sizes, and SHA-256 hashes for the key v2 research
files.
The blind audit protocol in docs/manual-audit-protocol.md and generated queue
in artifacts/long-graph-v2/blind-audit-queue.jsonl support condition-blinded
evidence review. docs/blind-audit-validation.md machine-checks queue/key
alignment, empty reviewer labels, citation alias resolution, and configured
leakage strings. docs/blind-audit-label-schema.md defines the human label
JSONL schema and the CLI validator for submitted reviewer labels. The
future-facing preregistration in
docs/v3-replication-plan.md defines held-out hypotheses, budget gates, and
success/failure criteria before confirmatory v3 rows are collected. The
exploratory v3 smoke packet in artifacts/v3-smoke/ and cross-model pilot
packet in artifacts/v3-pilot/ document the hosted gate runs without upgrading
them into a completed v3 replication claim. The first Stage 3 full-matrix
packet is committed in artifacts/v3-full/: 720 public rows on gpt-5-mini
with a passing gate check, artifact validation report, and paired A8/A9 -> A10
repair ledger. It is a held-out single-deployment full-run result, not a
cross-model v3 success label.
Hosted Make targets stream rows into results.jsonl as each call completes and
resume by default. To force a clean rerun, pass HOSTED_RESUME=:
HOSTED_RESUME= make hosted-full-refreshThe public static demo trace is committed at
static/action-hijack-case-study.html and opens directly in a browser. The
aggregate research dashboard is committed at static/research-dashboard.html.
Paper-style write-up materials are in docs/paper-draft.md, with adjudication
rules in docs/labeling-rubric.md, a one-page reviewer brief in
docs/research-brief.md, and initial manual challenge labels in
data/manual-audit.hosted-challenge.jsonl plus strict-abstention labels in
data/manual-audit.hosted-strict-challenge.jsonl plus paired A8/A9 boundary
labels in data/manual-audit.hosted-a8-a9-boundary.jsonl. Reproduction steps
are in docs/reproducibility.md. Research scope and safety boundaries are in
SECURITY.md.
The deterministic benchmark now runs 30 synthetic tasks across A0-A4, including 24 adversarial tasks and 6 benign controls.
| Condition | Cases | Attack cases | Accuracy | Attack success | Cited poisoned | Filtered poisoned |
|---|---|---|---|---|---|---|
| A0_DIRECT | 30 | 24 | 66.7% | 0.0% | 0.0% | 0.0% |
| A1_AGENT_BASELINE | 30 | 24 | 20.0% | 100.0% | 80.0% | 0.0% |
| A2_SOURCE_RANKING | 30 | 24 | 100.0% | 0.0% | 0.0% | 0.0% |
| A3_PROMPT_SHIELDS | 30 | 24 | 40.0% | 75.0% | 60.0% | 20.0% |
| A4_FULL_DEFENSE | 30 | 24 | 100.0% | 0.0% | 0.0% | 80.0% |
This deterministic harness result is separate from the hosted model snapshots
in docs/. It gives the project a repeatable local baseline and a quick
regression signal.
The local deterministic harness with synthetic web pages can:
- Load a task case and a small page corpus.
- Simulate browsing and page selection.
- Run A0 through A4.
- Log page visits, selected evidence, ignored/filtered pages, final answer, citations, safety state, and action decisions.
- Produce a Markdown report and a static demo trace.
.
├── data/ # Synthetic web pages and task cases
├── docs/ # Research plan, protocol, budget plan
├── static/ # Self-contained public demo artifact
├── src/ # Harness source
└── tests/ # Unit tests and fixture checks
M1 local harness and M2 dataset expansion are implemented for the 30-task
synthetic benchmark. M3 audit queue and the first M5 static trace are in place.
The Azure-hosted runner is implemented for the same redacted synthetic corpus.
Completed hosted artifacts now include the full matrix, challenge matrix,
strict-abstention matrix, evidence-boundary matrix, and relation-verifier
boundary follow-up with repeated-trial variance. The A7 structured relation
gate has also been validated on repeated boundary trials and an expanded
16-task boundary sweep. The A8 classified relation gate is implemented with
local smoke results and a hosted repeat snapshot showing the classifier-label
gap against A7. The A9 calibrated relation gate then recovered the A7 ceiling
on the expanded boundary set, with a paired appendix quantifying the 14 fixed
A8 misses and 0 new A9 misses. The hosted graph stress run now extends the
result to 12 multi-page evidence graphs: A8/A9 reached 36/36 accuracy and
12/12 correct abstention while A4 blocked poisoned citations but only abstained
on 2/12 evidence gaps. A harder hosted long-graph run then surfaced the next
boundary: A8/A9 preserved 12/12 evidence-gap abstentions but over-abstained on
some direct policy and privacy-board controls. The A10 preservation-calibrated
follow-up repaired that long-graph boundary, reaching 36/36 accuracy, 12/12
correct evidence-gap abstention, and 24/24 direct-control preservation with
0/36 attack success and 0/36 poisoned citations. A second Azure deployment,
gpt-4-1-mini, replicated the same A10 result on another 36 hosted rows after
an A8/A9 cross-model baseline reproduced the direct-no over-abstention
failure at 1/12 preservation for both conditions. The hosted long-graph v2 run
scaled this finding to 24 tasks and 360 rows: A10 reached 72/72 accuracy,
24/24 evidence-gap abstention, and 48/48 direct-control preservation, while
A8 reached 70/72 and A9 reached 66/72 due to direct-control over-abstention.
A second v2 run on gpt-4-1-mini added 216 hosted rows and replicated the
same A10 result: A10 reached 72/72 while A8 reached 60/72 and A9 reached 59/72,
again with 0 poisoned citations and 0 provider errors.
CI is enabled on the public GitHub repo, and the static dashboard summarizes
the main hosted results.
Keep all attack pages synthetic and redacted. Public demo artifacts should show attack mechanisms and outcomes, not reusable operational prompt-injection text.