Governed Multi-LLM Agent Architecture

Responsible AI control plane for a multi-LLM prior-authorization decision-support workflow.

This repo demonstrates how to keep the core multi-LLM engineering story intact while adding the governance controls expected in regulated healthcare AI: pre-router PHI/PII redaction, risk tiering, defense-in-depth redaction before third-party egress, approved-provider egress policy, structured governance explanations, prior-auth evidence coverage reports, human-in-the-loop escalation and closure, deterministic privacy and red-team evals, prior-auth invariance checks, retry/fallback safety, schema contracts for governance artifacts, observability helpers, evidence packet export, governance scorecards, CI evidence, and audit evidence.

This is an architecture MVP, not a production medical or coverage-decision system.

Architecture At A Glance

flowchart TD
    U[User prompt] --> BAA
    subgraph BAA[Managed GCP / ADK boundary]
        ADK[ADK runtime and session state]
        PR[before_model_callback PHI/PII redaction]
        GEM[Gemini orchestrator]
        ADK --> PR
        PR -->|redacted model request| GEM
    end
    GEM --> G[Third-party egress control plane]
    G --> R[Risk tiering]
    G --> P[Defense-in-depth PHI/PII redaction]
    G --> E[Approved-provider egress policy]
    G --> EX[Structured governance explanations]
    G --> EC[Prior-auth evidence coverage report]
    G --> H[HITL escalation decision]
    G --> A[Audit trace]
    G --> EV[Deterministic governance evals]
    G --> EP[Reviewer evidence packet]
    EV --> SC[Governance scorecard]
    EX --> A
    EC --> A
    EC --> H
    A --> EP
    E -->|redacted prompt only| O[OpenAI]
    E -->|redacted prompt only| C[Claude]
    E -->|redacted prompt only| X[Grok]
    O --> GEM
    C --> GEM
    X --> GEM
    GEM --> V[MVP boundary checks / planned output validation]
    V --> A
    V --> F[Final decision-support answer]

Loading

User request
  -> pre-router PHI/PII redaction
  -> ADK/Gemini orchestrator inside managed GCP boundary
  -> Responsible AI egress control plane
       -> risk tiering
       -> PHI/PII redaction for third-party providers
       -> approved-provider egress policy
       -> structured governance explanations
       -> evidence coverage report for prior authorization
       -> HITL escalation decision
       -> HITL assignment and review closure
       -> deterministic governance evals
       -> audit trace
       -> reviewer evidence packet and scorecard
  -> Multi-LLM data plane
       -> OpenAI / Claude / Grok perspectives
       -> Gemini synthesis
       -> retry/fallback and cost tracking
  -> decision-support output for human review

The first vertical slice is prior-authorization summarization:

Prior-auth request
  -> pre-router PHI/PII redaction
  -> Gemini router
  -> risk tiering
  -> defense-in-depth PHI/PII redaction before third-party egress
  -> approved-provider egress policy
  -> structured governance explanations
  -> evidence coverage report
  -> multi-LLM orchestration
  -> HITL escalation
  -> HITL assignment and closure
  -> sanitized audit events
  -> reviewer evidence packet
  -> governance scorecard
  -> red-team, privacy, and invariance eval evidence

Multi-LLM Data Plane

The agent still demonstrates modern AI orchestration:

• provider diversity through LiteLLM;
• Gemini as orchestrator and synthesizer;
• provider-specific strengths from OpenAI, Claude, and Grok;
• retry with backoff, fallback chains, and graceful degradation;
• token and cost tracking.

The local usage summary in metrics.py is ephemeral demo telemetry, not audit evidence. Durable metric events should flow through observability.record_metric(), which writes sanitized audit events with a trace ID.

Optional OpenTelemetry spans and operational metrics live behind multi_model_agent/telemetry.py. Local/dev mode remains no-op unless enabled by environment variables; prompts, responses, raw PHI, raw excerpts, and reviewer IDs are never attached as telemetry attributes.

Default model versions:

Role	Provider	LiteLLM model ID
Orchestrator / synthesizer	Gemini	gemini-3.5-flash
Structured implementation perspective	OpenAI	gpt-5.5
Deep reasoning perspective	Claude	claude-opus-4-8
Alternative exploration perspective	Grok	xai/grok-4.3

Responsible AI Control Plane

The MVP governance path is implemented in code:

Control	File
Pre-router PHI/PII redaction	multi_model_agent/pre_router.py
Risk tiering	multi_model_agent/risk.py
PHI/PII redaction before third-party egress	multi_model_agent/privacy.py
Provider egress policy	multi_model_agent/policy.py
Structured governance explanations	multi_model_agent/explainer.py
Prior-auth evidence coverage report	multi_model_agent/evidence_coverage.py
HITL escalation and review closure	multi_model_agent/escalation.py, multi_model_agent/review.py, scripts/record_human_review.py
PHI-safe audit trace and hash-chain verification	multi_model_agent/audit.py, multi_model_agent/audit_store.py, multi_model_agent/audit_hashing.py
Schema contracts	multi_model_agent/schemas.py
Retry/fallback safety	multi_model_agent/reliability.py
Provider tool integration	multi_model_agent/tools.py
Optional OpenTelemetry governance observability	multi_model_agent/telemetry.py, docs/observability.md
Reviewer evidence packets	multi_model_agent/evidence_packet.py, scripts/export_audit_packet.py
Governance scorecard	scripts/generate_governance_scorecard.py, governance/governance_scorecard.md
Repository governance CI	.github/workflows/governance-ci.yml
Deterministic red-team eval	scripts/run_redteam_eval.py, evals/redteam/prior_auth_redteam_cases.json
Privacy redaction benchmark	evals/privacy/run_redaction_benchmark.py, evals/privacy/labeled_phi_cases.jsonl
Prior-auth structured invariance regression	evals/fairness/run_invariance_eval.py, evals/fairness/prior_auth_invariance_cases.jsonl

The ADK agent uses before_model_callback=redact_before_model, which redacts text in the Gemini router request before the model call. External provider calls are then prepared through prepare_provider_request(), which redacts sensitive data again and records privacy, risk, policy, evidence coverage, and escalation events before egress to non-Google third-party LLMs.

Structured governance explanations are deterministic control-plane artifacts. They use reason codes, policy IDs, decision factors, redaction counts, source references, and model provenance; they do not ask models to reveal hidden Chain-of-Thought and do not persist hidden reasoning.

For prior-authorization workflows, evidence_coverage.py produces a deterministic EvidenceCoverageReport that labels required documentation elements as present, missing, insufficient, or not_applicable. The report stores source references and hashes over redacted excerpts, not raw excerpts. It is support for human review only; it must not approve care, deny coverage, determine medical necessity, diagnose, or recommend treatment.

Human review is represented as an append-only lifecycle. High-risk prior-authorization requests emit human_review_assigned; reviewers can close the loop with scripts/record_human_review.py, which HMACs the raw reviewer ID with MULTI_LLM_REVIEW_HMAC_KEY, redacts the rationale, appends human_review_completed, and records human_override_recorded for modified, rejected, or escalated outcomes. resolve_trace_state() replays sanitized audit events to derive the terminal trace status.

export MULTI_LLM_REVIEW_HMAC_KEY=local-demo-secret
python scripts/record_human_review.py --trace-id TRACE_ID --reviewer-id reviewer-123 --decision accepted --rationale "Reviewed against supplied documentation."

Reviewer-ready evidence packets can be exported for any trace present in a JSONL audit log:

python scripts/export_audit_packet.py --trace-id TRACE_ID --audit-log audit_logs/dev_audit.jsonl --output-dir .tmp_evidence_packet_check

Each packet folder contains sanitized audit events, chain verification, terminal trace state, governance explanations, evidence coverage, redaction summary, model provenance, human review status, and a reviewer summary. Packet export rebuilds the trace folder on each run and uses the canonical stored trace ID in the packet contents.

Trust boundary note: the local ADK process receives user input, but no model should receive raw PHI/PII by default. The MVP redacts the ADK model request before the Gemini router call and redacts again before third-party provider calls. Raw PHI/PII may still exist in ADK session state/history or platform logs before callback redaction; production requires ingest-time redaction before session write and strict content-logging controls.

MVP redaction is text-only and destructive. That is intentional for the prior-auth demo because the human reviewer retains the source document, but production may require stable pseudonymous tokens and managed inspection for files, images, and attachments.

Current MVP Limitations

The current controls are intentionally deterministic and testable, but they are not production-grade clinical or coverage-decision controls:

• privacy.py uses regular expressions and misses many real PHI forms, including unlabeled names, bare DOBs, international formats, context-free identifiers, homoglyphs, encoded/spaced values, and PII embedded in arbitrary JSON.
• risk.py uses lexical keyword heuristics and can be bypassed by paraphrase or indirect phrasing.
• evidence_coverage.py uses deterministic keyword heuristics. It is useful for reviewer-facing coverage checks, but it is not a payer-policy engine, clinical classifier, or medical-necessity model.
• Provider responses are plain text today. There is no enforced provider response_format, JSON schema, or post-response clinical-boundary validator on model outputs yet.
• Runtime objects such as GovernanceContext and PrivacyAssessment may contain raw PHI during request processing. Safe views and audit persistence prevent those values from being written to durable artifacts, but production needs stricter runtime data-minimization and logging controls.
• The deterministic red-team suite covers 36 MVP control-plane cases across prompt injection, PHI exfiltration, provider egress, autonomy-boundary, fallback, fanout, and hallucinated-evidence scenarios. It is still a fast regression suite, not proof of broad adversarial robustness.
• The privacy benchmark gates only identifier types the current regex redactor is expected to catch: email, formatted phone, SSN, and member ID. Bare-name and bare-date recall are reported as known limitations, not CI blockers.
• The prior-auth structured invariance regression uses synthetic demographic variants and compares evidence-coverage statuses, human-review requirement, and decision boundaries. It is not a production fairness validation.
• Evidence packet redaction totals are finding observations summed across audit events, not deduplicated per-request entity counts. The packet includes source_event_count and counting_strategy to make this explicit.
• The scorecard's sample_phi_regression_guard_passed field is a regression check over known synthetic sample values in committed artifacts, not a general PHI detector. General PHI safety still depends on sanitize-on-write, safe views, and targeted privacy tests.

PHI-Safe Audit Chain

Audit events are sanitized before persistence and before hashing. The persistence allowlist favors structured fields such as provider, action, risk tier, reason codes, policy IDs, model provenance, redaction summaries, token counts, evidence coverage reports, hashed redacted excerpts, and safe error categories. Generic free-text fields such as raw reasons, names, arbitrary values, prompts, responses, excerpts, and reviewer IDs are not persisted.

The default local sink writes JSONL to audit_logs/dev_audit.jsonl; tests can swap in an in-memory adapter through the audit facade. Verify a local log with:

python scripts/verify_audit_chain.py audit_logs/dev_audit.jsonl

The JSONL hash chain is integrity-verifiable, not tamper-proof. It detects accidental edits, event reordering, middle deletion, and insertion when the verifier checks the chain links and hashes. A process-local thread lock protects demo appends; multi-process writers need a file lock, database, append-only object store, or external audit service. Appending reloads and verifies the existing chain, which is simple and defensible for this MVP but O(n) per write. A partial final-line write from a crash requires manual log rotation or recovery before appends resume. A user with filesystem write access could still rewrite the whole file and recompute hashes unless production storage adds protected HMAC/signing keys, immutable storage, object-store versioning, append-only controls, or external digest anchoring.

If raw PHI is accidentally persisted, rotate the affected log, preserve any restricted incident copy only as policy requires, write a documented scrub event to a new log, and accept the chain break rather than silently rewriting history.

Evidence

High-signal artifacts:

• Architecture
• Observability
• ADR: governed prior-authorization slice
• Model risk tiering
• System card
• AI impact assessment
• Standards crosswalk
• EU AI Act applicability reasoning
• FDA SaMD boundary reasoning
• Governance operating model
• Board risk report
• Residual risk register
• Prior-auth walkthrough
• Evidence coverage sample
• Governance scorecard
• Red-team report
• Privacy redaction benchmark report
• Prior-auth invariance report

Deterministic Governance Evals

Run the focused MVP red-team suite:

python scripts/run_redteam_eval.py

It checks 36 deterministic cases across direct and indirect prompt injection, PHI exfiltration, redaction evasion, autonomy-boundary pressure, medical-necessity and coverage-decision pressure, system prompt leakage attempts, provider egress policy, fanout abuse, fallback safety, and hallucinated evidence. It does not call external LLM providers.

Run the privacy benchmark:

python evals/privacy/run_redaction_benchmark.py

The benchmark reports precision, recall, F1, false positives, false negatives, and metrics by identifier type. Fast gates currently apply only to email, formatted phone, SSN, and member ID recall. Bare names and bare dates are tracked as non-gated limitations.

Run the prior-auth structured invariance regression:

python evals/fairness/run_invariance_eval.py

The invariance regression uses synthetic demographic and identity variants of the same prior-auth packet. It compares structured evidence-coverage fields (requirement_id, status), human-review requirement, and prohibited decision boundaries rather than free-text rationales.

Run unit tests:

python -m unittest discover -s tests

Run the fast local governance evidence path:

python -m unittest discover -s tests
python scripts/run_redteam_eval.py
python evals/privacy/run_redaction_benchmark.py
python evals/fairness/run_invariance_eval.py
python scripts/verify_audit_chain.py examples/audit/sample_audit_chain.jsonl
python scripts/export_audit_packet.py --trace-id sample-trace-001 --audit-log examples/audit/sample_audit_chain.jsonl --output-dir .tmp_evidence_packet_check
python scripts/generate_governance_scorecard.py

Generate the governance scorecard from local reports:

python scripts/generate_governance_scorecard.py

GitHub Actions runs repository governance checks on pull requests and manual dispatches. The workflow runs unit tests, discrete human-review and observability test modules, the deterministic red-team eval, the privacy benchmark, the invariance eval, sample audit-chain verification, and scorecard generation. It uploads the red-team report, privacy benchmark report, invariance report, scorecard, and evidence packet folder as artifacts. Cloud Build remains the GCP deployment and deployment-evidence workflow; GitHub Actions does not replace it.

Optional Git Hooks

This repo includes opt-in hooks for local guardrails:

git config core.hooksPath .githooks

The pre-commit hook blocks generated audit logs by path and by parsing staged JSON/JSONL for audit.v1 stored events. It also requires at least one core reviewer-facing doc to be staged whenever governance or implementation files are staged: README.md, docs/architecture.md, examples/prior_authorization/governance_walkthrough.md, governance/system_card.md, governance/model_risk_tiering.md, or governance/ai_impact_assessment.md. Test-only commits do not trigger this documentation gate. The hook then runs compile checks. The pre-push hook runs unit tests and the deterministic red-team eval. Run the privacy and invariance evals locally before reviewer handoff when changing privacy.py, evidence_coverage.py, risk.py, policy.py, or eval fixtures. Set PYTHON=/path/to/python before invoking Git if your shell does not expose python on PATH. These hooks are useful local tripwires, but CI should remain the source of record for required governance checks.

Deployment Profile

The repo is designed for Google ADK and a managed GCP agent runtime, but product names are intentionally contained in deployment/gcp/ because Google Cloud agent services are evolving.

MVP deployment artifacts:

• cloudbuild.yaml
• deployment/gcp/managed_agent_runtime_profile.md
• deployment/gcp/cloud_build_pipeline.md
• deployment/gcp/agent_inventory_manifest.yaml
• deployment/gcp/safety_screening_policy.yaml

Cloud Build is validation-only in the MVP. Live deployment, agent inventory registration, governed capability registration, and managed safety-screening integration are expansion items until implemented for real.

Repository governance CI lives in .github/workflows/governance-ci.yml. It is scoped to fast deterministic checks and evidence artifact generation. Cloud Build remains the GCP deployment path.

Standards Scope

Core MVP mappings focus on:

• NIST AI RMF
• ISO/IEC 42001
• SOC 2 Type 2
• HIPAA
• OWASP Top 10 for LLM Applications
• EU AI Act applicability reasoning
• FDA SaMD / GMLP boundary reasoning
• FHIR provenance and interoperability considerations

CMMC, NIST SP 800-171, and FedRAMP are explicitly out of scope unless a federal/CUI scenario is added.

Local Setup

pip install -r multi_model_agent/requirements.txt
cp multi_model_agent/.env.example multi_model_agent/.env
adk web

Set real provider keys only in local secrets or a managed secret store. Do not commit .env. The default model IDs are also shown in multi_model_agent/.env.example and can be overridden with environment variables.