Skip to content

Remove database runtime implementations from core#199

Open
vaibhav-i wants to merge 1 commit into
codex/arena-separation-trusted-fanout-workflowfrom
codex/arena-separation-core-database-runtime
Open

Remove database runtime implementations from core#199
vaibhav-i wants to merge 1 commit into
codex/arena-separation-trusted-fanout-workflowfrom
codex/arena-separation-core-database-runtime

Conversation

@vaibhav-i

@vaibhav-i vaibhav-i commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • remove built-in SQL/Mongo verification and database reset implementations from ax-eval core
  • require explicit arena-owned providers for database verification and cleanup, and remove mongodb, mysql2, pg, and @types/pg from the root package
  • delegate both legacy DAEB runtime aliases through the explicit arena compatibility map to the trusted-workflow fail-closed boundary
  • retain generic HTTP/Asana reset compatibility while unsupported reset/reclaim fails at the provider boundary before resolving obsolete credentials, templates, or scope
  • harden import/package guards and preserve v1 SQL/Mongo declarations as provider inputs only

Verification

  • Node 22.23.1
  • npm test: 576 core tests and 208 arena tests
  • npm run typecheck
  • npm run build
  • npm run pack:check (34 core files, 86 arena files)
  • core and arena npm --cache .npm-cache pack --dry-run
  • git diff --check

All tests are offline and keyless. No live or credentialed evaluation was run.

Independent audit

The independent reviewer found one P2 contract bug: unsupported database reset eagerly resolved core credentials before reporting the missing provider. Client/scope resolution is now lazy, and auth-required ${MISSING_*} regressions cover both direct reset and exec-plan --reclaim.

The final exact-commit audit of 446df3bd2bf73d27f430f4c32051853a4e7fcf28 was clean.

Stack

Base: #198 trusted workflow cell fan-out.

Rollback: revert this PR to restore the built-in core database runtime while leaving the explicit arena providers intact.

Next slice: remove the remaining DAEB execution loops and policy from core.

@github-actions

Copy link
Copy Markdown

Normalized records diff

Overall is a macro-average of participating surface scores. Agents are never averaged together.

Overall by agent

Product Agent Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Surfaces
fixture-db claude-code 92.9% 92.9% +0.0 pp 78.6% (11/14) 78.6% (11/14) 2
fixture-db codex 78.6% 78.6% +0.0 pp 64.3% (9/14) 64.3% (9/14) 2

Surface cells

Product Agent Surface Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Latency ms Cost USD Tokens Harness version
fixture-db claude-code api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1100 → 1100 0.4200 → 0.4200 3350 → 3350 2.1.5 → 2.1.5
fixture-db claude-code cli 100.0% 100.0% +0.0 pp 85.7% (6/7) 85.7% (6/7) 1300 → 1300 0.5100 → 0.5100 3720 → 3720 2.1.5 → 2.1.5
fixture-db codex api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1200 → 1200 — → — 3600 → 3600 0.121.0 → 0.121.0
fixture-db codex cli 71.4% 71.4% +0.0 pp 57.1% (4/7) 57.1% (4/7) 1500 → 1500 — → — 4100 → 4100 0.121.0 → 0.121.0

Operational metrics are context only and never affect ranking.

@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-trusted-fanout-workflow branch from a117f2f to f38ede6 Compare July 22, 2026 15:00
@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-core-database-runtime branch from 95f3f83 to e5f87a8 Compare July 22, 2026 15:00
@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-trusted-fanout-workflow branch from f38ede6 to d3a1e54 Compare July 23, 2026 00:20
@vaibhav-i
vaibhav-i force-pushed the codex/arena-separation-core-database-runtime branch from e5f87a8 to 446df3b Compare July 23, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant