feat(embedding): add strategy-based fallback engine with provenance

[dubscode]

Summary

Implements embedding parity hardening with an explicit strategy engine, controlled Anthropic fallback policy, and provenance plumbing across indexing/retrieval.

This PR also archives the OpenSpec change and syncs its capability specs into main OpenSpec specs.

What changed

• Added embedding strategy V2 foundation:
  • typed strategy config schema (version: 1.0)
  • deterministic strategy resolution by strategy id
  • startup validation with structured config errors (unknown/missing/cyclic/default issues)
  • backward-compatible legacy default mapping
• Added Anthropic native-first fallback behavior:
  • fallback only when failure category is explicitly eligible
  • fallback order enforced from config
  • terminal failure behavior when fallback is disallowed/exhausted/absent
• Added normalized embedding provenance envelope:
  • strategy id, attempt path, resolved provider/model, fallback state
  • failure category + terminal reason on failure
• Wired provenance into runtime data flows:
  • persisted provenance JSON in chunk_embeddings
  • exposed embedding provenance metadata in retrieval results
  • optional provenance log emission via env flag
• Improved DB migration handling:
  • migration runner now executes all src/db/migrations/*.sql in sorted order
  • added 0002_embedding_provenance.sql
• Added test coverage:
  • strategy config/resolution conformance
  • Anthropic policy behavior
  • provenance completeness for success/failure
• Added rollout/rollback docs:
  • docs/embedding-strategy-rollout.md
  • README env var updates
• OpenSpec updates:
  • archived embedding-parity-hardening
  • created synced specs:
    • openspec/specs/embedding-strategy-configuration/spec.md
    • openspec/specs/anthropic-embedding-fallback-and-provenance/spec.md

Feature gate

Embedding strategy engine is gated by:

• DUBSBOT_EMBEDDING_STRATEGY_V2=1

Optional:

• DUBSBOT_EMBEDDING_STRATEGY_CONFIG_JSON
• DUBSBOT_EMBEDDING_PROVENANCE_LOG=1

Validation

• pnpm checks passed locally:
  • pnpm test
  • pnpm typecheck
  • pnpm lint
  • pnpm build

Notes

• This is stacked on top of PR feat(retrieval): add retrieval quality proofing workflow and archive change #6 (codex/feat/retrieval-quality-proofing).
• Anthropic adapter currently remains deterministic/local for embeddings in this codebase; policy/provenance logic is still exercised and tested through strategy execution behavior.

---

🥞 DubStack

• feat(embedding): add strategy-based fallback engine with provenance #7 feat(embedding): add strategy-based fallback engine with provenance 👈

[github-actions]

PR Checks Summary

• Status: passed
• Command: pnpm checks
• Workflow run: https://github.com/wiseiodev/dubsbot/actions/runs/22654382321
• Commit: df2b6a1

[Copilot]

Pull request overview

This PR introduces a strategy-based, configurable embedding fallback engine with full provenance tracking for the embedding pipeline. It adds explicit strategy selection (primary provider/model + ordered fallback list), startup validation, a provenance envelope persisted to the database, and a feature-flag-gated rollout path. The system is backward-compatible, defaulting to the legacy embedding behavior unless DUBSBOT_EMBEDDING_STRATEGY_V2=1 is set.

Changes:

• New src/context/embedding/ module: strategy.ts (schema/validation), engine.ts (execution with fallback + provenance), and config.ts (loading + legacy default mapping).
• Refactored src/db/migrate.ts from single-file to glob-based ordered migration runner; added src/db/migrations/0002_embedding_provenance.sql to add provenance JSONB column to chunk_embeddings.
• Integration into full-index.ts (indexing path), hybrid.ts (retrieval metadata), daemon/main.ts, and cli/runtime.ts; new conformance tests in tests/embedding-strategy.test.ts; rollout documentation.

Reviewed changes

Copilot reviewed 15 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/context/embedding/strategy.ts	Core schema (Zod), validation, cycle detection, and typed error classes
src/context/embedding/engine.ts	BFS fallback execution loop with provenance tracking
src/context/embedding/config.ts	Config loading from env, legacy default strategy construction
src/context/indexer/full-index.ts	Strategy V2 path for embedding during full indexing
src/context/retrieval/hybrid.ts	Exposes provenance fields from DB in retrieval results
src/db/migrations/0002_embedding_provenance.sql	Adds provenance JSONB column to chunk_embeddings
src/db/migrate.ts	Refactors migration runner from hardcoded single file to glob-ordered multi-file
src/daemon/main.ts	Calls loadEmbeddingStrategyConfig() at startup for early validation
src/cli/runtime.ts	Loads and exposes embeddingStrategyConfig from the runtime
tests/embedding-strategy.test.ts	Conformance tests for config, fallback policy, and provenance completeness
docs/embedding-strategy-rollout.md	Rollout and rollback procedure documentation
README.md	Documents new env vars and rollout guide link
openspec/specs/*/spec.md (×2)	Spec files for new capabilities (with placeholder Purpose sections)
openspec/changes/archive/*/	Archived change proposal, design, tasks documents

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.