perf(core): hoist ingest_priority lookup out of FilesystemPath insert

[andykais-claude]

Root cause

forager.filesystem.discover() degraded quadratically as the filesystem_path table grew. Each FilesystemPath.create() ran a CASE/subquery to compute the next ingest_priority:

CASE
  WHEN :priority_instruction = 'first'
    THEN COALESCE((SELECT MAX(ingest_priority) FROM filesystem_path WHERE directory = 0), 0) + 1000
  WHEN :priority_instruction = 'last'
    THEN COALESCE((SELECT MIN(ingest_priority) FROM filesystem_path WHERE directory = 0), 0) - 1000
  WHEN :priority_instruction = 'none'
    THEN NULL
  ELSE 1/0
END

The schema does have a partial unique index that should make this O(log n):

CREATE UNIQUE INDEX filesystem_path_priority
  ON filesystem_path (ingest_priority) WHERE directory = 0;

But SQLite's planner picks filesystem_path_directory (the plain index on directory) and does a full scan of every directory=0 row to find the MAX. EXPLAIN QUERY PLAN confirmed it:

SEARCH filesystem_path USING INDEX filesystem_path_directory (directory=?)

So every insert paid an O(n) probe, making bulk discovery O(n²).

Fix

Two layered changes:

1. Model (packages/core/src/models/filesystem_path.ts)

   • Remove the CASE/subquery from the INSERT — #create now binds ingest_priority directly as a parameter.
   • Add select_max_priority() / select_min_priority() helpers that pin the partial unique index via INDEXED BY filesystem_path_priority. With the hint, SQLite uses a covering-index lookup (SEARCH ... USING COVERING INDEX filesystem_path_priority), turning the probe into O(log n).
   • Expose PRIORITY_SPACER as a static so callers can replicate the existing "+ 1000 per row" stride.

2. Actions (packages/core/src/actions/filesystem_actions.ts)

   • Resolve MAX(ingest_priority) exactly once at the start of discover(), then hand out monotonically increasing priorities by simple in-memory increment per newly-created row. Within a single discover() pass we are the only writer, so the partial unique constraint is still preserved.
   • #add_filepath now returns whether it inserted (so the loop only advances the counter on actual creations).

Benchmark

Single discover loop, fresh DB, WAL on, Linux.

rows inserted	before (per-insert)	after (per-insert)
0 → 1k	0.84 ms	1.24 ms
1k → 5k	1.62 ms	1.20 ms
5k → 10k	1.18 ms	1.10 ms
10k → 20k	2.93 ms	0.59 ms
20k → 40k	10.26 ms	1.19 ms
40k → 80k	n/a	1.67 ms

Before: linear per-insert growth → O(n²) total. After: flat per-insert cost → O(n) total. At 40k rows the per-row cost dropped ~8.5x; the gap widens further with table size (the original 40k → 80k batch would have taken ~30+ minutes; the new path finishes in ~67s).

Standalone select_max_priority() benchmark on a populated table runs in ~0.004 ms thanks to:

SEARCH filesystem_path USING COVERING INDEX filesystem_path_priority

Testing

• ✅ deno check packages/core/src/mod.ts
• ✅ deno task --cwd packages/core lint
• ✅ deno test ... --filter 'ingest actions' (exercises the new discover flow end-to-end)
• ✅ Full deno test core suite: 26 tests pass; the 4 pre-existing failures are all duration/ffprobe floating-point precision issues called out in AGENTS.md ("Audio duration: 6.96 vs 6.92", keypoint timestamp precision) and reproduce on main unchanged.

Compatibility notes

• FilesystemPath.create({ priority_instruction: ... }) is gone. Callers pass ingest_priority: number | null directly. The two in-tree callers (#add_filepath for files, #add_directories for directories) are updated.
• No migration is needed — schema and indexes are unchanged. Only the SQL the model emits changed.