perf(vectors): adopt cocoindex native cross-component batching (#2219) when available; revert single-component loop

[HumanBean17]

Context — why this is a follow-up, not a rewrite

The vectors phase used to spend ~91s of kernel I/O on a 1167-file repo because coco.mount_each creates one Lance merge_insert transaction per file (1168 fragments, 1170 manifest commits). cocoindex flushes target writes once per processing component, and mount_each makes one component per source item. cocoindex does not yet batch target writes across mount_each components — this is upstream issue cocoindex#2219 ("Support batching for target state sinks from different components", filed 2026-06-25), which is literally this problem.

Current approach (this branch)

app_main now iterates the source in a single component (async for … in walk_dir().items() instead of coco.mount_each), so all declare_row calls flush as one merge_insert per table. process_*_file stay @coco.fn(memo=True), and _RowHandler.reconcile skips rows whose fingerprint is unchanged.

Measured on Shopizer (1167 files / ~3500 chunks):

metric	mount_each (before)	single-component (after)
vectors wall	119.7s	40.4s (CPU) / 37.5s (MPS)
kernel I/O (sys)	91.15s	4.2s
Lance fragments	1168	1
no-change increment	—	2.4s, 0 new fragments
1-file-change increment	—	10.2s, +1 fragment

Correctness holds: random uuid4() ids are only generated on a memo miss (changed file); on a memo hit coco replays the recorded target state (same ids, same content) and reconcile skips. No need to leave cocoindex or unify on FileHashTracker for vectors.

This issue — the residual cost + the upstream unlock

The single-component loop has one residual cost: on every increment it re-iterates all files so coco can fingerprint each for memo/reconcile (~3s fixed on 1167 files; grows linearly with repo size). mount_each avoids the re-visit (per-file source memo skips unchanged files entirely) — but mount_each is exactly what caused the per-file transaction storm.

When cocoindex#2219 ships and we adopt a compatible cocoindex version:

1. Revert app_main to mount_each (restores per-file source-memo granularity → unchanged files skipped entirely on increment, no re-visit, no re-fingerprint).
2. Enable native cross-component sink batching → few merge_insert transactions without sacrificing per-file memo granularity.

That gets the best of both: fast bulk init AND proportional, no-revisit increment. Until then, the single-component loop is the supported workaround and the init/reprocess win is already captured.

Related upstream: cocoindex#2220 (schema-aware LanceDB row fingerprinting) — would let reconcile skip unchanged rows even cheaper.

References

• cocoindex#2219 — [FEATURE]: Support batching for target state sinks from different components cocoindex-io/cocoindex#2219
• cocoindex#2220 — [FEATURE]: Add fingerprinting checks to prevent unnecessary recomputation for schema updates in LanceDB connector cocoindex-io/cocoindex#2220
• cocoindex target-state guide — https://cocoindex.io/docs/programming_guide/target_state/