Skip to content

Add worker-backed source scanning#26

Merged
V3RON merged 4 commits into
masterfrom
feat/workers-source-scanner
Jul 1, 2026
Merged

Add worker-backed source scanning#26
V3RON merged 4 commits into
masterfrom
feat/workers-source-scanner

Conversation

@V3RON

@V3RON V3RON commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

What is this?

This PR adds worker-backed source scanning for cache misses. Sniffler now keeps cache lookup, diagnostics, and graph construction on the main thread, but can parallelize file reads, import/export parsing, content hashing, and metadata collection when the filesystem supports worker scanning.

It also adds a top-level workers config option with auto as the default. workers: 0 keeps the current serial path, while any positive value enables worker scanning when Sniffler chooses that path.

How does it work?

selectImpact now walks source files in order, reuses cached entries for hits, and collects only misses for scanning. Miss results are merged back into the original source-file order before graph build so output stays deterministic.

Scanner selection now goes through a dedicated source-scanner abstraction:

  • serial scanner for the existing main-thread path
  • worker scanner backed by a per-run worker pool for cache misses only
  • automatic selection that stays serial for small miss counts and uses workers on Node FS when the miss set is large enough

The new FileSystem.supportsWorkerScanning flag makes the filesystem contract explicit. Node FS reports true; memory and fallback/custom FS implementations report false. If a worker scan is requested against an unsupported filesystem, Sniffler throws a clear error.

Cache config hashing ignores workers, so changing parallelism does not invalidate semantic cache data. Diagnostics now record scanner mode, worker count, job count, and worker failure count.

Why is this useful?

  • Large cold runs spend less time in source scanning by parallelizing only the expensive miss path.
  • Cache behavior stays stable because workers changes execution strategy, not analysis output.
  • Deterministic ordering is preserved, so graph and impact results remain predictable.
  • The filesystem contract is clearer, which makes worker support explicit instead of implicit.
  • Diagnostics get better signal for scan-path performance and failures.

Benchmark Notes

On Expensify cold runs, workers: auto reduced source scan time by about 52% and total run time by about 33% versus workers: 0.

V3RON added 3 commits July 1, 2026 10:04
Document scoped implementation tasks for impact workflow, cache lifecycle, test recommendation, graph edge semantics, and CLI execution refactors.
Scan cache misses in worker pool when FS supports it, keep cache and graph main-thread-owned, and document/configure workers.
Drop tracked task markdown files from branch contents.
@V3RON V3RON changed the title feat: add worker-backed source scanning Add worker-backed source scanning Jul 1, 2026
…-scanner

# Conflicts:
#	src/impact/impact-command.ts
@V3RON V3RON merged commit 4acc529 into master Jul 1, 2026
6 checks passed
@V3RON V3RON deleted the feat/workers-source-scanner branch July 1, 2026 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant