Indexing worker can freeze the whole OS on binary-heavy folders, and always dies with a 0-byte log — needs resource guards and early log flush

[junk151516]

Environment: v0.9.0, Windows amd64 (ui variant), 32GB RAM. mem.init prints budget_mb=11429.

Incident: index_repository --mode moderate over a 29GB non-git folder (~65k files: CCTV footage frames, a 535MB .mp4, .zip archives, YOLO .pt/.onnx weights, plus ~50 real .py files) hard-froze Windows twice (both times requiring a physical power-off, confirmed via Kernel-Power 41 events). The second freeze happened on a fresh boot with nothing else running — the index run was the only workload.

The diagnostic gap: every crashed/killed worker leaves its log at 0 bytes — not a single line is ever flushed (logs/.worker-*.log). I now have 4 separate incidents with this pattern: the two OS freezes above, plus two contained worker crashes in cross-repo mode (reported separately). The budget_mb printed at init evidently didn't constrain the worker before the OS became unresponsive.

Workaround that confirms the cause: adding a .cbmignore that excludes the footage dirs + media extensions makes the exact same folder index fine in seconds (1.3k nodes, worker RSS never visibly elevated).

Asks:

1. Enforce the memory budget in the worker itself — abort with a structured error well before the OS starts thrashing.
2. Skip obvious binary/media files (extension + magic bytes) before reading them into the parser pipeline.
3. Flush a log header (pid, repo, mode) synchronously at worker start, and the current file path per file — so postmortems are possible. Today a dead worker is forensically invisible.

Happy to run any diagnostic build against the same folder.

[DeusData]

Thanks for the careful incident report and for isolating the binary-heavy directories with .cbmignore. A workload that can make the host OS unresponsive is a high-priority 0.9.1-rc stability issue. Please do not rerun the unsafe corpus until resource containment is in place. The RED work should use bounded synthetic binary/media fixtures and separately cover early binary rejection, worker memory-budget enforcement, and a synchronously flushed startup/current-file diagnostic so a killed worker cannot leave a zero-byte log.