Skip to content

Preflight disk headroom per node; raise cleanly when it won't fit#182

Merged
bscholer merged 1 commit into
mainfrom
worktree-storage-headroom
May 15, 2026
Merged

Preflight disk headroom per node; raise cleanly when it won't fit#182
bscholer merged 1 commit into
mainfrom
worktree-storage-headroom

Conversation

@bscholer

Copy link
Copy Markdown
Owner

Summary

Closes the two failure modes the periodic auto-sweep (#178) couldn't address on its own:

  1. cache_max_bytes set to the whole disk. The 30s sweep finds nothing to evict (cache is under budget), but the next op still ENOSPCs because actual `disk_free` is way smaller than the budget headroom suggests. Veil Nebula at 1.5TB was this case.
  2. The op genuinely needs more bytes than we can possibly free. Should surface a clean "needs N GB, only M GB available" error before the job starts, not a half-stacked failure deep into Siril.

Shape

  • `Node.estimate_storage_bytes(inputs, params)` is a new opt-in method on the base class (`None` by default). Bulk-tier sequence nodes (`calibrate`, `convert_lights`, `seq_bg_extract`, `seq_offset`, `seq_register`, `seq_resample`) implement it using Siril's own size formula from `src/io/sequence.c`: `frame_bytes = rxrynb_layers*bps + FITS_HDU; total = frame_bytes * nb_frames`. We approximate by scaling the input directory's on-disk bytes for bit-depth (uint16→float32 is 2x) and spatial scale (drizzle 2x is 4x, resample 0.5 is 0.25x). One sample FITS header read tells us BITPIX.

  • `ensure_disk_headroom(cache, need_bytes, cache_max_bytes, db_path)` enforces `headroom = min(disk_free, cache_max_bytes - cache_used) >= need`. Both constraints are binding — whole-disk-cache case falls out naturally because `cache_max - used` becomes huge and `disk_free` binds.

  • Runtime calls the estimator right before `reserve()`, sweeps aggressively if needed, emits `node_failed` with a clean message on raise. Also logs `estimate=X actual=Y ratio=R` on every successful bulk node so you can validate the multipliers on real datasets and tune from data rather than guessing.

  • `JobWorker` passes `cache_max_bytes` + `db_path` into `run_job` so the headroom path activates. Tests that bypass JobWorker get the legacy no-op behavior.

Drizzle is handled

`seq_register` passes `scale=drizzle_scale` (1, 2, or 3) when drizzle is on. A drizzle-2 register against a 100GB input estimates 400GB (×4 area), so the preflight catches that before it ENOSPCs mid-job.

Test plan

  • 10 new tests in `tests/test_storage_headroom.py` covering estimator (frame count, bit-depth, scale), headroom helper (noop / sweep / raise / budget-binds vs disk-binds)
  • Full suite: 678 passed, 1 skipped
  • Ruff clean
  • After deploy: run a real Veil Nebula register on the Linux box, watch the logs for the estimate vs actual ratio. Tune the bit-depth multiplier if the ratio is consistently off.

Two failure modes the auto-sweep couldn't address on its own:
  1. cache_max_bytes set to the whole disk: the 30s sweep finds nothing
     to evict (cache is under budget), but the next op still ENOSPCs
     because actual disk_free is way smaller than the budget headroom.
  2. The op genuinely needs more bytes than we can possibly free.

Per-node estimator + storage helper close the loop:

  - Node.estimate_storage_bytes(inputs, params) is a new opt-in method
    on the base class (returns None by default). Bulk-tier sequence
    nodes (calibrate, convert_lights, seq_bg_extract, seq_offset,
    seq_register, seq_resample) implement it using Siril's own formula
    (frame_bytes = rx * ry * nb_layers * bps + FITS_HDU, total = per_frame
    * nb_frames). We approximate that by scaling the input directory's
    on-disk bytes for bit-depth (uint16 -> float32 is 2x) and spatial
    scale (drizzle 2x is 4x, resample 0.5 is 0.25x). One sample FITS
    header read tells us BITPIX.

  - ensure_disk_headroom(cache, need_bytes, cache_max_bytes, db_path)
    computes headroom = min(disk_free, cache_max_bytes - cache_used)
    and sweeps aggressively if headroom < need. Raises
    InsufficientStorageError when even an empty cache wouldn't be
    enough. Both binding constraints (physical disk and cache budget)
    are enforced; whole-disk-cache case falls out for free.

  - server/runtime.py calls the estimator right before reserve(),
    invokes the headroom helper when the worker passed cache_max_bytes,
    and emits node_failed with a human-readable message on raise. It
    also logs `estimate=X actual=Y ratio=R` on every successful node so
    the multiplier can be tuned against real datasets.

  - JobWorker looks up cache_max_bytes once per job and passes it +
    db_path into run_job. Tests that bypass JobWorker get the legacy
    no-op behavior, so existing test coverage isn't disrupted.

10 new tests in tests/test_storage_headroom.py cover the estimator
(frame-count, bit-depth, scale), the headroom helper (noop, sweep,
raise, budget-vs-disk), and the integration shape.
@bscholer
bscholer merged commit 002bbca into main May 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant