Preflight disk headroom per node; raise cleanly when it won't fit#182
Merged
Conversation
Two failure modes the auto-sweep couldn't address on its own:
1. cache_max_bytes set to the whole disk: the 30s sweep finds nothing
to evict (cache is under budget), but the next op still ENOSPCs
because actual disk_free is way smaller than the budget headroom.
2. The op genuinely needs more bytes than we can possibly free.
Per-node estimator + storage helper close the loop:
- Node.estimate_storage_bytes(inputs, params) is a new opt-in method
on the base class (returns None by default). Bulk-tier sequence
nodes (calibrate, convert_lights, seq_bg_extract, seq_offset,
seq_register, seq_resample) implement it using Siril's own formula
(frame_bytes = rx * ry * nb_layers * bps + FITS_HDU, total = per_frame
* nb_frames). We approximate that by scaling the input directory's
on-disk bytes for bit-depth (uint16 -> float32 is 2x) and spatial
scale (drizzle 2x is 4x, resample 0.5 is 0.25x). One sample FITS
header read tells us BITPIX.
- ensure_disk_headroom(cache, need_bytes, cache_max_bytes, db_path)
computes headroom = min(disk_free, cache_max_bytes - cache_used)
and sweeps aggressively if headroom < need. Raises
InsufficientStorageError when even an empty cache wouldn't be
enough. Both binding constraints (physical disk and cache budget)
are enforced; whole-disk-cache case falls out for free.
- server/runtime.py calls the estimator right before reserve(),
invokes the headroom helper when the worker passed cache_max_bytes,
and emits node_failed with a human-readable message on raise. It
also logs `estimate=X actual=Y ratio=R` on every successful node so
the multiplier can be tuned against real datasets.
- JobWorker looks up cache_max_bytes once per job and passes it +
db_path into run_job. Tests that bypass JobWorker get the legacy
no-op behavior, so existing test coverage isn't disrupted.
10 new tests in tests/test_storage_headroom.py cover the estimator
(frame-count, bit-depth, scale), the headroom helper (noop, sweep,
raise, budget-vs-disk), and the integration shape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the two failure modes the periodic auto-sweep (#178) couldn't address on its own:
Shape
`Node.estimate_storage_bytes(inputs, params)` is a new opt-in method on the base class (`None` by default). Bulk-tier sequence nodes (`calibrate`, `convert_lights`, `seq_bg_extract`, `seq_offset`, `seq_register`, `seq_resample`) implement it using Siril's own size formula from `src/io/sequence.c`: `frame_bytes = rxrynb_layers*bps + FITS_HDU; total = frame_bytes * nb_frames`. We approximate by scaling the input directory's on-disk bytes for bit-depth (uint16→float32 is 2x) and spatial scale (drizzle 2x is 4x, resample 0.5 is 0.25x). One sample FITS header read tells us BITPIX.
`ensure_disk_headroom(cache, need_bytes, cache_max_bytes, db_path)` enforces `headroom = min(disk_free, cache_max_bytes - cache_used) >= need`. Both constraints are binding — whole-disk-cache case falls out naturally because `cache_max - used` becomes huge and `disk_free` binds.
Runtime calls the estimator right before `reserve()`, sweeps aggressively if needed, emits `node_failed` with a clean message on raise. Also logs `estimate=X actual=Y ratio=R` on every successful bulk node so you can validate the multipliers on real datasets and tune from data rather than guessing.
`JobWorker` passes `cache_max_bytes` + `db_path` into `run_job` so the headroom path activates. Tests that bypass JobWorker get the legacy no-op behavior.
Drizzle is handled
`seq_register` passes `scale=drizzle_scale` (1, 2, or 3) when drizzle is on. A drizzle-2 register against a 100GB input estimates 400GB (×4 area), so the preflight catches that before it ENOSPCs mid-job.
Test plan