Skip to content

Remove the previous file's PAM sidecar when to_geotiff overwrites a path (#3595)#3599

Merged
brendancol merged 8 commits into
mainfrom
deep-sweep-metadata-geotiff-2026-07-01
Jul 2, 2026
Merged

Remove the previous file's PAM sidecar when to_geotiff overwrites a path (#3595)#3599
brendancol merged 8 commits into
mainfrom
deep-sweep-metadata-geotiff-2026-07-01

Conversation

@brendancol

Copy link
Copy Markdown
Contributor

Closes #3595.

  • _write_sidecars now deletes a pre-existing <path>.aux.xml on every successful string-path write before re-creating it, so an overwrite can't hand the old file's category_names / STATISTICS_* to the new pixels. This matches GDAL's GDALDriver::QuietDelete on dataset creation.
  • The .qml style sidecar is left alone unless a new color_ramp= write replaces it: QGIS treats .qml as user styling that persists across data updates.
  • All four write paths (eager, dask streaming, GPU dispatch, VRT) share the sidecar closure, so one removal covers them all. The VRT writer refuses same-path overwrites outright, so its stale case is a foreign sidecar at a fresh path.

Backend coverage: numpy, dask, GPU (nvCOMP), and VRT writes are each exercised in the new tests; the GPU test ran live on a CUDA host.

Test plan:

  • 10 new tests in xrspatial/geotiff/tests/write/test_stale_sidecar_overwrite_3595.py (categorical->plain, ramp->plain keeps .qml, ramp->categorical, categorical->ramp, multiband symbology no-op still removes, foreign sidecar at a fresh path, bare ndarray input, dask, VRT, GPU)
  • geotiff write suite: 1213 passed
  • round-trip + attrs suites: 63 passed
  • rasterize categorical + release gates: 182 passed

Found by the metadata propagation sweep (state CSV update included on this branch).

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: Remove the previous file's PAM sidecar when to_geotiff overwrites a path (#3595)

Blockers (must fix before merge)

None.

Suggestions (should fix, not blocking)

  • xrspatial/geotiff/_writers/eager.py:515-531: the removal only runs at the four success return points, so a failed write leaves the old file and its sidecar consistent with each other. That ordering matters and nothing pins it. A small test (patch the encoder to raise mid-write, assert the old .aux.xml survives) would keep a future refactor from moving the removal before the pixel write.

Nits (optional improvements)

  • xrspatial/geotiff/_writers/eager.py:421-427: the refresh contract is documented under the color_ramp parameter, but it applies to every string-path write, including plain ones with no ramp and no categories. A reader checking why their hand-placed .aux.xml vanished after a plain write won't look there. Consider a Notes section or a sentence near the top of the docstring.
  • xrspatial/geotiff/_writers/eager.py:529: except OSError: pass also swallows PermissionError, so on a locked sidecar (Windows) the write succeeds and the stale sidecar silently stays. That matches GDAL's QuietDelete, so it's a documented-behavior question rather than a bug; a comment word about it would do.

What looks good

  • The removal lives in the shared _write_sidecars closure, so all four write paths (eager, dask streaming, GPU dispatch, VRT) get it from one place, and the isinstance(path, str) guard keeps file-like destinations untouched.
  • Keeping the .qml is the right call: QGIS users expect styles to survive data updates, and the test pins that choice explicitly.
  • Test coverage is thorough for a sidecar fix: both sidecar kinds in both directions, the multiband symbology no-op, foreign sidecars at fresh paths, bare ndarray input, and dask/VRT/GPU paths. The VRT test correctly documents why it can't use a same-path overwrite (tiles-dir guard).

Checklist

  • Algorithm matches reference (GDAL GDALDriver::QuietDelete semantics)
  • All write paths produce consistent sidecar state
  • NaN handling: n/a (no pixel math)
  • Edge cases covered by tests (foreign sidecar, multiband no-op, ndarray, file-like unaffected by guard)
  • Dask path exercised
  • No premature materialization (one os.remove, no data reads)
  • Benchmark not needed (metadata-only path)
  • README feature matrix: no new function, no change needed
  • Docstring updated (see nit on placement)

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review: commit 5fed444

All three findings from the first pass are addressed:

  • Failed-write ordering is now pinned by test_failed_write_keeps_old_sidecar, which monkeypatches the module-level write seam to raise and asserts the old sidecar (and its category_names on re-read) survives.
  • The refresh contract now also appears in the top-level to_geotiff docstring, ahead of the Parameters section, so it's visible for plain writes with no ramp or categories.
  • The except OSError block explains the PermissionError/QuietDelete trade-off inline.

Test file is at 11 passing tests. No new findings; nothing blocking.

@brendancol

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Done. Merged origin/main into the branch. The only conflict was in the color_ramp parameter docstring in eager.py — main had updated the statistics-pass description (streaming dask accumulates from buffers; GPU/VRT re-read for stats), while this branch had added the PAM sidecar refresh note (#3595). The merged text keeps both. Merge commit: af18dfe.

…eotiff-2026-07-01

# Conflicts:
#	xrspatial/geotiff/_writers/eager.py
@brendancol brendancol merged commit d496289 into main Jul 2, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

to_geotiff leaves the previous file's PAM sidecar behind on overwrite

2 participants