[Phase 6] S1 GRD RTC productionization — integration tracker (DRAFT)

[lhoupert]

Living draft PR tracking the Phase-6 integration branch feat--s1_grd_phase6. Tracker issue: #226. Spec/plan: claude-docs/{specs,plans}/s1_grd_phase6_productionization.md (#224).

Note

Stacked on Phase 5 (#186). Base is feat--s1_grd_phase5 so the diff shows only the Phase-6 delta (the 4 merged PRs below), not the phase-5 changes. When #186 merges to main, GitHub retargets this PR to main and the diff stays phase-6-only. Not for merge yet — it's a progress view.

Goal

Automated, multi-tile, in-cluster S1 GRD RTC on staging: a data-driven trigger discovers new CDSE S1A/S1C products, auto-provisions the DEM per tile, runs s1tiling → ingest → quality-gate → register, storing each tile as a multi-temporal datacube (sentinel-1-grd-rtc-staging) with per-acquisition STAC items (sentinel-1-grd-rtc-acquisitions). Explorer/TiTiler rendering is deferred → #228; validation is via the validate_s1_grd_rtc notebook + STAC.

Consolidated in this branch (data-pipeline code) — 381 unit tests green

PR	Task	What
#224	—	spec + plan
#229	T1 + T5	quality-gate CLI validate_s1_rtc.py (tile-agnostic, verified on 31TDH) + per-acquisition register register_per_acquisition.py
#230	T3	ensure_dem.py — auto-fetch GLO-30 DEM per tile swath (convention ground-truthed vs real eotile gpkg)
#231	T4	append scenes to the per-tile cube, idempotently (ingest_v1_s1_rtc.py)

Task status

• T1 quality-gate CLI — code merged (feat(s1-phase6): quality-gate CLI + per-acquisition register (T1, T5) #229); Argo quality-gate step pending
• T2 platform render S1A+S1C + off-platform-download tolerance — platform-deploy
• T3 ensure-dem auto-fetch — code merged (feat(s1-phase6): ensure_dem.py — auto-fetch GLO-30 DEM per tile swath (T3) #230); Argo ensure-dem step + cluster run pending
• T4 datacube append + idempotency — code merged (feat(s1-phase6): append scenes to the per-tile cube, idempotently (T4) #231); per-tile write mutex (Argo) + cluster multi-time run pending
• T5 per-acquisition catalogue — code merged (feat(s1-phase6): quality-gate CLI + per-acquisition register (T1, T5) #229); multi-time-cube notebook check (plan P-3) pending
• T6 CDSE watcher → Argo CronWorkflow trigger — platform-deploy; blocked by 1–5 + OQ
• T7 multi-tile AOI iteration — blocked (OQ-1 AOI tiles)
• T8 bounded backfill — blocked (OQ-2 window)
• T9 acceptance soak (5×14×≥95% staging) — blocked (OQ-1, OQ-3)
• D1 deferred-rendering tracking issue → S1 GRD RTC — titiler rendering support for per-acquisition items (I2 / option 2) #228

Remaining work (not in this repo)

• platform-deploy Argo manifests: ensure-dem step, quality-gate step, per-tile synchronization mutex (T4), CronWorkflow trigger (T6); T2 platform render.
• Cluster runs: live new-tile DEM provision, multi-time S3 cube, end-to-end CP-A.
• Open questions (Loïc): OQ-1 AOI/soak tiles · OQ-2 backfill window · OQ-3 alerting path.

Checkpoints

• CP-A (after T1–T5): one product on any tile → gated → cube slice + per-acquisition item → notebook PASS. Code-side unblocked; CP-A-local runnable now (needs Docker + CDSE + ~2 GB DEM).
• CP-B (after T8): automated multi-tile trigger + backfill.
• CP-C = T9 soak.

🤖 Generated with Claude Code

[lhoupert]

lb10 6-way region concurrency test (overnight) + 31TBH issue

Test

Manual region run off the Pyrenees cron at lookback_days=10, parallelism=6, v1-s1rtc-limit=6 (s1rtc-test-lb10-6way-4jghc).

• Discover found 24 products across the 14-tile Pyrenees AOI (both orbits) and fanned out 24 product-chains (s1tiling → ingest).
• 6-way parallelism confirmed: peak running_children=6 (max_seen=6); total wall time ~5h55m (s1tiling is the long pole, ~23m–1h per product).
• Result: 47/48 steps ✓ — 23 of 24 products fully ingested. One s1tiling child failed (tile 31TBH), which marks the parent workflow Failed even though continueOn correctly skipped only that branch's ingest.

Issue found: AOI contained a non-existent S2 tile (31TBH)

Reproduced 31TBH alone and captured the s1tiling log live:

tile_to_product_overlap_ratio : 0%
WARNING - Tile '31TBH' does not exist, skipping ...
CRITICAL - No existing tiles found, exiting...   → exit 73

Root cause: gen_aoi_tiles builds the list from mgrs.toMGRS() (pure math) and filtered only by DEM/land coverage. Land coverage ≠ S2-granule existence. 31TBH (Pyrenees foothills, lon −0.70→0.57) straddles the 0° UTM zone-30/31 boundary; its ground is served by the zone-30 tile 30TYN (in the AOI, processed fine), so ESA's S2 grid omits the redundant zone-31 square. It's land, so the DEM filter kept it — only the authoritative S2 grid knows it isn't a real granule. Confirmed absent from both s1tiling's MGRS.gpkg and eotile's s2_no_overlap.gpkg (56,686 tiles); the other 13 Pyrenees tiles are all valid.

A single such tile fails the whole cron run, so this isn't 31TBH-specific — any math-valid-but-non-existent tile in an AOI would do the same.

Fix

• fix(s1-rtc): validate AOI against the S2 grid (drop 31TBH) + extend Pyrenees one row north #268 — gen_aoi_tiles now validates every generated tile against the eotile S2 grid (read_s2_tile_ids / resolve_s2_gpkg; tiles_for_region requires the valid-S2 set). Regenerating pyrenees yields exactly 13 tiles (31TBH removed), byte-identical to the corrected file. New adversarial test; 467 unit tests pass. Self-defending for the planned ~1000-tile redesign.
• platform-deploy#254 — drops 31TBH from the deployed AOI ConfigMap (14 → 13).

Net

Pipeline + 6-way concurrency validated at region scale; the only failure was a bad AOI entry, now fixed at the generator. Still open before un-suspend: confirm a clean full-AOI run, the v0.4.0 ingest-template pin, and parallelism 6-vs-3 for steady state.

[lhoupert]

Datetime-rendering test plan — one tile (31TEH) first, then scale

Both halves are merged into feat--s1_grd_phase6: CF-time ingest (data-model #180/#193, pin 16c5f14) + datetime-sel register (#271). Combined image: sha-d746b39. We start fresh (no legacy-cube migration; the cron stays suspended).

Phase A — validate 31TEH end-to-end

1. Wipe s1-rtc-31TEH.zarr + its STAC items (s1-rtc-31TEH-* in …-acquisitions-staging, s1-rtc-31TEH in …-grd-rtc-staging).
2. Re-ingest 31TEH with sha-d746b39 (full s1tiling → ingest → register), image overridden per-run.
3. Validate:
   • cube time decodes to datetime64 at every multiscale level (CF attrs present);
   • per-acquisition items carry sel=time={datetime} render links (not a positional index);
   • titiler renders each item 200 + correct timestamp — in particular the previously-400 descending 20260607t055223 slice; a non-monotonic (later-then-earlier) order still renders correctly.

Phase B — scale

On a clean 31TEH: deploy sha-d746b39 to the ingest template pipeline_image_version + cron trigger_image_version, wipe + re-ingest the remaining AOI tiles, then un-suspend the cron.

Why this is safe now: appending the CF image to a legacy cube KeyErrors on the missing coarse-level time, so we deliberately rebuild fresh rather than append/migrate. Tracking: data-model#192.

[lhoupert]

S1 RTC datetime-rendering — validated + provisional tag; titiler dependency + un-suspend gate.

Validation: ingested real GeoTIFFs into a never-cached store (valid31teh, image sha-d746b39 = #271 PR-merge build) → fresh CF cube rendered 10/10 stable HTTP 200 by exact datetime. titiler (0.9.1 / xarray 2026.4.0) CF-decodes time; exact .sel returns the right slice on a non-monotonic axis. 31TEH's earlier flapping was 100% a poisoned titiler cache, not the code.

Tag: cut provisional v0.6.0-s1rtc on feat--s1_grd_phase6@32a173f (the #271 merge commit; tag build → :v0.6.0-s1rtc on OVH). Provisional — pins eopf-geozarr 16c5f14, a squash commit on the unmerged data-model #180 branch (GC-eligible if #180 squash-merges). Final release gated on #180 landing.

Titiler dependency / un-suspend gate: deploying the tag + un-suspending the cron is gated on EOPF-Explorer/titiler-eopf#118? the datatree cache staleifies on every cube append (24h TTL_DATASETS, ~40 replicas, no reliable invalidation) so the newest acquisition's preview 500s/400s until TTL. Un-suspend exit criterion: after an append, the newest acquisition renders 200 within minutes across repeated requests (no flapping). Never-rendered tiles are unaffected; re-ingesting a previously-rendered tile needs the titiler cache handled.

[lhoupert]

Design proposal for the titiler blocker posted → EOPF-Explorer/titiler-eopf#118 (comment-4732177066).

Two fixes specified against titiler v0.9.1 reader.py, implementation-ready for the titiler team:

1. Stale-on-append datatree cache — version the redis cache key with the store's consolidated-metadata ETag (open_dataset keys on src_path only today). Verified these S1 RTC cubes are zarr v3 with consolidated metadata at root, so an append rewrites root zarr.json and bumps its ETag → new key → fresh read; stale keys self-expire; replica-independent; no admin invalidation. Also flagged a likely env-name mismatch (TITILER_EOPF_CACHE_TTL_DATASETS vs the metadata_ttl field actually read → deployed dataset TTL may be the 300s default).
2. sel int() 500 — datetime-aware cast + degrade to 422 on a stale int64 time axis.

Includes a test plan reusing the existing tests/test_reader.py / tests/test_eopf_cache_integration.py harness. Un-suspend remains gated on the titiler team landing these.

[lhoupert]

Superseded by #186, which now contains the full Phase 5 + Phase 6 diff against main in a single review surface (the head branch was fast-forwarded to this Phase-6 tip). Closing this progress draft — the review history here stays for reference. Continue review on #186.