Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: Publish docs via GitHub Pages
on:
push:
branches:
- main
- add-dse-architecture-vision-deck
pull_request:
branches:
- main
Expand Down
1 change: 1 addition & 0 deletions docs/dse-architecture-vision/diagrams_svg/07_reference.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
355 changes: 355 additions & 0 deletions docs/dse-architecture-vision/index.html

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions docs/dse-architecture-vision/todos.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
- [ ] add stac, stac-geoparquet to services
- [x] revise roles and responsibilities
- [x] review roadmap
43 changes: 43 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,49 @@ The **◆** designation represents a category of ongoing work.
| **Empowered users** | Cloud-native guidance · Science support · Format evaluation | In-browser rendering · Cloud-optimized decision framework · Improved access & auth libraries · Dataset + tooling coverage metrics | AI-assisted optimization (skills + tooling) · ESRI / ArcGIS integration |
| **Trusted & reliable data** | ◆ Transactional Zarr (Icechunk) | Virtual stores for ongoing datasets · Synchronized metadata + data | Event-driven (object store notifications) for near-real time (NRT) updates |

## Phases

While the grid above tracks *what* moves through our portfolio, the phases below sketch *when* — a notional sequence (timelines are notional, not concrete).

<svg viewBox="0 0 1200 380" width="100%" style="max-width:100%;height:auto;" xmlns="http://www.w3.org/2000/svg" font-family="Arial, Helvetica, sans-serif" role="img" aria-label="ODD phases notional timeline">
<text x="20" y="26" font-size="18" font-weight="bold" fill="#0b2545">ODD phases — notional timeline</text>
<text x="1180" y="26" font-size="12" fill="#64748b" text-anchor="end">timelines are notional, not concrete</text>
<g fill="#475569" font-size="13" font-weight="bold" text-anchor="middle">
<text x="152" y="56">FY26.4</text>
<text x="376" y="56">FY27.1</text>
<text x="600" y="56">FY27.2</text>
<text x="824" y="56">FY27.3</text>
<text x="1048" y="56">FY27.4</text>
</g>
<g stroke="#e2e8f0" stroke-width="1">
<line x1="264" y1="64" x2="264" y2="250"/>
<line x1="488" y1="64" x2="488" y2="250"/>
<line x1="712" y1="64" x2="712" y2="250"/>
<line x1="936" y1="64" x2="936" y2="250"/>
</g>
<line x1="40" y1="64" x2="40" y2="250" stroke="#cbd5e1"/>
<line x1="1160" y1="64" x2="1160" y2="250" stroke="#cbd5e1"/>
<rect x="40" y="80" width="448" height="48" rx="8" fill="#0e7490"/>
<text x="60" y="109" font-size="14" font-weight="bold" fill="#ffffff">Demonstrate the data lake &#8212; varied datasets</text>
<rect x="264" y="138" width="448" height="48" rx="8" fill="#0e7490"/>
<text x="284" y="167" font-size="13.5" font-weight="bold" fill="#ffffff">Demonstrate the query engine + service integration</text>
<rect x="712" y="196" width="448" height="48" rx="8" fill="#0e7490"/>
<text x="732" y="225" font-size="14" font-weight="bold" fill="#ffffff">Demonstrate caching + AI use</text>
<rect x="40" y="266" width="1120" height="46" rx="8" fill="#f1f5f9" stroke="#64748b" stroke-width="1.5"/>
<text x="56" y="294" font-size="13" fill="#0f172a"><tspan font-weight="bold">Throughout: </tspan>socialization of the plan &#183; external-team integration &#183; iterating on the plan as we incorporate varied datasets</text>
<rect x="40" y="322" width="1120" height="46" rx="8" fill="#f1f5f9" stroke="#64748b" stroke-width="1.5"/>
<text x="56" y="350" font-size="13" fill="#0f172a"><tspan font-weight="bold">Foundational libraries: </tspan>Zarr &#183; Icechunk &#183; obstore (IO) &#183; warp / resampling / projection performance &#183; in-browser Zarr + COG &#183; GeoZarr &amp; standards</text>
</svg>

**FY26.4–27.1 — Demonstrate the data lake.** Demonstrate the utility and performance of Icechunk stores as a data lake platform across varied data types (HLS, NISAR, GPM IMERG, NLDAS, TEMPO, ...). Leverage VEDA instances to demonstrate the value of the data lake through services, and direct access the value to scientists. Simultaneously, we will migrate the data services components, specifically TiTiler-CMR, to the Data Services team.

**FY27.1–27.2 — Demonstrate the query engine + service integration.** Showcase integrated discovery, query and access via the query engine. Integrate the query engine with data services so a single interface serves discovery, query, and access.

**FY27.3–27.4 — Demonstrate caching + AI use.** Demonstrate performance using multiscales and a *data cache* (i.e. a distributed in-memory store). Work with the AI/ML teams to demonstrate use of the data lake by AI (e.g. Water Insight or EIE); LLMs discover, reason about, and ingest data from the lake.

**Throughout — alongside every phase.** Socialize the vision with other teams and incoporate feedback. Iterate on the plan as we work to incorporate varied datasets. Continue foundational work in Zarr, Icechunk, and other underlying geospatial libraries: IO (obstore), warp / resampling / projection performance, reading and handling Zarr + COG directly in the browser, and geospatial data standards (GeoZarr).


## How we work

ODD is a research and development team, not an operations or continued-maintenance team. Success for any item on this roadmap is *graduating off of it* — not staying on it indefinitely.
Expand Down