From 836a22797297a81a80312211497646a1ea712d2a Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 10:34:28 +0200 Subject: [PATCH 01/22] =?UTF-8?q?docs:=20deltalake=20=E2=80=94=20drop=20ex?= =?UTF-8?q?ternal-direct;=20add=20\$viewdefinition-export?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trims the customer-facing Data Lakehouse Topic Destination tutorial to the two managed modes that we recommend (managed-zerobus + managed-sql); the external-direct writeMode stays in the module's code but no longer appears in the docs. ~270 lines removed (write-mode subsection, comparison-table column, Configuration parameters tab, SP-grants tab, Alternative-external-direct section, "How it works external-direct" subsection, maintenance subsection, manual schema-evolution subsection, two troubleshooting entries). New page docs/modules/sql-on-fhir/operation-viewdefinition-export.md documents the SQL-on-FHIR v2 \$viewdefinition-export operation that sansara now ships first-party; explains the kind-based backend registry, lists currently registered backends, the kick-off request shape, the spec-defined params (with their MVP status), the status polling protocol, and the failure model. Linked from the SoF README + SUMMARY navigation. The deltalake tutorial gains a new "Ad-hoc one-shot export" section pointing at the \$viewdefinition-export op page (since this module is the first backend, kind=data-lakehouse) and a Related-docs link. Co-Authored-By: Claude Opus 4.7 (1M context) --- SUMMARY.md | 1 + docs/modules/sql-on-fhir/README.md | 6 + .../operation-viewdefinition-export.md | 128 +++++++ .../data-lakehouse-aidboxtopicdestination.md | 334 ++++-------------- 4 files changed, 200 insertions(+), 269 deletions(-) create mode 100644 docs/modules/sql-on-fhir/operation-viewdefinition-export.md diff --git a/SUMMARY.md b/SUMMARY.md index b980c69cb..87de59dd1 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -327,6 +327,7 @@ * [$run operation](modules/sql-on-fhir/operation-run.md) * [$sqlquery-run operation](modules/sql-on-fhir/operation-sqlquery-run.md) * [$materialize operation](modules/sql-on-fhir/operation-materialize.md) + * [$viewdefinition-export operation](modules/sql-on-fhir/operation-viewdefinition-export.md) * [Defining flat views with view definitions](modules/sql-on-fhir/defining-flat-views-with-view-definitions.md) * [Migrate to the spec-compliant ViewDefinition format](modules/sql-on-fhir/migrate-to-the-spec-compliant-viewdefinition-format.md) * [Query data from flat views](modules/sql-on-fhir/query-data-from-flat-views.md) diff --git a/docs/modules/sql-on-fhir/README.md b/docs/modules/sql-on-fhir/README.md index 8d3b35d97..c4814843c 100644 --- a/docs/modules/sql-on-fhir/README.md +++ b/docs/modules/sql-on-fhir/README.md @@ -28,6 +28,12 @@ Bundle a SQL query, its ViewDefinition dependencies, and parameters into a SQLQu See [$sqlquery-run operation](./operation-sqlquery-run.md). +## Export a ViewDefinition's rows + +Run a one-shot ad-hoc export of a ViewDefinition's materialized rows to a backend-provided sink (e.g. Databricks Unity Catalog managed Delta) via the SQL-on-FHIR v2 `$viewdefinition-export` operation. + +See [$viewdefinition-export operation](./operation-viewdefinition-export.md). + ## De-identification Starting from version **2604**, ViewDefinition columns can be annotated with de-identification methods to transform sensitive data during SQL generation. Supported methods include redact, cryptoHash, dateshift, encrypt, substitute, perturb, and custom PostgreSQL functions. diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md new file mode 100644 index 000000000..c2c452bae --- /dev/null +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -0,0 +1,128 @@ +--- +description: Async bulk export of a ViewDefinition's materialized rows to a backend-provided sink (Databricks Delta, etc.) +--- +# `$viewdefinition-export` operation + +{% hint style="info" %} +Implements [SQL-on-FHIR v2 `$viewdefinition-export`](https://build.fhir.org/ig/FHIR/sql-on-fhir-v2/OperationDefinition-ViewDefinitionExport.html). Async pattern follows the FHIR async-request convention (HTTP `202` + `Content-Location` → polling URL). +{% endhint %} + +{% hint style="warning" %} +Requires **fhir-schema mode** (same as the other ViewDefinition operations). +{% endhint %} + +A one-shot ad-hoc export of a ViewDefinition's materialized rows into a backend-provided sink. Aidbox owns the FHIR-side wiring (route, Parameters parsing, async kick-off, status polling); the sink is contributed by an external Aidbox module that registers itself as a **backend** keyed by the `kind` input parameter. + +Use this when you need a periodic snapshot / backfill / ad-hoc dump and don't want to stand up an `AidboxTopicDestination` with its continuous-streaming worker. + +## Registered backends + +| `kind` | Sink | Module | +|---|---|---| +| `data-lakehouse` | Databricks Unity Catalog managed Delta table | [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) | + +Future BigQuery / ClickHouse backends would plug in with their own `kind`. Customers see "no backend registered for kind=X" if they invoke with an unsupported value. + +## Kick-off + +```http +POST /fhir/ViewDefinition/$viewdefinition-export +Content-Type: application/fhir+json +Prefer: respond-async + +{ + "resourceType": "Parameters", + "parameter": [ + {"name": "view", + "part": [{"name": "name", "valueString": "patient_flat"}, + {"name": "viewReference", "valueReference": {"reference": "ViewDefinition/patient_flat"}}]}, + {"name": "kind", "valueString": "data-lakehouse"}, + + {"name": "writeMode", "valueString": "managed-zerobus"}, + {"name": "databricksWorkspaceUrl", "valueString": "https://workspace.cloud.databricks.com"}, + {"name": "databricksWorkspaceId", "valueString": "1234567890123456"}, + {"name": "databricksRegion", "valueString": "us-east-1"}, + {"name": "tableName", "valueString": "catalog.schema.patient_flat"}, + {"name": "databricksWarehouseId", "valueString": "wh-abc"}, + {"name": "awsRegion", "valueString": "us-east-1"}, + {"name": "stagingTablePath", "valueString": "s3://bucket/staging/patient_flat/"} + ] +} +``` + +Response: + +``` +202 Accepted +Content-Location: /fhir/ViewDefinition/$viewdefinition-export/status/ + +{ + "resourceType": "Parameters", + "parameter": [ + {"name": "exportId", "valueString": ""}, + {"name": "status", "valueCode": "in-progress"}, + {"name": "location", "valueUri": "/fhir/ViewDefinition/$viewdefinition-export/status/"} + ] +} +``` + +## Spec-defined parameters + +| Parameter | Required | Notes | +|---|---|---| +| `view` | yes | Exactly one entry. `viewReference` must point at a server-stored ViewDefinition. Inline `viewResource` is not yet supported. | +| `kind` | yes | Selects the backend (e.g. `data-lakehouse`). | +| `clientTrackingId` | no | Echoed back in the status response. | +| `_format` | no | `ndjson`, `parquet`, `json`, or omitted. Functionally ignored — the sink format is determined by the backend (Delta for `kind=data-lakehouse`). | +| `header` | no | Echoed; not meaningful for non-CSV sinks. | +| `patient` (0..\*) | no | List of Patient references. Currently accepted but **not yet applied** to the underlying SQL — the full view is exported. | +| `group` (0..\*) | no | List of Group references. Same status as `patient` — accepted, not yet applied. | +| `_since` | no | Same — accepted, not yet applied. | +| `source` | no | External data source URI. **Not supported** — rejected. | + +Backend-specific parameters live alongside the spec ones in the same `Parameters` body. See the backend's docs for the full list. For `kind=data-lakehouse` see the [Data Lakehouse Topic Destination tutorial](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md). + +## Status polling + +```http +GET /fhir/ViewDefinition/$viewdefinition-export/status/ +``` + +Response codes: + +- `202 Accepted` — still in progress. The same `Content-Location` is returned so the client can keep polling. +- `200 OK` — terminal. Body is a `Parameters` resource with the final shape (`status=completed` or `status=failed`, plus `output[].location` on success). +- `404 Not Found` — unknown `export-id`. + +Completed output for `kind=data-lakehouse`: + +```json +{ + "resourceType": "Parameters", + "parameter": [ + {"name": "exportId", "valueString": ""}, + {"name": "status", "valueCode": "completed"}, + {"name": "clientTrackingId", "valueString": "..."}, + {"name": "exportStartTime", "valueInstant": "2026-05-22T00:00:00Z"}, + {"name": "exportEndTime", "valueInstant": "2026-05-22T00:01:30Z"}, + {"name": "output", + "part": [{"name": "name", "valueString": "patient_flat"}, + {"name": "location", "valueUri": "databricks-uc:catalog.schema.patient_flat"}]} + ] +} +``` + +The `output[].location` URI scheme is backend-specific (`databricks-uc:` for the data-lakehouse backend). + +## Failure model + +- **Input validation failures** (missing `view`, missing `kind`, multiple views, `source` set, etc.) — synchronous `400 OperationOutcome`. +- **No backend registered for `kind`** — same shape; `400` with `code=no-backend`. +- **Backend-side failures** (e.g., Databricks auth, missing target table, schema mismatch) — async. The kick-off returns `202`, then status polling reports `status=failed` with an `error` field carrying the message. + +## Limitations (current) + +- One `view` per request (spec allows `1..*`). +- `patient` / `group` / `_since` filters extracted but not yet applied to the SQL. +- Status registry is in-process — restarting the Aidbox node loses pending export status. Long-running exports across restarts will be tracked via a FHIR custom resource in a follow-up. +- Cancellation (`cancelUrl`) and `estimatedTimeRemaining` are not implemented. diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 1a74182d1..1fa96d215 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -56,12 +56,12 @@ Unity Catalog tables come in two flavours: | Storage location | Databricks-managed cloud storage (path picked by Unity Catalog) | Your bucket — declared with `LOCATION 's3://...' / 'gs://...' / 'abfss://...'` at `CREATE TABLE` | | Who owns the files | Unity Catalog — manages read, write, storage, and optimization | You — Unity Catalog manages metadata only | | `DROP TABLE` | Deletes the data | Drops metadata only — files stay in your bucket | -| Supported write paths from Aidbox | **Zerobus REST ingest** (Aidbox `managed-zerobus`), or **SQL warehouse INSERT** (Aidbox `managed-sql`) | **Direct Parquet + Delta commit** via STS-vended Unity Catalog creds (Aidbox `external-direct`) | +| Supported write paths from Aidbox | **Zerobus REST ingest** (Aidbox `managed-zerobus`), or **SQL warehouse INSERT** (Aidbox `managed-sql`) | Not supported by this module — use a managed table | | External STS credential vending| Not available for managed targets (`EXTERNAL USE SCHEMA` is only grantable on external schemas) | Allowed if the principal has `EXTERNAL USE SCHEMA` on the schema | | Predictive Optimization | Enabled by default for accounts created on or after **2024-11-11**; runs `OPTIMIZE` / `VACUUM` / `ANALYZE` automatically. Billed under the **Jobs Serverless** SKU. | **Not supported** — Predictive Optimization runs only on managed tables | | Liquid Clustering | Opt-in per table (automatic liquid clustering requires Predictive Optimization and is also opt-in) | Opt-in per table | -The "Supported write paths" row drives the module's three `writeMode` values — see [Overview](#overview) for the resulting write paths. +The "Supported write paths" row drives the module's `writeMode` values — see [Overview](#overview) for the resulting write paths. ## Overview @@ -81,7 +81,6 @@ graph LR Mod -- poll batch --> PG Mod -- REST ingest
(managed-zerobus) --> DBX Mod -- SQL INSERT
(managed-sql) --> DBX - Mod -- Delta write
(external-direct) --> FS DBX -- read / write Delta files --> FS ``` @@ -92,13 +91,12 @@ The flow: 3. The Data Lakehouse module polls the destination's batch from the same PostgreSQL queue. 4. For `managed-zerobus` mode (default): the module POSTs each batch as a JSON array to Databricks' Zerobus REST ingest endpoint, which writes directly to the managed table. No SQL parsing / planning per write. 5. For `managed-sql` mode: the module sends `INSERT` (and `ALTER` / `DESCRIBE` when needed) to the Databricks SQL warehouse; the warehouse writes the Delta files to storage. -6. For `external-direct` mode: the module gets short-lived storage credentials from Unity Catalog and writes Delta files directly to your bucket. The module may also perform an initial export of pre-existing resources at first start — see [Initial export](#initial-export) for when this runs and how to skip it. ### Write modes -The module supports three **write modes**, picked per-destination via the `writeMode` parameter (see the [Configuration](#configuration) section below for the full parameter list). +The module supports two **write modes**, picked per-destination via the `writeMode` parameter (see the [Configuration](#configuration) section below for the full parameter list). ### managed-zerobus mode (default) @@ -139,46 +137,23 @@ graph LR - Each batch becomes a single `INSERT INTO managed (cols) VALUES (...)` against the SQL warehouse. - Initial bulk export uses a one-shot staging Delta table + `MERGE INTO`. See [Initial export](#how-it-works-managed-modes). -### external-direct mode +## Choosing between the two modes -`writeMode=external-direct` targets a **non-managed external Delta table** that you own. +**Default to `managed-zerobus`.** Pick `managed-sql` when Zerobus isn't available on your Databricks SKU — same managed target, but every batch hits a warm SQL warehouse. -```mermaid -graph LR - A(FHIR resource POST / PUT / DELETE):::blue2 - B(Aidbox Topics API):::blue2 - C(PostgreSQL queue):::neutral2 - D(ViewDefinition flatten):::yellow2 - K(Direct Delta writer):::green2 - T2(External Delta table on S3 / GCS / ADLS):::violet2 - - A --> B --> C --> D --> K --> T2 -``` - -- The module writes Delta files straight to your bucket from the Aidbox process. No SQL warehouse, no Databricks compute on the write path. -- Storage backends: AWS S3, Google Cloud Storage, Azure ADLS Gen2. -- You own table maintenance — schedule `OPTIMIZE` and `VACUUM` yourself. See [Compaction and maintenance](#compaction-and-maintenance). - -## Choosing between the three modes - -**Default to `managed-zerobus`.** Pick a different mode only when one of these applies: - -- **Zerobus isn't available on your Databricks SKU** → `managed-sql`. Same managed target, but every batch hits a warm SQL warehouse. -- **You want the files in your own bucket and own table maintenance yourself** → `external-direct`. No Databricks compute on the write path. - -| | `managed-zerobus` (default) | `managed-sql` | `external-direct` | -| ------------------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------ | -| Table type | Unity Catalog **managed** (Databricks owns the files) | Unity Catalog **managed** (Databricks owns the files) | **External** (the User's bucket owns the files) | -| Hot-path transport | Zerobus REST ingest API | Databricks SQL warehouse (Statement Execution API) | Direct Delta commits via Hadoop FS | -| Who runs maintenance | Databricks (Predictive Optimization handles `OPTIMIZE` / `VACUUM`) | Databricks (Predictive Optimization handles `OPTIMIZE` / `VACUUM`) | The User schedules `OPTIMIZE` / `VACUUM` | -| Databricks compute cost surface| **No warm warehouse** — pay-per-row Zerobus + storage only | SQL warehouse must be running to accept INSERTs — Databricks bills uptime | No warehouse — no Databricks compute charge for write path | -| Schema drift handling | Auto-`ALTER` on mismatch | Auto-`ALTER` on mismatch | User runs `ALTER TABLE` and recreates the destination | -| Initial export path | Staging Delta on your bucket → `MERGE INTO` target | Staging Delta on your bucket → `MERGE INTO` target | Bulk written straight to the target in one Delta commit | -| Storage backends | Databricks-managed storage | Databricks-managed storage | AWS S3, GCS, Azure ADLS Gen2 | +| | `managed-zerobus` (default) | `managed-sql` | +| ------------------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------ | +| Table type | Unity Catalog **managed** (Databricks owns the files) | Unity Catalog **managed** (Databricks owns the files) | +| Hot-path transport | Zerobus REST ingest API | Databricks SQL warehouse (Statement Execution API) | +| Who runs maintenance | Databricks (Predictive Optimization handles `OPTIMIZE` / `VACUUM`) | Databricks (Predictive Optimization handles `OPTIMIZE` / `VACUUM`) | +| Databricks compute cost surface| **No warm warehouse** — pay-per-row Zerobus + storage only | SQL warehouse must be running to accept INSERTs — Databricks bills uptime | +| Schema drift handling | Auto-`ALTER` on mismatch | Auto-`ALTER` on mismatch | +| Initial export path | Staging Delta on your bucket → `MERGE INTO` target | Staging Delta on your bucket → `MERGE INTO` target | +| Storage backends | Databricks-managed storage | Databricks-managed storage | ## Authentication -All three modes authenticate to Databricks via [**OAuth Machine-to-Machine (M2M)**](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) with a service principal: the module exchanges `client_id` + `client_secret` at the workspace token endpoint for a ~1h bearer token, caches it, and re-issues a fresh one when fewer than 5 minutes remain. +Both modes authenticate to Databricks via [**OAuth Machine-to-Machine (M2M)**](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) with a service principal: the module exchanges `client_id` + `client_secret` at the workspace token endpoint for a ~1h bearer token, caches it, and re-issues a fresh one when fewer than 5 minutes remain. The bearer is sent on every Databricks call. What differs between modes is which Databricks surfaces see it: @@ -186,9 +161,8 @@ The bearer is sent on every Databricks call. What differs between modes is which |-----------------------------|-----------------------------------------------|--------------------------------------------|----------------------------|--------------------------------------| | `managed-zerobus` (default) | only during initial-export (staging vending) | bootstrap + initial-export only | Zerobus REST (every batch) | Zerobus ingest service, Databricks-side | | `managed-sql` | only during initial-export (staging vending) | every batch (`INSERT` / `ALTER` / `DESCRIBE`) | — | SQL warehouse compute | -| `external-direct` | every cred-refresh (~45 min) | none | — | sender process, with Unity-Catalog-vended STS | -In `external-direct` you can also skip Databricks entirely and authenticate against the bucket with static AWS keys (`awsAccessKeyId` + `awsSecretAccessKey`) or the [AWS default provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html). The service principal and the grants it needs are set up in the [Usage example](#usage-example-patient-data-export) below. +The service principal and the grants it needs are set up in the [Usage example](#usage-example-patient-data-export) below. ## Installation @@ -197,10 +171,10 @@ In `external-direct` you can also skip Databricks entirely and authenticate agai - Aidbox **2605** or newer ([install guide](../../getting-started/run-aidbox-locally.md)) - A Databricks workspace (Free Edition works for evaluation, paid for production) - The [Databricks CLI](https://docs.databricks.com/aws/en/dev-tools/cli/install) installed locally (`brew install databricks/tap/databricks` on macOS) — every Databricks-side operation in the tutorial uses it -- AWS CLI (only for `managed-*` modes that do initial-export — for the staging bucket + IAM role) -- A SQL warehouse (skip only for `external-direct`) +- AWS CLI (for initial-export staging — bucket + IAM role) +- A SQL warehouse - For `managed-zerobus`: Zerobus enabled on your SKU (Databricks Free Edition supports it; for paid plans confirm with Databricks support) -- For initial-export in the `managed-*` modes: an S3/GCS/ADLS bucket you control +- For initial-export: an S3/GCS/ADLS bucket you control The service principal that authenticates the module is created in step 3 of the usage example — you don't need it before you start. @@ -321,7 +295,7 @@ All requests in this tutorial use `Content-Type: application/json`. ParameterTypeDescription -writeModestringmanaged-zerobus (default), managed-sql, or external-direct. Omit to get managed-zerobus +writeModestringmanaged-zerobus (default) or managed-sql. Omit to get managed-zerobus skipInitialExportbooleanSkip initial export of existing data (default: false) targetFileSizeMbunsignedIntParquet target size during initial export (default: 128) initialExportParallelismunsignedIntTotal number of parallel chunks for hash-partitioned initial export (default 1 — sequential). Recommended 4-8 for ≥1M-row datasets; 16-32 for multi-Aidbox setups. See Large-scale initial export @@ -372,63 +346,6 @@ All requests in this tutorial use `Content-Type: application/json`. {% endtab %} -{% tab title="external-direct mode" %} -**Required:** - - - - - - - - - - - - - -
ParameterTypeDescription
viewDefinitionstringThe name field of the ViewDefinition resource (not id)
batchSizeunsignedIntRows per worker tick / batch commit
sendIntervalMsunsignedIntMax time between batched commits, in ms
writeModestringMust be external-direct (otherwise the default managed-zerobus path is used)
tablePathstrings3://... / gs://... / abfss://.... Required unless databricksWorkspaceUrl set (then resolved from Unity Catalog)
awsRegionstringRequired for real AWS / GovCloud (skip for MinIO / LocalStack)
- -
- -Authentication parameters - -Pick **one** of: Unity Catalog credential vending, static AWS keys, or default AWS provider chain. - - - - - - - - - - - - - -
ParameterTypeDescription
databricksWorkspaceUrlstringIf set: Unity Catalog credential vending; databricksClientId + databricksClientSecret + tableName must also be set
databricksClientIdstringSP client_id (required iff databricksWorkspaceUrl set)
databricksClientSecretstringSP client_secret; supports vault refs (required iff databricksWorkspaceUrl set)
tableNamestringUnity Catalog catalog.schema.table (when using Unity Catalog credential vending)
awsAccessKeyIdstringStatic IAM key (falls back to default provider chain when absent). Supports vault refs
awsSecretAccessKeystringStatic IAM secret. Supports vault refs
- -
- -
- -Advanced parameters - - - - - - - - - - - -
ParameterTypeDescription
skipInitialExportbooleanSkip initial export of existing data (default: false)
targetFileSizeMbunsignedIntParquet target size during initial export (default: 128)
initialExportParallelismunsignedIntTotal number of parallel chunks for hash-partitioned initial export (default 1 — sequential). Recommended 4-8 for ≥1M-row datasets; 16-32 for multi-Aidbox setups. See Large-scale initial export
s3EndpointstringMinIO / LocalStack endpoint (forces path-style URLs)
- -
-{% endtab %} {% endtabs %} ## Output semantics @@ -456,11 +373,10 @@ Use [the read-time projection below](#querying-the-table) to collapse history to ### At-least-once delivery -Messages are persisted in a PostgreSQL queue and retried on failure. The three modes differ on the crash-between-commit-and-ack window: +Messages are persisted in a PostgreSQL queue and retried on failure. Both write modes have the same crash-between-commit-and-ack semantics: -- **`managed-zerobus`** — initial export is idempotent; live writes are at-least-once (REST has no offset / transaction id, so a replay re-inserts). -- **`managed-sql`** — initial export is idempotent; live writes are at-least-once (SQL `INSERT` has the same constraint). -- **`external-direct`** — idempotent for both. Each Delta commit carries a transaction id; replays are silently deduped. +- Initial export is **idempotent** for both: rows are staged in an external Delta, then `MERGE INTO target USING staging ON t.id = s.id WHEN NOT MATCHED THEN INSERT *`. A replay finds the existing rows in the target and inserts zero new ones. +- Live writes are **at-least-once** for both: the Zerobus REST endpoint has no offset / transaction id; the SQL `INSERT` path has the same constraint. Use the read-time dedup pattern below to collapse duplicates. ### Querying the table @@ -480,7 +396,7 @@ This is one example, not the only approach — wrap it in a Databricks SQL view ## Usage example: patient data export -The example below uses `managed-zerobus` (the default). For non-default modes see [`managed-sql`](#alternative-managed-sql-configuration) or [`external-direct`](#alternative-external-direct-configuration). +The example below uses `managed-zerobus` (the default). For the alternative SQL-warehouse path, see [`managed-sql` configuration](#alternative-managed-sql-configuration). Authenticate the [Databricks CLI](https://docs.databricks.com/aws/en/dev-tools/cli/install) once — **as your own user** (PAT or `databricks auth login`). @@ -815,7 +731,7 @@ In both `managed-*` modes the module issues `ALTER TABLE ADD COLUMNS` automatica {% endstep %} {% hint style="info" %} -**The next step sets up initial-bulk staging.** Skip it (and the staging-specific grants in the next-but-one step) if you only need new data going forward — the destination has a parameter that turns the backfill off. `external-direct` doesn't use staging either. +**The next step sets up initial-bulk staging.** Skip it (and the staging-specific grants in the next-but-one step) if you only need new data going forward — the destination has a parameter that turns the backfill off. {% endhint %} {% step %} @@ -902,27 +818,6 @@ databricks grants update external-location "$EXTERNAL_LOCATION_NAME" --json '{ ``` {% endtab %} -{% tab title="external-direct" %} -Different — `EXTERNAL_USE_SCHEMA` is on the **target's** schema (writes go directly), and you grant against the External Location backing the target's bucket prefix (which can be the same one you registered above if both target and staging live under the same bucket): - -```sh -databricks grants update catalog "$CATALOG" --json '{ - "changes":[{"principal":"'"$SP_CLIENT_ID"'","add":["USE_CATALOG"]}]}' - -databricks grants update schema "$CATALOG.$TARGET_SCHEMA" --json '{ - "changes":[{"principal":"'"$SP_CLIENT_ID"'","add":["USE_SCHEMA","EXTERNAL_USE_SCHEMA"]}]}' - -databricks grants update table "$CATALOG.$TARGET_SCHEMA.$TARGET_TABLE" --json '{ - "changes":[{"principal":"'"$SP_CLIENT_ID"'","add":["SELECT","MODIFY"]}]}' - -databricks grants update external-location "$EXTERNAL_LOCATION_NAME" --json '{ - "changes":[{"principal":"'"$SP_CLIENT_ID"'","add":["READ_FILES","WRITE_FILES","CREATE_EXTERNAL_TABLE"]}]}' -``` - -{% hint style="warning" %} -`EXTERNAL_USE_SCHEMA` is **only grantable on external schemas** (their own `storage_root` set, no inherited managed location). UC managed schemas refuse this grant by design. -{% endhint %} -{% endtab %} {% endtabs %} {% endstep %} @@ -1062,99 +957,6 @@ POST /fhir/AidboxTopicDestination The Databricks setup is identical to `managed-zerobus` — same catalog, schema, target table, warehouse, staging chain, SP, and grants. The warehouse simply ends up servicing every batch instead of only the bootstrap. -## Alternative: `external-direct` configuration - -If you don't need Unity Catalog managed-table governance and want the highest throughput (direct-to-storage Parquet writes, zero Databricks compute cost), use `writeMode=external-direct`. The module commits Parquet + Delta transaction-log entries straight to your bucket via Unity Catalog credential vending. - -### Setup differences from the managed modes - -1. **Create the target schema as external**, not managed. `EXTERNAL USE SCHEMA` is grantable only on external schemas (their own `storage_root` set, no inherited managed location). On most Free Edition and recent paid workspaces (Default Storage enabled), a plain `CREATE SCHEMA $CATALOG.$TARGET_SCHEMA` produces a managed schema that silently refuses the grant later. Create it with an explicit storage root pointed at an External Location you own: - - ```sh - databricks schemas create "$TARGET_SCHEMA" "$CATALOG" \ - --storage-root "s3://$STAGING_BUCKET/target/" - ``` - - This replaces the plain `CREATE SCHEMA` from the catalog/schema step above. The bucket prefix must be covered by an External Location with `READ_FILES, WRITE_FILES, CREATE_EXTERNAL_TABLE` granted to the SP — the same `$EXTERNAL_LOCATION_NAME` you registered for the managed modes is fine if both target and staging live under the same bucket. - -2. **Create the table with `LOCATION`** so it's external: - - ```sh - databricks api post /api/2.0/sql/statements --json "$(jq -n \ - --arg wh "$WAREHOUSE_ID" \ - --arg stmt "CREATE TABLE $CATALOG.$TARGET_SCHEMA.$TARGET_TABLE (id STRING, ts TIMESTAMP, cts TIMESTAMP, gender STRING, birth_date DATE, family_name STRING, given_name STRING, is_deleted INT) USING DELTA LOCATION 's3://$STAGING_BUCKET/target/$TARGET_TABLE/'" \ - '{warehouse_id: $wh, wait_timeout: "30s", statement: $stmt}')" - ``` - -3. **No warehouse needed at runtime** — writes don't go through SQL compute. (The warehouse is still needed once for the `CREATE TABLE` above.) - -4. **Different grants** — `EXTERNAL USE SCHEMA` on the **target's** schema (now external thanks to step 1), and `READ FILES, WRITE FILES, CREATE EXTERNAL TABLE` on the External Location backing the target bucket. See the `external-direct` tab in [Grant the service principal](#grant-the-service-principal). - -5. **No `stagingTablePath`** — initial export writes directly to the final external table; no intermediate staging. - -6. **The User owns the schema** — there's no auto-`ALTER` in this mode. If you add a column to the ViewDefinition, you must `ALTER TABLE` yourself before recreating the destination, or initial validation will fail. - -### Destination configuration - -```http -POST /fhir/AidboxTopicDestination - -{ - "resourceType": "AidboxTopicDestination", - "id": "patient-databricks-external", - "topic": "http://example.org/subscriptions/patient-updates", - "kind": "data-lakehouse-at-least-once", - "meta": { - "profile": [ - "http://health-samurai.io/fhir/core/StructureDefinition/aidboxtopicdestination-dataLakehouseAtLeastOnceProfile" - ] - }, - "parameter": [ - {"name": "writeMode", "valueString": "external-direct"}, - {"name": "databricksWorkspaceUrl", "valueString": "$DATABRICKS_HOST"}, - {"name": "databricksClientId", "valueString": "$SP_CLIENT_ID"}, - {"name": "databricksClientSecret", "valueString": "$SP_CLIENT_SECRET"}, - {"name": "tableName", "valueString": "$CATALOG.$TARGET_SCHEMA.$TARGET_TABLE"}, - {"name": "awsRegion", "valueString": "$AWS_REGION"}, - {"name": "viewDefinition", "valueString": "patient_flat"}, - {"name": "batchSize", "valueUnsignedInt": 50}, - {"name": "sendIntervalMs", "valueUnsignedInt": 5000} - ] -} -``` - -### Static AWS keys (no Unity Catalog vending) - -`external-direct` can also write to a Delta table that isn't governed by Unity Catalog — for example, a bucket your own AWS account owns directly, or a MinIO / non-Databricks S3 deployment. Omit `databricksWorkspaceUrl` entirely and provide static AWS keys + `tablePath`: - -```http -POST /fhir/AidboxTopicDestination - -{ - "resourceType": "AidboxTopicDestination", - "id": "patient-deltalake-s3", - "topic": "http://example.org/subscriptions/patient-updates", - "kind": "data-lakehouse-at-least-once", - "meta": { - "profile": [ - "http://health-samurai.io/fhir/core/StructureDefinition/aidboxtopicdestination-dataLakehouseAtLeastOnceProfile" - ] - }, - "parameter": [ - {"name": "writeMode", "valueString": "external-direct"}, - {"name": "tablePath", "valueString": "s3://my-bucket/patients/"}, - {"name": "awsRegion", "valueString": "us-east-1"}, - {"name": "awsAccessKeyId", "valueString": ""}, - {"name": "awsSecretAccessKey", "valueString": ""}, - {"name": "viewDefinition", "valueString": "patient_flat"}, - {"name": "batchSize", "valueUnsignedInt": 50}, - {"name": "sendIntervalMs", "valueUnsignedInt": 5000} - ] -} -``` - -You can also omit `awsAccessKeyId` / `awsSecretAccessKey` to fall back to the [AWS SDK default credentials provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html) — env vars, EC2 instance profile / ECS task role, EKS IRSA, or shared profile from `~/.aws/credentials`. - ## Initial export When a new destination is created with `skipInitialExport` not set to `true`, the module exports the **current state** of every row in `sof.` — one row per resource the ViewDefinition matches. @@ -1219,20 +1021,6 @@ Recommended values: Independent of `initialExportParallelism`, the SQL warehouse running the final MERGE remains a separate axis — for ≥100M-row exports temporarily scale it to `M` or `L` while initial-export is running, then back down once `initialExportStatus=completed`. -### How it works — `external-direct` mode - -```mermaid -graph LR - PG[(Aidbox PostgreSQL
sof.<view>)]:::neutral2 - M[Aidbox sender]:::blue2 - Target[(External Delta target
on S3 / GCS / ADLS)]:::violet2 - - M -- 1. read rows --> PG - M -- 2. write Parquet + Delta commit
via Unity-Catalog-vended STS --> Target -``` - -No staging — the module writes `sof.` rows straight to the external target table. All rows land in one Delta commit at the end, so consumers see either zero rows or the full historical batch (all-or-nothing visibility). Requires `EXTERNAL USE SCHEMA` so Unity Catalog will vend write credentials for the target. - ## Monitoring ### Status endpoint @@ -1273,33 +1061,21 @@ The module automatically: 2. **Adds deletion flag**: Sets `is_deleted = 0` for create/update, `is_deleted = 1` for delete operations 3. **Batches messages**: Groups messages according to `batchSize` and `sendIntervalMs` parameters 4. **Coerces types per write path**: - - `managed-sql` / `external-direct` — Java SQL dates / timestamps are converted to ISO-8601 strings; the SQL warehouse (or the Delta-Kernel writer) parses them into `DATE` / `TIMESTAMP` columns. + - `managed-sql` — Java SQL dates / timestamps are converted to ISO-8601 strings; the SQL warehouse parses them into `DATE` / `TIMESTAMP` columns. - `managed-zerobus` — dates are encoded as `int32` epoch-days, timestamps as `int64` epoch-microseconds, as required by the Zerobus REST wire format. ISO strings would be rejected with a `400` from the endpoint. See [Output semantics](#output-semantics) for append-only behaviour, at-least-once delivery, and the recommended read-time dedup query. ## Compaction and maintenance -**Managed modes (`managed-zerobus` and `managed-sql`)** — Databricks runs maintenance for you: +Databricks runs maintenance for you: - [Predictive Optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization) is enabled by default for Databricks accounts created on or after **2024-11-11**. Older accounts can enable it manually at the catalog / schema level. - When enabled, it runs `OPTIMIZE`, `VACUUM`, and `ANALYZE` in the background. - Predictive Optimization runs against managed tables **only** and is billed under the **Jobs Serverless** SKU. -**`external-direct` mode** — you own the table and the maintenance: - -- Predictive Optimization does **not** apply to external tables (Databricks restricts it to managed tables). -- Recommended pattern: schedule a [Databricks SQL Job](https://docs.databricks.com/aws/en/jobs/) running - - ```sql - OPTIMIZE aidbox_export.fhir.patients; - VACUUM aidbox_export.fhir.patients RETAIN 168 HOURS; - ``` - ## Schema evolution -### Managed modes (auto-heal) - Both `managed-zerobus` and `managed-sql` auto-`ALTER TABLE ADD COLUMNS` when the ViewDefinition has new columns. Triggered at sender start and on per-batch schema-mismatch (retried once). To add a column: @@ -1314,16 +1090,38 @@ Existing rows will have `NULL` in the new column. The module only ADDS columns automatically. Column drops, renames, or narrowing type changes (e.g., `BIGINT` → `INT`) are not auto-applied — you must run the corresponding `ALTER TABLE` manually. {% endhint %} -### `external-direct` mode (manual) +## Ad-hoc one-shot export -The User owns the external table schema. If the ViewDefinition adds a column without a matching `ALTER TABLE` on the Databricks side, the destination's healthcheck will **fail at startup** with a clear error message pointing at the missing column. +Besides the continuous `AidboxTopicDestination` flow above, the module also serves as the `kind="data-lakehouse"` backend for the [SQL-on-FHIR v2 `$viewdefinition-export` operation](../../modules/sql-on-fhir/operation-viewdefinition-export.md) — a one-shot async export of a ViewDefinition's rows into the same Databricks managed UC table. Useful for periodic snapshots / backfills / ad-hoc dumps where standing up a continuous destination is overkill. -To add a column: +The Databricks-side setup (catalog, schema, target table, staging schema, SP, grants, warehouse) is identical to the continuous flow above. Invocation: + +```http +POST /fhir/ViewDefinition/$viewdefinition-export +Prefer: respond-async + +{ + "resourceType": "Parameters", + "parameter": [ + {"name": "view", + "part": [{"name": "name", "valueString": "patient_flat"}, + {"name": "viewReference", "valueReference": {"reference": "ViewDefinition/patient_flat"}}]}, + {"name": "kind", "valueString": "data-lakehouse"}, + {"name": "writeMode", "valueString": "managed-zerobus"}, + {"name": "databricksWorkspaceUrl", "valueString": "$DATABRICKS_HOST"}, + {"name": "databricksWorkspaceId", "valueString": "$WORKSPACE_ID"}, + {"name": "databricksRegion", "valueString": "$DATABRICKS_REGION"}, + {"name": "tableName", "valueString": "$CATALOG.$TARGET_SCHEMA.$TARGET_TABLE"}, + {"name": "databricksWarehouseId", "valueString": "$WAREHOUSE_ID"}, + {"name": "awsRegion", "valueString": "$AWS_REGION"}, + {"name": "stagingTablePath", "valueString": "s3://$STAGING_BUCKET/staging/$TARGET_TABLE/"} + ] +} +``` + +Returns `202 Accepted` + `Content-Location: /fhir/ViewDefinition/$viewdefinition-export/status/`. Poll that URL until you get `200 OK` with the final `Parameters` output. -1. Run `ALTER TABLE aidbox_export.fhir.patients ADD COLUMNS (new_col STRING)` in Databricks SQL. -2. Add the column to your ViewDefinition. -3. Re-materialize: `POST /fhir/ViewDefinition/{id}/$materialize`. -4. Delete and recreate the destination. +See the [`$viewdefinition-export` operation page](../../modules/sql-on-fhir/operation-viewdefinition-export.md) for the full parameter list, status response shape, and current limitations. ## Multiple destinations @@ -1340,22 +1138,20 @@ You can create multiple destinations for the same topic — for example, to mirr ### Common issues -1. **`EXTERNAL_WRITE_NOT_ALLOWED_FOR_TABLE`** (writeMode=external-direct against a managed table) — Unity Catalog vending refuses managed tables by design. Either recreate the table as external (with explicit `LOCATION '...'`), or switch the destination to `writeMode=managed`. -2. **`EXTERNAL_ACCESS_DISABLED_ON_METASTORE`** — your Unity Catalog metastore has external data access disabled (the Databricks Free Edition default). In Catalog Explorer → Metastore → enable **External data access**. -3. **`Privilege EXTERNAL USE SCHEMA is not applicable to this entity`** — you're trying to grant `EXTERNAL USE SCHEMA` on a managed schema. Either recreate the schema as external, or switch to `writeMode=managed`. -4. **`INSUFFICIENT_PRIVILEGES` on table or warehouse** — verify all grants in [Grant the service principal](#grant-the-service-principal). Don't forget `CAN_USE` on the warehouse. -5. **`DELTA_INSERT_COLUMN_ARITY_MISMATCH`** in managed mode — the module should auto-heal this once. If it persists, check that the schema diff is column-add only (drops / renames are not auto-applied). -6. **Schema mismatch in external-direct mode** — the module fails at startup with a clear message naming the missing columns. Run the corresponding `ALTER TABLE` and recreate the destination. -7. **Slow first write** — Serverless warehouses cold-start in 30-90s on first use after idle. The module's HTTP timeout is 120s for SQL Statement Execution and uses `wait_timeout=50s` polling, so cold starts succeed transparently but the first batch's latency is high. Keep the warehouse warm with a periodic ping if first-batch latency matters. -8. **Duplicate rows after recreating destination** — deleting and recreating a destination triggers initial export again. Set `skipInitialExport: true` when recreating a destination that already has its data exported. -9. **`LOCATION_OVERLAP` during initial export** — `stagingTablePath` either equals the staging schema's `storage_root` (which UC treats as the schema's own managed location) or doesn't sit under your External Location. Set it to a sub-prefix of the External Location, e.g. `s3:///staging/patient_flat/`, not the External Location root itself. -10. **`Unsupported table kind. Tables created in default storage are not supported` (Zerobus error 4024)** — the catalog backing your target table was created without `--storage-root`, so Unity Catalog placed it in the workspace's default-storage prefix. `managed-zerobus` refuses to write into default storage. Recreate the catalog with `databricks catalogs create --storage-root s3:///managed/` pointing inside a registered External Location (see [Create the catalog and target schema](#create-the-catalog-and-target-schema) in the usage example). -11. **`DELTA_CREATE_TABLE_SCHEME_MISMATCH` on initial export retry** — your `stagingTablePath` contains a `_delta_log/` from a previous initial-export run, and the new run has a different ViewDefinition schema (e.g. you added `ts`/`cts` columns). The module drops the UC staging table metadata on cleanup but does NOT delete S3 files, so the old `_delta_log/` survives and conflicts. Fix: either point `stagingTablePath` at a fresh sub-prefix (e.g. append a nonce: `s3:///staging/-v2/`), or manually `aws s3 rm --recursive` the old prefix. +1. **`Privilege EXTERNAL USE SCHEMA is not applicable to this entity`** — you're trying to grant `EXTERNAL USE SCHEMA` on a managed schema. The staging schema (`_staging`) must be external — create it with an explicit `storage_root` pointed at your staging External Location. +2. **`INSUFFICIENT_PRIVILEGES` on table or warehouse** — verify all grants in [Grant the service principal](#grant-the-service-principal). Don't forget `CAN_USE` on the warehouse. +3. **`DELTA_INSERT_COLUMN_ARITY_MISMATCH`** — the module should auto-heal this once. If it persists, check that the schema diff is column-add only (drops / renames are not auto-applied). +4. **Slow first write** — Serverless warehouses cold-start in 30-90s on first use after idle. The module's HTTP timeout is 120s for SQL Statement Execution and uses `wait_timeout=50s` polling, so cold starts succeed transparently but the first batch's latency is high. Keep the warehouse warm with a periodic ping if first-batch latency matters. +5. **Duplicate rows after recreating destination** — deleting and recreating a destination triggers initial export again. Set `skipInitialExport: true` when recreating a destination that already has its data exported. +6. **`LOCATION_OVERLAP` during initial export** — `stagingTablePath` either equals the staging schema's `storage_root` (which UC treats as the schema's own managed location) or doesn't sit under your External Location. Set it to a sub-prefix of the External Location, e.g. `s3:///staging/patient_flat/`, not the External Location root itself. +7. **`Unsupported table kind. Tables created in default storage are not supported` (Zerobus error 4024)** — the catalog backing your target table was created without `--storage-root`, so Unity Catalog placed it in the workspace's default-storage prefix. `managed-zerobus` refuses to write into default storage. Recreate the catalog with `databricks catalogs create --storage-root s3:///managed/` pointing inside a registered External Location (see [Create the catalog and target schema](#create-the-catalog-and-target-schema) in the usage example). +8. **`DELTA_CREATE_TABLE_SCHEME_MISMATCH` on initial export retry** — your `stagingTablePath` contains a `_delta_log/` from a previous initial-export run, and the new run has a different ViewDefinition schema (e.g. you added `ts`/`cts` columns). The module drops the UC staging table metadata on cleanup but does NOT delete S3 files, so the old `_delta_log/` survives and conflicts. Fix: either point `stagingTablePath` at a fresh sub-prefix (e.g. append a nonce: `s3:///staging/
-v2/`), or manually `aws s3 rm --recursive` the old prefix. ## Related documentation - [ViewDefinitions](../../modules/sql-on-fhir/defining-flat-views-with-view-definitions.md) - [`$materialize` operation](../../modules/sql-on-fhir/operation-materialize.md) +- [`$viewdefinition-export` operation](../../modules/sql-on-fhir/operation-viewdefinition-export.md) — the SQL-on-FHIR ad-hoc export this module backs as `kind=data-lakehouse` - [Topic-based Subscriptions](../../modules/topic-based-subscriptions/README.md) - [External Secrets (Vault)](../../configuration/secret-files.md) — storing sensitive parameters like `databricksClientSecret` as file-backed secrets - [HashiCorp Vault Integration](../../tutorials/other-tutorials/hashicorp-vault-external-secrets.md) — step-by-step tutorial for Kubernetes with Secrets Store CSI Driver From ea561527737cf079f28d7b5f361208847e557f75 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 10:58:39 +0200 Subject: [PATCH 02/22] docs: note $viewdefinition-export availability since Aidbox 2605 Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/modules/sql-on-fhir/operation-viewdefinition-export.md | 4 ++++ .../data-lakehouse-aidboxtopicdestination.md | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index c2c452bae..36784130c 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -3,6 +3,10 @@ description: Async bulk export of a ViewDefinition's materialized rows to a back --- # `$viewdefinition-export` operation +{% hint style="info" %} +Available in Aidbox versions **2605** and later. +{% endhint %} + {% hint style="info" %} Implements [SQL-on-FHIR v2 `$viewdefinition-export`](https://build.fhir.org/ig/FHIR/sql-on-fhir-v2/OperationDefinition-ViewDefinitionExport.html). Async pattern follows the FHIR async-request convention (HTTP `202` + `Content-Location` → polling URL). {% endhint %} diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 1fa96d215..16ab42f5a 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1092,6 +1092,10 @@ The module only ADDS columns automatically. Column drops, renames, or narrowing ## Ad-hoc one-shot export +{% hint style="info" %} +Available in Aidbox versions **2605** and later. +{% endhint %} + Besides the continuous `AidboxTopicDestination` flow above, the module also serves as the `kind="data-lakehouse"` backend for the [SQL-on-FHIR v2 `$viewdefinition-export` operation](../../modules/sql-on-fhir/operation-viewdefinition-export.md) — a one-shot async export of a ViewDefinition's rows into the same Databricks managed UC table. Useful for periodic snapshots / backfills / ad-hoc dumps where standing up a continuous destination is overkill. The Databricks-side setup (catalog, schema, target table, staging schema, SP, grants, warehouse) is identical to the continuous flow above. Invocation: From e61eabab537cdc896114c9686b398b8e57569bb6 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 12:59:56 +0200 Subject: [PATCH 03/22] docs: link $viewdefinition-export in sof README + AWS-only callout Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/modules/sql-on-fhir/README.md | 2 +- .../data-lakehouse-aidboxtopicdestination.md | 10 +++++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/modules/sql-on-fhir/README.md b/docs/modules/sql-on-fhir/README.md index c4814843c..5963ac2b6 100644 --- a/docs/modules/sql-on-fhir/README.md +++ b/docs/modules/sql-on-fhir/README.md @@ -30,7 +30,7 @@ See [$sqlquery-run operation](./operation-sqlquery-run.md). ## Export a ViewDefinition's rows -Run a one-shot ad-hoc export of a ViewDefinition's materialized rows to a backend-provided sink (e.g. Databricks Unity Catalog managed Delta) via the SQL-on-FHIR v2 `$viewdefinition-export` operation. +Run a one-shot ad-hoc export of a ViewDefinition's materialized rows to a backend-provided sink (e.g. Databricks Unity Catalog managed Delta) via the SQL-on-FHIR v2 [`$viewdefinition-export`](./operation-viewdefinition-export.md) operation. See [$viewdefinition-export operation](./operation-viewdefinition-export.md). diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 16ab42f5a..0071a40c8 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1,5 +1,5 @@ --- -description: Export FHIR resources to a Data Lakehouse — Databricks Unity Catalog managed tables or non-managed external Delta tables on S3 / GCS / Azure ADLS — using SQL-on-FHIR ViewDefinitions. +description: Export FHIR resources to Databricks Unity Catalog managed Delta tables using SQL-on-FHIR ViewDefinitions. --- # Data Lakehouse AidboxTopicDestination @@ -8,11 +8,15 @@ description: Export FHIR resources to a Data Lakehouse — Databricks Unity Cata This functionality is available starting from Aidbox version **2605**. {% endhint %} -This page sets up an `AidboxTopicDestination` that streams FHIR resource changes into Delta-Lake tables — Databricks-managed Unity Catalog tables, or external Delta tables on S3 / GCS / Azure ADLS that you own. Rows are flattened by a [ViewDefinition](../../modules/sql-on-fhir/defining-flat-views-with-view-definitions.md) so analytics consumers see columns, not nested FHIR JSON. +{% hint style="info" %} +**Cloud support: AWS only (today).** The initial-export staging Delta is written via Unity Catalog credential vending, and the module currently only consumes the `aws_temp_credentials` response (S3 / `s3a://` staging buckets). GCS and Azure ADLS Gen2 staging are not yet wired — adding them is tracked as a follow-up. +{% endhint %} + +This page sets up an `AidboxTopicDestination` that streams FHIR resource changes into a Databricks Unity Catalog managed Delta table. Rows are flattened by a [ViewDefinition](../../modules/sql-on-fhir/defining-flat-views-with-view-definitions.md) so analytics consumers see columns, not nested FHIR JSON. ## Background -"Data Lakehouse" is the generic name for the destination category — a hybrid of object-storage data lake and warehouse, implemented here on top of the Delta Lake table format. Concretely the module writes Delta-formatted tables that can live on plain cloud object storage you own, or in Databricks Unity Catalog managed storage; either way the destination kind is the same (`data-lakehouse-at-least-once`). +"Data Lakehouse" is the generic name for the destination category — a hybrid of object-storage data lake and warehouse, implemented here on top of the Delta Lake table format. The module writes a Delta-formatted Unity Catalog managed table; the destination kind is `data-lakehouse-at-least-once`. If you're already comfortable with Databricks, Unity Catalog, and Delta Lake, skip to [Overview](#overview). From 20aee532486c4ca34322979784ffb159a5dd6c98 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 13:25:14 +0200 Subject: [PATCH 04/22] docs: fix $viewdefinition-export failure-model + AWS-only callout on op page - 'no backend for kind' is async-failed (defmulti :default throws inside the future), not sync 400. Updated the Failure model section. - Added an AWS-only callout to the operation page itself so users reading just that page aren't surprised at runtime. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../sql-on-fhir/operation-viewdefinition-export.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index 36784130c..216d3dae1 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -120,9 +120,16 @@ The `output[].location` URI scheme is backend-specific (`databricks-uc:` for the ## Failure model -- **Input validation failures** (missing `view`, missing `kind`, multiple views, `source` set, etc.) — synchronous `400 OperationOutcome`. -- **No backend registered for `kind`** — same shape; `400` with `code=no-backend`. -- **Backend-side failures** (e.g., Databricks auth, missing target table, schema mismatch) — async. The kick-off returns `202`, then status polling reports `status=failed` with an `error` field carrying the message. +- **Input validation failures** (missing `view`, missing `kind`, multiple views, `source` set, etc.) — synchronous `400 OperationOutcome` returned from the kick-off `POST`. No `export-id` is allocated. +- **Backend-side failures** — async. The kick-off returns `202` with an `export-id`; status polling later reports `status=failed` with the error in the `error` parameter. Includes: + - **No backend registered for `kind`** (e.g., typo, module not deployed) — the defmulti's `:default` method raises a clear `ex-info`, so the polling output's `error` field reads `"No backend registered for $viewdefinition-export kind=..."`. + - **Databricks auth** (bad `client-id` / `client-secret`). + - **Missing target table** / **missing required Databricks parameter** (e.g., no `tableName`). + - **Schema mismatch** the module can't auto-`ALTER`. + +## Cloud support + +The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=data-lakehouse`, [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md)) currently supports AWS only**. The initial-export staging Delta is written via Unity Catalog credential vending and the module only consumes the `aws_temp_credentials` response (S3 / `s3a://` staging buckets). GCS and Azure ADLS Gen2 staging are tracked as follow-ups. ## Limitations (current) From efb7744eca25fc79eace5484f26aaf69a45c49ca Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 13:35:44 +0200 Subject: [PATCH 05/22] docs: note \$viewdefinition-export reuses topic-destination initial-export flow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For the data-lakehouse backend the operation is literally the same managed-initial-export! path the AidboxTopicDestination runs on its first start — same staging Delta + MERGE INTO + drop-staging, just exposed standalone with no continuous-streaming worker around it. Made that explicit on both pages so readers don't have to discover the relationship by reading source. - operation-viewdefinition-export.md: new "Relationship to AidboxTopicDestination's initial export" section with a comparison table. - data-lakehouse-aidboxtopicdestination.md ad-hoc section: short callout pointing at the existing "Initial export" section. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../sql-on-fhir/operation-viewdefinition-export.md | 13 +++++++++++++ .../data-lakehouse-aidboxtopicdestination.md | 4 +++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index 216d3dae1..7cba72a5c 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -19,6 +19,19 @@ A one-shot ad-hoc export of a ViewDefinition's materialized rows into a backend- Use this when you need a periodic snapshot / backfill / ad-hoc dump and don't want to stand up an `AidboxTopicDestination` with its continuous-streaming worker. +## Relationship to AidboxTopicDestination's initial export + +For `kind=data-lakehouse`, this operation reuses **the exact same flow** that the [Data Lakehouse Topic Destination](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) runs on its first start as the "initial export": read `sof.` rows → stage Parquet to an external Delta via Unity Catalog credential vending → `MERGE INTO target USING staging ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` → drop staging. The difference is what surrounds that flow: + +| | Topic destination initial export | `$viewdefinition-export` | +|---|---|---| +| Triggered by | `POST /AidboxTopicDestination` (once, on resource creation) | `POST /fhir/ViewDefinition/\$viewdefinition-export` (any time, repeatable) | +| Followed by | Continuous streaming worker that ingests new resource changes | Nothing — one-shot | +| Status surface | `$status` on the destination resource | This op's poll URL | +| `AidboxTopicDestination` row in PG | Yes — owns the destination's lifecycle | No — purely ad-hoc, no resource created | + +So if all you need is a periodic snapshot of a view and not continuous streaming, this operation gives you the same write path without the topic-destination plumbing. + ## Registered backends | `kind` | Sink | Module | diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 0071a40c8..040c66b67 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1102,7 +1102,9 @@ Available in Aidbox versions **2605** and later. Besides the continuous `AidboxTopicDestination` flow above, the module also serves as the `kind="data-lakehouse"` backend for the [SQL-on-FHIR v2 `$viewdefinition-export` operation](../../modules/sql-on-fhir/operation-viewdefinition-export.md) — a one-shot async export of a ViewDefinition's rows into the same Databricks managed UC table. Useful for periodic snapshots / backfills / ad-hoc dumps where standing up a continuous destination is overkill. -The Databricks-side setup (catalog, schema, target table, staging schema, SP, grants, warehouse) is identical to the continuous flow above. Invocation: +Architecturally this is **the initial export from [Initial export](#initial-export) above, exposed standalone** — same `sof.` → staging Delta → `MERGE INTO target` → drop-staging flow, no continuous-streaming worker afterwards, no `AidboxTopicDestination` row in PG. The Databricks-side setup (catalog, schema, target table, staging schema, SP, grants, warehouse) is therefore identical to the continuous flow's setup above. + +Invocation: ```http POST /fhir/ViewDefinition/$viewdefinition-export From 09afe23daead91a914cfb2689f4e64f462e82ed1 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 13:37:29 +0200 Subject: [PATCH 06/22] docs: tighten 'AWS S3 only' callout User feedback: the previous wording (\"AWS only\", \"aws_temp_credentials\") was technical and easy to miss. Replaces with explicit \"AWS S3 only; GCS and Azure ADLS Gen2 not supported\" callouts, escalated to 'warning' style on the tutorial, repeated in the operation page's Cloud support section, mentioned in Prerequisites and on the stagingTablePath parameter row. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../sql-on-fhir/operation-viewdefinition-export.md | 2 +- .../data-lakehouse-aidboxtopicdestination.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index 7cba72a5c..b1b946a56 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -142,7 +142,7 @@ The `output[].location` URI scheme is backend-specific (`databricks-uc:` for the ## Cloud support -The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=data-lakehouse`, [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md)) currently supports AWS only**. The initial-export staging Delta is written via Unity Catalog credential vending and the module only consumes the `aws_temp_credentials` response (S3 / `s3a://` staging buckets). GCS and Azure ADLS Gen2 staging are tracked as follow-ups. +The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=data-lakehouse`, [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md)) currently supports AWS S3 only** for the staging Delta path. **Google Cloud Storage** (`gs://...`) and **Azure ADLS Gen2** (`abfss://...`) are not yet supported — adding them is tracked as a follow-up. The Databricks Unity Catalog managed target table is unaffected (UC manages target storage internally). ## Limitations (current) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 040c66b67..e427896c7 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -8,8 +8,8 @@ description: Export FHIR resources to Databricks Unity Catalog managed Delta tab This functionality is available starting from Aidbox version **2605**. {% endhint %} -{% hint style="info" %} -**Cloud support: AWS only (today).** The initial-export staging Delta is written via Unity Catalog credential vending, and the module currently only consumes the `aws_temp_credentials` response (S3 / `s3a://` staging buckets). GCS and Azure ADLS Gen2 staging are not yet wired — adding them is tracked as a follow-up. +{% hint style="warning" %} +**Cloud support: AWS S3 only (today).** The initial-export staging bucket must be an **AWS S3** bucket (`s3://...`). **Google Cloud Storage** (`gs://...`) and **Azure ADLS Gen2** (`abfss://...`) are not yet supported as staging backends — adding them is tracked as a follow-up. The Databricks Unity Catalog managed target table is unaffected (UC manages target storage internally). {% endhint %} This page sets up an `AidboxTopicDestination` that streams FHIR resource changes into a Databricks Unity Catalog managed Delta table. Rows are flattened by a [ViewDefinition](../../modules/sql-on-fhir/defining-flat-views-with-view-definitions.md) so analytics consumers see columns, not nested FHIR JSON. @@ -178,7 +178,7 @@ The service principal and the grants it needs are set up in the [Usage example]( - AWS CLI (for initial-export staging — bucket + IAM role) - A SQL warehouse - For `managed-zerobus`: Zerobus enabled on your SKU (Databricks Free Edition supports it; for paid plans confirm with Databricks support) -- For initial-export: an S3/GCS/ADLS bucket you control +- For initial-export: an **S3 bucket** you control. GCS and Azure ADLS Gen2 are not supported for the staging path today (see the "Cloud support: AWS only" callout above). The service principal that authenticates the module is created in step 3 of the usage example — you don't need it before you start. @@ -286,7 +286,7 @@ All requests in this tutorial use `Content-Type: application/json`. - +
tableNamestringManaged table full name: catalog.schema.table
databricksWarehouseIdstringSQL warehouse ID — used at bootstrap for schema sync + (if initial-export runs) the final MERGE INTO. No warm-warehouse traffic during live writes.
awsRegionstringAWS region of the staging bucket
stagingTablePathstrings3://bucket/path/ for the staging Delta table created during initial export. Required when skipInitialExport is not true
stagingTablePathstrings3://bucket/path/ for the staging Delta table created during initial export (S3 only today). Required when skipInitialExport is not true
@@ -328,7 +328,7 @@ All requests in this tutorial use `Content-Type: application/json`. tableNamestringManaged table full name: catalog.schema.table databricksWarehouseIdstringSQL warehouse ID awsRegionstringAWS region of the staging bucket -stagingTablePathstrings3://bucket/path/ for the staging Delta table created during initial export. Required when skipInitialExport is not true +stagingTablePathstrings3://bucket/path/ for the staging Delta table created during initial export (S3 only today). Required when skipInitialExport is not true From 8515ef6a5baa6def5c00f94f7490389d9a99b703 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 13:44:34 +0200 Subject: [PATCH 07/22] docs: cross-link Initial-export section to $viewdefinition-export MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two flows are the same path — call that out at the top of the Initial-export section, not just bottom-up from the ad-hoc section. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../data-lakehouse-aidboxtopicdestination.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index e427896c7..09001d431 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -963,6 +963,10 @@ The Databricks setup is identical to `managed-zerobus` — same catalog, schema, ## Initial export +{% hint style="info" %} +The same flow described below is also exposed standalone as a FHIR operation: [`$viewdefinition-export`](../../modules/sql-on-fhir/operation-viewdefinition-export.md). Use that operation when you want a one-shot snapshot of a ViewDefinition without standing up a continuous `AidboxTopicDestination` — it reuses this module as the `kind=data-lakehouse` backend, and you get an async kick-off + status-poll URL instead of a destination's `$status`. See the [Ad-hoc one-shot export](#ad-hoc-one-shot-export) section below for a usage example. +{% endhint %} + When a new destination is created with `skipInitialExport` not set to `true`, the module exports the **current state** of every row in `sof.` — one row per resource the ViewDefinition matches. - **Updates after destination creation** append a new row each (`POST` / `PUT` / `DELETE`), accumulating a full audit trail. From e7b2c891f839f39bc3c4f417bf45ac02b100e3dc Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 13:47:10 +0200 Subject: [PATCH 08/22] docs: canonicalize 'how it works' on \$viewdefinition-export page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mermaid + step-by-step explanation of the staging-Delta + MERGE INTO flow moves from the topic-destination tutorial to the operation page — the operation is the canonical entry point for that flow (and the topic-destination's initial export reuses the same code path internally). Topic tutorial keeps only an inline pointer; 'Large-scale initial export' subsection stays in the tutorial since it's about a destination-only parameter (initialExportParallelism on AidboxTopicDestination, not on the operation). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operation-viewdefinition-export.md | 36 +++++++++++++++++++ .../data-lakehouse-aidboxtopicdestination.md | 34 ++---------------- 2 files changed, 38 insertions(+), 32 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index b1b946a56..d49bfc07a 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -140,6 +140,42 @@ The `output[].location` URI scheme is backend-specific (`databricks-uc:` for the - **Missing target table** / **missing required Databricks parameter** (e.g., no `tableName`). - **Schema mismatch** the module can't auto-`ALTER`. +## How it works (`kind=data-lakehouse`) + +The first-party backend uses a **staging Delta table** as a relay: it writes the `sof.` rows to an external Delta table at a customer-provided `stagingTablePath` (via Unity Catalog credential vending), then `MERGE INTO`s the managed target, then drops the staging table. Same flow for `writeMode=managed-zerobus` and `writeMode=managed-sql`. + +```mermaid +graph LR + PG[(Aidbox PostgreSQL
sof.<view>)]:::neutral2 + M[Aidbox sender]:::blue2 + Staging[Staging external Delta table
on stagingTablePath]:::yellow2 + WH[Databricks SQL warehouse]:::green2 + Target[(Unity Catalog managed Delta target)]:::violet2 + + M -- 1. read rows --> PG + M -- 2. write Parquet + Delta commit
via Unity-Catalog-vended STS --> Staging + M -- 3. MERGE INTO target USING staging ON id
WHEN NOT MATCHED THEN INSERT * --> WH + WH -- 4. read --> Staging + WH -- 5. write --> Target + M -- 6. DROP TABLE staging --> WH +``` + +Steps in detail: + +1. Register a temporary external Delta table at `stagingTablePath` with the same schema as `sof.`. +2. Unity Catalog vends short-lived STS credentials for the staging path. +3. The module writes all `sof.` rows to the staging path as one Delta commit. +4. The module issues `MERGE INTO {managed_target} USING {staging} ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` against the SQL warehouse. The MERGE reads the staging Delta snapshot through the Delta protocol and inserts any rows whose `id` is not yet present in the target. +5. The module drops the staging table. + +On failure the staging table is best-effort dropped, then the export retries up to 3 times with exponential backoff (1s → 2s → 4s). + +{% hint style="info" %} +The `MERGE` is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must have an `id` column. +{% endhint %} + +The Databricks-side setup (catalog, schema, target table, staging schema, SP, grants, warehouse) is documented in the [Data Lakehouse Topic Destination tutorial](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md). Same setup serves both the continuous topic destination and this ad-hoc operation — the operation reuses the topic destination's initial-export code path. + ## Cloud support The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=data-lakehouse`, [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md)) currently supports AWS S3 only** for the staging Delta path. **Google Cloud Storage** (`gs://...`) and **Azure ADLS Gen2** (`abfss://...`) are not yet supported — adding them is tracked as a follow-up. The Databricks Unity Catalog managed target table is unaffected (UC manages target storage internally). diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 09001d431..c7ed45db2 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -978,39 +978,9 @@ To skip the initial export (e.g., the table is already populated or you only nee { "name": "skipInitialExport", "valueBoolean": true } ``` -### How it works — managed modes +### How it works -Initial bulk export uses a **staging table** as a relay: the module writes Parquet to an external Delta table at `stagingTablePath` (via UC credential vending), then `MERGE INTO` the managed target on `id`, then drops the staging table. Identical for `managed-zerobus` and `managed-sql`. - -```mermaid -graph LR - PG[(Aidbox PostgreSQL
sof.<view>)]:::neutral2 - M[Aidbox sender]:::blue2 - Staging[Staging external Delta table
on stagingTablePath]:::yellow2 - WH[Databricks SQL warehouse]:::green2 - Target[(Unity Catalog managed Delta target)]:::violet2 - - M -- 1. read rows --> PG - M -- 2. write Parquet + Delta commit
via Unity-Catalog-vended STS --> Staging - M -- 3. MERGE INTO target USING staging ON id
WHEN NOT MATCHED THEN INSERT * --> WH - WH -- 4. read --> Staging - WH -- 5. write --> Target - M -- 6. DROP TABLE staging --> WH -``` - -Steps in detail: - -1. Register a temporary external Delta table at `stagingTablePath` with the same schema as `sof.`. -2. Unity Catalog vends short-lived STS credentials for the staging path. -3. The module writes all `sof.` rows to the staging path as one Delta commit. -4. The module issues `MERGE INTO {managed_target} USING {staging} ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` against the SQL warehouse. The MERGE reads the staging Delta snapshot through the Delta protocol and inserts any rows whose `id` is not yet present in the target. -5. The module drops the staging table. - -The whole sequence runs as one atomic operation from the destination's lifecycle perspective. On failure: best-effort drop of the staging table, retry up to 3 times with exponential backoff (1s → 2s → 4s). - -{% hint style="info" %} -The MERGE is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must have an `id` column. -{% endhint %} +The flow (mermaid diagram + step-by-step, with the staging Delta relay and idempotent `MERGE INTO`) is documented on the [`$viewdefinition-export` operation page → How it works](../../modules/sql-on-fhir/operation-viewdefinition-export.md#how-it-works-kind-data-lakehouse). The destination's initial export and the standalone operation share the same code path. ### Large-scale initial export From c14dda5a5bda4f9b0ae73e277909a417b1eff773 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 13:55:52 +0200 Subject: [PATCH 09/22] docs: clarify how initialExportParallelism works in a multi-pod cluster MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previous wording was vague — readers had no way to pick N when their deployment has multiple Aidbox pods. New wording spells out: - N is the cluster-wide chunk count, not a per-pod thread count. - Each pod auto-sizes its local worker pool to min(N, its cores). - Workers coordinate cluster-wide via PG advisory locks; no leader, no external service. - Effective concurrency is min(N, total cores across all pods); raising N higher than that wastes CPU on lock-claim spin. - Practical sizing rule: N ≈ sum of cores across all pods, capped by max_connections / ~2 (each worker holds ~2 PG connections). Suggested-values table now has rows for 1 pod / 2-4 pod HA / 4+ pod deployments instead of the single-node-centric values it had before. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../data-lakehouse-aidboxtopicdestination.md | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index c7ed45db2..aaaca1ce7 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -984,18 +984,30 @@ The flow (mermaid diagram + step-by-step, with the staging Delta relay and idemp ### Large-scale initial export -For >=1M-row datasets the single-cursor + single-Kernel-writer default is the wall-clock bottleneck. The `initialExportParallelism` parameter (default `1`) fans the staging write out across N hash-partitioned workers. They write into the **same** staging Delta table — Delta supports concurrent append-only writers natively (INSERT × INSERT does not conflict). The final `MERGE INTO target` runs as a single SQL statement after all chunks land. +For ≥1M-row datasets the single-cursor + single-Kernel-writer default is the wall-clock bottleneck. The `initialExportParallelism` parameter (default `1`) fans the staging write out across `N` hash-partitioned chunks. The chunks are written by parallel workers into per-chunk staging Delta tables, then a single `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL ...)` materializes everything into the managed target in one SQL statement. -Coordination across one or many Aidbox nodes uses **PostgreSQL advisory locks** (`pg_try_advisory_lock`). All Aidbox nodes share the same metastore by definition, so no external dependency is needed — each worker thread on each node tries to claim chunk ids `0..N-1`, runs the partition, marks the chunk completed in the destination resource extension, releases the lock, and moves to the next available chunk. PG drops advisory locks on connection disconnect, so a crashed worker frees its chunk for re-claim automatically. +**`N` is the cluster-wide chunk count, not a per-pod thread count.** You set it once on the destination; the module figures out how many workers each Aidbox pod runs. -Recommended values: +**How a multi-pod cluster cooperates:** every Aidbox pod sharing the same PG metastore participates automatically — there's no leader, no service-discovery, no external coordinator. -| N | Use case | -|---|---| -| `1` (default) | Backward-compatible, single-cursor sequential. | -| `4` | Single Aidbox node, ≥1M rows. ~3-4× speedup. | -| `8` | Larger Aidbox (≥8 cores) + plenty of PG read capacity. | -| `16-32` | Multi-Aidbox HA deployment, ≥100M-row scale. Chunks distribute across all nodes; total cap by your PG `max_connections` budget. | +- Each pod spawns up to `min(N, )` local worker threads. +- Every worker loops over chunk-ids `0..N-1` and tries `pg_try_advisory_lock(destination-hash, chunk-id)` on each one. The first pod's worker to claim a given chunk-id runs that chunk; other workers (on the same pod or any other pod) see `false` and move to the next chunk-id. +- When a chunk finishes the worker marks the chunk-id completed in the destination resource extension (a separate PG advisory lock serializes that read-modify-write across pods) and releases the chunk's lock. +- A crashed worker's PG connection drops → PG releases all its session-level advisory locks automatically → a sibling worker on any pod picks the chunk up on its next loop iteration. + +Effective wall-clock concurrency is therefore `min(N, sum_of_cores_across_all_pods)`. Raising `N` past total cores doesn't make it faster — extra workers spin on lock-claim with nothing to do. Lowering `N` below total cores wastes CPU. + +**Picking `N` for a multi-pod cluster:** + +| Cluster shape | Suggested `N` | Notes | +|---|---|---| +| 1 pod, ≤4 cores | `1` (default) | Single-cursor sequential. Fine for <1M rows. | +| 1 pod, 4-8 cores, ≥1M rows | `4` | ~3-4× speedup; one Kernel writer per worker, one PG cursor per worker. | +| 1 pod, ≥8 cores | `8` | Watch PG read capacity — each cursor holds one connection until its chunk finishes. | +| 2-4 pods (HA), ≥10M rows | `16` | Distributes evenly across pods; survives a pod restart mid-export. | +| 4+ pods, ≥100M rows | `32` | Cap by your PG `max_connections` budget — each worker holds at least one connection. | + +**Practical sizing rule:** set `N ≈ sum of cores across all Aidbox pods running the destination`. If PG `max_connections` is the bottleneck (each worker holds ~2 connections: one for the cursor, one for the lock), cap `N` at `(max_connections - other_load) / 2`. Independent of `initialExportParallelism`, the SQL warehouse running the final MERGE remains a separate axis — for ≥100M-row exports temporarily scale it to `M` or `L` while initial-export is running, then back down once `initialExportStatus=completed`. From fe3ccbbd93962ebf1848cee9b95aec9e75893a0e Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 15:06:59 +0200 Subject: [PATCH 10/22] docs: add explicit N-sizing formula + worked example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reader asked for a formula instead of a vague rule. Replaces the 'N ≈ sum of cores' sentence with a code block showing the two ceilings (core count and PG connection budget), what each variable means, what the per-worker connection cost actually is (2), and a worked example for a 4-pod × 8-core cluster. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../data-lakehouse-aidboxtopicdestination.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index aaaca1ce7..08a7d4ddb 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1007,7 +1007,24 @@ Effective wall-clock concurrency is therefore `min(N, sum_of_cores_across_all_po | 2-4 pods (HA), ≥10M rows | `16` | Distributes evenly across pods; survives a pod restart mid-export. | | 4+ pods, ≥100M rows | `32` | Cap by your PG `max_connections` budget — each worker holds at least one connection. | -**Practical sizing rule:** set `N ≈ sum of cores across all Aidbox pods running the destination`. If PG `max_connections` is the bottleneck (each worker holds ~2 connections: one for the cursor, one for the lock), cap `N` at `(max_connections - other_load) / 2`. +**Picking `N` — formula.** Two ceilings, take the smaller: + +``` +N = min( total_cores_across_all_pods, + (pg_max_connections - aidbox_baseline) / 2 ) +``` + +- `total_cores_across_all_pods` — sum of CPU cores allocated to every Aidbox pod that's connected to this metastore. Each pod auto-caps its local worker pool at its own core count (via `Runtime.availableProcessors()`); the cluster-wide sum is the maximum number of workers that will ever run concurrently. Setting `N` higher than that adds nothing — surplus workers spin on `pg_try_advisory_lock` returning `false`. +- `pg_max_connections` — your PostgreSQL `max_connections` setting (default `100`). +- `aidbox_baseline` — connections Aidbox uses for normal traffic (HikariCP pool size × pod count). Conservative default: `30 × pod_count`. Check your `pg_stat_activity` under steady-state load if you want a tighter number. +- `/ 2` — each worker holds at most two PG connections: one for the `sof.` server-side cursor, one short-lived for the lock-and-progress operations. + +Floor at `1` (default), don't bother going above `64`. + +Example, 4 pods × 8 cores each, PG `max_connections=200`: +- core ceiling: `32` +- connection ceiling: `(200 - 30·4) / 2 = (200 - 120) / 2 = 40` +- → `N = min(32, 40) = 32` Independent of `initialExportParallelism`, the SQL warehouse running the final MERGE remains a separate axis — for ≥100M-row exports temporarily scale it to `M` or `L` while initial-export is running, then back down once `initialExportStatus=completed`. From bb9656777ba4a6999f1e2a71a023707fdf9d4a14 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 15:08:08 +0200 Subject: [PATCH 11/22] docs: render initialExportParallelism formula in KaTeX Co-Authored-By: Claude Opus 4.7 (1M context) --- .../data-lakehouse-aidboxtopicdestination.md | 31 ++++++++++++------- 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 08a7d4ddb..7dee73d19 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1009,22 +1009,29 @@ Effective wall-clock concurrency is therefore `min(N, sum_of_cores_across_all_po **Picking `N` — formula.** Two ceilings, take the smaller: -``` -N = min( total_cores_across_all_pods, - (pg_max_connections - aidbox_baseline) / 2 ) -``` +$$ +N = \min\!\left(\, C_{\text{total}},\; \frac{M - B}{2} \,\right) +$$ + +where -- `total_cores_across_all_pods` — sum of CPU cores allocated to every Aidbox pod that's connected to this metastore. Each pod auto-caps its local worker pool at its own core count (via `Runtime.availableProcessors()`); the cluster-wide sum is the maximum number of workers that will ever run concurrently. Setting `N` higher than that adds nothing — surplus workers spin on `pg_try_advisory_lock` returning `false`. -- `pg_max_connections` — your PostgreSQL `max_connections` setting (default `100`). -- `aidbox_baseline` — connections Aidbox uses for normal traffic (HikariCP pool size × pod count). Conservative default: `30 × pod_count`. Check your `pg_stat_activity` under steady-state load if you want a tighter number. -- `/ 2` — each worker holds at most two PG connections: one for the `sof.` server-side cursor, one short-lived for the lock-and-progress operations. +- $$C_{\text{total}}$$ — sum of CPU cores allocated to every Aidbox pod connected to this metastore. Each pod auto-caps its local worker pool at its own core count (via `Runtime.availableProcessors()`); the cluster-wide sum is the maximum number of workers that will ever run concurrently. Setting $$N$$ higher than that adds nothing — surplus workers spin on `pg_try_advisory_lock` returning `false`. +- $$M$$ — your PostgreSQL `max_connections` setting (default `100`). +- $$B$$ — connections Aidbox uses for normal traffic (HikariCP pool size × pod count). Conservative default: $$30 \cdot \text{pod\_count}$$. Check your `pg_stat_activity` under steady-state load if you want a tighter number. +- The `/ 2` reflects that each worker holds at most two PG connections: one for the `sof.` server-side cursor, one short-lived for the lock-and-progress operations. Floor at `1` (default), don't bother going above `64`. -Example, 4 pods × 8 cores each, PG `max_connections=200`: -- core ceiling: `32` -- connection ceiling: `(200 - 30·4) / 2 = (200 - 120) / 2 = 40` -- → `N = min(32, 40) = 32` +**Worked example** — 4 pods × 8 cores each, $$M = 200$$: + +$$ +C_{\text{total}} = 4 \cdot 8 = 32, \qquad +B = 30 \cdot 4 = 120 +$$ + +$$ +N = \min\!\left(\, 32,\; \frac{200 - 120}{2} \,\right) = \min(32,\, 40) = 32 +$$ Independent of `initialExportParallelism`, the SQL warehouse running the final MERGE remains a separate axis — for ≥100M-row exports temporarily scale it to `M` or `L` while initial-export is running, then back down once `initialExportStatus=completed`. From f159705456b1052c40cef57fbe601f0873142be4 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 15:12:13 +0200 Subject: [PATCH 12/22] docs: drop 'Relationship to TopicDestination initial export' from op page Reviewer feedback: an op-page reader shouldn't need to know what an AidboxTopicDestination is to understand the operation. The cross-link remains in the other direction (the topic-destination tutorial says 'this same flow is exposed as \$viewdefinition-export') and a brief 'Databricks-side setup is documented in the topic tutorial' pointer stays inside How-it-works for the setup steps the reader genuinely needs to find. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operation-viewdefinition-export.md | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index d49bfc07a..e8ed8f58f 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -19,19 +19,6 @@ A one-shot ad-hoc export of a ViewDefinition's materialized rows into a backend- Use this when you need a periodic snapshot / backfill / ad-hoc dump and don't want to stand up an `AidboxTopicDestination` with its continuous-streaming worker. -## Relationship to AidboxTopicDestination's initial export - -For `kind=data-lakehouse`, this operation reuses **the exact same flow** that the [Data Lakehouse Topic Destination](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) runs on its first start as the "initial export": read `sof.` rows → stage Parquet to an external Delta via Unity Catalog credential vending → `MERGE INTO target USING staging ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` → drop staging. The difference is what surrounds that flow: - -| | Topic destination initial export | `$viewdefinition-export` | -|---|---|---| -| Triggered by | `POST /AidboxTopicDestination` (once, on resource creation) | `POST /fhir/ViewDefinition/\$viewdefinition-export` (any time, repeatable) | -| Followed by | Continuous streaming worker that ingests new resource changes | Nothing — one-shot | -| Status surface | `$status` on the destination resource | This op's poll URL | -| `AidboxTopicDestination` row in PG | Yes — owns the destination's lifecycle | No — purely ad-hoc, no resource created | - -So if all you need is a periodic snapshot of a view and not continuous streaming, this operation gives you the same write path without the topic-destination plumbing. - ## Registered backends | `kind` | Sink | Module | @@ -174,7 +161,7 @@ On failure the staging table is best-effort dropped, then the export retries up The `MERGE` is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must have an `id` column. {% endhint %} -The Databricks-side setup (catalog, schema, target table, staging schema, SP, grants, warehouse) is documented in the [Data Lakehouse Topic Destination tutorial](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md). Same setup serves both the continuous topic destination and this ad-hoc operation — the operation reuses the topic destination's initial-export code path. +The Databricks-side setup (catalog, schema, target table, staging schema, service principal, grants, warehouse) is documented in the [Data Lakehouse Topic Destination tutorial](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) — the same setup is reused here. ## Cloud support From e89dce6f0bacee1cbd0d923274aae8efdd5b7738 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 15:21:54 +0200 Subject: [PATCH 13/22] docs: consolidate write-modes section + tighten hints MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Drop 'destination kind is data-lakehouse-at-least-once' from Background (not needed there; the kind is in the configuration examples where it actually has to be typed). - Merge the two writeMode subsections into one. They share the same target, init flow, schema-drift handling and maintenance — only the hot-path transport differs. One mermaid diagram with both arms replaces two near-identical diagrams + the standalone 'Choosing between the two modes' table (most rows of that table were identical copies anyway). - Tighten the 'setting up initial-bulk staging' hint to one line. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../data-lakehouse-aidboxtopicdestination.md | 56 ++++--------------- 1 file changed, 11 insertions(+), 45 deletions(-) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 7dee73d19..7df325dda 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -16,7 +16,7 @@ This page sets up an `AidboxTopicDestination` that streams FHIR resource changes ## Background -"Data Lakehouse" is the generic name for the destination category — a hybrid of object-storage data lake and warehouse, implemented here on top of the Delta Lake table format. The module writes a Delta-formatted Unity Catalog managed table; the destination kind is `data-lakehouse-at-least-once`. +"Data Lakehouse" is the generic name for the destination category — a hybrid of object-storage data lake and warehouse, implemented here on top of the Delta Lake table format. The module writes a Delta-formatted Unity Catalog managed table. If you're already comfortable with Databricks, Unity Catalog, and Delta Lake, skip to [Overview](#overview). @@ -100,31 +100,7 @@ The module may also perform an initial export of pre-existing resources at first ### Write modes -The module supports two **write modes**, picked per-destination via the `writeMode` parameter (see the [Configuration](#configuration) section below for the full parameter list). - -### managed-zerobus mode (default) - -`writeMode=managed-zerobus` targets a **Databricks Unity Catalog managed table** via the [Zerobus REST ingest endpoint](https://docs.databricks.com/aws/en/ingestion/zerobus-ingest). - -```mermaid -graph LR - A(FHIR resource POST / PUT / DELETE):::blue2 - B(Aidbox Topics API):::blue2 - C(PostgreSQL queue):::neutral2 - D(ViewDefinition flatten):::yellow2 - Z(Zerobus REST ingest):::green2 - T0(Unity Catalog managed Delta table):::violet2 - - A --> B --> C --> D --> Z --> T0 -``` - -- Each batch is JSON-encoded as an array and POSTed to the Zerobus REST endpoint with an OAuth M2M bearer. No SQL parsing, no warehouse cold-start. -- Initial bulk export uses a one-shot staging Delta table + `MERGE INTO` — same path as `managed-sql`. -- Schema sync at sender bootstrap hits the SQL warehouse once (`INFORMATION_SCHEMA.COLUMNS` + optional `ALTER TABLE`); live writes don't. - -### managed-sql mode - -`writeMode=managed-sql` — same target as `managed-zerobus` (Unity Catalog **managed** table), but routes incoming batches through a Databricks SQL warehouse. Use this when Zerobus isn't available on your Databricks SKU. +The module supports two **write modes**, picked per-destination via the `writeMode` parameter. Both target the same Unity Catalog managed Delta table and share the same initial-bulk staging + `MERGE INTO` flow, the same auto-`ALTER` schema-drift handling, and the same Databricks-side Predictive Optimization. They differ only in **how live batches reach Databricks**: ```mermaid graph LR @@ -132,28 +108,18 @@ graph LR B(Aidbox Topics API):::blue2 C(PostgreSQL queue):::neutral2 D(ViewDefinition flatten):::yellow2 - M(Databricks SQL warehouse):::green2 - T1(Unity Catalog managed Delta table):::violet2 + Z(Zerobus REST ingest
managed-zerobus):::green2 + M(Databricks SQL warehouse
managed-sql):::green2 + T(Unity Catalog managed Delta table):::violet2 - A --> B --> C --> D --> M --> T1 + A --> B --> C --> D + D -- managed-zerobus --> Z --> T + D -- managed-sql --> M --> T ``` -- Each batch becomes a single `INSERT INTO managed (cols) VALUES (...)` against the SQL warehouse. -- Initial bulk export uses a one-shot staging Delta table + `MERGE INTO`. See [Initial export](#how-it-works-managed-modes). - -## Choosing between the two modes - -**Default to `managed-zerobus`.** Pick `managed-sql` when Zerobus isn't available on your Databricks SKU — same managed target, but every batch hits a warm SQL warehouse. +**Default to `managed-zerobus`.** Each batch is JSON-encoded and POSTed to the [Zerobus REST ingest endpoint](https://docs.databricks.com/aws/en/ingestion/zerobus-ingest) with an OAuth M2M bearer — no SQL parsing, no warm warehouse. The warehouse is hit once at sender bootstrap for `INFORMATION_SCHEMA.COLUMNS` + optional `ALTER TABLE`; live writes don't touch it. -| | `managed-zerobus` (default) | `managed-sql` | -| ------------------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------ | -| Table type | Unity Catalog **managed** (Databricks owns the files) | Unity Catalog **managed** (Databricks owns the files) | -| Hot-path transport | Zerobus REST ingest API | Databricks SQL warehouse (Statement Execution API) | -| Who runs maintenance | Databricks (Predictive Optimization handles `OPTIMIZE` / `VACUUM`) | Databricks (Predictive Optimization handles `OPTIMIZE` / `VACUUM`) | -| Databricks compute cost surface| **No warm warehouse** — pay-per-row Zerobus + storage only | SQL warehouse must be running to accept INSERTs — Databricks bills uptime | -| Schema drift handling | Auto-`ALTER` on mismatch | Auto-`ALTER` on mismatch | -| Initial export path | Staging Delta on your bucket → `MERGE INTO` target | Staging Delta on your bucket → `MERGE INTO` target | -| Storage backends | Databricks-managed storage | Databricks-managed storage | +**Use `managed-sql` only when Zerobus isn't available on your Databricks SKU.** Each batch becomes a single `INSERT INTO managed (cols) VALUES (...)` against a SQL warehouse — same target, same idempotent staging-MERGE init, just a warm warehouse on the hot path. The cost difference is the warehouse uptime billing. ## Authentication @@ -735,7 +701,7 @@ In both `managed-*` modes the module issues `ALTER TABLE ADD COLUMNS` automatica {% endstep %} {% hint style="info" %} -**The next step sets up initial-bulk staging.** Skip it (and the staging-specific grants in the next-but-one step) if you only need new data going forward — the destination has a parameter that turns the backfill off. +**Setting up initial-bulk staging next.** Skip this step and the staging grants below if you only need new data going forward — set `skipInitialExport=true` on the destination. {% endhint %} {% step %} From b95a7053f7729aeb28c52a07d568d84ffc6b7427 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 15:27:20 +0200 Subject: [PATCH 14/22] docs: spell out initial-export timing model + monitoring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User asked whether millions of rows cause an HTTP timeout. They don't — POST AidboxTopicDestination returns 201 after bootstrap (1-2 min worst case on a cold warehouse), initial-export runs in a background future sized only by the dataset. Made that explicit: - New 'Timing & monitoring' subsection in Initial export. - Phase table showing what runs sync inside POST vs in the future. - The \$status fields you read during init-export (status / rowsSent / error). - Retry behaviour (3× exp backoff, idempotent MERGE on id). - Note that continuous worker starts in parallel with init-export, not serialized after it. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../data-lakehouse-aidboxtopicdestination.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 7df325dda..f2a7c53eb 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -948,6 +948,35 @@ To skip the initial export (e.g., the table is already populated or you only nee The flow (mermaid diagram + step-by-step, with the staging Delta relay and idempotent `MERGE INTO`) is documented on the [`$viewdefinition-export` operation page → How it works](../../modules/sql-on-fhir/operation-viewdefinition-export.md#how-it-works-kind-data-lakehouse). The destination's initial export and the standalone operation share the same code path. +### Timing & monitoring + +The kick-off and the export are **decoupled** — `POST /AidboxTopicDestination` does not hold the HTTP connection open while billions of rows stream into Databricks. + +| Phase | Where it runs | Approx. duration | +|---|---|---| +| Aidbox writes the resource to PG | sync, inside the POST | <1s | +| Module bootstrap (validate, OAuth token, schema sync + optional `ALTER TABLE` via SQL warehouse) | sync, inside the POST | 1-2 min on a cold warehouse, <1s when warm | +| Initial export (cursor → staging Delta → `MERGE INTO` target → drop staging) | **async, in a background `future`** | seconds to hours, depending on row count and `initialExportParallelism` | +| Continuous worker (PG queue → batch → Zerobus or SQL) | async, runs forever | — | + +So `POST /AidboxTopicDestination` returns `201 Created` after **bootstrap** (1-2 minutes worst-case), not after initial-export. There's no HTTP timeout regardless of dataset size. + +Poll progress via the destination's `$status` endpoint: + +```sh +curl -u root:secret "$AIDBOX_URL/fhir/AidboxTopicDestination/patient-databricks/\$status" | jq +``` + +Relevant fields during initial export: + +- `initialExportStatus` — `not_started` / `export-in-progress` / `completed` / `skipped` / `failed`. +- `initialExportProgress_rowsSent` — running row count (updated every 10k rows). +- `initialExportError` — error message when `initialExportStatus=failed`. + +On failure the module retries up to 3 times with exponential backoff (1s → 2s → 4s). The `MERGE INTO ... ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` is idempotent on `id`, so a retry after a lost ack inserts zero new rows. + +The continuous worker starts polling the PG queue **immediately after destination creation**, in parallel with initial export — initial-export and live writes are not serialized. The MERGE keying on `id` means a continuous-stream row inserted before initial-export catches up just gets skipped by the eventual MERGE pass (idempotent). + ### Large-scale initial export For ≥1M-row datasets the single-cursor + single-Kernel-writer default is the wall-clock bottleneck. The `initialExportParallelism` parameter (default `1`) fans the staging write out across `N` hash-partitioned chunks. The chunks are written by parallel workers into per-chunk staging Delta tables, then a single `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL ...)` materializes everything into the managed target in one SQL statement. From 095d41e7ba0c3c98baf436743d4fcddba5047ae8 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 15:43:08 +0200 Subject: [PATCH 15/22] docs: doc-audit fixes (mermaid, stale N recommendations, leaks of impl detail) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tutorial: - Overview mermaid said 'Cloud storage S3 / GCS / ADLS' even though we now flag AWS-S3-only. Fixed to 'AWS S3 bucket'. - 'The flow' enumeration listed both modes inline (steps 4-5), redundant with the Write modes subsection that follows. Collapsed to one step that points at Write modes. - The two writeMode tabs in Configuration still had stale 'Recommended 4-8 for ≥1M-row datasets; 16-32 for multi-Aidbox setups' on the initialExportParallelism row. Replaced with a pointer to the formula section. Operation page: - Stacked three info-hints at the top were noisy. Merged into one. - 'Registered backends' line and Failure-model entry for 'no backend registered for kind' described the user-visible behaviour with an implementation-detail leak ('the defmulti's :default method raises a clear ex-info'). Reworded. - Step 1 of How-it-works referenced 'sof.' as jargon. Spelled out what that means ('the SQL-on-FHIR materialized view in Aidbox's PostgreSQL'). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operation-viewdefinition-export.md | 16 ++++------------ .../data-lakehouse-aidboxtopicdestination.md | 9 ++++----- 2 files changed, 8 insertions(+), 17 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index e8ed8f58f..8b4c8abff 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -4,15 +4,7 @@ description: Async bulk export of a ViewDefinition's materialized rows to a back # `$viewdefinition-export` operation {% hint style="info" %} -Available in Aidbox versions **2605** and later. -{% endhint %} - -{% hint style="info" %} -Implements [SQL-on-FHIR v2 `$viewdefinition-export`](https://build.fhir.org/ig/FHIR/sql-on-fhir-v2/OperationDefinition-ViewDefinitionExport.html). Async pattern follows the FHIR async-request convention (HTTP `202` + `Content-Location` → polling URL). -{% endhint %} - -{% hint style="warning" %} -Requires **fhir-schema mode** (same as the other ViewDefinition operations). +Available in Aidbox versions **2605** and later. Requires **fhir-schema mode**. Implements [SQL-on-FHIR v2 `$viewdefinition-export`](https://build.fhir.org/ig/FHIR/sql-on-fhir-v2/OperationDefinition-ViewDefinitionExport.html) — the FHIR async-request pattern (HTTP `202` + `Content-Location` → polling URL). {% endhint %} A one-shot ad-hoc export of a ViewDefinition's materialized rows into a backend-provided sink. Aidbox owns the FHIR-side wiring (route, Parameters parsing, async kick-off, status polling); the sink is contributed by an external Aidbox module that registers itself as a **backend** keyed by the `kind` input parameter. @@ -25,7 +17,7 @@ Use this when you need a periodic snapshot / backfill / ad-hoc dump and don't wa |---|---|---| | `data-lakehouse` | Databricks Unity Catalog managed Delta table | [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) | -Future BigQuery / ClickHouse backends would plug in with their own `kind`. Customers see "no backend registered for kind=X" if they invoke with an unsupported value. +Future BigQuery / ClickHouse backends would plug in with their own `kind`. An unsupported `kind` is reported as `status=failed` in the poll response with the error `"No backend registered for $viewdefinition-export kind=X"` — see [Failure model](#failure-model) below. ## Kick-off @@ -122,7 +114,7 @@ The `output[].location` URI scheme is backend-specific (`databricks-uc:` for the - **Input validation failures** (missing `view`, missing `kind`, multiple views, `source` set, etc.) — synchronous `400 OperationOutcome` returned from the kick-off `POST`. No `export-id` is allocated. - **Backend-side failures** — async. The kick-off returns `202` with an `export-id`; status polling later reports `status=failed` with the error in the `error` parameter. Includes: - - **No backend registered for `kind`** (e.g., typo, module not deployed) — the defmulti's `:default` method raises a clear `ex-info`, so the polling output's `error` field reads `"No backend registered for $viewdefinition-export kind=..."`. + - **No backend registered for `kind`** (e.g., typo, module not deployed) — the polling response's `error` field reads `"No backend registered for $viewdefinition-export kind=..."`. - **Databricks auth** (bad `client-id` / `client-secret`). - **Missing target table** / **missing required Databricks parameter** (e.g., no `tableName`). - **Schema mismatch** the module can't auto-`ALTER`. @@ -149,7 +141,7 @@ graph LR Steps in detail: -1. Register a temporary external Delta table at `stagingTablePath` with the same schema as `sof.`. +1. Register a temporary external Delta table at `stagingTablePath` with the same schema as the SQL-on-FHIR materialized view (`sof.` in Aidbox's PostgreSQL). 2. Unity Catalog vends short-lived STS credentials for the staging path. 3. The module writes all `sof.` rows to the staging path as one Delta commit. 4. The module issues `MERGE INTO {managed_target} USING {staging} ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` against the SQL warehouse. The MERGE reads the staging Delta snapshot through the Delta protocol and inserts any rows whose `id` is not yet present in the target. diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index f2a7c53eb..950ac66da 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -78,7 +78,7 @@ graph LR PG[(Aidbox PostgreSQL)]:::neutral2 Mod[Data Lakehouse module]:::yellow2 DBX[Databricks workspace]:::green2 - FS[(Cloud storage
S3 / GCS / ADLS)]:::violet2 + FS[(AWS S3 bucket)]:::violet2 Client -- FHIR POST / PUT / DELETE --> Aidbox Aidbox -- write resource +
enqueue topic event --> PG @@ -93,8 +93,7 @@ The flow: 1. A FHIR API client (a user, an integration, a backfill script) sends a `POST` / `PUT` / `DELETE` to Aidbox. 2. Aidbox persists the resource and enqueues a topic event for the destination in PostgreSQL. 3. The Data Lakehouse module polls the destination's batch from the same PostgreSQL queue. -4. For `managed-zerobus` mode (default): the module POSTs each batch as a JSON array to Databricks' Zerobus REST ingest endpoint, which writes directly to the managed table. No SQL parsing / planning per write. -5. For `managed-sql` mode: the module sends `INSERT` (and `ALTER` / `DESCRIBE` when needed) to the Databricks SQL warehouse; the warehouse writes the Delta files to storage. +4. The batch is sent to Databricks via one of the two paths picked by `writeMode` — see [Write modes](#write-modes) below. The module may also perform an initial export of pre-existing resources at first start — see [Initial export](#initial-export) for when this runs and how to skip it. @@ -268,7 +267,7 @@ All requests in this tutorial use `Content-Type: application/json`. writeModestringmanaged-zerobus (default) or managed-sql. Omit to get managed-zerobus skipInitialExportbooleanSkip initial export of existing data (default: false) targetFileSizeMbunsignedIntParquet target size during initial export (default: 128) -initialExportParallelismunsignedIntTotal number of parallel chunks for hash-partitioned initial export (default 1 — sequential). Recommended 4-8 for ≥1M-row datasets; 16-32 for multi-Aidbox setups. See Large-scale initial export +initialExportParallelismunsignedIntCluster-wide number of parallel chunks for hash-partitioned initial export (default 1 — sequential). See Large-scale initial export for the sizing formula. @@ -309,7 +308,7 @@ All requests in this tutorial use `Content-Type: application/json`. skipInitialExportbooleanSkip initial export of existing data (default: false) targetFileSizeMbunsignedIntParquet target size during initial export (default: 128) -initialExportParallelismunsignedIntTotal number of parallel chunks for hash-partitioned initial export (default 1 — sequential). Recommended 4-8 for ≥1M-row datasets; 16-32 for multi-Aidbox setups. See Large-scale initial export +initialExportParallelismunsignedIntCluster-wide number of parallel chunks for hash-partitioned initial export (default 1 — sequential). See Large-scale initial export for the sizing formula. From 6964f8bd462c55f59e05fbeda5439a3f3174975b Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 15:44:38 +0200 Subject: [PATCH 16/22] docs: drop stale 'external archive table' use-case from Multiple destinations Remnant of the external-direct writeMode that was removed earlier in this PR. Reframed the use-cases to ones that work with managed-only. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../data-lakehouse-aidboxtopicdestination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 950ac66da..12bda0d6e 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1139,7 +1139,7 @@ See the [`$viewdefinition-export` operation page](../../modules/sql-on-fhir/oper ## Multiple destinations -You can create multiple destinations for the same topic — for example, to mirror the same data into both a managed analytics table and an external archive table, or to use different ViewDefinitions for different downstream consumers. Each destination operates independently with its own queue, writer, and status. +You can create multiple destinations for the same topic — for example, to materialize the same source resources into different ViewDefinitions for different downstream consumers, or to mirror writes into separate managed tables (different catalogs / schemas). Each destination operates independently with its own queue, writer, and status. ## Retry behavior From a2c2d2f29b33a1cd074be0145d4a4ac4963da2f5 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 18:02:54 +0200 Subject: [PATCH 17/22] docs($viewdefinition-export): multi-pod parallelism + cross-pod status MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace the "Status registry is in-process" MVP limitation with the shipped architecture: canonical state lives in the data-lakehouse backend's Postgres `viewdefinition_export_jobs` table (shared across all Aidbox pods); pod-local cache holds only the echo-only spec fields plus the `kind` needed to route the status defmulti. Polls arriving on a different pod than the kick-off return 404 — call out the hostname-stickiness assumption explicitly so cluster operators configure their LB accordingly. - Document `initialExportParallelism` for `$viewdefinition-export`: the operation reuses the same chunked + per-pod-advisory-locked multi-pod path the AidboxTopicDestination initial export does, so a single export now scales across the whole cluster's cores. Sizing guidance cross-links to the tutorial's existing Large-scale initial export section instead of duplicating the formula. --- docs/modules/sql-on-fhir/operation-viewdefinition-export.md | 4 +++- .../data-lakehouse-aidboxtopicdestination.md | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index 8b4c8abff..3c83c6111 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -153,6 +153,8 @@ On failure the staging table is best-effort dropped, then the export retries up The `MERGE` is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must have an `id` column. {% endhint %} +For large views, set the backend-specific `initialExportParallelism > 1` parameter (default `1`, sequential): the backend hash-partitions `sof.` into `N` chunks, writes them in parallel into per-chunk staging tables (`/chunk-0/`, `/chunk-1/`, …), then materializes the union into the target via one `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL …)`. In a multi-pod Aidbox cluster the workload is shared across pods — each pod claims chunks via Postgres advisory locks, so a single export benefits from the whole cluster's compute. Sizing guidance lives in the tutorial's [Large-scale initial export](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#large-scale-initial-export) section. + The Databricks-side setup (catalog, schema, target table, staging schema, service principal, grants, warehouse) is documented in the [Data Lakehouse Topic Destination tutorial](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) — the same setup is reused here. ## Cloud support @@ -163,5 +165,5 @@ The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=d - One `view` per request (spec allows `1..*`). - `patient` / `group` / `_since` filters extracted but not yet applied to the SQL. -- Status registry is in-process — restarting the Aidbox node loses pending export status. Long-running exports across restarts will be tracked via a FHIR custom resource in a follow-up. +- Restart safety: the canonical job state (status, completed chunks, output location) is persisted by the backend — for the first-party `data-lakehouse` backend, in a Postgres `viewdefinition_export_jobs` table shared across all Aidbox pods. **A poll arriving on a different pod than the one that received the kick-off will return 404**: Aidbox keeps a small per-pod cache of the echo-only spec fields (`clientTrackingId`, `_format`) plus the `kind` needed to route the status dispatch, and that cache is in-process. In practice clients hit the same hostname for the life of an export, and Aidbox cluster load balancers commonly use hostname-sticky routing, so this is usually not visible. A FHIR-resource-backed status (so any pod can answer any poll) is a follow-up. - Cancellation (`cancelUrl`) and `estimatedTimeRemaining` are not implemented. diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 12bda0d6e..33c5d64ad 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1135,6 +1135,8 @@ Prefer: respond-async Returns `202 Accepted` + `Content-Location: /fhir/ViewDefinition/$viewdefinition-export/status/`. Poll that URL until you get `200 OK` with the final `Parameters` output. +For large views add `{"name": "initialExportParallelism", "valueUnsignedInt": }` to the Parameters body — same semantics as the AidboxTopicDestination parameter ([Large-scale initial export](#large-scale-initial-export)). Multi-pod Aidbox clusters cooperate automatically on the same export via Postgres advisory locks. Polling the status URL is hostname-sticky: the kick-off pod owns the small per-pod metadata cache needed to answer status requests, so always poll the same hostname you posted to (in a load-balanced cluster, ensure sticky-by-hostname or session affinity for the `/fhir/ViewDefinition/$viewdefinition-export/status/` path). + See the [`$viewdefinition-export` operation page](../../modules/sql-on-fhir/operation-viewdefinition-export.md) for the full parameter list, status response shape, and current limitations. ## Multiple destinations From 1aab2a94b48caa60cab8f322d12c046e092b54a1 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 18:23:09 +0200 Subject: [PATCH 18/22] docs($viewdefinition-export): multi-pod execution + troubleshooting - Add a "Multi-pod execution" section to the operation page (after "How it works", before "Cloud support") explaining the shipped architecture: canonical state in `tds.viewdefinition_export_jobs`, cross-pod fanout via `cache_replication_msgs` PG NOTIFY (same path as AidboxTopicDestination create/delete), chunk-claim via `pg_try_advisory_lock`, implicit crash recovery via session-level lock auto-release. Includes a sequence diagram showing two pods racing for chunks. Cross-links to the tutorial's existing sizing formula instead of duplicating it. - Promote the 404-on-non-kick-off-pod note from a buried Limitations bullet into a dedicated "Troubleshooting" section with concrete LB guidance (cookie affinity, source-IP hash, honouring Content-Location). Limitations now links to it. - In the tutorial's Ad-hoc one-shot export section, replace the dense parallelism+stickiness paragraph with a "Scaling and multi-pod execution" subsection naming the shipped mechanism explicitly and cross-linking the operation page's Troubleshooting. --- .../operation-viewdefinition-export.md | 42 ++++++++++++++++++- .../data-lakehouse-aidboxtopicdestination.md | 6 ++- 2 files changed, 45 insertions(+), 3 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index 3c83c6111..20d3c3b1c 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -153,17 +153,55 @@ On failure the staging table is best-effort dropped, then the export retries up The `MERGE` is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must have an `id` column. {% endhint %} -For large views, set the backend-specific `initialExportParallelism > 1` parameter (default `1`, sequential): the backend hash-partitions `sof.` into `N` chunks, writes them in parallel into per-chunk staging tables (`/chunk-0/`, `/chunk-1/`, …), then materializes the union into the target via one `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL …)`. In a multi-pod Aidbox cluster the workload is shared across pods — each pod claims chunks via Postgres advisory locks, so a single export benefits from the whole cluster's compute. Sizing guidance lives in the tutorial's [Large-scale initial export](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#large-scale-initial-export) section. +For large views, set the backend-specific `initialExportParallelism > 1` parameter (default `1`, sequential): the backend hash-partitions `sof.` into `N` chunks, writes them in parallel into per-chunk staging tables (`/chunk-0/`, `/chunk-1/`, …), then materializes the union into the target via one `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL …)`. In a multi-pod Aidbox cluster the workload is shared across pods — see [Multi-pod execution](#multi-pod-execution) below. Sizing guidance lives in the tutorial's [Large-scale initial export](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#large-scale-initial-export) section. The Databricks-side setup (catalog, schema, target table, staging schema, service principal, grants, warehouse) is documented in the [Data Lakehouse Topic Destination tutorial](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) — the same setup is reused here. +## Multi-pod execution + +Every Aidbox pod sharing the same PostgreSQL metastore participates in any in-flight export automatically — there's no leader election, no service-discovery, no external coordinator. + +Canonical state for an export lives in a backend-owned PostgreSQL table (`tds.viewdefinition_export_jobs` for `kind=data-lakehouse`, created lazily on first kick-off, sitting alongside the existing `tds.event_storage` that AidboxTopicDestinations use). The pod that received the `POST` inserts the jobs row and broadcasts a `::vd-export-new-job` event on the `cache_replication_msgs` PG NOTIFY channel — the same fan-out infrastructure Aidbox already uses to replicate `AidboxTopicDestination` create/delete across the cluster. Every pod's cache-listener thread picks the event up and spawns local worker threads. + +Workers on every pod then race for chunks via `pg_try_advisory_lock(export-id-hash, chunk-id)`. First worker to claim a given chunk-id runs that chunk; the others see `false` and move on to the next. The lock mechanism is identical to the one `AidboxTopicDestination`'s distributed initial-sync uses; the [tutorial's Large-scale initial export](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#large-scale-initial-export) section has the sizing formula. + +Crash recovery is implicit: PG session-level advisory locks auto-release on connection drop, so when a worker (or its whole pod) dies mid-chunk, a sibling worker on any surviving pod re-claims that chunk on its next loop iteration. + +```mermaid +sequenceDiagram + participant C as Client + participant P1 as Aidbox pod 1
(kick-off pod) + participant PG as PostgreSQL
tds.viewdefinition_export_jobs
+ cache_replication_msgs + participant P2 as Aidbox pod 2 + + C->>P1: POST $viewdefinition-export + P1->>PG: INSERT jobs row + P1->>PG: NOTIFY ::vd-export-new-job + P1-->>C: 202 + Content-Location + PG-->>P1: deliver event + PG-->>P2: deliver event + par chunk race on pg_try_advisory_lock + P1->>PG: claim chunk-0 → write chunk-0 staging + P2->>PG: claim chunk-1 → write chunk-1 staging + end + P1->>PG: claim merge-coordinator lock + P1->>P1: MERGE INTO target USING (UNION ALL stagings) + P1->>PG: UPDATE jobs SET status=completed +``` + ## Cloud support The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=data-lakehouse`, [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md)) currently supports AWS S3 only** for the staging Delta path. **Google Cloud Storage** (`gs://...`) and **Azure ADLS Gen2** (`abfss://...`) are not yet supported — adding them is tracked as a follow-up. The Databricks Unity Catalog managed target table is unaffected (UC manages target storage internally). +## Troubleshooting + +**Status poll returns `404` on a multi-pod cluster.** The canonical jobs row is shared across pods (see [Multi-pod execution](#multi-pod-execution)), but Aidbox additionally keeps a small per-pod in-memory cache of the echo-only spec fields (`clientTrackingId`, `_format`, the registered `kind` needed to route the status dispatch). That cache only lives on the pod that received the original `POST`. A `GET .../status/` arriving on any other pod returns `404`. + +Configure your load balancer with session affinity on the `/fhir/ViewDefinition/$viewdefinition-export/status/` path so clients keep hitting the same pod for the lifetime of the export. The kick-off response's `Content-Location` header already names the hostname the client should stick to — most LBs can be told to honour it (cookie-based affinity in nginx-ingress, source-IP hash, etc.). A FHIR-resource-backed status (so any pod can answer any poll) is tracked as a follow-up. + ## Limitations (current) - One `view` per request (spec allows `1..*`). - `patient` / `group` / `_since` filters extracted but not yet applied to the SQL. -- Restart safety: the canonical job state (status, completed chunks, output location) is persisted by the backend — for the first-party `data-lakehouse` backend, in a Postgres `viewdefinition_export_jobs` table shared across all Aidbox pods. **A poll arriving on a different pod than the one that received the kick-off will return 404**: Aidbox keeps a small per-pod cache of the echo-only spec fields (`clientTrackingId`, `_format`) plus the `kind` needed to route the status dispatch, and that cache is in-process. In practice clients hit the same hostname for the life of an export, and Aidbox cluster load balancers commonly use hostname-sticky routing, so this is usually not visible. A FHIR-resource-backed status (so any pod can answer any poll) is a follow-up. +- Status polling requires LB session affinity on the `Content-Location` hostname — see [Troubleshooting](#troubleshooting). - Cancellation (`cancelUrl`) and `estimatedTimeRemaining` are not implemented. diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index 33c5d64ad..f62f1fa4a 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1135,7 +1135,11 @@ Prefer: respond-async Returns `202 Accepted` + `Content-Location: /fhir/ViewDefinition/$viewdefinition-export/status/`. Poll that URL until you get `200 OK` with the final `Parameters` output. -For large views add `{"name": "initialExportParallelism", "valueUnsignedInt": }` to the Parameters body — same semantics as the AidboxTopicDestination parameter ([Large-scale initial export](#large-scale-initial-export)). Multi-pod Aidbox clusters cooperate automatically on the same export via Postgres advisory locks. Polling the status URL is hostname-sticky: the kick-off pod owns the small per-pod metadata cache needed to answer status requests, so always poll the same hostname you posted to (in a load-balanced cluster, ensure sticky-by-hostname or session affinity for the `/fhir/ViewDefinition/$viewdefinition-export/status/` path). +### Scaling and multi-pod execution + +For large views add `{"name": "initialExportParallelism", "valueUnsignedInt": }` to the Parameters body — same semantics, same sizing formula as the continuous-destination flow's [Large-scale initial export](#large-scale-initial-export). One ad-hoc export benefits from every Aidbox pod connected to the same metastore: the kick-off pod inserts a row into `tds.viewdefinition_export_jobs` and broadcasts a `cache_replication_msgs` PG NOTIFY event; every pod's cache listener picks it up and joins the chunk race via `pg_try_advisory_lock`. No leader, no service-discovery, no per-job pod assignment — identical mechanism to the continuous AidboxTopicDestination initial sync. + +Status polling is hostname-sticky: the kick-off pod keeps a small per-pod cache of the echo-only spec fields needed to assemble the status response, so always poll the same hostname you posted to. In a load-balanced cluster, configure session affinity on `/fhir/ViewDefinition/$viewdefinition-export/status/` (most LBs can honour the `Content-Location` header the kick-off returned). See the [operation page's Troubleshooting](../../modules/sql-on-fhir/operation-viewdefinition-export.md#troubleshooting) for details. See the [`$viewdefinition-export` operation page](../../modules/sql-on-fhir/operation-viewdefinition-export.md) for the full parameter list, status response shape, and current limitations. From 483f1cb66a5a2a4188d7bc431b240cf7dab367b2 Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 18:54:22 +0200 Subject: [PATCH 19/22] docs($viewdefinition-export): switch architecture to async-api + pluggable backend API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operation was re-architected away from a custom multi-pod implementation (module-owned tds.viewdefinition_export_jobs table + pg_try_advisory_lock + cache_replication_msgs NOTIFY fan-out + per-pod *export-meta cache with the 404-on-non-kick-off-pod bug) onto Aidbox's standard async-task engine — the same db-scheduler-backed engine that powers $purge, box.sdc.fhir.workflow, and box.operations. We were reinventing a wheel; async-api gives us cross-pod execution, restart safety, and lease-based crash recovery for free. Job state now lives in shared db_scheduler.scheduled_tasks, so any pod can answer any status poll — the LB-stickiness / cookie-affinity guidance is gone with it. Backend modules now plug in by implementing four mandatory defmultis on their kind (plan-export, setup-export, export-chunk, finalize-export) and one optional one (cancel-export). See aidbox-api io.healthsamurai.topics.api for the contract. - Operation page: rewrite Multi-pod execution section + sequence diagram; drop the 404-on-non-kick-off-pod Troubleshooting entry and the LB session-affinity guidance from Limitations. - Tutorial: simplify the Ad-hoc one-shot export → Scaling and multi-pod subsection — drop the jobs-table / NOTIFY / hostname-stickiness prose and cross-link to the operation page for the orchestration details. The continuous AidboxTopicDestination initial-sync still uses pg_try_advisory_lock — that path is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operation-viewdefinition-export.md | 42 +++++++++---------- .../data-lakehouse-aidboxtopicdestination.md | 4 +- 2 files changed, 21 insertions(+), 25 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index 20d3c3b1c..4341d8fa1 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -159,49 +159,45 @@ The Databricks-side setup (catalog, schema, target table, staging schema, servic ## Multi-pod execution -Every Aidbox pod sharing the same PostgreSQL metastore participates in any in-flight export automatically — there's no leader election, no service-discovery, no external coordinator. +Workflow orchestration runs on Aidbox's standard async-task engine — the same one that powers [`$purge`](../../api/bulk-api/purge.md) and other long-running operations. Job state lives in shared PostgreSQL tables (`db_scheduler.scheduled_tasks` + history), so every pod connected to the same metastore can pick up work and answer status polls. -Canonical state for an export lives in a backend-owned PostgreSQL table (`tds.viewdefinition_export_jobs` for `kind=data-lakehouse`, created lazily on first kick-off, sitting alongside the existing `tds.event_storage` that AidboxTopicDestinations use). The pod that received the `POST` inserts the jobs row and broadcasts a `::vd-export-new-job` event on the `cache_replication_msgs` PG NOTIFY channel — the same fan-out infrastructure Aidbox already uses to replicate `AidboxTopicDestination` create/delete across the cluster. Every pod's cache-listener thread picks the event up and spawns local worker threads. +On kick-off: -Workers on every pod then race for chunks via `pg_try_advisory_lock(export-id-hash, chunk-id)`. First worker to claim a given chunk-id runs that chunk; the others see `false` and move on to the next. The lock mechanism is identical to the one `AidboxTopicDestination`'s distributed initial-sync uses; the [tutorial's Large-scale initial export](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#large-scale-initial-export) section has the sizing formula. +1. The receiving pod synchronously validates inputs and calls the backend's `plan-export` (chunk discovery) and `setup-export` (e.g. allocate per-chunk staging tables) hooks. Config errors surface as `400` here, before any task is scheduled. +2. The pod schedules `N` chunk tasks + 1 supervisor task on the async-task engine and returns `202` to the client. +3. Chunk tasks run concurrently on any pod with a free async-task worker — chunks naturally spread across the cluster. The supervisor task polls chunk completion and dispatches the backend's `finalize-export` hook (for `kind=data-lakehouse`: the final `MERGE INTO target USING (UNION ALL stagings)` + staging drop). -Crash recovery is implicit: PG session-level advisory locks auto-release on connection drop, so when a worker (or its whole pod) dies mid-chunk, a sibling worker on any surviving pod re-claims that chunk on its next loop iteration. +Crash recovery is inherited from the async-task engine: tasks heartbeat into PostgreSQL, and a task whose worker dies is re-leased to another pod after the heartbeat timeout. No bespoke advisory-lock juggling, no `NOTIFY` fan-out, no kick-off-pod affinity. ```mermaid sequenceDiagram participant C as Client - participant P1 as Aidbox pod 1
(kick-off pod) - participant PG as PostgreSQL
tds.viewdefinition_export_jobs
+ cache_replication_msgs + participant P1 as Aidbox pod 1
(kick-off) + participant PG as PostgreSQL
(db_scheduler.scheduled_tasks) participant P2 as Aidbox pod 2 C->>P1: POST $viewdefinition-export - P1->>PG: INSERT jobs row - P1->>PG: NOTIFY ::vd-export-new-job + P1->>P1: plan-export + setup-export (sync) + P1->>PG: schedule N chunk tasks + 1 supervisor P1-->>C: 202 + Content-Location - PG-->>P1: deliver event - PG-->>P2: deliver event - par chunk race on pg_try_advisory_lock - P1->>PG: claim chunk-0 → write chunk-0 staging - P2->>PG: claim chunk-1 → write chunk-1 staging + par chunks lease from PG + P1->>PG: lease chunk-0 → write staging-0 + P2->>PG: lease chunk-1 → write staging-1 end - P1->>PG: claim merge-coordinator lock - P1->>P1: MERGE INTO target USING (UNION ALL stagings) - P1->>PG: UPDATE jobs SET status=completed + P2->>PG: supervisor task wakes → all chunks done + P2->>P2: finalize-export (MERGE + drop stagings) + P2->>PG: mark export completed + C->>P2: GET status/<id> → 200 completed ``` +Backends plug in by implementing four mandatory multimethods (`plan-export`, `setup-export`, `export-chunk`, `finalize-export`) and one optional one (`cancel-export`) — see `aidbox-api` `io.healthsamurai.topics.api` for the contract. + ## Cloud support The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=data-lakehouse`, [`topic-destination-deltalake`](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md)) currently supports AWS S3 only** for the staging Delta path. **Google Cloud Storage** (`gs://...`) and **Azure ADLS Gen2** (`abfss://...`) are not yet supported — adding them is tracked as a follow-up. The Databricks Unity Catalog managed target table is unaffected (UC manages target storage internally). -## Troubleshooting - -**Status poll returns `404` on a multi-pod cluster.** The canonical jobs row is shared across pods (see [Multi-pod execution](#multi-pod-execution)), but Aidbox additionally keeps a small per-pod in-memory cache of the echo-only spec fields (`clientTrackingId`, `_format`, the registered `kind` needed to route the status dispatch). That cache only lives on the pod that received the original `POST`. A `GET .../status/` arriving on any other pod returns `404`. - -Configure your load balancer with session affinity on the `/fhir/ViewDefinition/$viewdefinition-export/status/` path so clients keep hitting the same pod for the lifetime of the export. The kick-off response's `Content-Location` header already names the hostname the client should stick to — most LBs can be told to honour it (cookie-based affinity in nginx-ingress, source-IP hash, etc.). A FHIR-resource-backed status (so any pod can answer any poll) is tracked as a follow-up. - ## Limitations (current) - One `view` per request (spec allows `1..*`). - `patient` / `group` / `_since` filters extracted but not yet applied to the SQL. -- Status polling requires LB session affinity on the `Content-Location` hostname — see [Troubleshooting](#troubleshooting). - Cancellation (`cancelUrl`) and `estimatedTimeRemaining` are not implemented. diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index f62f1fa4a..da9a51dd5 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -1137,9 +1137,9 @@ Returns `202 Accepted` + `Content-Location: /fhir/ViewDefinition/$viewdefinition ### Scaling and multi-pod execution -For large views add `{"name": "initialExportParallelism", "valueUnsignedInt": }` to the Parameters body — same semantics, same sizing formula as the continuous-destination flow's [Large-scale initial export](#large-scale-initial-export). One ad-hoc export benefits from every Aidbox pod connected to the same metastore: the kick-off pod inserts a row into `tds.viewdefinition_export_jobs` and broadcasts a `cache_replication_msgs` PG NOTIFY event; every pod's cache listener picks it up and joins the chunk race via `pg_try_advisory_lock`. No leader, no service-discovery, no per-job pod assignment — identical mechanism to the continuous AidboxTopicDestination initial sync. +For large views add `{"name": "initialExportParallelism", "valueUnsignedInt": }` to the Parameters body — same chunking semantics and sizing formula as the continuous-destination flow's [Large-scale initial export](#large-scale-initial-export). -Status polling is hostname-sticky: the kick-off pod keeps a small per-pod cache of the echo-only spec fields needed to assemble the status response, so always poll the same hostname you posted to. In a load-balanced cluster, configure session affinity on `/fhir/ViewDefinition/$viewdefinition-export/status/` (most LBs can honour the `Content-Location` header the kick-off returned). See the [operation page's Troubleshooting](../../modules/sql-on-fhir/operation-viewdefinition-export.md#troubleshooting) for details. +Chunks run on Aidbox's standard async-task engine, so they spread across every pod sharing the metastore and status polls are answered by any pod. No load-balancer affinity is needed. See the [operation page's Multi-pod execution](../../modules/sql-on-fhir/operation-viewdefinition-export.md#multi-pod-execution) for the orchestration details. See the [`$viewdefinition-export` operation page](../../modules/sql-on-fhir/operation-viewdefinition-export.md) for the full parameter list, status response shape, and current limitations. From 8a953ea91752faebf205e351a78612c6aeac242a Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 19:22:24 +0200 Subject: [PATCH 20/22] docs: drop databricksClient{Id,Secret} per-destination params MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Service-principal credentials now come exclusively from the Aidbox-wide settings (env vars BOX_DATABRICKS_DATA_LAKEHOUSE_CLIENT_ID / _CLIENT_SECRET, or the equivalent `module.databricks.data-lakehouse.*` setting keys). Per-destination overrides on AidboxTopicDestination.parameter[] and per-request fields on the `$viewdefinition-export` body are no longer accepted — a resolved plaintext secret in the destination's `resource` jsonb or in `db_scheduler.scheduled_tasks.task_data` was readable by anyone with PG access. - Drop the `databricksClientId` / `databricksClientSecret` rows from the managed-zerobus and managed-sql parameter tables. - Drop the two cred entries from each example destination body (managed-zerobus + managed-sql). - Drop the "resolve client_secret from External Secrets" inline- vault example (no longer applicable — the secret is set on the Aidbox box, not on the resource). - Drop the External Secrets bullet referencing `databricksClientSecret` from "Related documentation". - Add a callout right under the managed-zerobus example noting the secret lives on the box itself, with the docker-run env snippet showing how to inject it. The system-resources reference dump (`core-module-resources.md`) is auto-regenerated from sansara's IG StructureDefinition — follow-up regen will pick up the schema slice removal there. --- .../data-lakehouse-aidboxtopicdestination.md | 28 +++++-------------- 1 file changed, 7 insertions(+), 21 deletions(-) diff --git a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md index da9a51dd5..127cd0b92 100644 --- a/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md +++ b/docs/tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md @@ -246,8 +246,6 @@ All requests in this tutorial use `Content-Type: application/json`. databricksWorkspaceUrlstringhttps://<workspace>.cloud.databricks.com databricksWorkspaceIdstringNumeric workspace ID (e.g. 1234567890123456). Composes the Zerobus REST endpoint host databricksRegionstringWorkspace AWS region (e.g. us-east-1). Composes the Zerobus REST endpoint host -databricksClientIdstringService principal client_id for OAuth M2M -databricksClientSecretstringService principal client_secret; supports vault refs tableNamestringManaged table full name: catalog.schema.table databricksWarehouseIdstringSQL warehouse ID — used at bootstrap for schema sync + (if initial-export runs) the final MERGE INTO. No warm-warehouse traffic during live writes. awsRegionstringAWS region of the staging bucket @@ -288,8 +286,6 @@ All requests in this tutorial use `Content-Type: application/json`. batchSizeunsignedIntRows per worker tick / batch commit sendIntervalMsunsignedIntMax time between batched commits, in ms databricksWorkspaceUrlstringhttps://<workspace>.cloud.databricks.com -databricksClientIdstringService principal client_id for OAuth M2M -databricksClientSecretstringService principal client_secret; supports vault refs tableNamestringManaged table full name: catalog.schema.table databricksWarehouseIdstringSQL warehouse ID awsRegionstringAWS region of the staging bucket @@ -815,8 +811,6 @@ curl -u root:secret -X POST "$AIDBOX_URL/fhir/AidboxTopicDestination" \ {"name": "databricksWorkspaceUrl", "valueString": "${DATABRICKS_HOST}"}, {"name": "databricksWorkspaceId", "valueString": "${WORKSPACE_ID}"}, {"name": "databricksRegion", "valueString": "${DATABRICKS_REGION}"}, - {"name": "databricksClientId", "valueString": "${SP_CLIENT_ID}"}, - {"name": "databricksClientSecret", "valueString": "${SP_CLIENT_SECRET}"}, {"name": "tableName", "valueString": "${CATALOG}.${TARGET_SCHEMA}.${TARGET_TABLE}"}, {"name": "databricksWarehouseId", "valueString": "${WAREHOUSE_ID}"}, {"name": "awsRegion", "valueString": "${AWS_REGION}"}, @@ -835,21 +829,16 @@ EOF With `initialExportParallelism > 1` the module treats this value as a **base prefix** and writes each chunk into a subfolder `/chunk-/` — registering `chunk-0/` … `chunk-{N-1}/` as separate per-chunk staging Delta tables, then dropping them all after the final MERGE. No customer-side setup change: the `CREATE_TABLE` grant on the staging schema (granted below) covers any number of per-chunk tables, and the chunk subfolders inherit the same External Location and storage credential. {% endhint %} -In production, resolve `databricksClientSecret` from Aidbox's [External Secrets](../../configuration/secret-files.md) instead of inlining it: +Notice the destination does NOT include `databricksClientId` / `databricksClientSecret`. The service-principal credentials live on the Aidbox box itself (env vars `BOX_DATABRICKS_DATA_LAKEHOUSE_CLIENT_ID` and `BOX_DATABRICKS_DATA_LAKEHOUSE_CLIENT_SECRET`, or the corresponding `module.databricks.data-lakehouse.client-id` / `.client-secret` settings) — set them once when you provision the Aidbox cluster: -```json -{ - "name": "databricksClientSecret", - "_valueString": { - "extension": [ - {"url": "http://hl7.org/fhir/StructureDefinition/data-absent-reason", "valueCode": "masked"}, - {"url": "http://health-samurai.io/fhir/secret-reference", "valueString": "dbx-sp-secret"} - ] - } -} +```bash +docker run \ + -e BOX_DATABRICKS_DATA_LAKEHOUSE_CLIENT_ID="${SP_CLIENT_ID}" \ + -e BOX_DATABRICKS_DATA_LAKEHOUSE_CLIENT_SECRET="${SP_CLIENT_SECRET}" \ + ... aidboxone/aidbox:edition-XYZ ``` -`dbx-sp-secret` is a key from your `BOX_VAULT_CONFIG` mapping. Same pattern works for any other credential parameter. +Per-destination overrides are intentionally not accepted: a resolved plaintext secret in `AidboxTopicDestination.resource` (or in `db_scheduler.scheduled_tasks.task_data` for `$viewdefinition-export`) is readable by anyone with PG read access. Aidbox settings keep the secret in the boot-config and the encrypted-at-rest `aidbox-settings` table instead. {% endstep %} @@ -911,8 +900,6 @@ POST /fhir/AidboxTopicDestination "parameter": [ {"name": "writeMode", "valueString": "managed-sql"}, {"name": "databricksWorkspaceUrl", "valueString": "$DATABRICKS_HOST"}, - {"name": "databricksClientId", "valueString": "$SP_CLIENT_ID"}, - {"name": "databricksClientSecret", "valueString": "$SP_CLIENT_SECRET"}, {"name": "tableName", "valueString": "$CATALOG.$TARGET_SCHEMA.$TARGET_TABLE"}, {"name": "databricksWarehouseId", "valueString": "$WAREHOUSE_ID"}, {"name": "awsRegion", "valueString": "$AWS_REGION"}, @@ -1173,7 +1160,6 @@ You can create multiple destinations for the same topic — for example, to mate - [`$materialize` operation](../../modules/sql-on-fhir/operation-materialize.md) - [`$viewdefinition-export` operation](../../modules/sql-on-fhir/operation-viewdefinition-export.md) — the SQL-on-FHIR ad-hoc export this module backs as `kind=data-lakehouse` - [Topic-based Subscriptions](../../modules/topic-based-subscriptions/README.md) -- [External Secrets (Vault)](../../configuration/secret-files.md) — storing sensitive parameters like `databricksClientSecret` as file-backed secrets - [HashiCorp Vault Integration](../../tutorials/other-tutorials/hashicorp-vault-external-secrets.md) — step-by-step tutorial for Kubernetes with Secrets Store CSI Driver - [Azure Key Vault Integration](../../tutorials/other-tutorials/azure-key-vault-external-secrets.md) — step-by-step tutorial for AKS with Azure Key Vault - [Databricks: Predictive Optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization) From 126e7195ce58e6af0618a1818f587d556b97c1fe Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 21:11:40 +0200 Subject: [PATCH 21/22] docs($viewdefinition-export): document DELETE cancel endpoint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to the sansara-side cancel handler (PR #7699 commit ffbfa71c04): document the DELETE route, its three response codes, and what cancellation does + doesn't roll back. Tighten the Limitations bullet — \`cancelUrl\` discovery in the kick-off response body is still unwired, but cancellation itself works. --- .../operation-viewdefinition-export.md | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index 4341d8fa1..d68de3816 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -110,6 +110,23 @@ Completed output for `kind=data-lakehouse`: The `output[].location` URI scheme is backend-specific (`databricks-uc:` for the data-lakehouse backend). +## Cancellation + +```http +DELETE /fhir/ViewDefinition/$viewdefinition-export/status/ +``` + +Stops an in-flight export and triggers backend cleanup (the data-lakehouse backend drops per-chunk staging tables it created). Response codes: + +- `202 Accepted` — cancel acknowledged, async cleanup running. Body reports `status=canceling`; poll the same status URL to see when chunks finish dropping out of `db_scheduler.scheduled_tasks` and the operation reaches a terminal state. +- `200 OK` — operation already terminal (`completed` / `failed`). DELETE is idempotent; the response is the same as a GET on the status URL. +- `404 Not Found` — unknown `export-id` (no tasks in `db_scheduler.scheduled_tasks{,_history}` for that id). + +What cancellation does **not** do: + +- It does not roll back rows already merged into the target managed table. The merge is the last step of finalize-export — if cancel arrives before finalize, no rows have landed; if after finalize, the operation is already terminal and cancel is a no-op. +- It does not delete history rows of chunks that already completed. The audit trail of partial work survives in `db_scheduler.scheduled_tasks_history` (cleaned up by db-scheduler's regular sweep). + ## Failure model - **Input validation failures** (missing `view`, missing `kind`, multiple views, `source` set, etc.) — synchronous `400 OperationOutcome` returned from the kick-off `POST`. No `export-id` is allocated. @@ -200,4 +217,5 @@ The Aidbox-side wiring is cloud-agnostic, but **the first-party backend (`kind=d - One `view` per request (spec allows `1..*`). - `patient` / `group` / `_since` filters extracted but not yet applied to the SQL. -- Cancellation (`cancelUrl`) and `estimatedTimeRemaining` are not implemented. +- `cancelUrl` (the spec's pointer to a cancel endpoint exposed in the kick-off response) is not yet returned. Cancellation itself works — `DELETE` on the status URL is supported (see [Cancellation](#cancellation) above); clients have to know that convention rather than discover it from the response body. +- `estimatedTimeRemaining` is not computed. From 4af8195b77cf5277c814a77637c219c2fe0e059c Mon Sep 17 00:00:00 2001 From: spicyfalafel <58147555+spicyfalafel@users.noreply.github.com> Date: Fri, 22 May 2026 22:36:36 +0200 Subject: [PATCH 22/22] docs($viewdefinition-export): per-chunk staging is the universal path (N=1 included) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous "How it works" section described two architectures side- by-side: a single staging table for the default case, plus a per-chunk variant when initialExportParallelism>1. That was historical accident, not what the code does. The shipped module ALWAYS uses per-chunk staging — N=1 is just the degenerate case (one chunk, no UNION ALL source). Realigning the prose + mermaid avoids future readers inferring two code paths that don't actually exist, and explains why we keep per-chunk even at N=1 (Delta-on-S3 has no atomic put-if-absent on _delta_log/N.json without S3DynamoDBLogStore, so one-writer-per- Delta is required for correctness — see Delta #1830 / #1410). Also notes the HikariCP pool clamp introduced in the module: requesting parallelism beyond the available pool slots is rejected up-front with parallelism-exceeds-pool 400. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operation-viewdefinition-export.md | 32 ++++++++++--------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md index d68de3816..f92517b53 100644 --- a/docs/modules/sql-on-fhir/operation-viewdefinition-export.md +++ b/docs/modules/sql-on-fhir/operation-viewdefinition-export.md @@ -138,39 +138,41 @@ What cancellation does **not** do: ## How it works (`kind=data-lakehouse`) -The first-party backend uses a **staging Delta table** as a relay: it writes the `sof.` rows to an external Delta table at a customer-provided `stagingTablePath` (via Unity Catalog credential vending), then `MERGE INTO`s the managed target, then drops the staging table. Same flow for `writeMode=managed-zerobus` and `writeMode=managed-sql`. +The first-party backend uses **per-chunk staging Delta tables** as a relay. With `initialExportParallelism = N` (default `N=1`), the backend hash-partitions `sof.` into `N` chunks, writes each chunk into its own external Delta table under `/chunk-K/`, then materializes the union into the managed target with one `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL …)` against the SQL warehouse and drops every staging. The `N=1` case is just the degenerate path of that flow — one chunk, no `UNION ALL` source — so a single architecture covers both modes. Same flow for `writeMode=managed-zerobus` and `writeMode=managed-sql`. ```mermaid graph LR PG[(Aidbox PostgreSQL
sof.<view>)]:::neutral2 - M[Aidbox sender]:::blue2 - Staging[Staging external Delta table
on stagingTablePath]:::yellow2 + M[Aidbox sender N pods]:::blue2 + S0[Staging chunk-0]:::yellow2 + SN[Staging chunk-N-1]:::yellow2 WH[Databricks SQL warehouse]:::green2 Target[(Unity Catalog managed Delta target)]:::violet2 - M -- 1. read rows --> PG - M -- 2. write Parquet + Delta commit
via Unity-Catalog-vended STS --> Staging - M -- 3. MERGE INTO target USING staging ON id
WHEN NOT MATCHED THEN INSERT * --> WH - WH -- 4. read --> Staging + M -- 1. read chunk rows
abs hashtext id mod N = K --> PG + M -- 2. write Parquet + Delta commit
via UC-vended STS, per chunk --> S0 + M -- 2. write Parquet + Delta commit
via UC-vended STS, per chunk --> SN + M -- 3. MERGE INTO target USING
UNION ALL stagings ON id
WHEN NOT MATCHED THEN INSERT * --> WH + WH -- 4. read all stagings --> S0 + WH -- 4. read all stagings --> SN WH -- 5. write --> Target - M -- 6. DROP TABLE staging --> WH + M -- 6. DROP TABLE every staging --> WH ``` Steps in detail: -1. Register a temporary external Delta table at `stagingTablePath` with the same schema as the SQL-on-FHIR materialized view (`sof.` in Aidbox's PostgreSQL). -2. Unity Catalog vends short-lived STS credentials for the staging path. -3. The module writes all `sof.` rows to the staging path as one Delta commit. -4. The module issues `MERGE INTO {managed_target} USING {staging} ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` against the SQL warehouse. The MERGE reads the staging Delta snapshot through the Delta protocol and inserts any rows whose `id` is not yet present in the target. -5. The module drops the staging table. +1. `plan-export` decides the chunk count from `initialExportParallelism`. `setup-export` syncs the target schema against `sof.` (auto-`ALTER ADD COLUMNS` if Aidbox added columns) and pre-computes the staging-column spec so chunks don't re-describe `sof.`. +2. Each chunk task (one of `N`, possibly on different Aidbox pods) creates its own external Delta table at `/chunk-K/`, vends short-lived STS credentials from Unity Catalog for that prefix, and streams its hash-partition of `sof.` (`abs(hashtext(id::text)) % N = K`) as one Delta commit. +3. Once all `N` chunks complete, the supervisor task invokes `finalize-export`. The module issues a single `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL …) ON t.id = s.id WHEN NOT MATCHED THEN INSERT *` to the SQL warehouse — Databricks `UNION ALL` planning is effectively free, so cost scales with row count, not chunk count. +4. The module drops every staging table. -On failure the staging table is best-effort dropped, then the export retries up to 3 times with exponential backoff (1s → 2s → 4s). +On failure the per-chunk stagings are best-effort dropped via `cancel-export`, then the chunk's async-task retries up to 2 times with a 30-second backoff. The supervisor task itself doesn't retry — a chunk-failure cascade is surfaced as `failed` status, with the failing task's error message echoed back to the caller. {% hint style="info" %} The `MERGE` is idempotent on `id` — a retried export after a lost response inserts nothing instead of duplicating. Your ViewDefinition must have an `id` column. {% endhint %} -For large views, set the backend-specific `initialExportParallelism > 1` parameter (default `1`, sequential): the backend hash-partitions `sof.` into `N` chunks, writes them in parallel into per-chunk staging tables (`/chunk-0/`, `/chunk-1/`, …), then materializes the union into the target via one `MERGE INTO target USING (SELECT * FROM staging_0 UNION ALL …)`. In a multi-pod Aidbox cluster the workload is shared across pods — see [Multi-pod execution](#multi-pod-execution) below. Sizing guidance lives in the tutorial's [Large-scale initial export](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#large-scale-initial-export) section. +Why per-chunk staging even at `N=1`? Delta-on-S3 has no atomic put-if-absent on `_delta_log/N.json` without an S3DynamoDBLogStore, so two writers committing to one staging Delta race and lose commits silently ([Delta #1830](https://github.com/delta-io/delta/issues/1830), [#1410](https://github.com/delta-io/delta/issues/1410)). Per-chunk staging = exactly one writer per Delta table = no race. The MERGE source picks up where Delta left off via the standard snapshot read. Sizing guidance lives in the tutorial's [Large-scale initial export](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md#large-scale-initial-export) section. The per-chunk Aidbox PG load scales with `N` — request more than your HikariCP pool can spare and `plan-export` returns a `400` with `parallelism-exceeds-pool` before scheduling any work. The Databricks-side setup (catalog, schema, target table, staging schema, service principal, grants, warehouse) is documented in the [Data Lakehouse Topic Destination tutorial](../../tutorials/subscriptions-tutorials/data-lakehouse-aidboxtopicdestination.md) — the same setup is reused here.