feat(dbt): schema_suffix namespacing for StarRocks b2b_analytics dev/CI targets#2455
feat(dbt): schema_suffix namespacing for StarRocks b2b_analytics dev/CI targets#2455blarghmatey wants to merge 3 commits into
Conversation
…CI targets
StarRocks targets hardcoded schema `b2b_analytics` with no per-run namespacing,
so interactive dev runs and concurrent CI PR builds against the shared cluster
collided with each other and with the real qa-tier data. This mirrors the Trino
dev_qa/dev_production pattern for StarRocks (P2a; unblocks Phase-2 slim data CI).
- profiles.yml: add `starrocks_dev_qa_vault` and `starrocks_ci` targets — the
namespaced counterparts of `starrocks_qa_vault`/`starrocks_production`, whose
schema appends `_{{ var("schema_suffix") }}` (defaults to `dev`; CI passes
--vars '{"schema_suffix": "pr_<n>"}'). The bare targets stay unsuffixed for
scheduled/production builds.
- dbt_starrocks.py: STARROCKS_DBT_TARGET_MAP dev→starrocks_dev_qa_vault,
ci→starrocks_ci (qa/production unchanged).
- starrocks.py: _ENVS qa/ci dbt_target repointed to the namespaced targets — the
`ol-dbt starrocks` interactive/CI commands must not write the scheduled schema.
- test_starrocks.py: dropped the assertion that encoded the collision
(ci target == production target); added a test that dev/ci are namespaced
distinctly from production.
- b2b_analytics model doc: point the example --target at the dev target.
356 ol_dbt_cli tests pass, pre-commit (yamlfmt/yamllint/ruff/mypy) clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🔎 ol-dbt impact — column-level blast radius✅ No column-level downstream impact detected for the changed models. Posted by |
There was a problem hiding this comment.
Pull request overview
This PR adds schema_suffix-based namespacing for StarRocks dbt dev/CI targets so concurrent interactive runs and per-PR CI builds don’t collide on the shared b2b_analytics schema, aligning StarRocks behavior with the existing Trino schema-suffix pattern.
Changes:
- Added namespaced StarRocks dbt targets (
starrocks_dev_qa_vault,starrocks_ci) that append_{{ var("schema_suffix") }}to the schema. - Updated StarRocks target resolution/wiring so
dev/ciuse the namespaced targets while scheduled/production targets remain unsuffixed. - Updated tests and model docs to reflect the new targets and to avoid encoding the previous collision behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/ol_dbt/profiles.yml | Adds namespaced StarRocks targets that append schema_suffix to the schema to prevent collisions. |
| src/ol_dbt/models/b2b_analytics/_b2b_analytics__models.yml | Updates the smoke-test example target to use the namespaced dev target. |
| src/ol_dbt_cli/tests/test_starrocks.py | Adjusts assertions to ensure qa/ci targets differ from production (no collision). |
| src/ol_dbt_cli/ol_dbt_cli/commands/starrocks.py | Repoints qa and ci env configs to namespaced dbt targets for interactive/CI usage. |
| dg_projects/lakehouse/lakehouse/assets/lakehouse/dbt_starrocks.py | Updates Dagster’s StarRocks dbt target map so dev/ci use namespaced targets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- b2b_analytics model doc: the smoke-test description still claimed the table lands in the bare "b2b_analytics" database; document that starrocks_dev_qa_vault/starrocks_ci write to the namespaced b2b_analytics_<schema_suffix> schema instead. - test_starrocks.py: rename test_dev_and_ci_targets_are_schema_suffix_namespaced to test_qa_and_ci_targets_are_schema_suffix_namespaced and fix its docstring — the test asserts qa/ci differ from production (there is no "dev" key in _ENVS), and qa is itself namespaced now, not a bare shared-schema target. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
profiles.yml is rendered before dbt_project.yml's `vars:` block is
available, so var("schema_suffix") with no default fails immediately —
"Required var 'schema_suffix' not found in config" — for anyone invoking
starrocks_dev_qa_vault/starrocks_ci without explicitly passing
--vars '{"schema_suffix": "..."}'. Verified empirically: `dbt parse
--target starrocks_dev_qa_vault` failed with exactly that error before this
fix, and succeeds after; --vars override still works for CI's per-PR suffix.
Default to "dev", matching dbt_project.yml's own default for the var.
Note: the same var("schema_suffix")-without-default pattern already exists
in the pre-existing Trino dev_qa/dev_production targets in this same file
(this PR was written to mirror that pattern) and has the identical latent
bug — confirmed via the same dbt parse test. Left those alone; that's
pre-existing behavior outside this PR's scope, not something introduced
here. Flagging for a separate follow-up.
Caught by Sentry bug-prediction review on #2455 (CRITICAL).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
| # connection shape as production -- matches _ENVS["ci"]["dbt_target"]. | ||
| "ci": "starrocks_production", | ||
| "ci": "starrocks_ci", | ||
| "qa": "starrocks_qa_vault", | ||
| "production": "starrocks_production", | ||
| } |
There was a problem hiding this comment.
Bug: The starrocks_dbt_assets asset fails to pass the schema_suffix to its dbt command, causing all CI builds to collide by writing to the same default _dev schema.
Severity: HIGH
Suggested Fix
In dbt_starrocks.py, update the starrocks_dbt.cli invocation to include the --vars argument with the schema_suffix. The suffix should be determined based on the environment, similar to the logic used for the Trino asset, ensuring a unique schema is used for each CI run.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: dg_projects/lakehouse/lakehouse/assets/lakehouse/dbt_starrocks.py#L38-L42
Potential issue: The `starrocks_dbt_assets` Dagster asset does not pass the
`schema_suffix` variable to its `dbt build` subprocess. The dbt profile for StarRocks is
configured to construct the schema name using this variable, with a default value of
`"dev"`. When running in a CI environment, the asset is intended to receive a unique
suffix (e.g., `pr_<n>`) to ensure build isolation. However, because the variable is not
passed, dbt falls back to the default. This will cause all concurrent CI builds to write
to the same `_dev` schema, leading to data collisions and negating the purpose of per-PR
namespacing.
What & why
StarRocks dbt targets hardcoded schema
b2b_analyticswith no per-run namespacing, so interactive dev runs and concurrent CI PR builds against the shared StarRocks cluster collided with each other and with the real qa-tier data. This mirrors the existing Trinodev_qa/dev_productionschema-suffix pattern for StarRocks.Part of the dbt warehouse CI/QA hardening & data-trust effort (tracking issue #2407); this is the P2a enabler that unblocks a slim, per-PR StarRocks data CI (each PR builds into
b2b_analytics_pr_<n>).Changes
profiles.yml— addstarrocks_dev_qa_vaultandstarrocks_ci, the namespaced counterparts ofstarrocks_qa_vault/starrocks_production. Theirschemaappends_{{ var("schema_suffix") }}(defaults todevviadbt_project.yml; CI passes--vars '{"schema_suffix": "pr_<n>"}'). The bare targets stay unsuffixed for scheduled/production builds.dbt_starrocks.py—STARROCKS_DBT_TARGET_MAP:dev → starrocks_dev_qa_vault,ci → starrocks_ci(qa/productionunchanged).starrocks.py—_ENVSqa/cidbt_targetrepointed to the namespaced targets, since theol-dbt starrocksinteractive/CI commands must not write into the scheduled schema.test_starrocks.py— dropped the assertion that encoded the collision (citarget ==productiontarget); added a test assertingdev/ciare namespaced distinctly fromproduction._b2b_analytics__models.yml— point the example--targetat the dev target.Testing
uv run pytestinsrc/ol_dbt_cli: 356 passed (StarRocks target/env tests updated){{ env_var("DBT_STARROCKS_SCHEMA", "b2b_analytics") }}_{{ var("schema_suffix") }}Live StarRocks connectivity is not exercised here (credential-free change); the target wiring and schema templating are covered by unit tests, matching how the bare targets are validated.
Note
Originally implemented pre-emptively but held until #2329 (
feat/dbt-starrocks-adapter) landed, since it builds on those targets. #2329 is now merged; this reconciles the change against currentmain.🤖 Generated with Claude Code