Skip to content

feat(dbt): schema_suffix namespacing for StarRocks b2b_analytics dev/CI targets#2455

Open
blarghmatey wants to merge 3 commits into
mainfrom
feat/starrocks-schema-suffix-namespacing
Open

feat(dbt): schema_suffix namespacing for StarRocks b2b_analytics dev/CI targets#2455
blarghmatey wants to merge 3 commits into
mainfrom
feat/starrocks-schema-suffix-namespacing

Conversation

@blarghmatey

Copy link
Copy Markdown
Member

What & why

StarRocks dbt targets hardcoded schema b2b_analytics with no per-run namespacing, so interactive dev runs and concurrent CI PR builds against the shared StarRocks cluster collided with each other and with the real qa-tier data. This mirrors the existing Trino dev_qa/dev_production schema-suffix pattern for StarRocks.

Part of the dbt warehouse CI/QA hardening & data-trust effort (tracking issue #2407); this is the P2a enabler that unblocks a slim, per-PR StarRocks data CI (each PR builds into b2b_analytics_pr_<n>).

Changes

  • profiles.yml — add starrocks_dev_qa_vault and starrocks_ci, the namespaced counterparts of starrocks_qa_vault/starrocks_production. Their schema appends _{{ var("schema_suffix") }} (defaults to dev via dbt_project.yml; CI passes --vars '{"schema_suffix": "pr_<n>"}'). The bare targets stay unsuffixed for scheduled/production builds.
  • dbt_starrocks.pySTARROCKS_DBT_TARGET_MAP: dev → starrocks_dev_qa_vault, ci → starrocks_ci (qa/production unchanged).
  • starrocks.py_ENVS qa/ci dbt_target repointed to the namespaced targets, since the ol-dbt starrocks interactive/CI commands must not write into the scheduled schema.
  • test_starrocks.py — dropped the assertion that encoded the collision (ci target == production target); added a test asserting dev/ci are namespaced distinctly from production.
  • _b2b_analytics__models.yml — point the example --target at the dev target.

Testing

  • uv run pytest in src/ol_dbt_cli: 356 passed (StarRocks target/env tests updated)
  • pre-commit (yamlfmt, yamllint, ruff, mypy) clean
  • Verified the folded YAML scalar renders the intended Jinja: {{ env_var("DBT_STARROCKS_SCHEMA", "b2b_analytics") }}_{{ var("schema_suffix") }}

Live StarRocks connectivity is not exercised here (credential-free change); the target wiring and schema templating are covered by unit tests, matching how the bare targets are validated.

Note

Originally implemented pre-emptively but held until #2329 (feat/dbt-starrocks-adapter) landed, since it builds on those targets. #2329 is now merged; this reconciles the change against current main.

🤖 Generated with Claude Code

…CI targets

StarRocks targets hardcoded schema `b2b_analytics` with no per-run namespacing,
so interactive dev runs and concurrent CI PR builds against the shared cluster
collided with each other and with the real qa-tier data. This mirrors the Trino
dev_qa/dev_production pattern for StarRocks (P2a; unblocks Phase-2 slim data CI).

- profiles.yml: add `starrocks_dev_qa_vault` and `starrocks_ci` targets — the
  namespaced counterparts of `starrocks_qa_vault`/`starrocks_production`, whose
  schema appends `_{{ var("schema_suffix") }}` (defaults to `dev`; CI passes
  --vars '{"schema_suffix": "pr_<n>"}'). The bare targets stay unsuffixed for
  scheduled/production builds.
- dbt_starrocks.py: STARROCKS_DBT_TARGET_MAP dev→starrocks_dev_qa_vault,
  ci→starrocks_ci (qa/production unchanged).
- starrocks.py: _ENVS qa/ci dbt_target repointed to the namespaced targets — the
  `ol-dbt starrocks` interactive/CI commands must not write the scheduled schema.
- test_starrocks.py: dropped the assertion that encoded the collision
  (ci target == production target); added a test that dev/ci are namespaced
  distinctly from production.
- b2b_analytics model doc: point the example --target at the dev target.

356 ol_dbt_cli tests pass, pre-commit (yamlfmt/yamllint/ruff/mypy) clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 15, 2026 20:54
@github-actions

Copy link
Copy Markdown

🔎 ol-dbt impact — column-level blast radius

✅ No column-level downstream impact detected for the changed models.

Posted by ol-dbt impact (annotate-only — does not block merge).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds schema_suffix-based namespacing for StarRocks dbt dev/CI targets so concurrent interactive runs and per-PR CI builds don’t collide on the shared b2b_analytics schema, aligning StarRocks behavior with the existing Trino schema-suffix pattern.

Changes:

  • Added namespaced StarRocks dbt targets (starrocks_dev_qa_vault, starrocks_ci) that append _{{ var("schema_suffix") }} to the schema.
  • Updated StarRocks target resolution/wiring so dev/ci use the namespaced targets while scheduled/production targets remain unsuffixed.
  • Updated tests and model docs to reflect the new targets and to avoid encoding the previous collision behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/ol_dbt/profiles.yml Adds namespaced StarRocks targets that append schema_suffix to the schema to prevent collisions.
src/ol_dbt/models/b2b_analytics/_b2b_analytics__models.yml Updates the smoke-test example target to use the namespaced dev target.
src/ol_dbt_cli/tests/test_starrocks.py Adjusts assertions to ensure qa/ci targets differ from production (no collision).
src/ol_dbt_cli/ol_dbt_cli/commands/starrocks.py Repoints qa and ci env configs to namespaced dbt targets for interactive/CI usage.
dg_projects/lakehouse/lakehouse/assets/lakehouse/dbt_starrocks.py Updates Dagster’s StarRocks dbt target map so dev/ci use namespaced targets.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/ol_dbt/models/b2b_analytics/_b2b_analytics__models.yml Outdated
Comment thread src/ol_dbt_cli/tests/test_starrocks.py Outdated
- b2b_analytics model doc: the smoke-test description still claimed the
  table lands in the bare "b2b_analytics" database; document that
  starrocks_dev_qa_vault/starrocks_ci write to the namespaced
  b2b_analytics_<schema_suffix> schema instead.
- test_starrocks.py: rename test_dev_and_ci_targets_are_schema_suffix_namespaced
  to test_qa_and_ci_targets_are_schema_suffix_namespaced and fix its docstring —
  the test asserts qa/ci differ from production (there is no "dev" key in
  _ENVS), and qa is itself namespaced now, not a bare shared-schema target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment thread src/ol_dbt/profiles.yml Outdated
profiles.yml is rendered before dbt_project.yml's `vars:` block is
available, so var("schema_suffix") with no default fails immediately —
"Required var 'schema_suffix' not found in config" — for anyone invoking
starrocks_dev_qa_vault/starrocks_ci without explicitly passing
--vars '{"schema_suffix": "..."}'. Verified empirically: `dbt parse
--target starrocks_dev_qa_vault` failed with exactly that error before this
fix, and succeeds after; --vars override still works for CI's per-PR suffix.

Default to "dev", matching dbt_project.yml's own default for the var.

Note: the same var("schema_suffix")-without-default pattern already exists
in the pre-existing Trino dev_qa/dev_production targets in this same file
(this PR was written to mirror that pattern) and has the identical latent
bug — confirmed via the same dbt parse test. Left those alone; that's
pre-existing behavior outside this PR's scope, not something introduced
here. Flagging for a separate follow-up.

Caught by Sentry bug-prediction review on #2455 (CRITICAL).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment on lines 38 to 42
# connection shape as production -- matches _ENVS["ci"]["dbt_target"].
"ci": "starrocks_production",
"ci": "starrocks_ci",
"qa": "starrocks_qa_vault",
"production": "starrocks_production",
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The starrocks_dbt_assets asset fails to pass the schema_suffix to its dbt command, causing all CI builds to collide by writing to the same default _dev schema.
Severity: HIGH

Suggested Fix

In dbt_starrocks.py, update the starrocks_dbt.cli invocation to include the --vars argument with the schema_suffix. The suffix should be determined based on the environment, similar to the logic used for the Trino asset, ensuring a unique schema is used for each CI run.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: dg_projects/lakehouse/lakehouse/assets/lakehouse/dbt_starrocks.py#L38-L42

Potential issue: The `starrocks_dbt_assets` Dagster asset does not pass the
`schema_suffix` variable to its `dbt build` subprocess. The dbt profile for StarRocks is
configured to construct the schema name using this variable, with a default value of
`"dev"`. When running in a CI environment, the asset is intended to receive a unique
suffix (e.g., `pr_<n>`) to ensure build isolation. However, because the variable is not
passed, dbt falls back to the default. This will cause all concurrent CI builds to write
to the same `_dev` schema, leading to data collisions and negating the purpose of per-PR
namespacing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants