Skip to content

feat: Expand dbt version support (1.7.x - 1.10.x) and fix dbt 1.8+ compatibility#127

Merged
dgniewek merged 2 commits intodevelopfrom
feature/support_for_other_dbt_versions
Nov 12, 2025
Merged

feat: Expand dbt version support (1.7.x - 1.10.x) and fix dbt 1.8+ compatibility#127
dgniewek merged 2 commits intodevelopfrom
feature/support_for_other_dbt_versions

Conversation

@dgniewek
Copy link
Collaborator

@dgniewek dgniewek commented Nov 7, 2025

Expand dbt version support (1.7.x - 1.10.x) and fix dbt 1.8+ compatibility

Changes

dbt Version Support Expansion

  • Expanded all adapter version ranges: >=1.7.x,<2.0.0 (Snowflake, BigQuery, Postgres, Redshift, Glue)
  • Removed dbt-core from base requirements (adapters provide it transitively)
  • Added Snowflake adapter to test matrix (tox.ini)
  • Relaxed Jinja2 constraint: ==3.1.2 → >=3.1.3,<4

dbt 1.8+ Compatibility Fix

  • Critical: Refactored publish.py to parse manifest.json as plain JSON instead of using dbt Python API
  • Removed imports: dbt.contracts.graph.manifest.Manifest, dbt.contracts.graph.nodes.*
  • Added defensive error handling with .get() for missing keys
  • Works across dbt 1.7.x through 1.10.x (verified with 70 test executions)

Test Coverage

  • Added 17 comprehensive unit tests for manifest parsing:
    • P0 (6 tests): Critical error cases (missing nodes, corrupt JSON, invalid structure)
    • P1 (4 tests): Important scenarios (full metadata, empty columns, multiple models)
    • P2 (7 tests): Edge cases (None values, mixed resource types, empty manifests)
  • All tests use plain JSON dicts instead of dbt Python objects

Documentation

  • Updated coverage badge (CodeClimate → shields.io, 95%)
  • Added troubleshooting section for dbt pre-release edge case
  • Created design doc: design/001-dbt-manifest-api-migration.md (complete ADR with test matrix)
  • Updated AGENTS.md with dbt 1.8+ fix details and PEP 440 pre-release behavior

Dependency Cleanup

  • Removed MarkupSafe pin (managed by Jinja2)
  • Removed Werkzeug (unused)
  • Fixed GitPython optional import

Breaking Changes

None. Users must continue installing with adapter extras:

pip install data-pipelines-cli[snowflake]  # Required (no change)

@dgniewek dgniewek requested review from kosmag and p-pekala November 7, 2025 10:41
@dgniewek dgniewek merged commit 3277ef8 into develop Nov 12, 2025
4 checks passed
@dgniewek dgniewek deleted the feature/support_for_other_dbt_versions branch November 12, 2025 14:56
dgniewek added a commit that referenced this pull request Nov 12, 2025
@github-actions @p-pekala
@pawelpinkos @Santhin @PiotrSierkin-Ki @paszynski @dependabot @jszafran @michalrudko @ptutak-getindata @dgniewek
Release 0.32.0 (#128)
24f7541 
* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* feat: looker integration for compile and deploy, new command: bi

* tests fix, looker git with ssh

* chore: organizing imports, removed not used variables

* chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path

* chore: formatting, types-requests dependency for mypy

* refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option

* Fix after rebase

* Fix after rebase

* Fix after rebase

* Fix after rebase

* Airbyte integration (#86)

* [DATA-366] Update datahub package and add dbt extra (#79)

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Documentation improvement (#80)

* Added IAP Airbyte authorization with service account file

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com>
Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com>
Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com>
Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com>

* Service account OIDC ID token in deploy command (#91)

* Added possibility to pass platform-agnostic service account OIDC ID token to deploy command

* Added changelog information

* added 'source freshness' command to dbt compile

* Bump copier (#93)

* [DATA-366] Update datahub package and add dbt extra (#79)

* feat: update datahub package and add dbt extra

* fix: replace datahub requirements

* fix: pre-commit coma

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Documentation improvement (#80)

* Documentation improvement

* dp compile default environment has been set to local

* GitPython is not required anymore (#83)

Installation docs upgrade

* chore: bump datahub (#84)

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* [DATA-366] Update datahub package and add dbt extra (#79)

* feat: update datahub package and add dbt extra

* fix: replace datahub requirements

* fix: pre-commit coma

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* chore: bump packages except copier

* fix: package name

* chore: add specific version for pyyaml

* fix: misspelling

* chore: test docs

* chore: test

* test

* test

* test

* test

* test

* test

* chore: bump packages

* chore: add renovate

* chore: bump copier

* chore: update exception

* fix: update by removing not existing flag

* chore: update tests

* feat: update schema on manifest to v7

* chore: bump packages

* chore: delete renovate

* chore: bump python version

* chore: remove junk

* chore: bring back old exception

* chore: add matrix to tox

* chore: add new require and classifier

* trigger

* chore(docs): update badge and changelog

* chore(docs): replace old extension

* chore: add description

* chore(docs): add old suffix and envops

* chore(docs): update changelog

* chore(docs): update changelog

* chore(docs): update copier

* chore: pinup version for dbt2looker

* chore(docs): update changelog

* chore: replace var for more meaningful

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com>

* chore: downgrade version to properly prepare release

* chore(docs): fix date in changelog

* BI deployment is disabled when config is missing (#95)

* DATA-629: Added Airbyte configuration documentation (#97)

* feat: disable sync bucket (#102)

* chore: add flag to turn off bucket syncing with artefacts

* chore: replace default value

* chore(docs): updat changelog

* fix: add readthedocs config file

* chore: repair failing tests

* feat: add test for sync_bucket

* chore(docs): cosmetic changes in changelog

* chore: rename `sync-bucket` to `disable-bucket-sync`

* chore(docs): update with flag

* chore: replace `sync_bucket` with `disable_bucket_sync`

* chore: update name for `_bucket_sync`

---------

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Fix failing readthedocs builds (#104)

* Removed sphinx action from prepare-release pipeline (#105)

* Bump werkzeug from 2.2.2 to 2.2.3 (#106)

* Bump werkzeug from 2.2.2 to 2.2.3

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@2.2.2...2.2.3)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* bump codeclimate-action to 4.0.0

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com>

* Airbyte fix creating multiple connections (#109)

* Fixing tests.

* Fix failing test.

* Remove redundant line.

* Add info about fix to CHANGELOG.md.

* Add workspace_id entry to config.

* Get rid of walrus operator & add test for missing workspace id.

* Fix wrong connection id in Airflow ingestion task.

* Fix typo in test name.

* Add fetching default workspace if not available in config.

* Increasign test coverage.

* Update workspace_id configuration description.

* Bumping Python version to 3.8 in prepare-release workflow.

* Pin pydantic version (#112)

* Pin Pydantic version until Copier fixes their dependency.

* Add info to changelog.

* Upgrade dbt to 1.5.4 (#114)

* Upgrade dbt to 1.5.4

* Fixing imports for publish.py

* Update of dbt_schema_version

* Updating manifest.json for tests

* Fix manifest for testing

* Fixing manifest cntd.

* Fixing manifest cntd.

* Replace example-project with exampleproject

* Adding tutorial for upgrading manifest in tests cntd.

* Fixing typos in the docs

* Version fixes before release

---------

Co-authored-by: Michal Rudko <michal.rudko@getindata.com>

* Databricks generate job (#116)

* new feature

* small fixes

* dbt databricks factory requirement

* python 3.9 onwards

* testing

* linting

* annotations fix

* linting

* no docs

* linting

* update docs

* testing

* tests with databricks

* testenv

* Upgrade pyyaml

* Dbt glue (#118)

* DBT glue

* DBT glue

* DBT glue

* DBT glue

* Dbt glue with spark (#120)

* Dbt glue with spark

* Fixing dp cli

* Dbt upgtade (#122)

* Dbt upgrade to 1.7

* Dbt upgrade to 1.7

* Fixing dp cli

* Upgrade versions

* Upgrade versions

* feat: Add support for python 3.11 and 3.12 (#125)

# Add Python 3.11 and 3.12 Support

  ## Summary
  Adds full support for Python 3.11 and 3.12 while maintaining Python 3.9/3.10 compatibility. Python 3.9 is deprecated (EOL Oct 2025) with support ending April 30, 2026.

  ## Changes

  **Dependencies**
  - Removed `pydantic<2` (unused, blocked Python 3.12)
  - Updated `packaging` to `>=23.0` (setuptools compatibility)
  - Updated `fsspec`, `gcsfs`, `s3fs` to `>=2024.6.0,<2025.0.0`
  - Added `setuptools<75.0.0` constraint in tox/requirements-dev

  **Testing**
  - Configured tox for `py39`, `py310`, `py311`, `py312`
  - All 112 tests pass on Python 3.10-3.12 (96% coverage)
  - Updated pre-commit: flake8 6.1.0, mypy 1.7.1

  **Code Quality**
  - Fixed exception handling: `ClientError` instead of bare `Exception`
  - Removed unnecessary `type: ignore` comments
  - Fixed line length violations

  **Documentation**
  - Updated `CONTRIBUTING.md` with dev setup and tox usage
  - Updated `.pre-commit-config.yaml` tool versions

  ## Testing
  ```bash
  tox              # All versions
  tox -e py312     # Specific version

  Results: 112 tests passed, 96% coverage on Python 3.9-3.12

  Release

  Version: 0.31.0 Type: Minor (non-breaking, adds Python 3.11/3.12 support) Target Branch: develop → main

  Breaking Changes

  None. Fully backward compatible.

* fix: rollback manual version bump and changelog edit

* fix: rollback another manual change in dp version

* ci: rollback missing default CHANGELOG configuration that was manually edited

* ci: fix release pipeline, add permissions to GH token

* ci: fix release pipeline, add permissions to GH token

* ci: fix publish pipeline - pin a proper version

* feat: Expand dbt version support (1.7.x - 1.10.x) and fix dbt 1.8+ compatibility (#127)

* feat: support other dbt versions

* chore: fix linter issue

* FIX #128  - Bump version and CHANGELOG for release 0.32.0

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com>
Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com>
Co-authored-by: Piotr Sierkin <psierkin@gmail.com>
Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com>
Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com>
Co-authored-by: jszafran <jszafran.pv@gmail.com>
Co-authored-by: michalrudko <michal.rudko@gmail.com>
Co-authored-by: Michal Rudko <michal.rudko@getindata.com>
Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com>
Co-authored-by: Dominik Gniewek-Węgrzyn <47598580+dgniewek@users.noreply.github.com>
Co-authored-by: Dominik Gniewek-Węgrzyn <dominik.gniewek@getindata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kosmag kosmag kosmag approved these changes

@p-pekala p-pekala Awaiting requested review from p-pekala

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dgniewek @kosmag