feat: Add support for python 3.11 and 3.12 by dgniewek · Pull Request #125 · getindata/data-pipelines-cli

dgniewek · 2025-10-31T14:15:34Z

Add Python 3.11 and 3.12 Support

Summary

Adds full support for Python 3.11 and 3.12 while maintaining Python 3.9/3.10 compatibility. Python 3.9 is deprecated (EOL Oct 2025) with support ending April 30, 2026.

Changes

Dependencies

Removed pydantic<2 (unused, blocked Python 3.12)
Updated packaging to >=23.0 (setuptools compatibility)
Updated fsspec, gcsfs, s3fs to >=2024.6.0,<2025.0.0
Added setuptools<75.0.0 constraint in tox/requirements-dev

Testing

Configured tox for py39, py310, py311, py312
All 112 tests pass on Python 3.10-3.12 (96% coverage)
Updated pre-commit: flake8 6.1.0, mypy 1.7.1

Code Quality

Fixed exception handling: ClientError instead of bare Exception
Removed unnecessary type: ignore comments
Fixed line length violations

Documentation

Updated CONTRIBUTING.md with dev setup and tox usage
Updated .pre-commit-config.yaml tool versions

Testing

tox              # All versions
tox -e py312     # Specific version

Results: 112 tests passed, 96% coverage on Python 3.9-3.12

Release

Version: 0.31.0 Type: Minor (non-breaking, adds Python 3.11/3.12 support) Target Branch: develop → main

Breaking Changes

None. Fully backward compatible.

….0.0 (initial governance framework - Spec Kit)

* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * Upgrade versions * Upgrade versions * feat: Add support for python 3.11 and 3.12 (#125) # Add Python 3.11 and 3.12 Support ## Summary Adds full support for Python 3.11 and 3.12 while maintaining Python 3.9/3.10 compatibility. Python 3.9 is deprecated (EOL Oct 2025) with support ending April 30, 2026. ## Changes **Dependencies** - Removed `pydantic<2` (unused, blocked Python 3.12) - Updated `packaging` to `>=23.0` (setuptools compatibility) - Updated `fsspec`, `gcsfs`, `s3fs` to `>=2024.6.0,<2025.0.0` - Added `setuptools<75.0.0` constraint in tox/requirements-dev **Testing** - Configured tox for `py39`, `py310`, `py311`, `py312` - All 112 tests pass on Python 3.10-3.12 (96% coverage) - Updated pre-commit: flake8 6.1.0, mypy 1.7.1 **Code Quality** - Fixed exception handling: `ClientError` instead of bare `Exception` - Removed unnecessary `type: ignore` comments - Fixed line length violations **Documentation** - Updated `CONTRIBUTING.md` with dev setup and tox usage - Updated `.pre-commit-config.yaml` tool versions ## Testing ```bash tox # All versions tox -e py312 # Specific version Results: 112 tests passed, 96% coverage on Python 3.9-3.12 Release Version: 0.31.0 Type: Minor (non-breaking, adds Python 3.11/3.12 support) Target Branch: develop → main Breaking Changes None. Fully backward compatible. * fix: rollback manual version bump and changelog edit * fix: rollback another manual change in dp version * ci: rollback missing default CHANGELOG configuration that was manually edited * ci: fix release pipeline, add permissions to GH token * ci: fix release pipeline, add permissions to GH token * FIX #126 - Bump version and CHANGELOG for release 0.31.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <47598580+dgniewek@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <dominik.gniewek@getindata.com>

* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <michal.rudko@getindata.com> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * Upgrade versions * Upgrade versions * feat: Add support for python 3.11 and 3.12 (#125) # Add Python 3.11 and 3.12 Support ## Summary Adds full support for Python 3.11 and 3.12 while maintaining Python 3.9/3.10 compatibility. Python 3.9 is deprecated (EOL Oct 2025) with support ending April 30, 2026. ## Changes **Dependencies** - Removed `pydantic<2` (unused, blocked Python 3.12) - Updated `packaging` to `>=23.0` (setuptools compatibility) - Updated `fsspec`, `gcsfs`, `s3fs` to `>=2024.6.0,<2025.0.0` - Added `setuptools<75.0.0` constraint in tox/requirements-dev **Testing** - Configured tox for `py39`, `py310`, `py311`, `py312` - All 112 tests pass on Python 3.10-3.12 (96% coverage) - Updated pre-commit: flake8 6.1.0, mypy 1.7.1 **Code Quality** - Fixed exception handling: `ClientError` instead of bare `Exception` - Removed unnecessary `type: ignore` comments - Fixed line length violations **Documentation** - Updated `CONTRIBUTING.md` with dev setup and tox usage - Updated `.pre-commit-config.yaml` tool versions ## Testing ```bash tox # All versions tox -e py312 # Specific version Results: 112 tests passed, 96% coverage on Python 3.9-3.12 Release Version: 0.31.0 Type: Minor (non-breaking, adds Python 3.11/3.12 support) Target Branch: develop → main Breaking Changes None. Fully backward compatible. * fix: rollback manual version bump and changelog edit * fix: rollback another manual change in dp version * ci: rollback missing default CHANGELOG configuration that was manually edited * ci: fix release pipeline, add permissions to GH token * ci: fix release pipeline, add permissions to GH token * ci: fix publish pipeline - pin a proper version * feat: Expand dbt version support (1.7.x - 1.10.x) and fix dbt 1.8+ compatibility (#127) * feat: support other dbt versions * chore: fix linter issue * FIX #128 - Bump version and CHANGELOG for release 0.32.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com> Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com> Co-authored-by: Piotr Sierkin <psierkin@gmail.com> Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com> Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com> Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com> Co-authored-by: jszafran <jszafran.pv@gmail.com> Co-authored-by: michalrudko <michal.rudko@gmail.com> Co-authored-by: Michal Rudko <michal.rudko@getindata.com> Co-authored-by: ptutak-getindata <133227844+ptutak-getindata@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <47598580+dgniewek@users.noreply.github.com> Co-authored-by: Dominik Gniewek-Węgrzyn <dominik.gniewek@getindata.com>

dgniewek added 2 commits October 27, 2025 15:40

docs: agentic coding setup - add CLAUDE.md and ratify constitution v1…

a947147

….0.0 (initial governance framework - Spec Kit)

feat: add support for python 3.11 and 3.12

c6bfc11

dgniewek changed the title ~~feat: add support for python 3.11 and 3.12~~ feat: Add support for python 3.11 and 3.12 Oct 31, 2025

dgniewek added 2 commits October 31, 2025 15:22

chore: remove unneeded files

000af3b

ci: fix GH action workflow

a70ec25

p-pekala approved these changes Oct 31, 2025

View reviewed changes

ci: remove codeclimate action, since it is not available anymore

1fef0bb

dgniewek merged commit 48f1951 into getindata:develop Oct 31, 2025
5 checks passed

dgniewek deleted the 001-python-311-312-support branch October 31, 2025 14:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add support for python 3.11 and 3.12#125

feat: Add support for python 3.11 and 3.12#125
dgniewek merged 5 commits intogetindata:developfrom
dgniewek:001-python-311-312-support

dgniewek commented Oct 31, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dgniewek commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add Python 3.11 and 3.12 Support

Summary

Changes

Testing

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dgniewek commented Oct 31, 2025 •

edited

Loading