-
Notifications
You must be signed in to change notification settings - Fork 724
[Refactor][CI/Docker] Replace poetry with uv for Python package management #8776
Copy link
Copy link
Open
Labels
component/pydevlakeRelates to the Python framework and/or plugin.Relates to the Python framework and/or plugin.devopsSomething about CI/CD (devops)Something about CI/CD (devops)type/refactorThis issue is to refactor existing codeThis issue is to refactor existing code
Description
What and why to refactor
Problem: CI job "build image" fails when installing Poetry via the official installer.
The issue occurs in backend/Dockerfile during the base stage build:
RUN curl -sSL https://install.python-poetry.org | python3 -which results in:
Installing Poetry (2.3.2): Creating environment
Installing Poetry (2.3.2): Installing Poetry
Installing Poetry (2.3.2): An error occurred. Removing partial environment.
Poetry installation failed.
Why refactor now:
- Poetry installer is unreliable in CI
- Alternative approach (pipx) also has issues
Describe the solution you'd like
Replace poetry with uv, a Rust-based Python package manager:
Advantages:
- 10-100x faster than poetry
- Compatible with pyproject.toml (no config changes needed)
- Minimal overhead in QEMU environments
- Actively maintained by Astral (ruff creators)
Related issues
- Original issue: poetry timeout in CI arm64 builds
- Python 3.9 EOL concerns
Additional context
Root cause identified: Poetry 2.3.x is incompatible with Python 3.9. The installer silently fails because it cannot create a working environment.
Attempted workarounds:
-
curl installer fails - Original method using
install.python-poetry.org:Installing Poetry (2.3.2): Creating environment Installing Poetry (2.3.2): Installing Poetry Installing Poetry (2.3.2): An error occurred. Removing partial environment. Poetry installation failed.- Root cause: Poetry 2.3.x requires Python >= 3.10, but the devlake base image uses Python 3.9
-
Version pinning to 2.2.1 - Locking to last Python 3.9 compatible version:
curl -sSL https://install.python-poetry.org | python3 - --version 2.2.1- Installation succeeds with Poetry 2.2.1
- However,
poetry installstill times out in CI arm64 builds:Creating virtualenv azuredevops-ANnMAkq9-py3.9 in /app/.cache/pypoetry/virtualenvs failed to query /usr/local/bin/python3.9 with code -1 err: 'timed out' - Timeout occurs during virtualenv creation when Poetry queries the Python interpreter
- This is exacerbated by QEMU emulation in cross-platform CI builds
3. pipx install poetry - Using pipx with version pinning:
- Installation succeeded:
python3 -m pipx install poetry - But
poetry installstill fails in CI arm64 builds:Creating virtualenv azuredevops-ANnMAkq9-py3.9 in /app/.cache/pypoetry/virtualenvs failed to query /usr/local/bin/python3.9 with code -1 err: 'timed out' - Root cause: Poetry queries Python interpreter during virtualenv creation, which times out in QEMU-emulated arm64 environments
- Local arm64 builds succeed because they run on native hardware
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
component/pydevlakeRelates to the Python framework and/or plugin.Relates to the Python framework and/or plugin.devopsSomething about CI/CD (devops)Something about CI/CD (devops)type/refactorThis issue is to refactor existing codeThis issue is to refactor existing code