Skip to content

[Refactor][CI/Docker] Replace poetry with uv for Python package management #8776

@narrowizard

Description

@narrowizard

What and why to refactor

Problem: CI job "build image" fails when installing Poetry via the official installer.

The issue occurs in backend/Dockerfile during the base stage build:

RUN curl -sSL https://install.python-poetry.org | python3 -

which results in:

Installing Poetry (2.3.2): Creating environment
Installing Poetry (2.3.2): Installing Poetry
Installing Poetry (2.3.2): An error occurred. Removing partial environment.
Poetry installation failed.

Why refactor now:

  • Poetry installer is unreliable in CI
  • Alternative approach (pipx) also has issues

Describe the solution you'd like

Replace poetry with uv, a Rust-based Python package manager:

Advantages:

  • 10-100x faster than poetry
  • Compatible with pyproject.toml (no config changes needed)
  • Minimal overhead in QEMU environments
  • Actively maintained by Astral (ruff creators)

Related issues

  • Original issue: poetry timeout in CI arm64 builds
  • Python 3.9 EOL concerns

Additional context

Root cause identified: Poetry 2.3.x is incompatible with Python 3.9. The installer silently fails because it cannot create a working environment.

Attempted workarounds:

  1. curl installer fails - Original method using install.python-poetry.org:

    Installing Poetry (2.3.2): Creating environment
    Installing Poetry (2.3.2): Installing Poetry
    Installing Poetry (2.3.2): An error occurred. Removing partial environment.
    Poetry installation failed.
    
    • Root cause: Poetry 2.3.x requires Python >= 3.10, but the devlake base image uses Python 3.9
  2. Version pinning to 2.2.1 - Locking to last Python 3.9 compatible version:

    curl -sSL https://install.python-poetry.org | python3 - --version 2.2.1
    • Installation succeeds with Poetry 2.2.1
    • However, poetry install still times out in CI arm64 builds:
      Creating virtualenv azuredevops-ANnMAkq9-py3.9 in /app/.cache/pypoetry/virtualenvs
      failed to query /usr/local/bin/python3.9 with code -1 err: 'timed out'
      
    • Timeout occurs during virtualenv creation when Poetry queries the Python interpreter
    • This is exacerbated by QEMU emulation in cross-platform CI builds

3. pipx install poetry - Using pipx with version pinning:

  • Installation succeeded: python3 -m pipx install poetry
  • But poetry install still fails in CI arm64 builds:
    Creating virtualenv azuredevops-ANnMAkq9-py3.9 in /app/.cache/pypoetry/virtualenvs
    failed to query /usr/local/bin/python3.9 with code -1 err: 'timed out'
    
  • Root cause: Poetry queries Python interpreter during virtualenv creation, which times out in QEMU-emulated arm64 environments
  • Local arm64 builds succeed because they run on native hardware

Metadata

Metadata

Assignees

No one assigned

    Labels

    component/pydevlakeRelates to the Python framework and/or plugin.devopsSomething about CI/CD (devops)type/refactorThis issue is to refactor existing code

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions