pip check fails due to dependency on ninja

[awvwgk]

Running pip check inside a conda environment currently fails because meson-python depends on the PyPI version of ninja and cannot detect the actually installed ninja version from conda-forge.

❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpep517:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
auditwheel                5.1.2            py39hf3d152e_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
cffi                      1.15.0           py39h4bc2ebd_0    conda-forge
dftd3                     0.5.1                    pypi_0    pypi
importlib-metadata        4.11.3           py39hf3d152e_1    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libblas                   3.9.0           14_linux64_openblas    conda-forge
libcblas                  3.9.0           14_linux64_openblas    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_16    conda-forge
libgfortran-ng            11.2.0              h69a702a_16    conda-forge
libgfortran5              11.2.0              h5c6108e_16    conda-forge
libgomp                   11.2.0              h1d223b6_16    conda-forge
liblapack                 3.9.0           14_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.0             pyhd8ed1ab_0    conda-forge
meson-python              0.3.0                    pypi_0    pypi
mesonpep517               0.2                pyhd8ed1ab_0    conda-forge
minpack                   2.0.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.10.2               h4bd325d_1    conda-forge
numpy                     1.22.3           py39h18676bf_2    conda-forge
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep517                    0.12.0           py39hf3d152e_2    conda-forge
pint                      0.19.1             pyhd8ed1ab_0    conda-forge
pip                       22.0.4             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0            py39hf3d152e_3    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.9.0            py39hb9d737c_1    conda-forge
pyelftools                0.28             py39hf3d152e_2    conda-forge
pyparsing                 3.0.8              pyhd8ed1ab_0    conda-forge
pytest                    7.1.2            py39hf3d152e_0    conda-forge
python                    3.9.12          h2660328_1_cpython    conda-forge
python-build              0.7.0              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
qcelemental               0.24.0             pyhd8ed1ab_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.1.0           py39hf3d152e_0    conda-forge
sqlite                    3.38.2               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
typing-extensions         4.2.0                hd8ed1ab_1    conda-forge
typing_extensions         4.2.0              pyha770c72_1    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
❯ pip check
meson-python 0.3.0 requires ninja, which is not installed.

[rgommers]

That seems simply expected to me, given thatpip cannot know about conda-installed packages.

This will resolve itself as soon as we package meson-python for conda-forge (and defaults will follow later). Till then, either:

• don't worry about it (I've never found a reason to use pip check, since it cannot know about my most important dependencies anyway)
• install ninja and meson from PyPI if you really want pip check to pass

I propose to close this, since this is normal and not a problem.

[awvwgk]

This will resolve itself as soon as we package meson-python for conda-forge

That's why I'm asking actually, see conda-forge/staged-recipes#19033.

[rgommers]

That's why I'm asking actually, see conda-forge/staged-recipes#19033.

Thanks for doing that packaging work. I commented on your PR there.

Running pip check inside a conda environment currently fails because meson-python depends on the PyPI version of ninja and cannot detect the actually installed ninja version from conda-forge.

Can you post the actual traceback you're seeing, and/or how to reproduce the problem?

[awvwgk]

This should more or less match the used check in the recipe

❯ mamba create -n mesonpy colorama meson pep621 tomli --override-channels --channel conda-forge
... snip ...
❯ mamba activate mesonpy
❯ pip install meson-python==0.4.0 --no-deps
Collecting meson-python==0.4.0
  Using cached meson_python-0.4.0-py3-none-any.whl (14 kB)
Installing collected packages: meson-python
Successfully installed meson-python-0.4.0
❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.1             pyhd8ed1ab_0    conda-forge
meson-python              0.4.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep621                    0.4.0              pyhd8ed1ab_0    conda-forge
pip                       22.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
python                    3.10.4          h2660328_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
❯ pip check
meson-python 0.4.0 requires ninja, which is not installed.
❯ echo $?
1

[rgommers]

That all makes sense. It may be the case that the pip shipped by conda-forge is patched so that it can recognize Python packages installed by conda/mamba, but that patches is failing to pick up a non-Python package on PyPI.

How about just asking on the conda-forge PR how to bypass pip check?

[awvwgk]

How about just asking on the conda-forge PR how to bypass pip check?

I can simply remove the pip check test from the recipe, still I would prefer to pass this check as a safe-guard should dependencies change with a future update.

[rgommers]

Then maybe a patch is needed for pip itself. I don't think it can be fixed in meson-python.

[awvwgk]

It may be the case that the pip shipped by conda-forge is patched so that it can recognize Python packages installed by conda/mamba

Python packages installed by conda / mamba are not fundamentally different from other Python packages installed by system package managers or pip. In fact most Python package are packaged via pip install on conda-forge.

The issue is that meson-python expects a non-Python package (ninja) to be installed and findable via pip, which is generally not true. The Kitware distribution of ninja on PyPI is to my knowledge the only ninja distribution which provides Python metadata, however depending on this can be dangerous because it cannot be installed in all environments, especially on Windows this dependency is not 100% reliable.

[rgommers]

The issue is that meson-python expects a non-Python package (ninja) to be installed and findable via pip, which is generally not true.

I'm not sure why you say that meson-python expects ninja to be "findable" via pip, and what that means? I think for an isolated build (with pip or build), ninja will be installed in the isolated venv, and for a build without build isolation, ninja from conda-forge works perfectly fine. Can you elaborate?

[awvwgk]

Installing meson-python with pip will always pull in ninja from PyPI, even if the environment actually provides a version, but not the metadata required for pip to recognize the installation. This leads to an inevitable additional installation of ninja as demonstrated here:

❯ mamba activate mesonpy
❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.1             pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep621                    0.4.0              pyhd8ed1ab_0    conda-forge
pip                       22.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
python                    3.10.4          h2660328_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
❯ pip install .
Processing /home/awvwgk/projects/src/git/meson-python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: meson>=0.60.0 in /home/awvwgk/software/opt/conda/envs/mesonpy/lib/python3.10/site-packages (from meson-python==0.4.0) (0.62.1)
Collecting ninja
  Using cached ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
Requirement already satisfied: tomli>=1.0.0 in /home/awvwgk/software/opt/conda/envs/mesonpy/lib/python3.10/site-packages (from meson-python==0.4.0) (2.0.1)
Building wheels for collected packages: meson-python
  Building wheel for meson-python (pyproject.toml) ... done
  Created wheel for meson-python: filename=meson_python-0.4.0-py3-none-any.whl size=14560 sha256=566f3387e324164dc31d4658f5da20cdc6fd56668db02b6971c2228cc8af6f6e
  Stored in directory: /home/awvwgk/.cache/pip/wheels/1e/e7/8d/f91cbe34dbc546a667d1e8b03e26c415a829c511e6d1e22f6e
Successfully built meson-python
Installing collected packages: ninja, meson-python
Successfully installed meson-python-0.4.0 ninja-1.10.2.3
❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.1             pyhd8ed1ab_0    conda-forge
meson-python              0.4.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.10.2.3                 pypi_0    pypi
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep621                    0.4.0              pyhd8ed1ab_0    conda-forge
pip                       22.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
python                    3.10.4          h2660328_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge

If I can see this correctly the meson package on PyPI does not depend on Kitware's ninja distribution, therefore I'm wondering why meson-python is adding it as dependency. Keep in mind that ninja is a C++ program and not a Python package, pip is not supposed to manage it.

[rgommers]

even if the environment actually provides a version, but not the metadata required for pip to recognize the installation. This leads to an inevitable additional installation of ninja as demonstrated here:

So I think that is conda-forge/ninja-feedstock#26. If the ninja conda-forge package would declare it provides ninja, things would work as expected here.

The dependency is there, and saying it's not is kinda backwards. The problem with PyPI is not that ninja is available there, it's that other (native) dependencies are not available.

Installing meson-python with pip

This is never necessary anymore once meson-python is packaged for conda.

[eli-schwartz]

If I can see this correctly the meson package on PyPI does not depend on Kitware's ninja distribution, therefore I'm wondering why meson-python is adding it as dependency.

Meson does have an optional extra, pip install 'meson[ninja]' will also guarantee that ninja support is resolved via PyPI rather than being assumed to exist via the OS repositories. And it is frequently used by linux distros which do not install PyPI-repackaged versions of system C++ command-line tools, so it cannot be a hard dependency.

meson-python is obviously making the tradeoff that pip install foo where foo is a package that utilizes this build backend, cannot work reliably if it cannot install ninja, and it cannot do that via pacman, emerge, dnf, apt, xbps... so better to install an extra copy than to break pip install. Obviously, this will mean distribution via channels that install ninja outside of pip, will need to patch meson-python to not depend on ninja.

[rgommers]

So I think that is conda-forge/ninja-feedstock#26. If the ninja conda-forge package would declare it provides ninja, things would work as expected here.

After some more digging in that repo, the correct fix is to create a new python-ninja package that installs the equivalent of the Python part of the ninja PyPI package. And then the meson-python conda-forge package should depend on python-ninja rather than on ninja, that will make everything work.

[rgommers]

And it is frequently used by linux distros which do not install PyPI-repackaged versions of system C++ command-line tools, so it cannot be a hard dependency.

I think this is the difference between meson and meson-python; the former is very Linux-oriented, while the latter is by nature cross-platform. On macOS and Windows users will typically not have ninja installed, so a hard dependency in meson-python seems like the right choice to me.