pip, setup, build_external, etc

[dengwirda]

Are the method(s) used to install binaries sufficient? Currently, two options are supported:

1. setup.py build_external, etc builds and installs the jigsaw-cpp binaries within jigsawpy itself.
2. If no local binaries are found, a global search of the machine path is done to pick them up in, e.g. usr/local/bin, etc --- used by the conda-based workflow.

@jreniel has suggested installing to the python environment directory, either as a 3rd option, or, perhaps(?) a replacement for 1.

Should also work out what's needed to make jigsawpy available for direct install via pip.

[daniellivingston]

Bumping this! Would be great. This SO post might be helpful:

Once the setup.py has been authored this way, building the python module is as simple as running py setup.py, which will run the build and produce the outfiles.
It is recommended that you produce a wheel for users over slow internet or who do not want to build from sources. To do that, you will want to install the wheel package (py -m pip install wheel) and produce a wheel distribution by performing py setup.py bdist_wheel, and then upload it using twine like any other package.

The above is for distribution. Below, is some hacky-ness for local pip installation.

Based on this post, it should be valid to use build_ext as a pip command:

$ pip install --help
....
  --use-pep517                Use PEP 517 for building source distributions (use --no-use-pep517 to force legacy behaviour).
  --install-option <options>  Extra arguments to be supplied to the setup.py install command (use like --install-option="--install-
                              scripts=/usr/local/bin"). Use multiple --install-option options to pass multiple options to setup.py install.
                              If you are using an option with a directory path, be sure to use absolute path.
  --global-option <options>   Extra global options to be supplied to the setup.py call before the install or bdist_wheel command.
....

However, when I try to run the following,

$ python -m pip install \
    --global-option=build_ext --verbose --verbose --verbose \
    git+https://github.com/dengwirda/jigsaw-python.git@f87571945975b4882d8c27bdfff9275f6e46baa8

It does not work:

$ python -c "import jigsawpy"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/jigsawpy/__init__.py", line 58, in <module>
    from jigsawpy import jigsaw, libsaw
  File "/usr/local/lib/python3.8/dist-packages/jigsawpy/libsaw.py", line 76, in <module>
    JLIB = ct.cdll.LoadLibrary(str(JLIBNAME))
  File "/usr/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libjigsaw.so: cannot open shared object file: No such file or directory

Output from the pip install command above:

/usr/lib/python3/dist-packages/pip/_internal/commands/install.py:255: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-wisxfcn5
Created temporary directory: /tmp/pip-req-tracker-_bdtkadl
Initialized build tracking at /tmp/pip-req-tracker-_bdtkadl
Created build tracker: /tmp/pip-req-tracker-_bdtkadl
Entered build tracker: /tmp/pip-req-tracker-_bdtkadl
Created temporary directory: /tmp/pip-install-vnubvzhc
Collecting git+https://github.com/dengwirda/jigsaw-python.git@f87571945975b4882d8c27bdfff9275f6e46baa8
  Created temporary directory: /tmp/pip-req-build-b5j2k7da
  Cloning https://github.com/dengwirda/jigsaw-python.git (to revision f87571945975b4882d8c27bdfff9275f6e46baa8) to /tmp/pip-req-build-b5j2k7da
  Running command git clone -q https://github.com/dengwirda/jigsaw-python.git /tmp/pip-req-build-b5j2k7da
  Running command git show-ref f87571945975b4882d8c27bdfff9275f6e46baa8
  Running command git rev-parse HEAD
  f87571945975b4882d8c27bdfff9275f6e46baa8
  Added git+https://github.com/dengwirda/jigsaw-python.git@f87571945975b4882d8c27bdfff9275f6e46baa8 to build tracker '/tmp/pip-req-tracker-_bdtkadl'
    Running setup.py (path:/tmp/pip-req-build-b5j2k7da/setup.py) egg_info for package from git+https://github.com/dengwirda/jigsaw-python.git@f87571945975b4882d8c27bdfff9275f6e46baa8
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info
    writing /tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info/requires.txt
    writing top-level names to /tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info/top_level.txt
    writing manifest file '/tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    adding license file 'LICENSE.md'
    writing manifest file '/tmp/pip-req-build-b5j2k7da/pip-egg-info/jigsawpy.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-b5j2k7da has version 0.3.6, which satisfies requirement jigsawpy==0.3.6 from git+https://github.com/dengwirda/jigsaw-python.git@f87571945975b4882d8c27bdfff9275f6e46baa8
  Removed jigsawpy==0.3.6 from git+https://github.com/dengwirda/jigsaw-python.git@f87571945975b4882d8c27bdfff9275f6e46baa8 from build tracker '/tmp/pip-req-tracker-_bdtkadl'
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from jigsawpy==0.3.6) (1.17.4)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from jigsawpy==0.3.6) (1.7.0)
Skipping wheel build for jigsawpy, due to binaries being disabled for it.
Installing collected packages: jigsawpy
  Created temporary directory: /tmp/pip-record-aajjn8dd
    Running command /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-b5j2k7da/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-b5j2k7da/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext install --record /tmp/pip-record-aajjn8dd/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/jigsawpy
    running build_ext
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/jigsawpy
    copying jigsawpy/loadmsh.py -> build/lib/jigsawpy
    copying jigsawpy/msh_l.py -> build/lib/jigsawpy
    copying jigsawpy/__init__.py -> build/lib/jigsawpy
    copying jigsawpy/msh_t.py -> build/lib/jigsawpy
    copying jigsawpy/savejig.py -> build/lib/jigsawpy
    copying jigsawpy/jigsaw.py -> build/lib/jigsawpy
    copying jigsawpy/certify.py -> build/lib/jigsawpy
    copying jigsawpy/def_t.py -> build/lib/jigsawpy
    copying jigsawpy/prj_t.py -> build/lib/jigsawpy
    copying jigsawpy/project.py -> build/lib/jigsawpy
    copying jigsawpy/loadjig.py -> build/lib/jigsawpy
    copying jigsawpy/jig_l.py -> build/lib/jigsawpy
    copying jigsawpy/libsaw.py -> build/lib/jigsawpy
    copying jigsawpy/bisect.py -> build/lib/jigsawpy
    copying jigsawpy/jig_t.py -> build/lib/jigsawpy
    copying jigsawpy/savemsh.py -> build/lib/jigsawpy
    creating build/lib/jigsawpy/parse
    copying jigsawpy/parse/__init__.py -> build/lib/jigsawpy/parse
    copying jigsawpy/parse/savevtk.py -> build/lib/jigsawpy/parse
    copying jigsawpy/parse/savewav.py -> build/lib/jigsawpy/parse
    copying jigsawpy/parse/saveoff.py -> build/lib/jigsawpy/parse
    creating build/lib/jigsawpy/tools
    copying jigsawpy/tools/mathutils.py -> build/lib/jigsawpy/tools
    copying jigsawpy/tools/__init__.py -> build/lib/jigsawpy/tools
    copying jigsawpy/tools/orthoball.py -> build/lib/jigsawpy/tools
    copying jigsawpy/tools/meshutils.py -> build/lib/jigsawpy/tools
    copying jigsawpy/tools/scorecard.py -> build/lib/jigsawpy/tools
    copying jigsawpy/tools/projector.py -> build/lib/jigsawpy/tools
    copying jigsawpy/tools/predicate.py -> build/lib/jigsawpy/tools
    running install_lib
    creating /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/loadmsh.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/msh_l.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/__init__.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/msh_t.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/savejig.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/jigsaw.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    creating /usr/local/lib/python3.8/dist-packages/jigsawpy/parse
    copying build/lib/jigsawpy/parse/__init__.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/parse
    copying build/lib/jigsawpy/parse/savevtk.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/parse
    copying build/lib/jigsawpy/parse/savewav.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/parse
    copying build/lib/jigsawpy/parse/saveoff.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/parse
    copying build/lib/jigsawpy/certify.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/def_t.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/prj_t.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/project.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    creating /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/tools/mathutils.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/tools/__init__.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/tools/orthoball.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/tools/meshutils.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/tools/scorecard.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/tools/projector.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/tools/predicate.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy/tools
    copying build/lib/jigsawpy/loadjig.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/jig_l.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/libsaw.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/bisect.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/jig_t.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    copying build/lib/jigsawpy/savemsh.py -> /usr/local/lib/python3.8/dist-packages/jigsawpy
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/loadmsh.py to loadmsh.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/msh_l.py to msh_l.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/__init__.py to __init__.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/msh_t.py to msh_t.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/savejig.py to savejig.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/jigsaw.py to jigsaw.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/parse/__init__.py to __init__.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/parse/savevtk.py to savevtk.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/parse/savewav.py to savewav.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/parse/saveoff.py to saveoff.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/certify.py to certify.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/def_t.py to def_t.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/prj_t.py to prj_t.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/project.py to project.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/tools/mathutils.py to mathutils.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/tools/__init__.py to __init__.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/tools/orthoball.py to orthoball.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/tools/meshutils.py to meshutils.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/tools/scorecard.py to scorecard.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/tools/projector.py to projector.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/tools/predicate.py to predicate.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/loadjig.py to loadjig.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/jig_l.py to jig_l.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/libsaw.py to libsaw.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/bisect.py to bisect.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/jig_t.py to jig_t.cpython-38.pyc
    byte-compiling /usr/local/lib/python3.8/dist-packages/jigsawpy/savemsh.py to savemsh.cpython-38.pyc
    running install_egg_info
    running egg_info
    creating jigsawpy.egg-info
    writing jigsawpy.egg-info/PKG-INFO
    writing dependency_links to jigsawpy.egg-info/dependency_links.txt
    writing requirements to jigsawpy.egg-info/requires.txt
    writing top-level names to jigsawpy.egg-info/top_level.txt
    writing manifest file 'jigsawpy.egg-info/SOURCES.txt'
    reading manifest file 'jigsawpy.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    adding license file 'LICENSE.md'
    writing manifest file 'jigsawpy.egg-info/SOURCES.txt'
    Copying jigsawpy.egg-info to /usr/local/lib/python3.8/dist-packages/jigsawpy-0.3.6-py3.8.egg-info
    running install_scripts
    writing list of installed files to '/tmp/pip-record-aajjn8dd/install-record.txt'
    Running setup.py install for jigsawpy ... done
Successfully installed jigsawpy-0.3.6
Cleaning up...
  Removing source in /tmp/pip-req-build-b5j2k7da
Removed build tracker: '/tmp/pip-req-tracker-_bdtkadl'

[dengwirda]

@daniellivingston, we currently have a few options for jigsaw/py installation:

1. The python3 setup.py build_external; [sudo] python3 setup.py install combination should build the underlying C++ layer via cmake and copy the binaries into the jigsawpy folder structure in the first step, and then install the python package (with the binaries embedded inside) in the second. This is often what I use to install on local machines.
2. There is also a pair of conda packages available for jigsaw and jigsawpy that have everything prebuilt. For E3SM + HPC, etc we usually create a conda environment that includes these jigsaw packages.
3. There is (possibly, maybe!) the option of adding a third pip-based install option if there's a need, though I believe this will require a little additional fiddling around. I believe it's necessary to build out the set of binary wheels for all of the different OS + python version combinations that'd be supported — I've done something like this before by (mis)using a CI workflow to automate all of the builds and upload to the pip server.

If there's a need for (3) I'd be happy to look into it — it'd be nice to have support for pip...

(Also, I don't believe it'll be possible/easy to use some of the standard build_ext style support for python extensions with jigsaw, as it's a standalone C++ library rather than a python-only extension — at least this is my current understanding!)

[jreniel]

3. [...] by (mis)using a CI workflow to automate all of the builds and upload to the `pip` server.

@dengwirda IMO, I wouldn't exactly call that a misuse. Build + push is a very legitimate use for CI systems. =)

[SorooshMani-NOAA]

@dengwirda I was wondering if you have any plans for releasing the jigsawpy 0.3.6 on conda?

[dengwirda]

@SorooshMani-NOAA yes, definitely! I'm working through the test + verification suite at the moment, and will build the releases when this is in good shape. Updates to the conda package should flow on from there.

[SorooshMani-NOAA]

Thank you. Recently I've been trying to create an environment with 0.3.6 (which fixes the tiny elements issue) on one of the platforms that has gcc 6.5.0 and 9.2.0 and the jigsaw code doesn't compile in either, at least on this platform. So having the option to install that version from conda would be great

[jreniel]

Should also work out what's needed to make jigsawpy available for direct install via pip.

Don't mean to necro-bump, but this is exactly the problem that poetry solves!

[ChrisBarker-NOAA]

Don't mean to necro-bump, but this is exactly the problem that poetry solves!

well, not really -- I dont think poetry handles C++ externl libs, etc.

However, this is a great case for scikit-build-core:

https://scikit-build-core.readthedocs.io/en/latest/

It was designed explicitly to easy the bulidng of packages that mix Python and compiled code.

It's built on top of cmake -- and since this project uses cmake already, it's a natural fit.

I've recently converted a couple projects to it -- I think this one would be pretty straightforward.

And these kinds of full-function setup.py builds are deprecated, so you'll need to update sometime before too long.

[jreniel]

well, not really -- I dont think poetry handles C++ externl libs, etc.

@ChrisBarker-NOAA

See: https://github.com/jreniel/jigsaw-python/blob/add_pyproject/pyproject.toml#L37-L39 and https://github.com/jreniel/jigsaw-python/blob/add_pyproject/build.py

The most interesting part is that the users don't even need poetry, pip install will still trigger the build.py afaik.
This is the only reason why I maintain that fork, because it makes it easy to use jigsawpy as a direct dependency in other pyproject.toml based projects without having to manually compile the C/C++ code.

[ChrisBarker-NOAA]

I see -- poetry doesn't help with the building, but it does let you write a totally custom build script.

Which is one step better than setup.py, but I still suggest scikit build as the "right" way to do it -- rather than a custom build script.

see: https://github.com/NOAA-ORR-ERD/cell_tree2d for a simple example.

(or any number of examples in the scikit-build docs and other projects).

In any case -- one way or another the setup.py needs to be retired ...

[jreniel]

Poetry is not the only way to achieve it, that's for sure.

[dengwirda]

Folks, I've included a minimal pyproject.toml + setuptools workflow in #28 and am closing this as a result.
In terms of the various "build infrastructure" options discussed here (poetry, sckit, etc) I don't feel this is heading in the simpler-is-better, minimal-dependencies direction that I'm keen to preserve with jigsaw.