Installation tweaks by wdeconinck · Pull Request #32 · GridTools/atlas4py

wdeconinck · 2026-01-29T09:10:36Z

I have made time available this year to work on our plan to extend atlas4py and hand over maintenance.
As a first step I tried the new build-system, and noticed that I needed to tweak the installation a little bit.

Fix RPATH handling where atlas library is external
Faster compilation, removing unneeded parts
Reinstallation, e.g. when developing, no longer picks up a previously installed atlas4py-provided atlas/eckit library
Added more tests, and a few cosmetic changes
Bugfix in RectangularDomain construction
Changes to avoid using deprecated eckit functionality to be removed with eckit 2.0.0.

wdeconinck · 2026-02-09T09:44:34Z

Thank you for merging #30 @tehrengruber , @havogt .
I have rebased this on latest master; It should be ready for review.

havogt · 2026-02-12T10:10:46Z

I only did a brief walk through the code so far. One question popped up and I didn't follow the code to figure it out myself: how is the version managed? Is it taking always the version of atlas?

wdeconinck · 2026-02-12T13:42:16Z

I only did a brief walk through the code so far. One question popped up and I didn't follow the code to figure it out myself: how is the version managed? Is it taking always the version of atlas?

atlas4py version is not managed by the found atlas library, but that could be a strategy. I did not deviate from what was in place. I removed however duplicate places where the version was defined with only a single place remaining, in the pyproject.toml file.
This version is then defined as a CMake variable SKBUILD_PROJECT_VERSION_FULL by scikit-build-core. I then add that as a preprocessor definition into the _atlas4py.cpp __version__ attribute of the module. This means that the manually managed __version__ attribute is no longer needed.

There is still the extra variable ATLAS4PY_ATLAS_VERSION defined in pyproject.toml, which is only used to download atlas if atlas was not found. This could be introspected from the atlas4py version instead of course, if the format is clear.
With the use of externally installed atlas versions, the version of atlas4py and the actual atlas (commit hash, branch, or tag) used may differ, as then ATLAS4PY_ATLAS_VERSION variable is ignored. I added a CMake configuration warning however when the version is mismatched.

… library

…f/eckit#105)

wdeconinck · 2026-02-13T16:04:19Z

Note, just rebased again on master.

tehrengruber · 2026-02-04T14:46:07Z

requirements-dev.txt

@@ -1,5 +1,6 @@
 black>=21.12b0
 bump2version>=1.0


Suggested change

bump2version>=1.0

tehrengruber · 2026-02-04T15:20:48Z

src/atlas4py/CMakeLists.txt

-    find_package(ecbuild)
-    if(NOT ecbuild_FOUND)
-        FetchContent_Declare(
+if(build_atlas)


The changes in this if-branch are fine with me, but my tendency is rather to simplify than to extend here. In case one want's to build atlas the motivation is likely: I want to use or develop atlas4py, don't bother me with installing atlas, just set me up. Being able to say build atlas for me, but please use this eckit version I provide appears a rather narrow use case that can easily be handled by just building everything manually.

if(build_atlas) # Build ecbuild set ( ecbuild_SOURCE_DIR ${CMAKE_BINARY_DIR}/_deps/ecbuild ) message( STATUS "Downloading ecbuild version \"${ATLAS4PY_ECBUILD_VERSION}\" to ${ecbuild_SOURCE_DIR}" ) FetchContent_Populate( ecbuild GIT_REPOSITORY https://github.com/ecmwf/ecbuild.git GIT_TAG ${ATLAS4PY_ECBUILD_VERSION} SOURCE_DIR ${ecbuild_SOURCE_DIR} QUIET ) set( ecbuild_ROOT ${ecbuild_SOURCE_DIR} CACHE INTERNAL "Found ecbuild" ) # Build eckit message( STATUS "Downloading and building eckit version \"${ATLAS4PY_ECKIT_VERSION}\"" ) # Disable unused features for faster compilation set(ECKIT_ENABLE_TESTS OFF) set(ECKIT_ENABLE_DOCS OFF) set(ECKIT_ENABLE_PKGCONFIG OFF) set(ECKIT_ENABLE_ECKIT_GEO OFF) set(ECKIT_ENABLE_ECKIT_SQL OFF) set(ECKIT_ENABLE_ECKIT_CMD OFF) FetchContent_Declare( eckit GIT_REPOSITORY https://github.com/ecmwf/eckit.git GIT_TAG ${ATLAS4PY_ECKIT_VERSION} ) FetchContent_MakeAvailable(eckit) set( eckit_ROOT ${eckit_BINARY_DIR} CACHE INTERNAL "Found eckit" ) # build atlas message( STATUS "Downloading and building atlas version \"${ATLAS4PY_ATLAS_VERSION}\"" ) # Disable unused features for faster compilation set(ATLAS_ENABLE_TESTS OFF) set(ATLAS_ENABLE_DOCS OFF) set(ATLAS_ENABLE_PKGCONFIG OFF) set(ECKIT_ENABLE_ECKIT_GEO OFF) set(ECKIT_ENABLE_ECKIT_SQL OFF) set(ATLAS_ENABLE_ECKIT_CMD OFF) FetchContent_Declare( atlas GIT_REPOSITORY https://github.com/ecmwf/atlas.git GIT_TAG ${ATLAS4PY_ATLAS_VERSION} ) FetchContent_MakeAvailable(atlas)

Thanks for this insight and I agree: Let's build all atlas dependencies with exact version requirements, in case atlas was not detected. I will adapt.

tehrengruber · 2026-02-04T15:31:56Z

src/atlas4py/_atlas4py.cpp

 PYBIND11_MODULE( _atlas4py, m ) {
    m.def("_initialise", atlasInitialise)
     .def("_finalise",   atlas::finalise);
    m.attr("version") = atlas::Library::instance().version();


Suggested change

m.attr("version") = atlas::Library::instance().version();

m.attr("atlas_version") = atlas::Library::instance().version();

or just removing it altogether?

Suggested change

m.attr("version") = atlas::Library::instance().version();

I suppose in time, __version__ and version/atlas_versionshould be the same. We might as well remove it then indeed.

tehrengruber · 2026-02-04T15:44:39Z

src/atlas4py/CMakeLists.txt

 else()
    set(rpath_origin_install_libdir "$ORIGIN/${CMAKE_INSTALL_LIBDIR}")
 endif()
+set( CMAKE_INSTALL_RPATH "${rpath_origin_install_libdir}" ) # A semicolon-separated list specifying the RPATH to use in installed targets


Can you add a motivation for the 3 additional options here?

Consider the use case without these added lines, and where we have atlas detected and not being built.
I then have this error at runtime (for instance for macOS):

from ._atlas4py import * E ImportError: dlopen(/path/to/venv/lib/python3.13/site-packages/atlas4py/_atlas4py.cpython-313-darwin.so, 0x0002): Library not loaded: @rpath/libatlas.0.45.dylib

It is clear that the rpath of the found atlas libraries is not added. CMake provides the option

set( CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE ) # add the automatic parts to RPATH which point to dirs outside build tree

to automatically add the rpaths of libraries which are not built within the build-tree, introspected from the location of the library. This then adds the rpath to the external atlas library. We could probably guard this with if (build_atlas) to make sure no extra rpaths are inserted accidentally.

The next options probably do not play a role because it is more for a standard package with multiple libraries and executables involving running executables from within the build-tree and saving time on installation. But it is the good practice that we always set by default for all our software.

set( CMAKE_SKIP_BUILD_RPATH FALSE ) # use RPATHs for the build tree

When CMAKE_SKIP_BUILD_RPATH is set to FALSE, then the executables in the build-tree etc. are also built with RPATHS.

set( CMAKE_BUILD_WITH_INSTALL_RPATH TRUE ) # build with *relative* rpaths by default

That means that the build-tree executables have been built with the same rpaths we encoded for the install. Usually we have compile destinations within the build-tree in a similar structure as install: <build>/lib, <build>/bin, and the rpaths are relative "$ORIGIN/../lib". This saves time on relinking on installation, and is then just a copy.

We could omit these last two for atlas4py for now.

tehrengruber · 2026-02-17T13:08:43Z

src/atlas4py/_atlas4py.cpp

    atlas::initialise(argc, argv);
 }

+py::object toPyObject( eckit::Configuration const& v );


If you want a textbook implementation you could also use py::cast and a type caster for eckit::Configuration.

Ow I'll see how to do it; that will be much nicer indeed! Thanks for the hint!

wdeconinck mentioned this pull request Jan 29, 2026

CI issues #33

Closed

wdeconinck force-pushed the installation-tweaks branch 2 times, most recently from 597b52a to 6d74645 Compare February 4, 2026 14:44

wdeconinck mentioned this pull request Feb 5, 2026

Add python 3.13, 3.14, drop < 3.10 #30

Merged

wdeconinck force-pushed the installation-tweaks branch 4 times, most recently from 6d1d6b1 to 1c7d1cb Compare February 9, 2026 09:34

wdeconinck force-pushed the installation-tweaks branch 2 times, most recently from b5f514e to 3281313 Compare February 11, 2026 15:23

wdeconinck added 6 commits February 13, 2026 17:02

Installation tweaks, robust to reinstallations and externalised atlas…

2af653d

… library

Fix bug in arguments of RectangularDomain

1578d37

Fix eckit deprecation warnings related to acessing eckit::Value (ecmw…

ddc5864

…f/eckit#105)

Update examples/Atlas4Py-Test.ipynb

e8d955e

Improve testing

ec7f1c4

Bump dev version to 0.41.1.dev2

7ef5085

wdeconinck force-pushed the installation-tweaks branch from 3281313 to 7ef5085 Compare February 13, 2026 16:03

tehrengruber reviewed Feb 17, 2026

View reviewed changes

	m.attr("version") = atlas::Library::instance().version();
	m.attr("atlas_version") = atlas::Library::instance().version();

Comments

Conversation

wdeconinck commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wdeconinck commented Feb 9, 2026

Uh oh!

havogt commented Feb 12, 2026

Uh oh!

wdeconinck commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wdeconinck commented Feb 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wdeconinck Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wdeconinck commented Jan 29, 2026 •

edited

Loading

wdeconinck commented Feb 12, 2026 •

edited

Loading

wdeconinck Feb 19, 2026 •

edited

Loading