Add catalyst-dev install config by joeycarter · Pull Request #3023 · PennyLaneAI/catalyst

joeycarter · 2026-07-14T19:32:11Z

Context: Developing plugins for Catalyst has, in the past, required implementing and building them either within or beside an existing Catalyst installation built from source (see for instance our MLIR plugin documentation. This poses a challenge to plugin developers since it requires building Catalyst and its dependencies (like llvm-project and stablehlo) from source—a long and sometimes difficult process for non-experts. Having a pre-built "development" package of Catalyst and its dependencies would help alleviate some of these difficulties, allowing plugin developers to focus on their new dialects and passes without having to worry about building Catalyst from source.

Description of the Change: This PR adds and updates the following:

CMake configs to generate installation rules for the MLIR layer of Catalyst.
CPack configs to package the Catalyst installation targets into a catalyst-dev package, which contains all headers and libraries necessary for a Catalyst plugin (e.g. to access Catalyst's dialects and passes from the plugin).
Updated LLVM CMake configs to set the desired installation targets of LLVM/MLIR. Only the LLVM/MLIR headers and libraries are included, as well as a minimal set of binaries necessary for development, such as FileCheck, tblgen, etc.
Two new Makefile rules, make pack-catalyst-dev and make pack-mlir-dev, to invoke the CMake/CPack commands to build the Catalyst and MLIR development packages, respectively.
Updated GitHub workflows to build the catalyst-dev package with every nightly wheels build, and make them available for download (at this time, they are not published anywhere automatically in the CI pipeline).

Benefits: With these dev packages, a plugin developer can now extract the archives into a desired location, e.g.

$ tar -xf catalyst-mlir-dev*.tar.gz -C $MLIR_DEV_INSTALL_DIR
$ tar -xf catalyst-dev*.tar.gz -C $CATALYST_DEV_INSTALL_DIR

Then build their plugin by adding the following CMake arguments:

$ cmake -B build -S . -G Ninja \
    -DMLIR_DIR=$MLIR_DEV_INSTALL_DIR/lib/cmake/mlir \
    -DCatalyst_DIR=$CATALYST_DEV_INSTALL_DIR/lib/cmake/catalyst \
    ...

[sc-125319] [sc-125321]

And change its component to `catalyst-bin`

codecov · 2026-07-22T21:28:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.96%. Comparing base (02d616e) to head (99c6c3f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3023   +/-   ##
=======================================
  Coverage   96.96%   96.96%           
=======================================
  Files         165      165           
  Lines       19100    19100           
  Branches     1818     1818           
=======================================
  Hits        18521    18521           
  Misses        424      424           
  Partials      155      155

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dime10

Thanks Joey! Looking pretty good

dime10 · 2026-07-22T21:30:37Z

  `-stage` naming convention used when invoking them from the command line (e.g. `quantum-compilation-stage`).
  [#3002](https://github.com/PennyLaneAI/catalyst/pull/3002)

+* New build configurations have been added to allow building "development" packages of Catalyst and


We don't normally document CI/build changes in our release notes. Unless we think this is a feature we'd like to advertise?

I think it warrants a changelog entry, it's not only a change to the build system, but an addition that plugin developers (internal and external) could be interested in.

If we want to advertise it then it shouldn't be under internal changes imo

It makes no difference to me what section it's in, but Internal Changes seems like the best fit to me. It's not a new feature or improvement that an external user of Catalyst would care about, it's new functionality that a Catalyst developer would care about.

mlxd

Thanks @joeycarter

Nice work. Just a few questions and minor suggestions from my side

mlxd · 2026-07-22T21:19:25Z

+      with:
+        name: catalyst-dev-archives
+        path: catalyst-dev/
+        retention-days: 14


Do we think this will be a limit we want to increase? Should we preserve a quarter's length, or is 2 weeks sufficient?

Same applies to all other uses of this below.

Given that we're building nightly, two weeks is probably sufficient. During plugin development we'll likely have to regularly grab the latest one anyway. Once we have a proper hosting system set up they can be uploaded there for persistent storage.

mlxd · 2026-07-22T21:23:06Z

 @CATALYST_CONFIG_INCLUDE_EXPORTS@
+
+# --- rebuild ExternalStablehloLib for build-tree consumers ---
+set(_shlo_libs ChloCAPI ChloOps StablehloAssemblyFormat StablehloBase


Do we need this set if it is only used in the NOT pathway below? Would it be better within the if-statement?

mlxd · 2026-07-22T21:30:59Z

+set_and_check(CATALYST_INCLUDE_DIRS "@PACKAGE_CMAKE_INSTALL_INCLUDEDIR@")
+set_and_check(CATALYST_LIBRARY_DIR  "@PACKAGE_CMAKE_INSTALL_LIBDIR@")
+
+# Reconstruct the bundled StableHLO interface target.


Same q as above.

Though, is this already defined somewhere that we can use, or is this the trial to define the pieces we only care about?

mlxd · 2026-07-22T21:32:35Z

+file(READ "${PROJECT_SOURCE_DIR}/../frontend/catalyst/_version.py" _ver_py)
+string(REGEX MATCH "__version__ = \"([^\"]+)\"" _m "${_ver_py}")
+set(CATALYST_PACKAGE_VERSION "${CMAKE_MATCH_1}" CACHE INTERNAL "Catalyst version")
+
+string(REGEX MATCH "^[0-9]+\\.[0-9]+\\.[0-9]+" CATALYST_NUMERIC_VERSION "${CATALYST_PACKAGE_VERSION}")


Rather than a scrape and regex, is it possible to call Python directly to output the version string from this file?

That's probably a good idea, let me try. For reference, this was some quick-and-dirty plagiarism from here:

catalyst/mlir/CMakeLists.txt

Lines 149 to 160 in 58a1a11

function(make_version_header_file input_file output_file)

set(version_number "unknown")

if (NOT EXISTS ${input_file})

message(WARNING "Could not find ${input_file}")

else()

file(READ ${input_file} input_content)

if (NOT input_content MATCHES "__version__ = \"([^\"]+)\"")

message(WARNING "Could not find Catalyst version in ${input_file}")

else()

set(version_number "${CMAKE_MATCH_1}")

endif()

endif()

mlxd · 2026-07-22T21:36:10Z

+
+write_basic_package_version_file(
+    "${catalyst_cmake_builddir}/CatalystConfigVersion.cmake"
+    VERSION ${CATALYST_NUMERIC_VERSION}


We've chatted about versioning the libs in the past, so I presume this handles it?

If so, is this following https://www.gnu.org/software/libtool/manual/libtool.html#Updating-version-info / https://www.gnu.org/software/libtool/manual/libtool.html#Managing-release-information ?

If not, fine for now, but we should discuss how to tag these to be versioned right for LNX/MACOS (and the other one)

I'm not entirely sure, but I don't think so. What this command does is make a CatalystConfigVersion.cmake file containing the following (for example):

# This is a basic version file for the Config-mode of find_package(). # It is used by write_basic_package_version_file() as input file for configure_file() # to create a version-file which can be installed along a config.cmake file. # # The created file sets PACKAGE_VERSION_EXACT if the current version string and # the requested version string are exactly the same and it sets # PACKAGE_VERSION_COMPATIBLE if the current version is >= requested version, # but only if the requested major version is the same as the current one. # The variable CVF_VERSION must be set before calling configure_file(). set(PACKAGE_VERSION "0.16.0") if(PACKAGE_VERSION VERSION_LESS PACKAGE_FIND_VERSION) set(PACKAGE_VERSION_COMPATIBLE FALSE) else() if("0.16.0" MATCHES "^([0-9]+)\\.") set(CVF_VERSION_MAJOR "${CMAKE_MATCH_1}") if(NOT CVF_VERSION_MAJOR VERSION_EQUAL 0) string(REGEX REPLACE "^0+" "" CVF_VERSION_MAJOR "${CVF_VERSION_MAJOR}") endif() else() set(CVF_VERSION_MAJOR "0.16.0") endif() if(PACKAGE_FIND_VERSION_RANGE) # both endpoints of the range must have the expected major version math (EXPR CVF_VERSION_MAJOR_NEXT "${CVF_VERSION_MAJOR} + 1") if (NOT PACKAGE_FIND_VERSION_MIN_MAJOR STREQUAL CVF_VERSION_MAJOR OR ((PACKAGE_FIND_VERSION_RANGE_MAX STREQUAL "INCLUDE" AND NOT PACKAGE_FIND_VERSION_MAX_MAJOR STREQUAL CVF_VERSION_MAJOR) OR (PACKAGE_FIND_VERSION_RANGE_MAX STREQUAL "EXCLUDE" AND NOT PACKAGE_FIND_VERSION_MAX VERSION_LESS_EQUAL CVF_VERSION_MAJOR_NEXT))) set(PACKAGE_VERSION_COMPATIBLE FALSE) elseif(PACKAGE_FIND_VERSION_MIN_MAJOR STREQUAL CVF_VERSION_MAJOR AND ((PACKAGE_FIND_VERSION_RANGE_MAX STREQUAL "INCLUDE" AND PACKAGE_VERSION VERSION_LESS_EQUAL PACKAGE_FIND_VERSION_MAX) OR (PACKAGE_FIND_VERSION_RANGE_MAX STREQUAL "EXCLUDE" AND PACKAGE_VERSION VERSION_LESS PACKAGE_FIND_VERSION_MAX))) set(PACKAGE_VERSION_COMPATIBLE TRUE) else() set(PACKAGE_VERSION_COMPATIBLE FALSE) endif() else() if(PACKAGE_FIND_VERSION_MAJOR STREQUAL CVF_VERSION_MAJOR) set(PACKAGE_VERSION_COMPATIBLE TRUE) else() set(PACKAGE_VERSION_COMPATIBLE FALSE) endif() if(PACKAGE_FIND_VERSION STREQUAL PACKAGE_VERSION) set(PACKAGE_VERSION_EXACT TRUE) endif() endif() endif() # if the installed or the using project don't have CMAKE_SIZEOF_VOID_P set, ignore it: if("${CMAKE_SIZEOF_VOID_P}" STREQUAL "" OR "8" STREQUAL "") return() endif() # check that the installed version has the same 32/64bit-ness as the one which is currently searching: if(NOT CMAKE_SIZEOF_VOID_P STREQUAL "8") math(EXPR installedBits "8 * 8") set(PACKAGE_VERSION "${PACKAGE_VERSION} (${installedBits}bit)") set(PACKAGE_VERSION_UNSUITABLE TRUE) endif()

I believe its main purpose is to handle find_package(Catalyst ...) commands, not library versioning.

By the way, the files under lib/ are all static libraries, like

libcatalyst-transforms.a libMLIRCatalyst.a libMLIRQRef.a libMLIRQuantum.a ...

so they wouldn't follow the shared-library versioning convention (I think?).

mlxd · 2026-07-22T21:44:07Z

+# Re-fetch the exported targets and strip the same third-party ones the
+# build-tree export already excludes (see cmake/modules/CMakeLists.txt).
+get_property(CATALYST_INSTALL_TARGETS GLOBAL PROPERTY MLIR_EXPORTS)
+set(TARGETS_TO_REMOVE


Do we want any of these?

I wondered this too, but opted to keep them excluded since I ran into issues with external dependencies that didn't seem worth pursuing right now. We don't need any of these libs for the internal plugin we're currently working on. Eventually we may want to include quantum-transforms, for instance, since this is a fairly core piece of the compiler, but generally the most important components for a plugin are the dialects rather than the transforms (since any plugin that depends on a Catalyst transform pass could just run it from Catalyst rather than from the plugin).

mlxd · 2026-07-22T21:48:40Z

+
+set(CPACK_PACKAGE_NAME "catalyst-dev")
+set(CPACK_PACKAGE_VERSION "${CATALYST_PACKAGE_VERSION}")
+set(CPACK_GENERATOR "TGZ")


If this is set to ZIP_ZSTD, could we just relabel the output as whl and call it a day?
It'll be larger, but it may be installable from our own PyPI mirror if it works.

We can set the generator to ZIP, but we can't pip install from a bare ZIP file, no? The wheels would require the appropriate metadata files.

One option might be to create a new pyproject.toml file for the catalyst-dev package with a scikit-build backend. Do you think it's worth exploring or it's okay to stick with a simple archive for now?

mlxd · 2026-07-22T21:49:01Z

+set(CPACK_PACKAGE_NAME "catalyst-dev")
+set(CPACK_PACKAGE_VERSION "${CATALYST_PACKAGE_VERSION}")
+set(CPACK_GENERATOR "TGZ")
+set(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}_${CMAKE_SYSTEM_NAME}_${CMAKE_SYSTEM_PROCESSOR}_${CATALYST_PACKAGE_VERSION}")


All good on aarch64 too?

I assume so, my understanding is that the CMAKE_SYSTEM_PROCESSOR variable takes the output of

uname -m

mlxd · 2026-07-22T21:51:14Z

+set(CPACK_COMPONENTS_GROUPING ALL_COMPONENTS_IN_ONE) # single archive, no per-component split
+
+message(STATUS "Preparing CPack package ${CPACK_PACKAGE_FILE_NAME}")
+include(CPack)


The variables are set before the inclusion? Just wondering if there's reason to this?

mlxd · 2026-07-22T21:52:59Z

+    DESTINATION ${CMAKE_INSTALL_LIBDIR}
+    COMPONENT catalyst-dev
+    FILES_MATCHING PATTERN "lib*.a")
+


Worth adding the license file here too via CPACK_RESOURCE_FILE_LICENSE if we want to distribute this. Same with CPACK_PACKAGE_VENDOR with the name from the headers.

Not needed, it was just downloading and re-uploading the package

joeycarter added 11 commits July 14, 2026 15:31

[WIP] Add catalyst-dev install config

5df1243

Add catalyst-dev to nightly build workflows

61a2f94

Add pack-catalyst-dev Makefile rule

7bd7bb3

Uncomment tools install command

d1828b7

And change its component to `catalyst-bin`

Also install LICENSE to catalyst-dev

bed0412

Remove pack-catalyst-dev from 'make all' rule

fd4b91f

Configure make rules for catalyst and mlir dev packages

988ed65

Remove dev package builds from linux-arm64 workflows

70fcf17

Update build_dev_package description

de85d24

Merge branch 'main' into joeycarter/add-catalyst-dev-install-config

525c5bf

Add help message for pack-mlir-dev Makefile rule

b31a62f

joeycarter marked this pull request as ready for review July 22, 2026 20:53

joeycarter added CI/Build Pull requests that update CI or build scripts author:build-wheels Run the wheel building workflows on this Pull Request labels Jul 22, 2026

joeycarter requested review from a team, dime10 and mlxd July 22, 2026 20:54

Add changelog entry

4c975df

dime10 reviewed Jul 22, 2026

View reviewed changes

mlxd reviewed Jul 22, 2026

View reviewed changes

joeycarter added 3 commits July 23, 2026 09:49

Remove collect-dev-package action from build-nightly-release

387e1e3

Not needed, it was just downloading and re-uploading the package

Use LLVM_INSTALL_STAGING_DIR for CMAKE_INSTALL_PREFIX

316aca0

Merge branch 'main' into joeycarter/add-catalyst-dev-install-config

99c6c3f

	function(make_version_header_file input_file output_file)
	set(version_number "unknown")
	if (NOT EXISTS ${input_file})
	message(WARNING "Could not find ${input_file}")
	else()
	file(READ ${input_file} input_content)
	if (NOT input_content MATCHES "__version__ = \"([^\"]+)\"")
	message(WARNING "Could not find Catalyst version in ${input_file}")
	else()
	set(version_number "${CMAKE_MATCH_1}")
	endif()
	endif()

Uh oh!

Conversation

joeycarter commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dime10 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dime10 Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlxd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joeycarter Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joeycarter commented Jul 14, 2026 •

edited

Loading

codecov Bot commented Jul 22, 2026 •

edited

Loading

dime10 Jul 23, 2026 •

edited

Loading

joeycarter Jul 23, 2026 •

edited

Loading