Migrate build configuration to pyproject.toml#3600
Conversation
Consolidate build, packaging, and tooling configuration into a single pyproject.toml and remove the setup.py/versioneer machinery: - Add pyproject.toml with the setuptools build backend. Project metadata, pytest, coverage, and ruff config all live here. Dependencies and optional-dependencies are read dynamically from the *requirements.txt files, preserving them as the source of truth for CI/dependabot. - Replace versioneer with setuptools-scm for git-tag-based versioning. Delete versioneer.py and the generated, tracked _version.py (now a build-time artifact, gitignored). __version__ is imported from the generated _version.py with an importlib.metadata fallback for un-built source trees. - Fold ruff.toml into [tool.ruff]; delete ruff.toml. - Trim setup.cfg to only the [pycodestyle] section, since pycodestyle cannot be configured from pyproject.toml (it reads setup.cfg/tox.ini). - Update the CI build step to `python -m build`, and update docs conf.py and MANIFEST.in accordingly. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the pyproject.toml migration, addressing setuptools-scm edge cases that versioneer previously masked: - Add fallback_version = "0.0.0" so any environment lacking SCM info (shallow clones without a reachable tag, git-archive tarballs) still produces an installable version instead of failing. - Remove the now-dead `.gitattributes` export-subst entry, which referenced the previously committed _version.py (now a build-time, gitignored artifact). - test-release.yml: check out with fetch-depth 0 so the tagged release build can always resolve the version from git. - .appveyor.yml: deepen clone_depth from 15 to 200 so setuptools-scm can reach a release tag when building the editable install. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
Add setuptools-scm's git-archive integration so builds from a source tarball produced by `git archive` (e.g. GitHub's auto-generated "Source code" assets) can still determine a version without a .git directory: - .git_archival.txt holds $Format:$ placeholders that git expands at archive time (commit hash, date, describe name). - .gitattributes marks it export-subst so the substitution happens. This replaces the equivalent support that versioneer previously provided through the committed _version.py. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This PR causes the following changes to the html docs (ubuntu-latest-3.11): See CI logs for the full diff. |
The docs comparison workflow pinned the version rendered into the generated HTML by creating a temporary 'v0.0.0.0' git tag, relying on versioneer resolving the version live (via git describe) at import time. setuptools-scm resolves the version at build/install time instead, so the tag - which is created after the PNL install step, deliberately, to avoid interfering with dependency resolution - no longer had any effect. The real version leaked into every generated page and polluted the docs diff on pull requests. Pin the version through a PNL_DOCS_VERSION environment variable read by docs/source/conf.py instead. This is independent of how the installed package determines its version, and removes the fragile requirement that the tag be created at a specific point relative to installation. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This PR causes the following changes to the html docs (ubuntu-latest-3.11): See CI logs for the full diff. |
The previous commit replaced the temporary 'v0.0.0.0' git tag with the PNL_DOCS_VERSION environment variable, but removing the tag broke the 'base' half of the docs comparison. The two docs builds are asymmetric while this change is in flight: the workflow definition comes from the PR merge commit and so applies to both, but the 'base' build checks out the base branch, whose conf.py resolves the version live from git and knows nothing about PNL_DOCS_VERSION. The tag was the only mechanism that build responds to; without it the base docs rendered the real version and every page differed. Keep both mechanisms so both halves are pinned to 0.0.0.0: the tag covers builds that resolve the version from git, PNL_DOCS_VERSION covers builds that resolve it at install time. The tag steps can be dropped once the base branch also supports PNL_DOCS_VERSION. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This PR causes the following changes to the html docs (ubuntu-latest-3.11): See CI logs for the full diff. |
jvesely
left a comment
There was a problem hiding this comment.
pycodestyle cannot be removed until ruff supports the same checks. Otherwise it would have been replaced at the same time as pydocstyle.
| # install time rather than live from git, which the tag above cannot | ||
| # affect. Read by docs/source/conf.py. | ||
| env: | ||
| PNL_DOCS_VERSION: ${{ github.event_name == 'pull_request' && '0.0.0.0' || '' }} |
There was a problem hiding this comment.
Why is this needed in addition to the tag?
There was a problem hiding this comment.
So we don't really need the tag anymore for this. With the switch to setuptools_scm, the tag here didn't work to spoof the version because it is only read at package install time when _version.py is generated. Based on the comment that says moving the tag before the install would mess up mdf's dependency resolution when the package is really (0.0.0.0). We can drop the tag completely now. I did that first but it caused big one-time diff on this PR because the base build we are comparing against is still using the versioneer and without the tag it gets the correct version. So the tag is really vestigial at the point, it can be removed.
There was a problem hiding this comment.
ah, makes sense, so removing the tag is a follow up?
| try: | ||
| from ._version import __version__ # noqa: E402 | ||
| except ImportError: | ||
| from importlib.metadata import version as _pkg_version, PackageNotFoundError |
There was a problem hiding this comment.
Why would this import ever fail?
and why should we silently change the version number instead of reporting failure in that case.
There was a problem hiding this comment.
_version.py is generated by setuptools-scm at build/install time and is gitignored now, so it's absent when importing straight from a source tree that puts the repo root on sys.path ahead of any installed copy. This is what we are doing in docs conf.py actually. The fallback gets it from package metadata in that case. If the package is not actually installed, then we fallback to unknown version.
There was a problem hiding this comment.
Why should it work when the package is not actually installed? Why can't it fail?
kmantel
left a comment
There was a problem hiding this comment.
I think it would also be good to compare the distributions produced with and without these changes, and twine check, if you haven't.
Also, I'm not sure if it's expected or not, but it seems odd looking at the doc difference comments on the PR: for some of them, the version is 0.0.0.0 on the base and 0.19.x.x on the head, and for others it's 0.19.x.x on the base and 0.0.0.0 on the head.
| # PNL_DOCS_VERSION pins the version for builds where it is resolved at | ||
| # install time rather than live from git, which the tag above cannot |
There was a problem hiding this comment.
Does this mean that the versioning doesn't change with editable installs?
There was a problem hiding this comment.
Yes, this is probably the one big difference between setuptools_scm and versioneer. The installed version is stuck at whatever it was when pip install -e . It does not run git describe every time you import to figure out the version.
| version_file = "psyneulink/_version.py" | ||
| # Guarantee an installable version even when SCM info is unavailable | ||
| # (shallow clones without a reachable tag, git-archive tarballs, etc.) | ||
| fallback_version = "0.0.0" |
There was a problem hiding this comment.
Why is this different than the other numbers including .github/workflows/pnl-ci-docs.yml, docs/source/conf.py:82, and psyneulink/__init__.py:61?
There was a problem hiding this comment.
Whoops, yeah, I think fallback_version in pyproject.toml and the init.py fallback should be 0+unknown. This is what versioneer returned when it couldn't figure out the version. I changed them to be consistent now. The docs ones are just 0.0.0.0 to stay fixed so that the diff doesn't get polluted. Lets leave them the same I think.
- Remove .appveyor.yml; the AppVeyor CI has been unused for a long time, so deleting it is cleaner than deepening its clone for setuptools-scm. - docs/source/conf.py: psyneulink.__version__ is always set, so drop the redundant getattr() fallback and reference the attribute directly. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
@jvesely, It seems like most of the pycodestyle checks we are supported by ruff? Which missing ones are you most concerned with? I think it is missing some of the format based ones but we have most of these ignored anyway it seems. |
@kmantel, good point on the twine check. I just ran it locally and it seems to pass. I will try to compare the distributions.
These difference were tied to a quirk of how we were spoofing the version number for the docs builds so that we wouldn't pollute the diff with a bunch of lines showing different version number differences each PR. The old approach used a v0.0.0.0 tag placed after package install but before docs build. versioneer would pickup this tag and render the docs with v0.0.0.0 for the diff base. However, with setuptools_scm, version is determined at package install time from the tag, not and import time. So this spoof didn't work anymore. I replaced that tag with an environment variable for the docs version spoof. I think this has been resolved now with the follow on commits I made, take a look at the last diff. |
setuptools-scm's file finder includes every git-tracked file in the sdist by default, unlike the previous setup.py/MANIFEST.in flow which only added an explicit set. This ballooned the source distribution from ~2.6 MB to ~115 MB, bundling docs/source/_static (multi-MB images/PDFs/GIFs), Scripts/, Matlab/, test_results/, .github/ and other non-distribution material. Add prune/exclude directives to MANIFEST.in (applied after the scm finder) to drop that content. The sdist returns to ~3.5 MB, in line with the old build, and still passes `twine check --strict`. The wheel is unaffected (it only ships the package + declared data), and CI testing is unaffected (test-release.yml runs tests from a checkout, not from the sdist). Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
Good call on comparing the distributions. The wheel was the same but the sdist was way bigger. It seems the setuptools_scm default is to include any git tracked file in the sdist. With setuptools_scm, MANIFEST.in is used to opt things out with prune and exclude directives. This is the opposite of the current approach so large binary assets from docs and Scripts were getting put in the sdist. I have modified the MANIFEST.in to exclude this stuff so now the sdist is about the same as before. |
|
This PR causes the following changes to the html docs (ubuntu-latest-3.11): See CI logs for the full diff. |
Align the build-time version fallback with the runtime fallback in psyneulink/__init__.py (and with the value versioneer previously reported when it could not determine a version). Both "version unavailable" paths now yield the same string instead of "0.0.0" vs "0+unknown". Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This PR causes the following changes to the html docs (ubuntu-latest-3.11): See CI logs for the full diff. |
| uses: actions/checkout@v7 | ||
| # Full history + tags needed for setuptools-scm to determine the version | ||
| with: | ||
| fetch-depth: 0 |
There was a problem hiding this comment.
This will take a long time. The full history is ~1.5GB and counting. Can it be restricted to ~1000? now that there's an automated PR to master every month, the gaps between tags shouldn't be that big.
From the docs [1,2] at least E12?, E13?, W503, W504. I haven't checked if they're present under different code in other categories. Rather than checking if they're ignored now, the question is whether it'd be useful to check in the future. [1] https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes |
This PR is an effort to move the package to the modern python project TOML. I also decided to swap out
versioneerforsetuptools_scmsince it would let us get rid ofsetup.pyentirely.setuptools_scmdynamically generates _version.py on package install rather than have it tracked insetup.cfgis only needed now forpycodestylesince it doesn't supportpyproject.tomlfor whatever reason. My vote would be to droppycodestylefor ruff in another PR.🤖 AI-Generated Description Below
Consolidate build, packaging, and tooling configuration into a single pyproject.toml and remove the setup.py/versioneer machinery:
python -m build, and update docs conf.py and MANIFEST.in accordingly.Assisted-By: Claude Opus 4.8 noreply@anthropic.com