Skip to content

Add Copier migration support and update template docs#393

Open
steve-downey wants to merge 30 commits into
bemanproject:mainfrom
steve-downey:copier
Open

Add Copier migration support and update template docs#393
steve-downey wants to merge 30 commits into
bemanproject:mainfrom
steve-downey:copier

Conversation

@steve-downey

Copy link
Copy Markdown
Member

Summary

This switches exemplar's template workflow over to Copier in a way that is usable for both new stamped repositories and older forks that started life as a plain GitHub template copy.

Copier is a modern project templating tool for keeping a generated repository tied back to its source template. In practice, that gives us two things we want here:

  • a declarative template definition in copier.yml
  • a persistent answers file that lets generated repositories run copier update later

Why

The previous flow was effectively a one-shot stamp-out process. It could generate a project, but it did not leave enough metadata behind for template-driven updates afterward.

This change closes that gap by making stamped projects retain the information Copier needs to understand:

  • which template they came from
  • which template revision they were generated from
  • which answers were used during generation

That lets older exemplar-based repositories be rebased onto a Copier-generated baseline and then participate in template updates going forward.

What changed

  • add a rendered .copier-answers.yml template so generated repositories keep Copier metadata
  • seed stable template source and commit information during stamp.sh and local template validation
  • update the self-check to account for the answers file while preserving the existing round-trip checks
  • document how to migrate an already-copied exemplar repository onto a Copier-managed base
  • sync the generated exemplar docs with the template sources

Parity

At this point the Copier-based workflow is functionally at parity with the prior cookiecutter-style stamping flow:

  • new projects can still be stamped from exemplar with the same practical outcome
  • local template validation still round-trips exemplar and checks for leakage into non-exemplar output
  • maintainers now also get persisted template metadata, which unlocks future copier update support

So this is not a reduction in template capability; it preserves the existing stamp-out behavior and adds an update path that the earlier flow did not provide.

Validation

  • ./copier/check_copier.sh

@steve-downey

Copy link
Copy Markdown
Member Author

A few reviewer notes to make the diff easier to scan:

  • The functional change is centered on template/.copier-answers.yml.jinja plus the hidden template_src_path and template_commit questions in copier.yml. Those are what make generated repositories updateable by Copier later.
  • stamp.sh and copier/check_copier.sh now explicitly seed canonical template metadata because both render from a .git-free snapshot. Without that, stamped repos would not get a usable _src_path / _commit pair.
  • The README/CONTRIBUTING changes are mostly there to document the migration path for older exemplar clones and to keep the generated exemplar docs in sync with the template sources.
  • The parity claim is intentionally narrow: stamp-out behavior and local validation still work as before, while Copier now also persists enough metadata for a future copier update flow.

Validation run locally:

  • ./copier/check_copier.sh

@ednolan

ednolan commented May 8, 2026

Copy link
Copy Markdown
Member

I need to review this more thoroughly but on a first skim-through this looks good

@coveralls

coveralls commented May 17, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 100.0%. remained the same — steve-downey:copier into bemanproject:main

Accidentally recreated the cookiecutter CODEOWNERS file. This should not
be part of the copier branch, but it was accidentally added in a merge
commit. This commit removes the file from the copier branch, which is
the correct state for that branch.
Some yaml files were slightly reformatted, leaving beman-submodule
failing in CI.
Use the {%- raw -%} forms to control whitespace. Also use -u for diff
to show the "unified" format, which is easier to read.
@steve-downey steve-downey marked this pull request as draft May 22, 2026 15:48
@steve-downey

Copy link
Copy Markdown
Member Author

Moved to Draft: Not high risk that anyone would merge, but there are a few round-trip issues I'm cleaning up, as well as docs and scripts on how to use the copier infrastructure.
In particular, keeping the connection and history from exemplar as stamp.sh does shouldn't be necessary as copier has its own tools for working with git to update with new features from the template.

@ednolan

ednolan commented May 22, 2026

Copy link
Copy Markdown
Member

I do want to move from cookiecutter to copier, but I would really prefer to keep around the .exemplar_version file/mechanism, since I need it for some of my own internal tooling that keeps repositories up to date, which copier doesn't entirely replace for me.

Move the randomization into copier.yml and remember the results for future
updates. Move the randomization out of the github action templates.
Provide new directions to replace `stamp.sh` and describe the intented
lifecycle workflow of a generated Beman project.
Add the new README text
Fix whitespace
Fix quote style to match
@steve-downey

Copy link
Copy Markdown
Member Author

Reviewer Guide: Copier Automation and Testing

These latest changes finalize the template maintenance pipeline to ensure the template never drifts from the reference implementation and doesn't break downstream users.

What to look at:

update_templates.py: A new automation script that synchronizes root codebase changes directly into the template Jinja files.
test_standard_project.sh & ci_tests.yml: I've added active testing for the template output in CI. It generates, builds, and tests downstream standalone projects across a matrix of configurations (Catch2/GTest and Modules ON/OFF).
MAINTAINERS.md: New documentation explaining the update loop and troubleshooting steps.
What to look for (and why this branch is ahead of main):

You will notice an infra submodule bump and a CODEOWNERS tweak in this PR that aren't on main yet. I deliberately included these as the payload to test the Copier update machinery.
Validation:

To prove the update path, I executed a live copier update against the downstream transcode project. It safely delivered the infra changes without touching transcode's custom Catch2 logic. The project built flawlessly, passed its test suite, and I have successfully pushed the update.

Comment thread .github/workflows/vcpkg-release.yml Outdated
@steve-downey steve-downey marked this pull request as ready for review May 23, 2026 21:53
@steve-downey

steve-downey commented May 23, 2026

Copy link
Copy Markdown
Member Author

Pull Request Summary

This PR migrates the Exemplar templating from Cookiecutter to Copier. Over the course of stabilizing this migration, several underlying infrastructure and CI testing issues became hard blockers for the matrix pipeline. As a result, this PR includes some vital infrastructure fixes that are strictly outside the scope of Copier, but were integral to getting the 120+ CI checks consistently completely green.

Here is the breakdown of the changes:

1. Primary Copier Migration (Core PR Scope)

  • Template Engine Transition: Migrated from Python Cookiecutter to Copier, adopting copier.yml for explicit configuration, lifecycle tracking, and template updates.
  • Sync Synchronization Scripts: Added update_templates.py to seamlessly sync changes from the repo root directly into the .jinja templates natively, alongside check_copier.sh to enforce parity.
  • Streamlined Validation Tooling: Updated test_standard_project.sh to leverage uvx copier to generate and test variants dynamically (GTest, Catch2, Modules).

2. Infrastructure & CI Matrix Fixes (Integral Fixes)

To achieve a passing pipeline, the following underlying testing infrastructure bugs were patched out (which otherwise would have been separate PRs):

  • Robust CMake Version Validation Matrix: Added a new explicit CI job (copier-cmake-matrix) and test_cmake_matrix.sh wrapper. This fetches pristine CMake binaries across a matrix of versions (3.30 to 4.3.2) dynamically via uv virtual environments, guaranteeing that templates natively compile correctly across all consumer CMake patch boundaries.
  • MSVC Module & CMake 4.3.3 Support: The CI runners pulling latest CMake binaries functionally broke CXX_MODULE_STD toolchain detection for CMake 4.3.3. I patched infra/cmake/enable-experimental-import-std.cmake to specifically map the correct experimental UUIDs for 4.3.3.
  • Submodule Tracking Mitigation: Because the aforementioned CMake toolchain patch is still pending upstream in bemanproject/infra#62, infra/.beman_submodule in this PR has been temporarily pointed to a local fork commit to bypass the strict beman-submodule check sync drift during workflows.
  • Jinja Action Formatting & Linting Parity: Resolved latent end-of-file-fixer hook failures in .github/workflows/vcpkg-release.yml caused by Jinja whitespace eaters ({%- endraw %}) dropping necessary EOF newlines, along with escaping nested ${{ matrix }} evaluation hooks in workflows so copier doesn't mangle GitHub Action executions.

Merge Note: Once this merges and upstream bemanproject/infra#62 resolves, we can point the infra submodule configuration completely back to bemanproject/infra on main.

@ClausKlein ClausKlein left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it really necessary that a copy of infra submodule is under templates?
It may be created with Beman-submodule, or not?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants