Skip to content

Python: turn code_projects into the installable rst_code_example_pipeline package#1367

Open
gusthoff wants to merge 36 commits into
AdaCore:mainfrom
gusthoff:dev/topic/infrastructure/python/rst-pipeline-refactor/2026-06-12/main
Open

Python: turn code_projects into the installable rst_code_example_pipeline package#1367
gusthoff wants to merge 36 commits into
AdaCore:mainfrom
gusthoff:dev/topic/infrastructure/python/rst-pipeline-refactor/2026-06-12/main

Conversation

@gusthoff

@gusthoff gusthoff commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Restructures the Python tooling under frontend/py_modules/code_projects/ into
a real, importable, installable package at
frontend/python/rst_code_example_pipeline/ (src layout, editable install,
console-script entry points), replacing the fragile sys.path[0] /
flat-import assumptions. Structural only — no behavior change.

Changes

  • Rename container py_modules/ → python/; package code_projects →
    rst_code_example_pipeline (distribution rst-code-example-pipeline).
  • Flat/implicit imports → explicit package-relative imports.
  • CLIs exposed as entry points: learn-extract-code, learn-check-code,
    learn-check-block (also runnable via python -m …).
  • Editable install wired into the VM provisioning and requirements.txt.
  • chop + resource moved into the package; sphinx/widget/ now depends on
    the library (dependency direction inverted).
  • toolchain.ini → package data; pyrightconfig.json → [tool.pyright].
  • Updated the full external seam: Makefile, compile_blocks, CI workflows,
    install_toolchain.sh, docs.

Testing / Validation

  • Import smoke from an unrelated cwd; entry points resolve.
  • compile_blocks.py on intro-to-ada — unchanged pass/fail set.
  • make local UNIT=courses/intro-to-ada builds and renders code widgets.
  • pyright green in the package; all CI suites green.

Notes

Build-time optimizations remain out of scope (tracked in
LEARN_INFRASTRUCTURE_IMPROVEMENTS.md). Existing VMs need
pip install -e frontend/python/rst_code_example_pipeline applied (handled by
re-provisioning).

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com

gusthoff and others added 30 commits June 12, 2026 20:28
@gusthoff @claude
Python: remove dead Py2 __future__ import from colors.py
d24b605 
colors.py carried `from __future__ import absolute_import, division,
print_function`, which is inert on Python 3.10+ (the only Python this
project runs on). Remove it. The `from __future__ import annotations`
lines in blocks.py/checks.py/extract_projects.py are NOT touched — those
are live (deferred annotation evaluation).

Part of the py_modules -> python/rst_code_example_pipeline refactor
(structural only, no behavior change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gusthoff
Merge feat/remove-py2-future: drop inert Py2 compatibility import
9d813c5
@gusthoff @claude
Python: convert library modules to explicit relative imports
3ef0b56 
Replace bare `import <module>` with `from . import <module>` in the three
library-only modules (blocks, fmt_utils, toolchain_setup).  This is the
prerequisite step for turning code_projects into a proper importable package.
Bare imports relied on sys.path[0] injection and break as soon as the
directory is not at the front of sys.path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: convert CLI modules to explicit relative imports
5dedc13 
Replace bare `import <module>` with `from . import <module>` in the three
CLI entry-point modules (check_code_block, check_projects, extract_projects).
Intentionally excludes the cross-package import
`from widget.chop import ...` in extract_projects — that is addressed when sphinx/widget/chop.py is moved into the package.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Test script: invoke code_projects CLIs via python -m in compile_blocks
6035947 
Switch compile_blocks.py from path-based subprocess calls
  python3 <abs-path>/extract_projects.py ...
to module invocations
  python3 -m code_projects.extract_projects ...
  python3 -m code_projects.check_projects ...

This is the bridge that makes the test script compatible with explicit
relative imports: `python3 -m package.module` sets __package__ so
relative imports resolve, while PYTHONPATH=…:py_modules still makes the
package findable.  Remove the now-unused PATH_CODE_PROJECTS variable.

Validated: compile_blocks on content/courses/intro-to-ada/ (25 RST
files) exits 0 on the epub VM with these changes applied.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Merge feat/explicit-imports: make intra-package imports explicit
c03b652 
All imports within code_projects now use explicit relative form
(from . import ...).  The test harness (compile_blocks) now invokes
the CLIs via `python3 -m code_projects.<mod>`, which is the bridge
needed until the package is installed in editable mode.

Validated: compile_blocks on intro-to-ada (25 RST files): exit 0 on epub VM.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add rst_code_example_pipeline project and move code_projects …
7d466e2 
…modules

Create the new package project at frontend/python/rst_code_example_pipeline/
with src layout and pyproject.toml, and move all 11 modules + toolchain.ini
from frontend/py_modules/code_projects/ via git mv (history preserved).

- pyproject.toml: name rst-code-example-pipeline, version 0.2.0,
  requires-python >=3.10, console-script entry points
  (learn-extract-code / learn-check-code / learn-check-block),
  setuptools src layout, [tool.pyright] replacing pyrightconfig.json.
- __init__.py __title__: code_projects -> rst_code_example_pipeline.
- pyrightconfig.json and README.md moved to the project root.
- py_modules/code_projects/ is now empty (removed from the tree).

Intra-package imports remain `from . import …` (converted in the preceding explicit-imports commit); the
widget.chop cross-package import in extract_projects remains (addressed when sphinx/widget/chop.py is moved into the package).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add cli/ subpackage with console-script main() wrappers
da5f073 
Add rst_code_example_pipeline/cli/ with three entry-point modules:
  cli/extract.py     -> learn-extract-code
  cli/check.py       -> learn-check-code
  cli/check_block.py -> learn-check-block

Each wrapper calls runpy.run_module(..., run_name='__main__',
alter_sys=True), which triggers the existing if __name__ == '__main__'
block in the target module.  This avoids touching the module-level global
variables those blocks rely on, keeping the change minimal.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Vagrant: install rst_code_example_pipeline editable in both VMs
ee3a02a 
After the frozen-requirements install, install the pipeline package in
editable mode so the console-script entry points (learn-extract-code,
learn-check-code, learn-check-block) are available on PATH inside the VM
venv.  Also update the toolchain.ini file-provision source path for both
VMs from the old py_modules/code_projects/ location to the new
python/rst_code_example_pipeline/src/rst_code_example_pipeline/ location.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Makefile: drop py_modules from PYTHONPATH (editable install replaces it)
7bd5114 
The rst_code_example_pipeline package is now installed in the VM venv via
`pip install -e`, so Python's import system finds it without a manual
PYTHONPATH entry.  Remove :py_modules from TEST_LOCAL_DRIVER and
TEST_LAMBDA_DRIVER.  The :sphinx entry remains (sphinx/widget is still a
non-installed module imported via PYTHONPATH until sphinx/widget/chop.py is moved into the package).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Test script: use entry points and rst_code_example_pipeline in compil…
11f8827 
…e_blocks

Three changes:
1. Remove dead `import colors as C` (colors.py is now inside the package;
   the import was unused in compile_blocks anyway).
2. `import code_projects.fmt_utils` -> `import rst_code_example_pipeline.fmt_utils`.
3. Replace `python3 -m code_projects.{extract,check}_projects` subprocess
   commands (the temporary `python3 -m code_projects.X` invocation introduced while the package was being set up) with the installed entry points
   `learn-extract-code` / `learn-check-code`.  These are available once
   the editable install runs in the VM venv.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
CI: point type-check workflow at frontend/python/rst_code_example_pip…
0dbf100 
…eline

Update the code_projects Type Check workflow:
- Both push/pull_request paths: triggers now watch
  frontend/python/rst_code_example_pipeline/** instead of the old
  frontend/py_modules/code_projects/**.
- working-directory updated accordingly so pyright finds pyproject.toml
  (which now carries [tool.pyright] in place of the old pyrightconfig.json).

The editable-install step before pyright will be added in a follow-up commit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: use setuptools.build_meta backend (compatible with older setu…
e2a2e8c 
…ptools)

setuptools.backends.legacy:build was added in setuptools 67.8; the VM's
venv has an older version.  Switch to setuptools.build_meta which is
available from setuptools 40.8 onwards.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Merge feat/package-move-and-install: real importable installed package
2ef5c41 
frontend/py_modules/code_projects/ → frontend/python/rst_code_example_pipeline/
(src layout, editable install, console-script entry points).

Validated on the epub VM:
- import smoke from /tmp ✅
- learn-extract-code/check-code/check-block --help ✅
- compile_blocks on intro-to-ada (25 RST files): exit 0 ✅

Note: widget.chop cross-package import in extract_projects remains; addressed when sphinx/widget/chop.py is moved into the package.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Test script: add import and entry-point smoke tests for the package
8b28a45 
Add rst_code_example_pipeline/tests/test_smoke.py (unittest, 14 tests):
- TestPackageMetadata: __title__ and __version__ assertions.
- TestModuleImports: each of the 9 package modules is importable;
  extract_projects and check_projects are skipped when widget.chop
  is not on PYTHONPATH (sphinx/widget/chop.py has not yet been moved into the package), all others run unconditionally.
- TestEntryPoints: each cli/main() accepts --help and exits 0;
  extract and check are similarly skipped when widget.chop is not importable.

Run: python -m unittest discover -s tests/
(from frontend/python/rst_code_example_pipeline/, with venv active;
add PYTHONPATH=../../sphinx to enable tests that import extract_projects).

Verified: 14/14 pass on epub VM with PYTHONPATH=sphinx.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Merge feat/package-smoke-tests: add package smoke tests
a0d8375 
14 unittest cases: package metadata, module imports, entry-point --help.
Tests that import extract_projects/check_projects skip when widget.chop is not on PYTHONPATH; all 14 pass on the epub VM with
PYTHONPATH=sphinx.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: move chop + resource into rst_code_example_pipeline (git mv)
da77b9a 
Move sphinx/widget/chop.py and sphinx/widget/resource.py into the
package via git mv (history preserved):
  frontend/python/rst_code_example_pipeline/src/rst_code_example_pipeline/

chop.py's `from .resource import Resource` is already a relative import
and continues to work in its new location within the package.

Update extract_projects.py: `from widget.chop import …` → `from .chop import …`
— the cross-package import is now an intra-package relative import.

Drop `extraPaths = ["../../sphinx"]` from [tool.pyright] in pyproject.toml;
the package no longer has any external dependency on sphinx/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Sphinx: import chop/resource from rst_code_example_pipeline in widget
745f88b 
Update sphinx/widget/widget.py to import ChopStrategy, manual_chop,
cheapo_gnatchop, real_gnatchop, and Resource from the package rather than
from sibling modules within sphinx/widget/ (which no longer exist).
This inverts the dependency direction: sphinx/widget/ now depends on the
library instead of the other way around.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Test script: update execute_rst_code_blocks chop import
376bd34 
chop.py is no longer in sphinx/widget/; update the import to
rst_code_example_pipeline.chop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Sphinx: update widget chop/resource tests to new import path
83e706e 
test_chop.py and test_resource.py imported from widget.chop / widget.resource;
both modules now live in rst_code_example_pipeline.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Test: drop widget.chop import guard from smoke tests (chop now in pac…
2db0674 
…kage)

chop.py and resource.py are now inside rst_code_example_pipeline, so
extract_projects.py uses an intra-package import.  The four
@unittest.skipUnless decorators that guarded tests requiring widget.chop
on PYTHONPATH are no longer needed; all 14 tests now run unconditionally.

Remove _widget_available(), the unused `sys` import, and the now-stale
docstring note about sphinx/ PYTHONPATH.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Merge feat/move-chop-resource: invert widget→pipeline dependency
49833d6 
Move sphinx/widget/chop.py and sphinx/widget/resource.py into the
rst_code_example_pipeline package (src layout, history preserved via
git mv).  sphinx/widget/widget.py now imports ChopStrategy, the chop
functions, and Resource from the package — dependency direction inverted.

All consumers updated: extract_projects (intra-package relative import),
widget.py (absolute cross-package import), execute_rst_code_blocks,
sphinx/tests/test_chop, sphinx/tests/test_resource.

package/pyproject.toml: extraPaths=[../../sphinx] removed — no external
sphinx dependency remains in the package.

smoke tests: @unittest.skipUnless widget.chop guards removed; all 14
tests now run unconditionally.

Validated on the epub VM:
- 14/14 package smoke tests pass (no skips)
- 6/6 sphinx widget tests pass (test_chop + test_resource)
- compile_blocks on content/courses/intro-to-ada/ (25 RST files): exit 0
- make local UNIT=courses/intro-to-ada: build succeeded, widgets render

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: move toolchain.ini into data/ subdirectory (git mv)
ef4e101 
Relocate the configuration file:
  src/rst_code_example_pipeline/toolchain.ini
→ src/rst_code_example_pipeline/data/toolchain.ini

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: load toolchain.ini via importlib.resources; declare as packag…
7ca089f 
…e data

Switch toolchain_info.py from os.path-based file discovery to
importlib.resources.files(), which is the idiomatic Python 3.10+ way to
locate data files bundled inside a package. The config is now read with
  files('rst_code_example_pipeline').joinpath('data/toolchain.ini').read_text()
instead of constructing an absolute path relative to __file__.

Add [tool.setuptools.package-data] to pyproject.toml so the .ini file
under src/rst_code_example_pipeline/data/ is included in the installed
distribution.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Vagrant: update toolchain.ini path after data/ subdirectory move
324596e 
The file was moved from src/rst_code_example_pipeline/toolchain.ini
to src/rst_code_example_pipeline/data/toolchain.ini. Update the
provision source path for both the web and epub VMs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
CI: update toolchain.ini path in install_toolchain.sh
a9e00c0 
Point TOOLCHAIN_CONFIG at the new location under the Python package
after the file was moved into the data/ subdirectory and the package
renamed from py_modules/code_projects to
python/rst_code_example_pipeline.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: remove pyrightconfig.json (superseded by pyproject.toml)
1450e02 
[tool.pyright] in pyproject.toml now carries the pyright configuration.
The standalone pyrightconfig.json is redundant and was carrying a stale
extraPaths entry that no longer applies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Docs: refresh package README with entry-point commands
8eab0c3 
Update all script invocations from
  python3 py_modules/code_projects/<script>.py
to the console-script entry points installed by the package:
  learn-extract-code, learn-check-code, learn-check-block

All original sections are preserved (Introduction, Simple usage, Verbose
mode, JSON file, JSON file and build directory, Checking single code
blocks, Forced check, Testing single code block, Style check).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Docs: add python/README.md index for Python packages
fa8a9b1 
Provides a brief overview and table of Python packages under
frontend/python/ so the directory purpose is clear at a glance.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Merge feat/relocate-config-docs: move toolchain.ini into data/ subdir
4d1aa10 
Move toolchain.ini into data/, switch toolchain_info.py to
importlib.resources, declare package-data, and update all callers
(Vagrantfile, install_toolchain.sh). Also removes the stale
pyrightconfig.json and refreshes the package README with the new
entry-point command names.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gusthoff and others added 6 commits June 13, 2026 02:28
@gusthoff @claude
CI: install package before running pyright type check
187f307 
The type checker needs the package installed so that imports resolve
correctly. Add an editable install step before the pyright run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Merge feat/ci-type-check: install package before pyright run
aa9bb56 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Docs: update root README and CONTRIBUTING for renamed package
985f56d 
Replace py_modules/code_projects references with
rst_code_example_pipeline and update entry-point commands
(learn-extract-code / learn-check-code) in the root README. Drop the
manual PYTHONPATH export — the package is installed during provisioning.

Update CONTRIBUTING.md toolchain.ini path to reflect the file's new
location under data/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Merge feat/docs-refresh: update root docs for renamed package
32a14b6 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: drop prefix from entry-point command names
6328225 
Rename learn-extract-code/learn-check-code/learn-check-block to
extract-code/check-code/check-block. The commands run inside an isolated
venv, so a namespace prefix adds no value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff
Editorial: align line escapes
f94e21e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

website infrastructure

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gusthoff