Skip to content

ci: standardize m-stdlib CI onto the Go m toolchain #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rafael5 merged 5 commits into from
Jun 15, 2026
Merged

ci: standardize m-stdlib CI onto the Go m toolchain #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
453ee86
ci: standardize m-stdlib CI onto the Go `m` toolchain
rafael5 Jun 15, 2026
a343484
docs(memory): record the Go-`m` CI standardization (PR #12)
rafael5 Jun 15, 2026
7a5b289
ci: use the reusable m-ci.yml instead of an inline job
rafael5 Jun 15, 2026
a617d5b
docs(memory): note the move to the reusable m-ci.yml caller
rafael5 Jun 15, 2026
fa433aa
ci: pin m-ci.yml reusable workflow to @main
rafael5 Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
253 changes: 21 additions & 232 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
# m-stdlib CI.
# m-stdlib CI — standardized onto the **Go `m`** toolchain
# (github.com/vista-cloud-dev/m-cli), via the reusable **m-ci.yml**
# (vista-cloud-dev/.github). The reusable workflow provisions `m` + the
# m-test-engine container; the actual gating runs THIS repo's Makefile targets
# (engine-free: drift gates + fmt/lint; engine-bound: test/coverage/ci-json),
# so all repo-specific knowledge (core suites, byte-mode `--chset m`,
# `--routines src`, the lint-severity gate, optional-module exclusions, the 85%
# aggregate coverage gate) stays in the Makefile — shared by CI and local `make`.
#
# Container pinned to yottadb/yottadb-base:latest-master per Phase 0 §6.0
# verification (2026-04-30). YottaDB r2.07 at pin time; mumps/mupip live
# at /opt/yottadb/current/ and are reachable after sourcing ydb_env_set.
#
# IRIS portability check (auxiliary track A5) reintroduced at v0.0.4 as
# a fail-soft (continue-on-error) job. See impl-plan §4 (vendor scope)
# and §8.6 (STDLOG ships alongside the IRIS re-add). The job's purpose
# is to surface portability regressions over time, not to gate merges.
# The Python drift-gate generators (tools/gen-*.py) stay Python and run as the
# `check-manifest`/`skill-check`/`doctest-check`/`check-docs-prose` engine-free
# targets below.

name: CI
on:
Expand All @@ -16,229 +18,16 @@ on:
pull_request:

jobs:
# m/v waterline G1 gate (engine-free) — m-stdlib is layer m; the gate
# scans src/*.m for any ^VSL* (v-layer) reference.
# m/v waterline gate (G1–G4 + meta-shape) — m-stdlib is layer m.
arch:
uses: vista-cloud-dev/.github/.github/workflows/arch-waterline.yml@main

m-stdlib:
runs-on: ubuntu-latest
container:
image: yottadb/yottadb-base:latest-master
options: --user root
strategy:
fail-fast: false
matrix:
ydb-image: ["latest-master"] # add tagged release lines as they're vetted

steps:
- name: Install git before checkout (so checkout does a real clone)
# Critical: the container ships without git. When
# actions/checkout@v4 runs inside the container and can't find
# git, it falls back to GitHub's REST API and downloads the
# source as a tarball — no .git directory. Every later step
# that calls git (manifest-check, skill-check, doctest-check)
# then errors with "Not a git repository".
#
# Installing git BEFORE checkout lets the action do a real
# `git clone` and leave a working .git in $GITHUB_WORKSPACE.
# Diagnosed 2026-05-09 via the previous safe.directory step's
# diagnostic listing of /__w/m-stdlib/m-stdlib/.git — file not
# found, but src/ etc. all present (tarball download).
run: |
apt-get update
apt-get install -y --no-install-recommends git ca-certificates

- uses: actions/checkout@v4

- name: Install Python toolchain
run: |
apt-get install -y --no-install-recommends \
python3.12 python3.12-venv python3.12-dev gcc make
python3.12 -m venv /tmp/venv
/tmp/venv/bin/pip install --upgrade pip

- name: Trust workspace inside the container
# Wildcard safe.directory entry — fine on an ephemeral CI
# runner. Avoids "dubious ownership" errors when later git
# operations run as root against the checked-out workspace.
run: git config --global --add safe.directory '*'

- name: Install m-cli + tree-sitter-m from git checkouts
run: |
# Distribution model: clone-and-install (no package registry).
# tree-sitter-m must land before m-cli so its local checkout
# satisfies m-cli's dependency declaration.
git clone --depth=1 https://github.com/m-dev-tools/tree-sitter-m /tmp/tree-sitter-m
git clone --depth=1 https://github.com/m-dev-tools/m-cli /tmp/m-cli
/tmp/venv/bin/pip install /tmp/tree-sitter-m
/tmp/venv/bin/pip install -e "/tmp/m-cli[lsp]"

- name: Manifest drift check
# Regenerates dist/stdlib-manifest.json + dist/errors.json from
# src/STD*.m via tools/gen-manifest.py and fails on diff against
# the committed artefacts. Mirrors `make fmt-check` — drift means
# somebody touched a `; doc:` block (or src/ structure) without
# re-running `make manifest`. Engine-free — only needs python3
# and git, both already installed above.
#
# `check-manifest` wraps `manifest-check` and additionally
# asserts dist/repo.meta.json is tracked and committed (Phase 0
# tier-1 contract — the org catalog generator fetches that file
# by raw URL and a missing/stale copy would break the smoke
# test in m-dev-tools/.github).
run: |
export PATH="/tmp/venv/bin:$PATH"
make check-manifest

- name: AI skill drift check
# Regenerates dist/skill/{SKILL.md,manifest-index.md,patterns.md,
# error-codes.md} from the manifest + tools/skill-patterns.md
# (WD1). Same drift model as manifest-check: any `; doc:` change
# that flows into the skill artefacts forces a regenerate-and-
# commit. Engine-free.
run: |
export PATH="/tmp/venv/bin:$PATH"
make skill-check

- name: Doctest drift check
# Regenerates tests/STD*DOCTST.m from the manifest's @example
# tags (WD2). Drift means someone added or changed a Pattern-A
# @example without running `make doctest`. Engine-free —
# actually executing the generated doctests is folded into the
# regular `make test` step below, which globs `tests/` and so
# picks up the committed STD*DOCTST.m suites alongside the
# hand-written STD*TST.m ones.
run: |
export PATH="/tmp/venv/bin:$PATH"
make doctest-check

- name: docs/ prose-only gate
# Cross-repo guardrail: docs/ holds only human-readable prose.
# Non-prose artifacts (generated data, JSON/TSV output, examples,
# scaffolding templates) belong elsewhere (dist/, examples/,
# templates/, top-level domain dirs). Engine-free find-based check.
run: make check-docs-prose

# The historical `make setup-ydb` step was removed: the Track A3
# Makefile refactor (commit 6ff7c6d) dropped that target along
# with the vista-meta seeding model. The new flow runs `m test`
# via m-cli's LocalEngine transport, which derives ydb_routines
# from the workspace at runtime — no separate "initialise YDB
# workspace" step required.

- name: Format check
run: |
. /opt/yottadb/current/ydb_env_set
export PATH="/tmp/venv/bin:$PATH"
make fmt-check

- name: Lint
run: |
. /opt/yottadb/current/ydb_env_set
export PATH="/tmp/venv/bin:$PATH"
make lint

- name: Test (TAP)
run: |
. /opt/yottadb/current/ydb_env_set
export PATH="/tmp/venv/bin:$PATH"
m test --format=tap | tee test-results.tap

- name: Upload TAP
if: always()
uses: actions/upload-artifact@v4
with:
name: tap-${{ matrix.ydb-image }}
path: test-results.tap

- name: Coverage (lcov)
run: |
. /opt/yottadb/current/ydb_env_set
export PATH="/tmp/venv/bin:$PATH"
m coverage --format=lcov > coverage.lcov || true

- name: Upload coverage
if: matrix.ydb-image == 'latest-master'
uses: codecov/codecov-action@v4
with:
files: coverage.lcov
fail_ci_if_error: false # flip to true once the corpus is non-empty

iris-portability-check:
# Auxiliary track A5: fail-soft IRIS portability surfacing.
# Runs on PRs only — push-to-main runs are reserved for the gating
# YottaDB job above. continue-on-error means a red IRIS run never
# blocks a merge; the artifact + check status make the regression
# visible in the PR view.
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
continue-on-error: true
container:
image: intersystemsdc/iris-community:latest
options: --user root
steps:
- name: Install git before checkout (so checkout does a real clone)
# Critical: the container ships without git. When
# actions/checkout@v4 runs inside the container and can't find
# git, it falls back to GitHub's REST API and downloads the
# source as a tarball — no .git directory. Every later step
# that calls git (manifest-check, skill-check, doctest-check)
# then errors with "Not a git repository".
#
# Installing git BEFORE checkout lets the action do a real
# `git clone` and leave a working .git in $GITHUB_WORKSPACE.
# Diagnosed 2026-05-09 via the previous safe.directory step's
# diagnostic listing of /__w/m-stdlib/m-stdlib/.git — file not
# found, but src/ etc. all present (tarball download).
run: |
apt-get update
apt-get install -y --no-install-recommends git ca-certificates

- uses: actions/checkout@v4

- name: Install Python toolchain
run: |
apt-get install -y --no-install-recommends \
python3.12 python3.12-venv python3.12-dev gcc make
python3.12 -m venv /tmp/venv
/tmp/venv/bin/pip install --upgrade pip

- name: Trust workspace inside the container
# Wildcard safe.directory entry — fine on an ephemeral CI
# runner. Avoids "dubious ownership" errors when later git
# operations run as root against the checked-out workspace.
run: git config --global --add safe.directory '*'

- name: Install m-cli + tree-sitter-m from git checkouts
run: |
# Distribution model: clone-and-install (no package registry).
# tree-sitter-m must land before m-cli so its local checkout
# satisfies m-cli's dependency declaration.
git clone --depth=1 https://github.com/m-dev-tools/tree-sitter-m /tmp/tree-sitter-m
git clone --depth=1 https://github.com/m-dev-tools/m-cli /tmp/m-cli
/tmp/venv/bin/pip install /tmp/tree-sitter-m
/tmp/venv/bin/pip install -e "/tmp/m-cli[lsp]"

- name: Format check (engine-agnostic)
run: /tmp/venv/bin/m fmt --check src/ tests/

- name: Lint (--target-engine=any)
# The --target-engine=any profile flags constructs that don't
# work on every supported engine. Treat findings here as the
# IRIS portability signal.
run: /tmp/venv/bin/m lint --error-on=error --target-engine=any src/ tests/

- name: Test under IRIS (best-effort)
# m-cli's IRIS runner support is itself work-in-progress. Until
# it lands fully, this step's purpose is to record the failure
# mode in the TAP artifact so the gap is visible. continue-on-error
# at the job level means a non-zero exit here never blocks a PR.
run: /tmp/venv/bin/m test --format=tap tests/ | tee iris-test-results.tap

- name: Upload IRIS TAP
if: always()
uses: actions/upload-artifact@v4
with:
name: iris-tap
path: iris-test-results.tap
# M-library CI on the Go `m` (build m → engine-free gates → m-test-engine →
# test/coverage → fail-soft IRIS).
ci:
uses: vista-cloud-dev/.github/.github/workflows/m-ci.yml@main
with:
engine-free-targets: "check-manifest skill-check doctest-check check-docs-prose fmt-check lint"
engine-targets: "test coverage ci-json"
codecov: true
run-iris: true
55 changes: 40 additions & 15 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,37 +125,62 @@ implement it here first; m-cli imports.
## Setup

m-stdlib ships pure-M source; there is no compiled artifact to install.
The toolchain dependencies are `m-cli` (Python) and the YottaDB runtime.
The toolchain is the **Go `m`** (github.com/vista-cloud-dev/m-cli) — one
binary for fmt / lint / test / coverage — plus a YottaDB runtime (the
`m-test-engine` Docker container by default). The Python drift-gate
generators under `tools/` need only system `python3` (pure-stdlib).

```bash
# Clone m-cli alongside m-stdlib and install it into a venv.
git clone https://github.com/m-dev-tools/tree-sitter-m ~/projects/tree-sitter-m
git clone https://github.com/m-dev-tools/m-cli ~/projects/m-cli
cd ~/projects/m-cli
python3 -m venv .venv
.venv/bin/pip install -e ".[lsp]" ../tree-sitter-m
# Build the Go `m` from m-cli (one static binary; no venv).
git clone https://github.com/vista-cloud-dev/m-cli ~/vista-cloud-dev/m-cli
cd ~/vista-cloud-dev/m-cli && go build -o dist/m .

# Engine: start m-test-engine for `make test` / `make coverage`.
make -C ~/projects/m-test-engine up # provides DockerEngine for m-cli
docker run -d --name m-test-engine ghcr.io/m-dev-tools/m-test-engine:0.1.0
```

The `M` Makefile variable defaults to `$HOME/projects/m-cli/.venv/bin/m`;
override it if your checkout lives elsewhere.
The `M` Makefile variable defaults to `m` on `$PATH`; override it to point at
the build, e.g. `make check M=$HOME/vista-cloud-dev/m-cli/dist/m`. The engine
transport is `$(M_ENGINE_FLAGS)` (default `--engine ydb --docker m-test-engine
--chset m`); override for a host-YDB box, e.g.
`make test M_ENGINE_FLAGS='--engine ydb --chset m'`.

## Test

```bash
make test # engine-bound; needs YDB transport (LocalEngine, DockerEngine, or SSHEngine)
make safe-test # same suites with auto-recovery from vista-meta failure modes
make coverage # gated at 85% per module
make test # core suites via $(M_ENGINE_FLAGS) (default: --docker m-test-engine, byte mode)
make safe-test # legacy vista-meta auto-recovery wrapper (not the Go-m path)
make coverage # aggregate line coverage gated ≥85% (--lcov coverage.lcov)
make check # fmt-check + lint + test (fast dev loop)
make ci # check + TAP + JSON coverage (CI-shaped invocation)
make ci # check + coverage + ci-json (CI-shaped; writes *.json + coverage.lcov)
```

Engine-free checks (`fmt-check`, `lint`, `manifest-check`,
`skill-check`, `doctest-check`) work on a fresh clone without YDB
`skill-check`, `doctest-check`) work on a fresh clone without an engine
configured.

## CI / toolchain (Go `m`)

CI (`.github/workflows/ci.yml`) runs on a plain `ubuntu-latest` runner: it
builds the Go `m` from m-cli (`M_CLI_REF`, default `main` — pin to a tag at
VSL Phase B item 5), runs the Python drift gates + `m` fmt/lint (engine-free),
then brings up the `m-test-engine` container and runs `make test` / `make
coverage` over `--docker` (the locally-proven path). The shared `arch`
waterline gate and a fail-soft IRIS portability job round it out.

Two reconciliations from the legacy Python m-cli are worth knowing:

- **Lint severity.** The Go `m` has no `--error-on=<severity>` flag (its
`--check` fails on *any* finding). The house gate — zero **error**-severity
findings, style/warning advisory — lives in `scripts/m-lint-gate.sh`
(reads `m lint -o json`). `make lint` calls it.
- **Aggregate coverage.** `m coverage --min-percent` gates **aggregate** line
coverage across the measured set, not per-module. m-stdlib already treated
the few sub-85% modules (STDFS, STDHARN, …) as documented exceptions, so the
aggregate gate (currently ~93%) is the faithful continuation. The optional
modules (STDCOMPRESS/STDCRYPTO/STDHTTP — no core suites) are excluded from
the measured set (`CORE_SRC`) so they don't drag the aggregate down.

## Build / generate

The repo commits its own generated artefacts under `dist/` — every
Expand Down
Loading
Loading