Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dist/repo.meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"layer": "m",
"license": "AGPL-3.0",
"agent_instructions": "AGENTS.md",
"verified_on": "2026-06-13",
"verified_on": "2026-06-14",
"exposes": {
"modules": "dist/stdlib-manifest.json",
"errors": "dist/errors.json"
Expand Down
46 changes: 23 additions & 23 deletions dist/stdlib-manifest.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/memory/MEMORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

One line per memory file. Content lives in the files, not here.

- [iris-native-backends](iris-native-backends.md) — PR #1: the 3 optional modules' IRIS dispatch arm uses the **inlined `$zversion["IRIS"` probe** (not a public engine helper — that part of the PR was dropped as superseded); dual-engine local-test runbook (**YDB needs `--chset m`**, rebuild `/tmp/m` for the flag); `m-test-iris` embedded-Python is non-functional so STDCOMPRESS-IRIS is unverifiable locally; how the stale PR was landed without merge/rebase/force-push (forward-commit-to-master-tree).
- [waterline-g1-gate](waterline-g1-gate.md) — the m/v waterline **G1 gate** (`m arch check` in m-cli) — dependency-direction (v→m only); how `layer` is declared (dist/ meta vs root `repo.meta.json` for m-cli's gitignored dist/), check-manifest doesn't schema-validate the meta, and the v-cli registry-regen `go mod tidy` dep. Built s12 (loose end C).
- [t0b2-msl-kids-base](t0b2-msl-kids-base.md) — VSL T0b.2 (MSL KIDS-install-as-green): **YDB leg GREEN — 15/15 test-in-place** after the m-ydb gbldir (`e5dcf85`) + v-pkg streamed-install (`aa1991f`) fixes. **IRIS leg (s9):** `raises^STDASSERT` **now ported to IRIS** (try/catch `irisRaises` branch; YDB byte-identical) → STDFMT/STDARGS clean + STDASSERTTST 40/40 both engines + STDUUID P2 gone; remaining IRIS crashes are **non-raises**. **(s10):** file I/O made dual-engine — STDFS portable facade (`$$openRead/Write/Append`+`readLn`) + STDOS.env IRIS arm + 5 consumers migrated; **STDFSTST 50/50 both engines, YDB full 2098/0**. **But the consumer SUITES still don't go green on IRIS** — separate non-file blockers (STDJSON **byte-mode** parser, STDCSV **`@cb@` indirection**, **wide-char** descriptions). file-I/O ≠ green suites; see §s10. Full 15/15 needs byte-mode + callback-idiom + wide-char work (out of file-I/O scope). ≤8-char-name decision keeps STDASSERT/STDSEMVER as a rename follow-up.
- [vsl-doc-gaps-v0.2](vsl-doc-gaps-v0.2.md) — how the VistA Standard Library architecture doc's §12 VDL gaps resolved at v0.2; the vdocs `XU:XU:UG` over-collapse defect that blocks gold-promotion of the Kernel feature guides.
Expand Down
55 changes: 55 additions & 0 deletions docs/memory/iris-native-backends.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
name: iris-native-backends
description: PR #1 reconciliation — the 3 optional modules' IRIS dispatch arm uses the inlined `$zversion["IRIS"` probe (not a public engine helper); dual-engine local-test runbook; m-test-iris embedded-Python gap; how a stale PR was landed without merge/rebase/force-push.
metadata:
type: project
---

**IRIS-native backends for the 3 optional modules** (STDCRYPTO / STDCOMPRESS /
STDHTTP) landed via **PR #1** ("B2"), reconciled onto the s9–s12 IRIS sweep on
2026-06-14. The stale PR predated that sweep and was CONFLICTING + partially
superseded; reconciliation kept only the novel payload.

**The engine seam is the inlined `$zversion["IRIS"` probe — the house idiom
for runtime modules.** Each module's dispatch helper gets an
`if $zversion["IRIS" quit $$iris…(…)` arm ahead of the YDB `$&pkg.fn` path
(STDCRYPTO → `$SYSTEM.Encryption.SHAHash/.HMACSHA`; STDHTTP →
`%Net.HttpRequest`; STDCOMPRESS → embedded-Python zlib + ctypes/zstd). The
PR's own `$$engine^STDOS()` helper was **dropped as superseded** — master's
STDOS is already IRIS-ported by inlining the same probe, STDASSERT already has
its `irisRaises` try/catch arm (s9), and `$$engine^STDHARN()` (internal) exists
if a helper is ever wanted. **Future IRIS arms in m-stdlib runtime code: inline
`$zversion["IRIS"`, do not add a cross-module engine helper.** See
[[t0b2-msl-kids-base]] for the s9–s12 idioms ($ZTIMESTAMP clock, xecute-built
dispatch, STDFS facade).

**Dual-engine local-test runbook** (the canonical invocations — the Makefile
targets omit these flags because they target host-YDB CI / the Python m-cli):
- Build a current `m`: the committed `m-cli/dist/m` can be stale; rebuild with
`cd m-cli && GOPROXY=file://$HOME/go/pkg/mod/cache/download GOSUMDB=off GOFLAGS=-mod=mod go build -o /tmp/m .`
(the `--chset` flag was missing from the committed binary).
- **YDB:** `m test tests/X.m --engine=ydb --docker=m-test-engine --routines src --chset m`
— **`--chset m` is mandatory**: the m-test-engine container defaults to
`ydb_chset=UTF-8`, but the byte/binary modules need byte mode (else
`%YDB-E-BADCHAR` on raw digest/compress bytes).
- **IRIS:** `m test tests/X.m --engine=iris --docker=m-test-iris --routines src --namespace USER`
(byte mode is inherent on IRIS).
- Results: STDCRYPTOTST 23/23, STDHTTPTST 68(YDB)/67(IRIS), STDCRYPTODOCTST 1/1,
STDCOMPRESSTST 59/59 (YDB) all green.

**Gap — `m-test-iris` (iris-community image) has non-functional embedded
Python**: `%SYS.Python` class exists but `Import("sys").version`→0, and the
STDCOMPRESS IRIS path aborts non-trappably (0/0). So **STDCOMPRESS-IRIS can't
be verified locally**; the PR validated it on `vista-iris` (working embedded
Python). The reconciled code is the PR's vista-iris-validated logic with only
the (proven-correct) seam changed. Discoveries register has the detail.

**Landing a stale PR under this sandbox** (`git merge`, `git rebase`,
`git clean`, `rm`, and force-push are all denied): don't rebase. Make a
**forward commit on the branch** whose tree equals `master + additive
backends` — `git checkout origin/master -- .` (sync whole tree to master),
re-`git rm` master's deletions, `git checkout HEAD -- <the backend files>` to
restore the wanted changes, regenerate `dist/`. GitHub's 3-way merge is then
clean (master's changes appear identically on both sides; backends are purely
additive in files master never touched) and the PR diff is minimal. Verify
with `git diff --cached --stat origin/master` before committing.
7 changes: 5 additions & 2 deletions docs/tracking/discoveries.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
created: 2026-05-10
last_modified: 2026-06-07
revisions: 3
last_modified: 2026-06-14
revisions: 4
doc_type: [NOTES]
---

Expand Down Expand Up @@ -71,6 +71,9 @@ requires "no open P0/P1 entries against those subjects."
| 2026-06-13 | P2 | m-stdlib | Consumer SUITES stay IRIS-red for **non-file** reasons (byte-mode · `@cb@` indirection · wide-char) | After the file-I/O facade, the consumer suites still 0/0 on IRIS — **the file I/O was only part of why.** Confirmed by isolated probes: **STDJSON** — `$$parse^STDJSON` (no file) crashes on IRIS = the **byte-mode** assumption (STDJSON/STDB64/STDHEX/STDCSPRNG treat 1 char == 1 byte; IRIS strings are 16-bit Unicode) — documented charset constraint; **STDCSV** — `do @callback@(rownum,.fields)` (indirection-with-args) crashes on IRIS even with ASCII (`parseFile` callback dispatch); **STDCSV/STDSEED/STDLOG/STDXML** descriptions also carry non-ASCII (wide-char output, the m-iris GetOut/session-capture lane). STDCSV's *parser core* (`$$parse` of a string) DOES pass on IRIS (2/2), so it's the callback + wide-char, not the parser. | Out of file-I/O scope — separate follow-ups: byte-mode portability (big; affects the byte-oriented modules), an IRIS-portable callback-dispatch idiom for `parseFile^STDCSV` (replace `@cb@(args)` with an `xecute`d call or a fixed dispatch), and the wide-char capture path. Tracked so the next session knows the file-I/O refactor alone does NOT green these suites. | **superseded 2026-06-13 (s11)** — the "STDJSON byte-mode" half was a MISDIAGNOSIS (see s11 row below); the `@cb@` + wide-char halves stand (wide-char now fixed). |
| 2026-06-13 | P1 | m-stdlib | **RE-BASELINE (s11): T0b.2 IRIS leg is 10/15; the gap is 4 code fixes, NONE byte-mode** | Rebuilt `m-iris/dist/m-iris` from `m-iris-driver@49a5b00` (the GetOut wide-char fix; the prior dist binary predated it) and ran `kids-test-in-place.sh iris` on foia. **10/15** (was 6/15). **The GetOut wide-char fix works on remote** — STDURL 150/0, STDREGEX 102/0, STDFMT 62/0 all green now. **Crucially STDB64 (55/0) + STDHEX (49/0) — the byte family — PASS on IRIS**, so there is **NO byte-mode blocker**; the s10 "STDJSON byte-mode" claim was wrong (it never isolated the crash). The 5 reds, each a DISTINCT cause: **(1) STDJSON** crash `<SYNTAX> parse+12` for ALL inputs = the **unguarded `zgoto`-`$etrap`** idiom (parse + encode) — IRIS rejects YDB `zgoto LEVEL:label`; STDFS/STDHARN/STDASSERT.raises already guard the same idiom with an `if $zversion["IRIS" … quit` arm, STDJSON is the only base module that didn't. **(2) STDXML** crash `<SUBSCRIPT> parseElement+20 myNs("")` = **null/empty-string subscripts** (`myNs("")`/`nsMap("")` for the default namespace) — IRIS rejects null subscripts, YDB allows. **(3) STDCSV** won't COMPILE on IRIS: `<COMPILE> #5475 Expected end of line : '@callback@(curRow,.fields)'` — **IRIS has no ARGUMENT indirection** (`do @cb@(args)`), only name-indirection; whole routine fails → 0/0. **(4) STDDATE** 1 fail "year in plausible range": `now()` → `3567-05-6.157218T…` because it reads `$ZHOROLOG` as YDB's 4-comma `d,s,u,t` but **IRIS `$ZHOROLOG` is single elapsed-seconds**. **(5) STDUUID** 2 fails were COLLATERAL from the crashers in the same sequential process — source AND installed both 131/131 in isolation; no fix. | **4 code fixes (this session, TDD dual-engine, YDB byte-identical):** #1 STDJSON IRIS try/catch arm (parse+encode); #2 STDXML null-subscript fix [USER FORK → **sentinel-key in code**, chosen 2026-06-13] **[LANDED — `$$dfltNsKey()`=single space; STDXMLTST 209/209 both engines]**; #3 STDCSV `xecute`-built callback dispatch **[LANDED — + a latent STDFS readLn `$ECODE`-on-EOF bug; STDCSVTST 59/59 & STDFSTST 50/50 both engines]**; #4 STDDATE `$ZTIMESTAMP` IRIS arm in now() **[LANDED — STDDATETST 66/66 both engines]**. #5 STDUUID none. Then re-run on foia → expect 15/15. **#1 COMPLETE — STDJSONTST 209/209 BOTH engines (YDB + foia remote).** Beyond the etrap port: fixed TWO latent UTF-8 OPERATOR-PRECEDENCE bugs in emitUtf8 + the surrogate combine (M has no precedence — `$char(192+cp\\64)` evaluated as `(192+cp)\\64`; garbage on BOTH engines, latent because old tests used literal-byte passthrough); rewrote the 2 byte tests to `\u`-escapes+`$char`; graceful empty-key reject on IRIS (user decision — null local subscript is unconditional on IRIS incl. foia; documented in-code + stdjson.md + users-guide); fixed irisParse `$ECODE`-on-failure pollution (same class as STDFS readLn EOF); migrated STDJSONTST file tests off raw YDB `open:(newversion)` to the STDFS facade — **but STDJSONTST has 2 further IRIS tail issues exposed once the crash clears: (a) byte-exact UTF-8 tests use `$zchar` (unsupported on IRIS) + literal-multibyte source (byte-mode boundary) → needs the byte-mode decision; (b) `tParseObjectEmptyKeyAllowed` stores `root("")` = null subscript → same sentinel-key class as STDXML.** Lesson: each crashing suite stacks several IRIS issues; 10/15 undercounts per-suite work. | **CLOSED (s11): IRIS leg 17/15→17/17 on foia** (`suites=17 pass=1483 fail=0`). All 4 code fixes + STDUUID `unixMs` `$ztimestamp` IRIS arm (v7 time prefix was loosely-monotonic on IRIS via the YDB `$zhorolog` assumption) landed; STDFS+STDOS added to the base (STDOS ported to IRIS dual-engine first). YDB per-suite green; vehu in-place loop re-run owed. |
| 2026-06-07 | P2 | vdocs | `consolidate` over-collapses the Kernel-8.0 per-feature User Guides into one anchor | Surfaced fetching the dedicated KIDS / Device Handler / TaskMan guides to fill the [VistA Standard Library architecture §12](../plans/msl-vsl-architecture.md) doc gaps. All ~41 distinct Kernel-8.0 `krn_8_0_{dg,sm}_*_ug` feature guides (KIDS, Device Handler, TaskMan, Alerts, Common Services, …) are assigned the **same `XU:XU:UG` anchor key** by the catalog/identity logic. `consolidate` treats them as one version group, keeps a single "winner", and demotes the other ~40 to `is_latest=0` — so they are fetched/converted/enriched/normalized and present in `index.db`, but **excluded from the FTS gold-search surface** (`vdocs ask` can't see them) and have no `documents/gold/consolidated/.../body.md` anchor. Same defect blocks `VIAB/via_vip_user_guide` (a separate fetch, also stuck at convert/promote). | m-stdlib side: the architecture doc's findings were read **directly from the normalized silver bodies** (`documents/silver/text/03-normalized/XU/<slug>/body.md`), so the doc does not depend on the fix. **vdocs side (open, upstream):** the `catalog`/identity stage must derive a per-document anchor key for granular feature guides (e.g. include `doc_subject`/slug, not just `doc_code=UG`), then re-run `consolidate`→`index`→`relate`→`manifest`. Until then these guides are cited by their fetched body paths and marked 🟡 gold-promotion-pending in the architecture doc §12/§13. | open (in vdocs) |
| 2026-06-14 | docs | m-stdlib | PR #1 IRIS backends reconciled — engine seam is the inlined `$zversion["IRIS"` probe, not a public `$$engine^STDOS()` | Stale feature PR #1 ("B2: IRIS-native backends") predated the s9–s12 IRIS sweep. It added its own `$$engine^STDOS()` engine-detect helper + a STDASSERT `irisCapture` try/catch arm. **Both were superseded:** master's STDOS is already IRIS-ported (inlines `$zversion["IRIS"` per function, no public engine helper) and STDASSERT already has the `irisRaises` try/catch arm (s9). Reconciliation **dropped** the PR's STDOS + STDASSERT changes and **rewired** the 3 backends' 6 dispatch call-sites from `$$engine^STDOS()="iris"` → `if $zversion["IRIS"` (master's established runtime idiom, used in STDOS + STDASSERT + STDHARN.engine). The `$ZCHAR`→`$CHAR` change in STDCOMPRESSTST was kept (IRIS has no `$ZCHAR`; `$CHAR`≡`$ZCHAR` for 0..255 under `ydb_chset=M`, matching the rest of the byte suites). | Branch made mergeable without merge/rebase/force-push (all sandbox-denied): a forward commit set the branch tree to `master + additive backends` (revert STDOS/STDASSERT to master, regenerate dist), so GitHub's 3-way merge is clean. PR diff is now just the 3 backend modules + STDCOMPRESSTST + their manifest entries. | resolved (PR #1) |
| 2026-06-14 | P3 | m-stdlib | `m-test-iris` (iris-community image) has **non-functional embedded Python** → STDCOMPRESS-IRIS unverifiable locally | The `intersystemsdc/iris-community:latest` container used for `m test --docker=m-test-iris` ships the `%SYS.Python` class (exists=1) but its embedded-Python runtime is not wired up: `##class(%SYS.Python).Import("sys").version` returns `0`, and calling the STDCOMPRESS IRIS path (`irisInit`→`b.exec`→helper) aborts the suite non-trappably (0/0). STDCRYPTO-IRIS (23/23) and STDHTTP-IRIS (67/67) verify fine — they use built-in classes (`$SYSTEM.Encryption` / `%Net.HttpRequest`), no Python. STDCOMPRESS-YDB is green (59/59 via libz/libzstd callout). The PR's original STDCOMPRESS-IRIS 59/59 was on `vista-iris` (working embedded Python). | The reconciled STDCOMPRESS IRIS logic is the PR's vista-iris-validated code, unchanged except the proven-correct seam — so the gap is the local engine image, not the code. Verify STDCOMPRESS-IRIS on a `vista-iris`-class instance (or once embedded Python is enabled in m-test-iris). Local-test runbook: `--docker=m-test-engine --routines src --chset m` (YDB byte mode — the container defaults to UTF-8) / `--docker=m-test-iris --routines src --namespace USER` (IRIS). Robustness follow-up (out of reconciliation scope): make the STDCOMPRESS IRIS path fail gracefully (return "FAIL"/0) when embedded Python is absent instead of a non-trappable abort. | open (env / follow-up) |
| 2026-06-14 | P2 | tree-sitter-m | pinned grammar (CI's `m-dev-tools/tree-sitter-m`) can't parse `^||` process-private globals | STDCOMPRESS's IRIS backend memoized its embedded-Python init with a `^||STDCOMPRESS("py")` process-private global. The local Go `m` (newer bundled grammar) parses it fine, but CI's Python m-cli + pinned tree-sitter-m emits **2 ERROR nodes** on the two `^||` references → `m fmt --check` fails (`source did not parse cleanly`), red-gating the YDB job. (master uses no `^||`, so master is green.) | m-stdlib side: **dropped the `^||` memo** — `irisInit^STDCOMPRESS` now re-execs the Python defs per call (idempotent; `ctypes.CDLL` refcounts the same .so, redefining helpers is free). tree-sitter-m side (upstream, open): add `^||` process-private-global syntax to the grammar so the memo idiom is available to M code. | resolved (m-stdlib); open (tree-sitter-m lane) |

## Cross-references

Expand Down
17 changes: 13 additions & 4 deletions docs/tracking/module-tracker.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,16 +114,25 @@ current state.
| [x] | P4 | L25 | 27 | [`STDXML`](../modules/stdxml.md) | `v0.4.0` | 14d | none (completed) | none | XML 1.0 + Namespaces 1.0 + XPath 1.0 + DTD envelope | n/a |
| [x] | P4 | L26 | 28 | [`STDMATH`](../modules/stdmath.md) | `v0.4.0` | 1d | none (completed) | none | Numeric helpers — clamp / min / max / sum / count / mean | n/a |
| [x] | P4 | L27 | 29 | [`STDXFRM`](../modules/stdxfrm.md) | `v0.4.0` | 1d | none (completed) | none | Higher-order array transforms — map / filter / reduce | n/a |
| [x] | P3 | H1 | 30 | [`STDCRYPTO`](../modules/stdcrypto.md) | `v0.4.0` | 2d | none (completed) | `$&stdcrypto.fn → libcrypto`; A6 | SHA-256/384/512 + HMAC-SHA-256/384/512 | 🟡 C12 |
| [x] | P3 | H2 | 31 | [`STDCOMPRESS`](../modules/stdcompress.md) | `v0.4.0` | 6d | none (completed) | `$&stdcompress.fn → libz + libzstd`; A6 | gzip / gunzip / deflate / inflate / zstdCompress / zstdDecompress | 🟡 C13 |
| [x] | P3 | H3 | 32 | [`STDHTTP`](../modules/stdhttp.md) | `v0.4.0` | 4d | none (options) | STDURL; `$&stdhttp.fn → libcurl`; A6 | HTTP/1.1 client + pure-M wire-format helpers | 🟡 C14 |
| [x] | P3 | H1 | 30 | [`STDCRYPTO`](../modules/stdcrypto.md) | `v0.4.0` | 2d | none (completed) | `$&stdcrypto.fn → libcrypto`; A6 | SHA-256/384/512 + HMAC-SHA-256/384/512 (+ IRIS-native arm: `$SYSTEM.Encryption`) | 🟡 C12 |
| [x] | P3 | H2 | 31 | [`STDCOMPRESS`](../modules/stdcompress.md) | `v0.4.0` | 6d | none (completed) | `$&stdcompress.fn → libz + libzstd`; A6 | gzip / gunzip / deflate / inflate / zstdCompress / zstdDecompress (+ IRIS-native arm: embedded Python zlib+ctypes/zstd) | 🟡 C13 |
| [x] | P3 | H3 | 32 | [`STDHTTP`](../modules/stdhttp.md) | `v0.4.0` | 4d | none (options) | STDURL; `$&stdhttp.fn → libcurl`; A6 | HTTP/1.1 client + pure-M wire-format helpers (+ IRIS-native arm: `%Net.HttpRequest`) | 🟡 C14 |
| [ ] | — | T1 | 33 | [`STDHARN`](../modules/stdharn.md) | — | 3d | P2 `^%MONLBL` coverage (STDCOV) · P4 STDWATCH hooks | STDASSERT (no-halt orchestration mode) | Resident pure-M test/coverage harness — frames `^STDASSERT` suites for m-cli 5.1 (server-side delegation) | ✅ `internal/harness` (P0–P1) |

**Aggregate.** ~108d shipped across all 32 landed modules (sum of
the Effort column above). **Full engine suite green on `main`
2026-05-08: 32 suites, 2483/2483 assertions.** All three Phase 3
modules engine-green: STDCRYPTO H1 (23/23), STDCOMPRESS H2 (59/59),
STDHTTP H3 (68/68). **All numbered tickets T1–T30 closed.** Optional
STDHTTP H3 (68/68). **IRIS-native backends landed (PR #1, 2026-06-14,
reconciled onto the s9–s12 IRIS sweep):** each optional module gained
an `if $zversion["IRIS"` arm in its dispatch helper (STDCRYPTO →
`$SYSTEM.Encryption`, STDHTTP → `%Net.HttpRequest`, STDCOMPRESS →
embedded-Python zlib+zstd) — the engine seam is the inlined `$zversion`
probe (master's idiom), **not** a public `$$engine^STDOS()` helper (that
part of the PR was dropped as superseded). Dual-engine verified locally:
STDCRYPTO 23/23 and STDHTTP 67/67 green on IRIS (m-test-iris) as well as
YDB; STDCOMPRESS-IRIS needs working embedded Python (the iris-community
image lacks it — see discoveries 2026-06-14). **All numbered tickets T1–T30 closed.** Optional
add-ons (rows tagged `none (options)`: T15 / T16 / T17 / T18 / T19 /
T22 / STDHTTP iter 3) sit behind concrete-consumer drivers and are
not gating any release. Per-module deep history (scaffolding, migrations,
Expand Down
Loading
Loading