diff --git a/docs/README.md b/docs/README.md index 23efdd5..163bb22 100644 --- a/docs/README.md +++ b/docs/README.md @@ -61,7 +61,7 @@ Each doc is labeled `[TYPE · type? · connection · connection?]`. ## `plans/` — Implementation and roadmap plans -- **`plans/discoverability-and-tooling-plan.md`** — `[PLAN · planning]` Phased plan to turn m-stdlib into a first-class discoverable surface across the source, CLI, VS Code, and AI consumption channels. +- **`plans/historical/discoverability-and-tooling-plan.md`** — `[PLAN · planning]` Phased plan to turn m-stdlib into a first-class discoverable surface across the source, CLI, VS Code, and AI consumption channels. - **`plans/future-modules-plan.md`** — `[PROPOSAL · planning]` Parking lot for module candidates that haven't crossed TDD-red yet, with priority and promotion process. - **`plans/m-libraries-remediation.md`** — `[ROADMAP · planning · design]` Background remediation strategy and prioritised roadmap derived from the M libraries survey — decisions locked. - **`plans/m-stdlib-implementation-plan.md`** — `[PLAN · planning · implementation]` Live per-module work plan with phase status, non-negotiables, and per-module specs for v0.0.1 through v0.4.0. diff --git a/docs/guides/m-doc-grammar.md b/docs/guides/m-doc-grammar.md index c62bd23..5c7b976 100644 --- a/docs/guides/m-doc-grammar.md +++ b/docs/guides/m-doc-grammar.md @@ -5,7 +5,7 @@ audience: m-stdlib maintainers writing or editing `; doc:` blocks; toolchain authors implementing the manifest generator (WA4), the `M-DOC-001` lint rule (WA3), the `m doc` family (Wave B), the VS Code extension (Wave C), the AI skill (Wave D), or any other consumer of m-stdlib metadata. -plan: docs/plans/discoverability-and-tooling-plan.md (the design rationale — +plan: docs/plans/historical/discoverability-and-tooling-plan.md (the design rationale — this guide is the normative spec implementing § 3.1) tracker: docs/tracking/discoverability-tracker.md (WA1 closes when this guide is reviewed; WA2 is the backfill that brings src/ into compliance) @@ -478,7 +478,7 @@ codebase. ## 10. Cross-references -- [Discoverability & tooling plan, § 3.1](../plans/discoverability-and-tooling-plan.md#31-formalise-an-m-doc-grammar-extends-does-not-replace--doc) — the design rationale this guide implements. +- [Discoverability & tooling plan, § 3.1](../plans/historical/discoverability-and-tooling-plan.md#31-formalise-an-m-doc-grammar-extends-does-not-replace--doc) — the design rationale this guide implements. - [Discoverability tracker, WA1–WA4](../tracking/discoverability-tracker.md#wa1--specify-m-doc-tag-grammar) — work items: WA1 ships this guide; WA2 backfills tags into src/; WA3 implements `M-DOC-001`; WA4 ships the manifest generator that consumes the grammar. - [Module tracker D2](../tracking/module-tracker.md#deferred-decisions--revisit-triggers) — the `@stable` SemVer CI gate is deferred; annotating now keeps the option open. - [STDJSON source — `parse` label](../../src/STDJSON.m) — the worked example in § 6 is sourced from this file. diff --git a/docs/modules/index.md b/docs/modules/index.md index a3cac3a..bfdbe57 100644 --- a/docs/modules/index.md +++ b/docs/modules/index.md @@ -17,7 +17,7 @@ this file is the released-module catalogue. Beyond the human-readable tables below, the released stdlib also ships a structured surface that downstream tools (m-cli `m doc`, the planned VS Code extension, the planned AI skill) read at runtime. -Source: [`docs/plans/discoverability-and-tooling-plan.md`](../plans/discoverability-and-tooling-plan.md). +Source: [`docs/plans/historical/discoverability-and-tooling-plan.md`](../plans/historical/discoverability-and-tooling-plan.md). | Artefact | Path | Purpose | |---|---|---| diff --git a/docs/plans/discoverability-and-tooling-plan.md b/docs/plans/historical/discoverability-and-tooling-plan.md similarity index 98% rename from docs/plans/discoverability-and-tooling-plan.md rename to docs/plans/historical/discoverability-and-tooling-plan.md index 98782c0..5a1cdc1 100644 --- a/docs/plans/discoverability-and-tooling-plan.md +++ b/docs/plans/historical/discoverability-and-tooling-plan.md @@ -778,18 +778,18 @@ Tracker note logged. ## 12. Cross-references -- [`README.md`](../../README.md) — top-level project overview, phase +- [`README.md`](../../../README.md) — top-level project overview, phase plan, module inventory. -- [`CHANGELOG.md`](../../CHANGELOG.md) — release history; the +- [`changelog.md`](../../tracking/changelog.md) — release history; the manifest's `since:` fields must match. -- [`docs/modules/index.md`](../modules/index.md) — phase-keyed +- [`docs/modules/index.md`](../../modules/index.md) — phase-keyed catalogue; will gain a "manifest" link in Wave A. -- [`docs/plans/m-stdlib-implementation-plan.md`](m-stdlib-implementation-plan.md) +- [`docs/plans/m-stdlib-implementation-plan.md`](../m-stdlib-implementation-plan.md) — the live build plan (modules and phases). This plan runs in parallel. -- [`docs/guides/m-tdd-guide.md`](../guides/m-tdd-guide.md) — TDD +- [`docs/guides/m-tdd-guide.md`](../../guides/m-tdd-guide.md) — TDD workflow; doctests in §3.4 land additional `*TST.m` routines into the same workflow. -- [`docs/tracking/module-tracker.md`](../tracking/module-tracker.md) +- [`docs/tracking/module-tracker.md`](../../tracking/module-tracker.md) — the work board; manifest stability tier (§3.6) feeds promotion decisions on this board. diff --git a/docs/plans/https-stack-spec.md b/docs/plans/https-stack-spec.md new file mode 100644 index 0000000..fc11eca --- /dev/null +++ b/docs/plans/https-stack-spec.md @@ -0,0 +1,401 @@ +# VistA-Native HTTPS Stack — Architecture Specification + +**Working name:** `VWEB` — "VistA Web Stack" (working routine namespace; final prefix DBA-assigned — see §13). `V*` = VistA-coupled package; portable layers live in m-stdlib under `STD*`. +**Version:** v0.1 (architecture / contracts only) +**Status:** Draft for review +**Target platforms:** IRIS *and* YottaDB/GT.M — single pure-M codebase +**Distribution:** KIDS build, deployable across VA VistA sites +**Hard constraint:** No ObjectScript, no CSP, no `%`-class dependencies. Standard M routines only. Anything platform-specific is isolated behind an adapter. + +> **One-line summary:** A Broker-pattern pure-M TCP listener that speaks HTTP/1.1 over a TLS socket, dispatches to FileMan/RPC handlers through a portable router, and includes a symmetric pure-M outbound HTTP client — giving VistA bi-directional web services with zero non-M components. + +--- + +## Table of Contents + +1. [Goals & Non-Goals](#1-goals--non-goals) +2. [Architectural Principles](#2-architectural-principles) +3. [Layered Architecture & Module Map](#3-layered-architecture--module-map) +4. [Inbound — Listener & Concurrency](#4-inbound--listener--concurrency) +5. [HTTP Parse/Serialize Contract](#5-http-parseserialize-contract-the-portability-seam) +6. [Router & Dispatch](#6-router--dispatch) +7. [Identity & Authorization](#7-identity--authorization) +8. [Outbound HTTP Client](#8-outbound-http-client) +9. [TLS Configuration (per engine)](#9-tls-configuration-per-engine) +10. [JSON Codec](#10-json-codec) +11. [Configuration Model (XPAR)](#11-configuration-model-xpar) +12. [Audit & Observability](#12-audit--observability) +13. [Naming & Namespace](#13-naming--namespace) +14. [KIDS Packaging](#14-kids-packaging) +15. [Open Decisions](#15-open-decisions) +16. [Roadmap](#16-roadmap) + +> Each numbered section is written to be extractable as a standalone module document later; cross-references use section numbers, not page positions. + +--- + +## 1. Goals & Non-Goals + +### Goals +- **Bi-directional.** VistA acts as both an HTTP **server** (inbound, exposing services) and an HTTP **client** (outbound, calling external services). +- **VistA-native.** Implemented as portable M routines, installable and patchable via KIDS, owned by a VistA namespace. +- **Portable.** One codebase runs on IRIS and YottaDB. Engine differences live only in named adapters (§3). +- **Modern auth.** OAuth 2.0 Bearer as the default credential; pluggable provider interface. +- **Reuses existing security model.** Maps onto NEW PERSON (#200), `^XUSEC` keys, and context options (OPTION #19) rather than inventing a parallel one. + +### Non-Goals (v0.1) +- Not replacing the RPC Broker or CPRS transport (Law 3 stands; this is additive). +- Not replacing HWSC/XOBW wholesale — outbound client may *coexist with* or later supersede it; not in scope to migrate existing XOBW consumers. +- No WebSockets, HTTP/2, or server-push in v0.1 (HTTP/1.1 request/response only). +- No HTML/UI rendering layer — this is a services stack (JSON-first). + +--- + +## 2. Architectural Principles + +These are the load-bearing decisions. Everything downstream follows from them. + +**P1 — The Broker is the precedent, not a dependency.** +The RPC Broker (`XWB`, file #8994, default TCP port 9200) is already a pure-M TCP listener that *jobs off each connection to a separate M process* and runs unmodified on both IRIS and YottaDB. This stack reuses that proven concurrency pattern but speaks **HTTP/1.1** on the wire instead of the Broker's (redacted, proprietary) framing. We are not extending or modifying `XWB` — we mirror its shape in a new namespace. + +**P2 — The only portable layer is M.** IRIS web tooling (CSP, `%CSP.REST`, `%Net.HttpRequest`, `%DynamicObject`) is ObjectScript and IRIS-only. None of it is permitted. Transport, TLS, and JSON are therefore implemented in M, with engine-specific mechanics isolated behind adapters. + +**P3 — Authenticate modern, authorize native.** A request carries a modern credential (OAuth2 Bearer). After validation it is *bound to a NEW PERSON #200 IEN* and the standard ambient `DUZ` is set. Authorization then reuses VistA's existing boundary: **OAuth scope → context option** (Law 5 — calls outside the context option's RPC/route surface are rejected regardless of identity). + +**P4 — Symmetric transforms.** The same request/response contract (§5) and JSON codec (§10) serve both inbound and outbound. The outbound client is the inbound parser run in reverse. + +**P5 — Configuration is data, not code.** All tunables (ports, TLS config names, AS introspection URL, timeouts) live in XPAR parameters (§11) with standard precedence, never hard-coded. + +--- + +## 3. Layered Architecture & Module Map + +``` + INBOUND OUTBOUND + ┌──────────────────────┐ ┌──────────────────────┐ + │ TLS Socket (server) │ │ TLS Socket (client) │ ← Edge / TLS adapter + ├──────────────────────┤ ├──────────────────────┤ (per-engine) + │ Listener + job-off │ │ Connection open │ + ├──────────────────────┤ ├──────────────────────┤ + │ HTTP parse → REQ │ │ RSP ← HTTP parse │ + │ RSP → HTTP serialize │ │ REQ → HTTP serialize │ ← HTTP codec (portable) + ├──────────────────────┴───┬──────────┴──────────────────────┤ + │ Request/Response contract (§5) │ ← Portability seam + ├─────────────────────────────────────────────────────────────┤ + │ Router & Dispatch (§6) │ Auth middleware (§7) │ ← Portable M + ├─────────────────────────────────────────────────────────────┤ + │ Handlers → FileMan DBS APIs / existing RPCs │ ← Application logic + └─────────────────────────────────────────────────────────────┘ + JSON codec (§10) and Config (§11) span all layers +``` + +**Module map.** Per **D0**, the map splits along the m-stdlib ↔ VistA +line. *Truly portable, engine-agnostic* layers are **imported from +m-stdlib (`STD*`)**; only the VistA-coupled and engine-adapter layers +are routines *in this package* (`VWEB*`, working name — §13). + +**Imported from m-stdlib (portable, YottaDB + IRIS, no VistA dependency):** + +| m-stdlib module | Role here | Status | +|---|---|---| +| `STDHTTPMSG` | HTTP/1.1 parse + serialize — request line, headers, body, chunked (§5) | propose (plan Pri 2) | +| `STDNET` | Portable socket open / read / write under the listener and outbound client | propose (plan Pri 1) | +| `STDHTTPD` | Generic HTTP **server framework** — accept-loop / per-connection worker model, route-matching engine, middleware chain (the engine-agnostic half of §4 + §6) | propose (plan Pri 4) | +| `STDJSON` | JSON encode/decode (§10) | exists — adopt + cross-engine conformance test (D5) | +| `STDJWT` | Local-JWT auth provider verify path (§7.1, D6) | propose (plan Pri 3) | +| `STDURL` / `STDB64` / `STDSTR` | Percent-decode, base64, CRLF / header-casing utilities (the old `ZHWSU`) | exist — import | +| `STDCRYPTO` | Introspection token hashing + `$$ctEquals` compare (§7.2) | exists (+ ctEquals ticket) | + +**This package (`VWEB*` — VistA-coupled or engine-adapter; KIDS-distributed):** + +| Module | Role | Portable? | +|---|---|---| +| `VWEBL` | Listener **launcher** — TaskMan startup OPTION + engine socket handoff, then hands the connection to `STDHTTPD`'s worker loop | Adapter (socket handoff differs per engine) | +| `VWEBIO` | Engine TLS/device adapter over `STDNET` (device params, TLS config ref §9) | Adapter | +| `VWEBR` | Route **table** + FileMan/RPC handler dispatch, registered into `STDHTTPD`'s router engine | VistA-coupled | +| `VWEBA` | Auth middleware (plugs into `STDHTTPD`'s middleware chain) + provider dispatch; DUZ / #200 binding (§7) | VistA-coupled | +| `VWEBAOI` | OAuth2 introspection provider (default); calls the outbound client | VistA-coupled | +| `VWEBCL` | Outbound HTTP client glue (`STDHTTPMSG` codec over `STDNET` / `VWEBIO`) | Thin; codec portable | +| `VWEBCFG` | Config accessor over XPAR | VistA-coupled | +| `VWEBLOG` | Audit / access-log writer (FileMan file / SIGN-ON LOG §12) | VistA-coupled | + +> **The generic server skeleton (accept-loop, worker model, route matching, +> middleware chain) is `STDHTTPD` in m-stdlib** — extracted so any pure-M +> service, VistA or not, can reuse it (plan Pri 4, Option-B). `VWEB*` is then a +> **thin VistA-binding shell**: *how* the listener launches (TaskMan), *how* the +> socket opens (`^%ZIS`/TLS), *what* the routes dispatch to (FileMan handlers), +> and *who* the caller is (`DUZ`/#200, context options). + +Engine specifics are confined to `VWEBL`, `VWEBIO`, and the socket-open +path of `VWEBCL` (all over `STDNET`). The server framework (`STDHTTPD`), HTTP +codec, JSON, JWT, and utility layers are write-once in m-stdlib and shared +verbatim by both engines and by any non-VistA consumer. + +--- + +## 4. Inbound — Listener & Concurrency + +**Model:** Broker-style listener + per-connection worker (P1). + +1. **Listener process** (started by TaskMan or a startup option) opens a *listening* TCP socket on the configured port, with TLS enabled by referencing a named server TLS config (§9). +2. On each accepted connection it **jobs off** a worker M process bound to that connection, then returns to accept the next. (This mirrors the Broker's "jobs-off each connection to a separate M process.") +3. The **worker** handles the full HTTP exchange for that connection: parse → auth → route → dispatch → serialize → write. With keep-alive, the worker loops over multiple request/response cycles until close or idle timeout. +4. Worker exit closes the socket and the process. + +**Per-engine open item — socket handoff.** How the accepted connection is transferred from listener to jobbed worker differs between IRIS and YottaDB (socket inheritance vs. re-attach vs. accept-in-child). This is the single most engine-sensitive piece and is isolated in `VWEBL`. Candidate patterns to evaluate in v0.2: +- *Accept-in-child:* listener only owns the listening socket; each worker performs the `accept`. (Cleanest handoff, needs a serialization/lock so only one worker accepts at a time, or an engine accept-queue.) +- *Inherit-on-job:* listener accepts, then jobs a child that inherits the connected device. + +**Limits & safety (config-driven, §11):** max concurrent workers, per-connection idle timeout, max request line / header / body sizes (DoS guards), max requests per keep-alive connection. + +--- + +## 5. HTTP Parse/Serialize Contract (the portability seam) + +This is the contract every layer above the socket depends on. Defined as M local-array shapes (data contracts, not code). + +### 5.1 Request structure — `REQ` +| Subscript | Contents | +|---|---| +| `REQ("method")` | Upper-case verb (`GET`,`POST`,…) | +| `REQ("path")` | Decoded path, no query string | +| `REQ("rawpath")` | Raw path as received | +| `REQ("query",name)` | Percent-decoded query param (last-wins or list — TBD §15) | +| `REQ("hdr",lowername)` | Header value, name lower-cased for case-insensitive lookup | +| `REQ("body")` | Raw body (single node if small) | +| `REQ("body",n)` | Body chunk `n` (large/streamed bodies) | +| `REQ("body","len")` | Decoded content length | +| `REQ("json",…)` | Parsed JSON tree (populated lazily by handler via §10) | +| `REQ("peer")` | Remote address | +| `REQ("tls","cn")` | Client-cert CN, if mTLS provider used | + +### 5.2 Response structure — `RSP` +| Subscript | Contents | +|---|---| +| `RSP("status")` | Integer status code | +| `RSP("hdr",Name)` | Response header (original casing preserved) | +| `RSP("body")` / `RSP("body",n)` | Body content | +| `RSP("json")` | If set, codec serializes it to `body` and sets `Content-Type` | + +### 5.3 Parser obligations (`STDHTTPMSG`, in m-stdlib) +- Read request line; reject malformed / oversized (414/400). +- Read headers until CRLFCRLF; enforce header count/size caps. +- Body framing: honor `Content-Length`; support `Transfer-Encoding: chunked` (de-chunk into `REQ("body",…)`). +- Connection management: detect `Connection: close` vs keep-alive; HTTP/1.1 default keep-alive. +- Never trust client `Content-Length` beyond configured max. + +### 5.4 Serializer obligations (`STDHTTPMSG`, in m-stdlib) +- Emit status line, headers, blank line, body, with correct CRLF. +- Always set `Content-Length` (no server-side chunking required in v0.1). +- Set `Date`, `Server`, `Connection` headers; `Content-Type` defaulted from `RSP("json")`. + +**Why this seam matters:** the parser/serializer are *fully portable* and identical on both engines — which is exactly why they live in m-stdlib (`STDHTTPMSG`), not this package. Only the bytes feeding them (from the socket layer — `STDNET` + the `VWEBIO` engine adapter) are engine-specific. + +--- + +## 6. Router & Dispatch + +**Route table** is data (a global / FileMan file — TBD §15), so routes are installable and patchable, not compiled in. + +| Field | Meaning | +|---|---| +| method | Verb or `*` | +| pattern | Path pattern with named segments, e.g. `/patients/:dfn/meds` | +| handler | `TAG^ROUTINE` entry point invoked with `REQ`/`RSP` | +| scope | Required OAuth scope(s) → mapped to a context option (§7) | +| auth | `required` \| `optional` \| `none` (health checks) | + +**Dispatch sequence (per request):** +1. Match `REQ("method")` + `REQ("path")` → route (extract path params into `REQ("param",name)`). +2. No match → 404. Method mismatch on existing path → 405. +3. If `auth=required`, invoke auth middleware (§7). Failure → 401/403. +4. Invoke handler `TAG^ROUTINE`. Handler reads `REQ`, populates `RSP`. +5. Uncaught error → 500 with a sanitized body; full detail to audit log (§12). +6. Serializer writes `RSP`. + +Handlers call FileMan DBS APIs (`GETS^DIQ`, `FILE^DIE`, `UPDATE^DIE`, `LIST^DIC`, `FIND^DIC`) or existing RPCs — i.e., this stack is a new *transport* in front of unchanged application logic. + +--- + +## 7. Identity & Authorization + +The most consequential section. Authentication is modern and pluggable; authorization is native. + +### 7.1 Pluggable provider interface +Auth is a provider abstraction (`VWEBA` dispatches to a configured provider). Each provider, given `REQ`, returns either a failure or a **principal** `{subject, scopes, claims}`. + +| Provider | Mechanism | v0.1 | +|---|---|---| +| **OAuth2 Introspection** (`VWEBAOI`) | RFC 6750 Bearer + RFC 7662 introspection call to configured AS | **Default** | +| Local JWT | Verify signature in-M (needs crypto) | Deferred — depends on §15 crypto decision | +| mTLS | Client cert (TLS config requires cert); CN/SAN → principal | Optional | +| Access/Verify | Legacy VistA pair (XUS) | Optional fallback | + +### 7.2 Default flow — OAuth2 Bearer + Introspection +1. Extract `Authorization: Bearer ` from `REQ("hdr","authorization")`. +2. Call the configured AS introspection endpoint **using the outbound client (§8)** — VistA validating an inbound token by acting as an HTTP client is the dogfooding payoff. +3. AS returns `active`, `sub`, `scope`, expiry, etc. Inactive/expired → 401. +4. Cache introspection results briefly (config TTL) keyed by token hash to avoid an AS round-trip per request. + +> No private signing keys or JWT crypto live in VistA in this default path — which is exactly what keeps it portable and KIDS-clean. + +### 7.3 The DUZ binding (mandatory) +A valid token authenticates a *subject*; it does not grant the ability to touch FileMan. Per Kernel invariants, `DUZ` (caller's IEN in NEW PERSON #200) is ambient and required for nearly all VistA operations. The stack must therefore, after authentication: + +1. Map `principal.subject` → a NEW PERSON #200 IEN via a configurable identity map (claim → #200 lookup; candidate keys: secid, NPI, network username, or a dedicated mapping file — TBD §15). +2. No mapping → 403 (authenticated but not provisioned in VistA). +3. Set `DUZ` and establish standard context (`DUZ(0)`, division, etc.) for the worker process. +4. Optionally check `^XUSEC("KEY",DUZ)` for handlers gated on security keys. + +### 7.4 Authorization — scope → context option (Law 5) +Rather than a bespoke permission layer, map each OAuth **scope** to a **context option (OPTION #19)** whose RPC/route surface defines what that scope may do. A request is authorized only if its principal's scopes cover the route's required scope, *and* the dispatched operation lies within the mapped context option's surface. This reuses VistA's existing, audited security boundary instead of duplicating it. + +### 7.5 Audit +Every authentication decision (success/failure, subject, mapped DUZ, route, scope) is logged (§12). Consider correlating with SIGN-ON LOG (#3.081) conventions for consistency with existing authentication auditing. + +--- + +## 8. Outbound HTTP Client + +Symmetric with inbound (P4): build a `REQ`, serialize with `STDHTTPMSG`, open a client TLS socket (`STDNET` + `VWEBIO`), write, read response with `STDHTTPMSG` into `RSP`. + +**Capabilities (v0.1):** methods GET/POST/PUT/PATCH/DELETE; request headers + body; JSON convenience via §10; TLS verification against a named client TLS config (§9); connect/read timeouts; basic retry policy (config-driven). Used internally by the introspection provider (§7.2) and available to any M caller needing to reach an external service. + +**Relationship to HWSC/XOBW:** XOBW provides outbound web-service calls today but is Java-middleware-backed and outside the portable-M goal (and outside the documented VDL corpus). This client is a pure-M alternative. v0.1 coexists; a migration assessment for existing XOBW consumers is out of scope and noted for a later module. + +**Engine-specific:** only the client socket *open* (device params + TLS reference) lives in the adapter; request building, serialization, and response parsing are portable. + +--- + +## 9. TLS Configuration (per engine) + +TLS stays out of M *logic* but is invoked from M via the device `OPEN` referencing an **externally defined named TLS configuration**. The shape is symmetric across engines; the config mechanism differs. + +| | IRIS | YottaDB/GT.M | +|---|---|---| +| Config definition | SSL/TLS Configuration created administratively (Management Portal / security config) — **not** ObjectScript | TLS config entry in the encryption plugin config file (`ydb_crypt_config`), OpenSSL-backed | +| Referenced from M | TCP device `OPEN` with `/SSL="ConfigName"` | Socket device with `tls` deviceparameter naming the config | +| Server (inbound) | Server-role SSL config on the listening socket | TLS-enabled server socket (same plugin used for replication TLS) | +| Client (outbound) | Client-role SSL config, verify peer | Client TLS config, CA bundle for verification | +| mTLS | Require client certificate in config | Configure client-cert requirement | + +**Design rule:** the M code references a TLS config *name* pulled from XPAR (§11). Operators provision the actual cert/key/CA out-of-band per engine. No certs or crypto material in M, none in the KIDS build. This is what makes native TLS viable without ObjectScript *and* without an external proxy. + +> Note: the listening-socket TLS handoff to jobbed workers interacts with §4's socket-handoff open item — confirm the chosen handoff pattern preserves the TLS session per engine. + +--- + +## 10. JSON Codec + +IRIS `%DynamicObject` and YottaDB have no shared JSON facility, so a **portable pure-M JSON codec** (`STDJSON`, from m-stdlib) is a required build component, not an afterthought. + +- **Decode:** JSON text → M local array tree (objects as named subscripts, arrays as numeric subscripts, with type markers for null/bool/number/string). +- **Encode:** the inverse, with correct escaping and UTF-8 handling. +- **Interface:** `REQ("json",…)` and `RSP("json")` are the codec's I/O surfaces (§5). +- **Reuse vs. build:** evaluate existing open-source pure-M JSON implementations before writing one; whichever is chosen must pass a shared conformance test on both engines. (Decision §15.) +- IRIS-native JSON is explicitly **not** used (breaks portability, violates P2). + +--- + +## 11. Configuration Model (XPAR) + +All tunables are XPAR parameters under the package, honoring precedence **USR > LOC > SRV > DIV > SYS > PKG**. + +| Parameter (illustrative) | Scope typical | Purpose | +|---|---|---| +| `VWEB LISTEN PORT` | SYS | Inbound TLS port | +| `VWEB TLS SERVER CONFIG` | SYS | Named server TLS config (§9) | +| `VWEB TLS CLIENT CONFIG` | SYS | Named client TLS config (outbound) | +| `VWEB MAX WORKERS` | SYS | Concurrency cap | +| `VWEB IDLE TIMEOUT` | SYS | Keep-alive idle seconds | +| `VWEB MAX BODY` | SYS | Max inbound body bytes | +| `VWEB AUTH PROVIDER` | SYS | Active provider (introspection/mtls/…) | +| `VWEB OAUTH INTROSPECT URL` | SYS | AS introspection endpoint | +| `VWEB OAUTH CACHE TTL` | SYS | Introspection cache seconds | +| `VWEB IDENTITY MAP KEY` | SYS | Which claim maps to #200 | + +--- + +## 12. Audit & Observability + +- **Access log:** one record per request — timestamp, peer, method, path, status, bytes, latency, mapped DUZ, route, scope. +- **Auth log:** auth decisions per §7.5. +- **Error log:** uncaught handler errors with full stack/context (never leaked in the HTTP body). +- **Health endpoint:** `GET /healthz` (auth `none`) for liveness; optional readiness check that verifies listener + AS reachability. +- Storage mechanism (FileMan file vs. flat global vs. MailMan alert thresholds) — decision §15. + +--- + +## 13. Naming & Namespace + +`ZHWS` was the original **placeholder** — now **retired** (D0/D7); `MWEB` +was a brief interim pick, also superseded. The `Z*` prefix signals a +*local/non-distributed* namespace, which is wrong for a package committed +to VA distribution (D0). The working namespace must: + +- be **distinct from `STD*`** — the portable, engine-agnostic primitives + live in m-stdlib under `STD*`; this package is the VistA-coupled + consumer that *imports* them, and sharing the prefix would blur the + load-bearing line drawn in D0; and +- **signal the VistA coupling** with a `V`-prefix — this package *is* the + VistA side (FileMan handlers, `DUZ`/#200 auth, KIDS), so making that + visible in every routine name reinforces the D0 line right in the code. + This matches the sibling VistA-bridge layer `VBL` + ([`m-stdlib-vista-bridge-library-architecture.md`](m-stdlib-vista-bridge-library-architecture.md)): + **`V*` = VistA-coupled package, `STD*` = portable library.** + +**Working namespace:** `VWEB` ("VistA Web Stack"); package title +*"VistA Web Services (on m-stdlib)."* The **final** routine namespace is +still **DBA-assigned** before any VA pilot (the working name is what the +spec and source carry until then). Routine names, options, parameters, +RPCs, security keys, and FileMan files all carry that namespace prefix +per §5 package invariants. + +> The split: `STD*` (m-stdlib) = portable / YottaDB+IRIS / no VistA +> dependency / library-distributed. `VWEB*` (this package) = VistA-coupled +> / KIDS-distributed / depends on m-stdlib being present at build time. + +--- + +## 14. KIDS Packaging + +- **Build:** a KIDS distribution containing the `VWEB*` routines, route-table file (if FileMan-backed), XPAR parameter definitions, context option(s) for scope mapping (§7.4), and any security keys. It declares the **`VBL` and m-stdlib base builds as KIDS Required Builds** — `STD*`/`VBL*` are installed once and reused, never bundled into this build (anti-duplication rule). +- **Patch identity:** `NAMESPACE*VERSION*PATCH` (e.g. `VWEB*1.0*1`). +- **Environment check / pre-install:** verify engine type and minimum version, TLS plugin/config availability, port availability, required Kernel patch level. +- **Post-install:** create XPAR defaults, register the startup option (listener launch via TaskMan), seed the route table. +- **No external artifacts in the build** — TLS material and AS endpoints are operator-provisioned (§9, §11), keeping the distributable pure-M and ATO-friendly. +- **Both engines:** the build installs identically on IRIS and YottaDB FOIA VistA; only the operator-provisioned TLS config mechanism differs. + +--- + +## 15. Open Decisions + +| # | Decision | Default leaning | +|---|---|---| +| **D0** | **Distribution & portability model (upstream of D7 and most rows below).** Is this a VA-distributable VistA package, or a private/greenfield service? | **RESOLVED 2026-06-06 — committed to VA distribution.** Ships as a KIDS build, installable and usable on VA VistA systems, IRIS-compatible. This makes the **m-stdlib ↔ VistA-package separation load-bearing**: truly portable, engine-agnostic (YottaDB *and* IRIS) code lives in **m-stdlib under the `STD*` namespace**; the VistA/KIDS-coupled stack (listener, router, auth, DUZ binding, FileMan handlers, KIDS/XPAR) is a **distinct VistA package** that depends on the `STD*` (and `VBL`) **base builds** via KIDS **Required Builds** — installed once and reused, **never vendored/copied in** (the anti-duplication rule; see [`m-stdlib-vista-bridge-library-architecture.md` §3.1/§4](m-stdlib-vista-bridge-library-architecture.md)). The package namespace must therefore **not** be `STD*` (that would blur the line) but be `V`-prefixed to signal VistA coupling (see D7 + §13). | +| D1 | Socket handoff pattern listener→worker (§4) | Accept-in-child, pending per-engine spike | +| D2 | Query/header multi-value handling (§5.1) | Last-wins + list node for repeats | +| D3 | Route table store: FileMan file vs. global (§6) | FileMan file (installable/auditable) | +| D4 | Identity map claim → #200 (§7.3) | Dedicated mapping file keyed on a stable claim (secid?) | +| D5 | JSON: adopt existing pure-M lib vs. build (§10) | Adopt + conformance-test if license/quality fit | +| D6 | Add local-JWT provider (needs M crypto) (§7.1) | Defer to v0.2+; introspection covers v0.1 | +| D7 | Namespace (§13): working name + path to official assignment, given D0 | **Resolved — `VWEB`** ("VistA Web Stack"). Retired the `ZHWS` placeholder (`Z*` = local/non-distributed, wrong for VA distribution) and the interim `MWEB`. Working namespace is (a) **distinct from `STD*`** to preserve the D0 line, and (b) **`V`-prefixed** to signal VistA coupling in every routine name — matching the sibling `VBL` bridge (`V*` = VistA package, `STD*` = portable). Final routine namespace **DBA-assigned** before VA pilot; verify `VWEB` against the namespace registry (no collision found vs. the 196-package KIDS list + gold-docs corpus). | +| D8 | Audit storage mechanism (§12) | FileMan file for access/auth; error to log global | +| D9 | mTLS as primary or fallback only (§7.1) | Fallback/optional; Bearer primary | + +--- + +## 16. Roadmap + +| Phase | Deliverable | +|---|---| +| **v0.1 (this doc)** | Architecture + contracts agreed | +| **v0.2** | Per-engine socket/TLS spikes resolving D1; finalized §4/§9 adapter contracts | +| **v0.3** | HTTP codec + JSON codec spec finalized (D5); conformance test suite defined | +| **v0.4** | Auth module spec finalized (D4, scope↔option map); audit spec (D8) | +| **v1.0 build** | KIDS package: listener, router, default introspection provider, outbound client, health endpoint | +| **Later** | Local-JWT provider (D6); HWSC/XOBW migration assessment; HTTP/2 / streaming | + +--- + +*End v0.1. Sections are modular by design — each can be promoted to a standalone spec document as detail accretes.* diff --git a/docs/plans/m-stdlib-vista-bridge-library-architecture.md b/docs/plans/m-stdlib-vista-bridge-library-architecture.md new file mode 100644 index 0000000..b3c0a97 --- /dev/null +++ b/docs/plans/m-stdlib-vista-bridge-library-architecture.md @@ -0,0 +1,847 @@ +--- +title: m-stdlib ↔ VistA Bridge Library Architecture +status: draft +version: v0.1 +tracker: docs/tracking/module-tracker.md +created: 2026-06-06 +last_modified: 2026-06-06 +revisions: 1 +doc_type: [ARCHITECTURE, DRAFT] +--- + +# m-stdlib ↔ VistA Bridge Library Architecture — **DRAFT v0.1** + +> **Status:** DRAFT v0.1 — architecture and contracts only; nothing here has +> crossed TDD-red. Names marked *(working)* are placeholders pending the +> decisions in [§13 Open Questions](#13-open-questions). +> +> **One-line summary:** Draw a sharp, tested seam between *engine-agnostic* +> m-stdlib (the `STD*` library, YottaDB **and** IRIS, zero VistA dependency) +> and a **separately-tracked, KIDS-installable VistA integration layer** +> (working name **`VBL` / "vista-bridge-library"**) that binds every side-effecting +> m-stdlib primitive to its real VistA back end (FileMan, Kernel, XPAR, +> TaskMan, the Device Handler, the RPC Broker), proven end-to-end against a +> live VistA with **no mocking or spoofing**. The bidirectional HTTP +> web-services stack ([`https-stack-spec.md`](https-stack-spec.md), `VWEB`) +> is carried here as the **end-to-end smoke test** of the whole vertical. + +--- + +## Table of Contents + +1. [Purpose, Scope & the Sharp-Line Principle](#1-purpose-scope--the-sharp-line-principle) +2. [Part A — m-stdlib Functionality & Architecture Review](#2-part-a--m-stdlib-functionality--architecture-review) +3. [Part B — VistA Integration Points Review](#3-part-b--vista-integration-points-review) +4. [Part C — The Separation Architecture (`VBL` / vista-bridge-library)](#4-part-c--the-separation-architecture-vbl--vista-bridge-library) +5. [The Vertical Integration Stack — Summary Table](#5-the-vertical-integration-stack--summary-table) +6. [Architecture Diagram (Mermaid)](#6-architecture-diagram-mermaid) + - [6.1 Example application #1 — FHIR R4 façade](#61-example-application-1--fhir-r4-façade) + - [6.2 Example application #2 — VistA log streaming to AWS S3](#62-example-application-2--vista-log-streaming-to-aws-s3) +7. [Part D — The VistA Integration Layer as an Independent Track](#7-part-d--the-vista-integration-layer-as-an-independent-track) +8. [Part E — End-to-End Testing Against VistA (No Mocking)](#8-part-e--end-to-end-testing-against-vista-no-mocking) +9. [Part F — Worked Example: Bidirectional HTTP Web Services (`VWEB`) as the Smoke Test](#9-part-f--worked-example-bidirectional-http-web-services-vweb-as-the-smoke-test) +10. [Risks](#10-risks) +11. [Open Questions](#11-open-questions) +12. [Documentation Gaps — Proposed VDL Fetches](#12-documentation-gaps--proposed-vdl-fetches) +13. [References (VistA gold-docs)](#13-references-vista-gold-docs) + +--- + +## 1. Purpose, Scope & the Sharp-Line Principle + +### 1.1 The problem + +m-stdlib fills the gaps in M's standard library as a **pure-M, engine-agnostic** +library: its core is dependency-free and byte-mode-correct on both YottaDB and +IRIS, with three optional `$ZF`-callout modules. It has **no VistA dependency** +and is tested on a bare engine. That portability is the whole value proposition +— `STDHTTPMSG` must be reusable by a non-VistA service exactly as easily as by a +VistA one. + +But a real VistA application cannot live on portable primitives alone. The +moment a primitive has a **side effect with a VistA-native home** — persisting a +record (FileMan, not a flat file), reading config (XPAR, not a `.env`), +authenticating a caller (Kernel `DUZ`/`^XUSEC`, not a bare token), opening a +listening socket (the Device Handler `^%ZIS` with a Kernel TLS config), writing +an audit record (a FileMan file / MailMan alert, not stdout), or shipping +(a KIDS build to a DBA-assigned namespace, not a `git clone`) — the portable +abstraction needs a **VistA-flavored back end**. + +### 1.2 The principle + +> **Engine-agnostic capability lives in `STD*` (m-stdlib). VistA-specific +> binding of that capability lives in a separate, independently-tracked, +> KIDS-installable package (`VBL` *(working)*). Nothing VistA leaks up into +> `STD*`; nothing portable is reimplemented in `VBL`.** + +This is the same load-bearing line drawn for the web stack in +[`https-stack-spec.md` D0](https-stack-spec.md): the commitment to **VA KIDS +distribution + IRIS compatibility** is what makes the separation mandatory +rather than stylistic. This document generalizes that line from "the web stack" +to **all** of m-stdlib. + +### 1.3 What "seamless" means here + +The requirement is that *all* m-stdlib functionality works against the VistA +back end. It does **not** mean every `STD*` module gets a VistA twin. It means: + +- **Pure/computational modules** (`STDJSON`, `STDREGEX`, `STDB64`, `STDHEX`, + `STDURL`, `STDDATE`, `STDCSV`, `STDXML`, `STDFMT`, `STDSTR`, `STDMATH`, + `STDCOLL`, `STDXFRM`, `STDSEMVER`, `STDUUID`, `STDCSPRNG`, the proposed + `STDHTTPMSG`/`STDHTTPD`/`STDJWT`/`STDVALID`/`STDESCAPE`) run **unchanged** on a VistA + engine — they touch no VistA seam, so the adapter layer is empty for them. + "Seamless" = *they already work*; `VBL` proves it under the VistA test + transport. +- **Side-effecting / environment modules** (`STDFS`, `STDOS`, `STDENV`, + `STDLOG`, `STDNET`, `STDCACHE`, `STDFIX`/`STDSEED`, and the Kernel-crypto + surface of `STDCRYPTO`) get a **VistA-backed adapter** in `VBL` that satisfies + the same contract using FileMan / XPAR / Kernel / Device-Handler instead of + POSIX. "Seamless" = *the contract is identical; only the back end differs*. + +The deliverable is **two deterministic, gated, fully-tested artifacts**: portable +`m-stdlib` (as today) and `VBL`/vista-bridge-library (new), the latter managed to a VA +VistA spec and **end-to-end tested against a live VistA** with no mocks. + +--- + +## 2. Part A — m-stdlib Functionality & Architecture Review + +### 2.1 Module inventory (33 modules, `v0.5.0`) + +| Group | Modules | VistA seam? | +|---|---|---| +| **Encoding / binary** | STDB64, STDHEX, STDCSPRNG | none (pure) | +| **Data formats** | STDJSON, STDCSV, STDTOML, STDXML, STDURL, STDFMT | none (pure) | +| **Text / collections / numeric** | STDSTR, STDREGEX, STDCOLL, STDXFRM, STDMATH, STDSEMVER, STDUUID, STDDATE | none (pure) | +| **Config / environment** | STDENV, STDOS | **XPAR / Kernel** | +| **Filesystem** | STDFS | **FileMan / HFS** | +| **Observability** | STDLOG, STDPROF | **FileMan file / MailMan** (log sink) | +| **Caching** | STDCACHE | optional global-backed variant | +| **Crypto (optional)** | STDCRYPTO (libcrypto), STDCOMPRESS (libz/zstd), STDHTTP (libcurl) | **Kernel `^XUSHSH` / TLS** alt-backend | +| **Test infrastructure** | STDASSERT, STDMOCK, STDFIX, STDSEED, STDSNAP, STDHARN | VistA test transport | +| **Proposed (web/security wave)** | STDNET, STDHTTPMSG, STDHTTPD, STDJWT, STDVALID, STDRATELIMIT, STDESCAPE, … | **Device Handler / Kernel** (STDNET) | + +### 2.2 Architectural invariants (carried into `VBL` unchanged) + +- **Byte mode (`ydb_chset=M`).** One M character == one byte; binary routines + are `$ZCHAR`/`$ZASCII`-exact. `VBL` inherits this — FileMan/global I/O is + byte-faithful. +- **Core dependency-free; 3 optional callout modules** (`STDCRYPTO`, + `STDCOMPRESS`, `STDHTTP`) tagged `; doc: @tier optional` → + `"tier":"optional"` in the manifest. `VBL` is, by construction, a **fourth + dependency class**: "requires a VistA-bearing engine," and is tracked as such. +- **Generated artifacts under `dist/`** (`stdlib-manifest.json`, `errors.json`, + `skill/`, `tests/STD*DOCTST.m`) regenerated from `; doc:` blocks by + `make manifest` / `make skill` / `make doctest`, gated by CI drift checks. + `VBL` adopts the same doc-comment → manifest → skill → doctest toolchain. +- **`^STDASSERT` test protocol** (`start` / `eq` / `report`), `*TST.m` + hand-written suites + generated `*DOCTST.m`, **85%-per-module coverage gate**, + TDD-red-first as a hard rule. +- **Multi-transport test runner** (m-cli): `LocalEngine` (host YDB), + `DockerEngine` (m-test-engine container), `SSHEngine` (legacy vista-meta over + SSH). The resident harness `STDHARN` already runs **dual-engine (YDB + IRIS)** + with an IRIS line-monitor coverage path (recent work on this branch). **This + runner is the hook for the new VistA engine transport** (§8). +- **Priority rule:** m-stdlib has architectural priority over consumers + (m-cli, and now `VBL`/`VWEB`) — utilities are implemented in `STD*` first and + imported, never duplicated downstream. + +### 2.3 The seams that matter for VistA + +Five m-stdlib seams have a VistA-native back end and therefore define the `VBL` +surface: + +1. **Storage** — `STDFS` (file read/write/exists/remove/size) → FileMan DBS. +2. **Config** — `STDENV` (typed key/value) + `STDOS` (env) → XPAR parameters. +3. **Identity/crypto** — `STDCRYPTO`/`STDCSPRNG` token compare & hashing → + Kernel `^XUSHSH` + `DUZ`/`^XUSEC` authorization context. +4. **Sockets** — `STDNET` (open/read/write) → Kernel Device Handler `^%ZIS` + with a named Kernel/engine TLS config. +5. **Observability** — `STDLOG` (structured records) → a FileMan audit file / + MailMan alert sink. + +Everything else is pure and crosses the line untouched. + +--- + +## 3. Part B — VistA Integration Points Review + +All entry points below are drawn from the VistA gold-docs corpus; citations are +keyed to [§13](#13-references-vista-gold-docs). + +### 3.1 Packaging & distribution — KIDS + +VistA packages are built, transported, and installed via **KIDS** (Kernel +Installation & Distribution System): + +- **BUILD (#9.6)** defines a package version (`NAMESPACE VERSION`) and its + components: routines, FileMan files + data dictionaries, OPTIONs, SECURITY + KEYs, PARAMETERs, templates, protocols, REMOTE PROCEDUREs, mail groups, HL7 + links. **INSTALL (#9.7)** records each install event; **PACKAGE (#9.4)** holds + version + patch-application history. +- A **Transport Global** carries the BUILD + components in KIDS-closed-reference + form, staged under `^XTMP`, distributed as a Host-File-Server file or PackMan + message. +- **Install phases:** load distribution → run **environment-check routine** + (abort/warn/proceed) → answer pre/post-install questions → run pre-install → + install components → run post-install → re-enable options. Programmatic hooks: + `XPD*` utilities (`MES^XPDUTL`, `$$VERSION^XPDUTL`, checkpoint APIs), + `EN^XPDIJ` to task the install via TaskMan. +- **Patch identity:** `NAMESPACE*VERSION*PATCH` (e.g. `VWEB*1.0*1`); the + routine **namespace is DBA-assigned** for VA distribution — `Z*` is local-only + and not distributable. + +> **`VBL` consequence — shared base packages, not vendoring.** A library only +> earns its keep if it is installed **once** and reused by every routine on the +> system; copying its routines into each consumer's build is exactly the +> copy/paste duplication VistA already suffers from. So each layer ships as its +> **own KIDS base package**, and consumers declare dependencies via KIDS +> **Required Builds** (the BUILD #9.6 mechanism patches already use to require +> prior builds) — the same way every VistA package already requires Kernel (`XU`) +> and FileMan (`DI`): +> +> - **m-stdlib base build** — the `STD*` routines, installed once (DBA-assigned +> namespace for the VistA form; see [§11 Q9](#11-open-questions)). +> - **`VBL` base build** — `VBL*` adapters + FileMan file definitions +> (audit/config) + XPAR parameter definitions; **Requires** the m-stdlib base. +> - **`VWEB` (and any other app)** — `VWEB*` + OPTIONs (listener startup, context +> options) + an environment-check routine (engine type/version, TLS, Kernel +> patch level); **Requires** the `VBL` base (and m-stdlib transitively). +> +> Result: `STD*` and `VBL*` exist **once** on a VistA system and are callable by +> *any* M routine or package — m-stdlib/`VBL` become infrastructure base packages, +> peers of `XU`/`DI` in the dependency graph. No duplicated routines, no +> per-consumer drift. + +### 3.2 Data — FileMan Database Server (DBS) silent API + +VistA's rule is **no direct global access — all file I/O goes through FileMan +DBS calls or a documented DBIA.** The callable surface `VBL` will wrap: + +- **Read:** `GETS^DIQ(file,iens,fields,flags,target[,msg])` (multi-field into an + FDA-shaped target array; flags `E`/`I`/`N`/`R`/`Z`), `$$GET1^DIQ(...)` (single + field), `LIST^DIC` / `FIND^DIC` / `$$FIND1^DIC` (lookup/finder, LAYGO). +- **Write:** `UPDATE^DIE([flags,]fda,ien[,msg])` (add/file from an FDA, `+n` + add-nodes resolved to real IENs), `FILE^DIE` (file into known records), + `WP^DIE` (word-processing fields). +- **Validate:** `VAL^DIE` / `CHK^DIE` (external→internal validation without + filing). +- **Errors:** structured in `^TMP("DIERR",$J,...)` with `TEXT`/`PARAM` subnodes; + the `DIERR` local holds `count^lines`. `VBL` maps `DIERR` → m-stdlib's + `$ECODE` convention (`,U-VBL-…,`) so callers wrap with one `$ETRAP`. +- **IENS** = comma-terminated internal-entry-number string; **FDA** = + `FDA(file,"iens",field)=value`. + +> **`VBL` consequence:** the `STDFS`-shaped storage seam, when bound to VistA, +> is **not** a filesystem — it is FileMan. `VBL` exposes a FileMan-native +> storage adapter (`$$get/$$set/$$exists/$$kill` over `(file,iens,field)`), +> *not* a fake POSIX path layer. Code that genuinely needs a host file uses the +> HFS path of `STDFS` unchanged; code that needs persistence uses the FileMan +> adapter. + +### 3.3 Config — XPAR Parameter Tools + +The **PARAMETER (#8989.5)** + **PARAMETER DEFINITION (#8989.51)** files, accessed +via `$$GET^XPAR(entity,param,instance,format)`, `EN^XPAR`, `CHG^XPAR`, +`ENVAL^XPAR`, with an **entity-precedence hierarchy** (typ. `USR > LOC > SRV > +DIV > SYS > PKG`). This is VistA's native, audited, hierarchical config store. + +> **`VBL` consequence:** the `STDENV`/`STDOS` config seam binds to XPAR. `VWEB`'s +> tunables (`VWEB LISTEN PORT`, `VWEB TLS SERVER CONFIG`, …) are XPAR parameters, +> not a `.env` file — already reflected in [`https-stack-spec.md` §11](https-stack-spec.md). + +### 3.4 Identity & authorization — Kernel + +- **Ambient identity:** `DUZ` = caller's IEN in **NEW PERSON (#200)**; `DUZ(0)` + = FileMan access code; required for nearly all FileMan operations. +- **Authorization:** **security keys** in `^XUSEC(KEY,DUZ)`; **context options** + (OPTION #19, "B"-type) gate the callable RPC/route surface — the Broker's + `XWB CREATE CONTEXT` / `CRCONTXT^XWBSEC` handshake validates the user holds the + context option before any RPC runs. +- **Credential hashing:** Kernel `^XUSHSH` / `$$ENCRYP^XUSHSH` (one-way hash for + Access/Verify codes). +- **Sign-on / audit:** `XUS SIGNON SETUP`, `XUS GET USER INFO`, BSE token APIs; + the **SIGN-ON LOG (#3.081)**. + +> **`VBL` consequence:** the auth/crypto seam binds `STDCRYPTO`'s +> compare/hash to Kernel where appropriate, and provides the **principal → +> `DUZ`/#200 binding** + **scope → context-option** authorization that +> [`https-stack-spec.md` §7](https-stack-spec.md) requires. Portable token +> crypto (HMAC, base64url, constant-time compare) stays in `STD*`; the VistA +> *authorization decision* lives in `VBL`. + +### 3.5 Process & scheduling — TaskMan + +`^%ZTLOAD` queues background jobs; startup OPTIONs launch persistent processes. +A long-running listener (the `VWEB`/`VBL` socket listener) is started at system +startup as a TaskMan-launched OPTION, mirroring `XWB LISTENER STARTER`. + +### 3.6 I/O & TLS — Device Handler + +Kernel's **`^%ZIS`** device handler opens/uses/closes devices from the **DEVICE +(#3.5)** file (incl. TCP socket devices: address/port/open-params). **TLS is +configured out-of-band** (engine-level: YottaDB `ydb_crypt_config` / IRIS SSL +config) and referenced **by name** from the device `OPEN` — *no certificate +material in M or in the KIDS build*. This matches +[`https-stack-spec.md` §9](https-stack-spec.md). + +> **`VBL` consequence:** `STDNET`'s portable socket primitives are wrapped by an +> `VBL` device adapter (`VWEBIO` in the web stack) that performs the +> engine-specific `^%ZIS`/TLS `OPEN`. `STDNET` stays portable; the TLS/device +> mechanics are the adapter's only engine-specific code. + +### 3.7 The existing connectivity precedent — RPC Broker (XWB) + +The **RPC Broker** is already a pure-M TCP listener (default port **9200**) that +**jobs off a separate M process per connection** and runs unmodified on IRIS and +YottaDB; **REMOTE PROCEDURE (#8994)** is the callable registry; **Broker Security +Enhancement (BSE)** adds mutual-TLS/IAM. The **M-to-M Broker** (XWB*1.1*34) does +VistA→VistA. This is the **concurrency precedent** `VWEB` mirrors (it speaks +HTTP/1.1 instead of the Broker's framing). + +### 3.8 Outbound web services today — HWSC / VistaLink (and why pure-M wins) + +VistA's current outbound web-service client (**HWSC / XOBW**, **WEB SERVER +#18.12**) and **VistaLink** are **middleware-backed** (J2EE/WebLogic, KAAJEE +auth) — *not* pure-M, and outside the portability goal. **No VistA-native OAuth +2.0 / SMART-on-FHIR / token-introspection / in-M HTTP server is documented in +the gold corpus** (see [§12](#12-documentation-gaps--proposed-vdl-fetches)). +This is precisely the gap the pure-M `VWEB` stack + `VBL` adapter close. + +### 3.9 Prior art — VIA / HWSC / VistaLink / MDWS vs. `VBL` + +VistA already has several integration layers. **None of them is what `VBL` is**, +and understanding why is the clearest justification for the bridge. The recurring +pattern in VA's existing answers is **a middleware tier (Java/J2EE) fronting a +set of M RPCs** — exactly the non-pure-M dependency this architecture removes. + +| Layer | What it is | Tier / language | Direction & scope | Relationship to `VBL` | +|---|---|---|---|---| +| **VIA** — VistA Integration Adapter (`VIAB`) | Enterprise **data-exposure API**: curated RPCs (e.g. the VIAB BMS `LISTORDERS`/`LISTORDERACTIONS`) fronted by a web-service tier; successor to MDWS (the BMS migration was mandated). | **Middleware** (SOAP/REST service) **+ KIDS-installed M RPCs** (`VIAB`) | Outbound, **domain-specific** read of clinical data to enterprise consumers (BMS, JLV) | **Different layer.** VIA is a consumer-facing *product* exposing business operations; `VBL` is a developer-facing *library* exposing technical primitives. A VIA-style API could be **rebuilt on `VWEB`+`VBL`** as pure-M, dropping the middleware. | +| **MDWS** — Medical Domain Web Services | Older enterprise web-service layer VIA replaced. | Middleware | Outbound, domain-specific | Superseded by VIA; same middleware pattern `VBL` avoids. | +| **HWSC / XOBW** | Outbound web-service **client** framework (WEB SERVER #18.12). | Middleware-backed (J2EE/WebLogic) | Outbound calls from VistA | `VBL`+`STDHTTPMSG`/`STDNET` are the **pure-M outbound client** alternative — no app server. | +| **VistaLink** | J2EE **connector** binding Java apps to VistA; KAAJEE auth. | Middleware (WebLogic, J2EE) | Inbound from Java apps | A connector for *foreign* (Java) callers; `VBL` is for *native M* callers, no JVM. | +| **RPC Broker (XWB)** | Pure-M TCP listener, job-off per connection, REMOTE PROCEDURE #8994; BSE/IAM TLS. | **Pure-M** | Bidirectional RPC framing (proprietary wire) | The **concurrency precedent** `VWEB` mirrors — but Broker speaks RPC framing, not HTTP, and is a transport, not an m-stdlib binding. `VBL` is complementary infrastructure, not a Broker replacement. | +| **`VBL` + `VWEB`** (this work) | Pure-M **adapter library** (`VBL`) + pure-M **HTTP services stack** (`VWEB`) on m-stdlib. | **Pure-M, zero non-M components** | Bidirectional, **domain-agnostic** foundation | The thing none of the above is: a reusable, KIDS-clean, middleware-free integration substrate. | + +**The throughline:** VA's enterprise integration story (MDWS → VIA, HWSC, +VistaLink) is consistently **middleware fronting M**. `VBL`/`VWEB` collapse that +into **pure M** — which is what makes it KIDS-clean, ATO-friendly, and portable +across YottaDB and IRIS without a JVM or app server. `VBL` does not compete with +the RPC Broker (a transport) or replace VIA (a data product); it is the +**library layer beneath** any future native service, including a pure-M +re-implementation of a VIA-class API. + +> **Naming note:** because **VIA / "VistA Integration Adapter"** is an active, +> mandated VA product (namespace `VIAB`), that name is unavailable for this layer +> — both a namespace collision and a category error (VIA is middleware; `VBL` is +> a pure-M library). This is the concrete basis for the `VBL` naming decision in +> [§11 Q1](#11-open-questions). + +--- + +## 4. Part C — The Separation Architecture (`VBL` / vista-bridge-library) + +### 4.1 Four layers, two repos, one wire + +``` +Layer 4 VistA internals FileMan (DI) · Kernel (XU) · XPAR · TaskMan · Device Handler · RPC Broker +Layer 3 VistA integration VBL / vista-bridge-library ← KIDS-installable, VistA-coupled, DBA namespace +Layer 2 m-stdlib STD* ← portable, YDB+IRIS, zero VistA dependency +Layer 1 M engine YottaDB | IRIS for Health (byte mode) +``` + +- **Layer 2 (`STD*`)** is the existing m-stdlib repo: portable, engine-agnostic. + Distributed two ways from one source — as **library source** for non-VistA + consumers, and as a **KIDS base package** installed *once* on a VistA system. + **Never imports VistA.** +- **Layer 3 (`VBL`)** is a **new, separately-tracked package/repo** + (`vista-bridge-library` *(working)*): VistA-coupled adapters that satisfy the + m-stdlib side-effect contracts using Layer-4 APIs. A **KIDS base package** + (own DBA-assigned namespace) that **Requires** the m-stdlib base. +- **Consumers** (e.g. `VWEB`) sit on Layers 2 + 3 and **Require** those base + builds — they never copy `STD*`/`VBL*` routines in. + +**Install once, reuse everywhere (the anti-duplication rule).** The point of a +library is that it lives on the system *once* and every routine calls it; copying +its routines into each consumer is the copy/paste sprawl VistA already has too +much of. So `STD*` and `VBL*` are **shared base builds**, depended on via KIDS +**Required Builds** — never vendored. This is the same dependency discipline by +which every VistA package already builds on `XU` (Kernel) and `DI` (FileMan). + +### 4.2 The adapter contract + +For each side-effecting m-stdlib seam, `VBL` provides a routine implementing the +**same public signature** the portable module exposes, backed by VistA: + +| m-stdlib seam (Layer 2) | `VBL` adapter (Layer 3, *working* names) | VistA back end (Layer 4) | +|---|---|---| +| `STDFS` storage / persistence | `VBLFS` (FileMan DBS storage) | `GETS^DIQ` / `UPDATE^DIE` / `FILE^DIE` (FileMan #) | +| `STDENV` / `STDOS` config | `VBLCFG` | `$$GET^XPAR` / `EN^XPAR` (#8989.5/.51) | +| `STDLOG` / `STDPROF` sink | `VBLLOG` | FileMan audit file / MailMan alert | +| `STDNET` socket open | `VBLIO` | `^%ZIS` device + named TLS config | +| `STDCRYPTO` hash / auth | `VBLSEC` | `^XUSHSH`; `DUZ`/#200 binding; `^XUSEC`; OPTION #19 | +| process / scheduling | `VBLTASK` | `^%ZTLOAD` startup OPTION | +| packaging / install | `VBLBLD` (build manifest + env-check) | KIDS BUILD #9.6 / INSTALL #9.7 | + +**Rule:** an `VBL` adapter contains *only* the VistA binding. Any logic that is +not VistA-specific (parsing, formatting, framing, encoding) stays in `STD*` and +is called by the adapter — never copied. + +### 4.3 Why a *separate package*, not a tier inside m-stdlib + +Same reasoning as [`https-stack-spec.md` D0](https-stack-spec.md): + +- m-stdlib's core is **dependency-free and testable on a bare engine**; folding + VistA-coupled routines in would make the core untestable without FileMan + + Kernel and break the portability promise. +- m-stdlib is **library-distributed**; `VBL` is **KIDS-distributed to a + DBA-assigned namespace** under VA governance. Different mechanism, different + governance — they cannot share a namespace. +- m-stdlib's manifest/skill/doctest/coverage gates assume **documented portable + API modules**; `VBL` adapters need a **live-VistA test transport**. Separate + tracks keep each gate honest. + +The identity lives in the **project name** (`vista-bridge-library`) and a +**`V`-prefixed routine namespace** (`VBL*` *(working)*), **distinct from `STD*`** +so the line stays sharp. This is the project-wide naming convention: **`V*` = +VistA-coupled package, `STD*` = portable library.** Both VistA-coupled packages +follow it — `VBL` (this bridge) and `VWEB` (the web stack on top of it) — so the +VistA-coupling is visible in every routine name, reinforcing the line directly in +the code (`STD*` = portable, `V*` = VistA-bound). + +--- + +## 5. The Vertical Integration Stack — Summary Table + +| # | Layer | Artifact / namespace | Engine dependency | Distribution | Test transport | Gate | +|---|---|---|---|---|---|---| +| **L1** | **M engine** | YottaDB · IRIS for Health (`ydb_chset=M`) | — | OS / IRIS install | n/a | byte-mode invariant | +| **L2** | **m-stdlib (portable)** | `STD*` (this repo) | YDB **or** IRIS; **no VistA** | library source **+ KIDS base build** (installed once, shared) | `LocalEngine` / `DockerEngine` (bare YDB), dual-engine IRIS via `STDHARN` | 85% cov · fmt · lint · manifest/skill/doctest drift | +| **L3** | **VistA integration** | `VBL` / vista-bridge-library *(working)* | VistA-bearing engine (FileMan + Kernel) | **KIDS** build, DBA namespace | **`VistaEngine` (new)** — live FOIA VistA on YDB **and** IRIS | 85% cov · same drift gates · **KIDS install/back-out verify** · VA VistA spec conformance | +| **L4** | **VistA internals** | FileMan (DI) · Kernel (XU) · XPAR · TaskMan · `^%ZIS` · RPC Broker (XWB) | — | (platform) | (system under test) | DBIA-conformant; no direct-global access | +| **W** | **Web-services stack** | `VWEB` HTTPS stack | L2 + L3 | KIDS build | `VistaEngine` end-to-end HTTP | bidirectional wire smoke test (§9) | +| **A** | **External application (real-world consumer)** | FHIR application — third-party FHIR R4 REST client | none — **connects only over HTTPS to W** | independent (not VistA-distributed) | `VistaEngine` FHIR conformance (`/metadata`, `/Patient`) | FHIR R4 resource validation | + +**The external app touches nothing but `VWEB`.** A third-party FHIR application is +a standard FHIR R4 client; it has **no knowledge of and no path into** the M +layers (L2–L4). Its single interface is the HTTPS endpoint `VWEB` exposes — that +boundary is the whole point of the stack. + +**Read top-down for a request:** the FHIR app issues `GET /Patient/123` over +**HTTPS** → arrives on L1's TLS socket → `VWEB` listener (L3) hands bytes to +`STDHTTPMSG` (L2) → router (L3) dispatches to **`VWEB`'s Patient handler** → auth +via `VBLSEC` (L3, validates the SMART-on-FHIR Bearer scope, binds `DUZ`) → handler +calls `VBLFS` (L3) → FileMan DBS (L4) reads PATIENT (#2) → handler maps the record +to a FHIR `Patient` resource and serializes it with `STDJSON` (L2) → `STDHTTPMSG` +(L2) frames the response → written to L1's socket → back to the external FHIR app. +**Every portable hop is `STD*`; every VistA hop is `VBL*`; they never blur.** + +--- + +## 6. Architecture Diagram (Mermaid) + +```mermaid +flowchart TB + subgraph APP["External application — third-party consumer (over HTTPS only)"] + FHIR["FHIR application
standard FHIR R4 REST client
GET /metadata · /Patient · /Observation
SMART-on-FHIR Bearer token"] + end + + subgraph CONS["Web-services stack (end-to-end smoke test)"] + VWEB["VWEB — bidirectional HTTPS stack
listener · router · auth · handlers
(KIDS package)"] + end + + subgraph L2["Layer 2 — m-stdlib · STD* · portable, YDB+IRIS, NO VistA"] + direction LR + PURE["Pure / computational
STDJSON STDHTTPMSG STDHTTPD STDJWT STDURL
STDB64 STDCSPRNG STDVALID STDREGEX"] + SEAM["Side-effect seams (contracts)
STDFS · STDENV/STDOS · STDLOG
STDNET · STDCRYPTO"] + end + + subgraph L3["Layer 3 — VistA integration · VBL / vista-bridge-library · KIDS, DBA namespace"] + direction LR + VBLFS["VBLFS
storage"] + VBLCFG["VBLCFG
config"] + VBLLOG["VBLLOG
audit"] + VBLIO["VBLIO
socket+TLS"] + VBLSEC["VBLSEC
authz / DUZ"] + VBLTASK["VBLTASK
listener"] + VBLBLD["VBLBLD
KIDS build"] + end + + subgraph L4["Layer 4 — VistA internals"] + direction LR + FM["FileMan DBS
GETS^DIQ · UPDATE^DIE"] + XPAR["XPAR
#8989.5/.51"] + KERN["Kernel
DUZ · #200 · ^XUSEC · ^XUSHSH"] + ZIS["Device Handler
^%ZIS + TLS config"] + TASK["TaskMan
^%ZTLOAD"] + KIDS["KIDS
BUILD #9.6 / INSTALL #9.7"] + XWB["RPC Broker (XWB)
precedent: job-off listener"] + end + + L1["Layer 1 — M engine: YottaDB | IRIS for Health (byte mode)"] + + FHIR ==>|"HTTPS · FHIR R4 REST (the only entry point)"| VWEB + + VWEB --> PURE + VWEB --> SEAM + SEAM -. "same contract, VistA back end" .-> VBLFS & VBLCFG & VBLLOG & VBLIO & VBLSEC + VWEB -. listener/startup .-> VBLTASK + VWEB -. packaged by .-> VBLBLD + + VBLFS --> FM + VBLCFG --> XPAR + VBLSEC --> KERN + VBLIO --> ZIS + VBLLOG --> FM + VBLTASK --> TASK + VBLBLD --> KIDS + VWEB -. mirrors concurrency of .-> XWB + + L4 --> L1 + L3 --> L1 + L2 --> L1 + + classDef app fill:#f3e6ff,stroke:#9933ff,stroke-dasharray: 5 3; + classDef portable fill:#e6f2ff,stroke:#3399ff; + classDef vista fill:#fff0e6,stroke:#ff8c1a; + classDef internals fill:#eaffea,stroke:#33cc33; + class FHIR app; + class PURE,SEAM portable; + class VBLFS,VBLCFG,VBLLOG,VBLIO,VBLSEC,VBLTASK,VBLBLD vista; + class FM,XPAR,KERN,ZIS,TASK,KIDS,XWB internals; +``` + +### 6.1 Example application #1 — FHIR R4 façade + +The diagram above uses a **third-party FHIR application** as its canonical +consumer: a standard FHIR R4 REST client that reaches VistA **only** through the +`VWEB` HTTPS endpoint, never touching the M layers. It is the inbound, request/ +response shape — the world *pulling* clinical data out of VistA over an open +standard. The §9 smoke test exercises this path end-to-end. + +### 6.2 Example application #2 — VistA log streaming to AWS S3 + +The complementary shape: an **outbound, fire-and-forget data egress** that proves +the same `STD*`/`VBL*` seams in the *write* direction. Where the FHIR app is a +client of VistA, the log shipper is VistA acting as a client of an external +service (AWS S3). + +**The need — why VistA barely logs today.** VistA has no mainstream, always-on +logging facility, and the reason is structural: the only place a pure-M routine +can durably write is a **global**, and a global lives in the *same database that +holds patient data*. A verbose log global (`^XTMP`, a package log file, …) grows +without bound, inflates journaling and backups, competes for the same disk, and +can threaten the live clinical system — so sites keep logging sparse, ad-hoc, or +switched off entirely. Yet modern operations need the opposite: comprehensive, +durable, externalized logs for security monitoring (SIEM), HIPAA audit trails, +debugging, and observability. The fix is to **stop writing logs into the M +database at all** and stream them to cheap, immutable, lifecycle-managed object +storage instead. + +**The application.** A log shipper binds the portable `STDLOG` sink seam (Layer 2) +to an S3 streaming sink (a `VBL*` adapter, Layer 3). VistA code anywhere calls the +ordinary `STDLOG` API; the records are batched, serialized to newline-delimited +JSON with `STDJSON`, signed with AWS SigV4 (`STDCRYPTO` HMAC-SHA256 + `STDHEX`), +framed by `STDHTTPMSG`, and `PUT` to an S3 bucket over TLS via `VBLIO`. A +background `TaskMan` job (`VBLTASK`) drives the periodic flush; bucket/region/ +credential references are XPAR parameters (`VBLCFG`). **No log ever lands in a +MUMPS global** — the legacy `^XTMP`-style sink is exactly what this replaces. + +The diagram stacks the levels as **separate bands, each in its own color**, with +the external **AWS Cloud** as a distinct dashed band at the bottom — the +whitespace and the colour break show the physical separation. The only thing that +leaves the M system is the single bold arrow that **crosses the network / trust +boundary**. + +```mermaid +flowchart TB + subgraph PROD["VistA log producers (any package)"] + APPCODE["VistA application code
do log^STDLOG(level,msg,...)"] + end + + OLD["✗ legacy sink: ^XTMP / log global
grows in the live DB — why VistA avoids logging"] + + subgraph L2["Layer 2 — m-stdlib · STD* (portable, no VistA)"] + direction LR + STDLOG["STDLOG
logging API + sink seam"] + STDJSON["STDJSON
record → NDJSON batch"] + SIG["STDCRYPTO + STDHEX
AWS SigV4 HMAC-SHA256"] + STDHTTP["STDHTTPMSG / STDHTTP
PUT request builder"] + end + + subgraph L3["Layer 3 — VBL (VistA bridge · KIDS, DBA namespace)"] + direction LR + SINK["VBLS3
S3 log sink: batch · spool · ship"] + VBLIO["VBLIO
socket + TLS"] + VBLCFG["VBLCFG
bucket / region / creds"] + VBLTASK["VBLTASK
background flusher"] + end + + subgraph L4["Layer 4 — VistA internals"] + direction LR + XPAR["XPAR
S3 config params"] + TASK["TaskMan
^%ZTLOAD flush job"] + ZIS["Device Handler
^%ZIS + TLS config"] + end + + subgraph CLOUD["☁ AWS Cloud — external service (separate trust domain · off the M database)"] + S3["S3 bucket
NDJSON log objects
lifecycle policy · Athena query"] + end + + APPCODE -->|"do log^STDLOG()"| STDLOG + STDLOG -. "sink bound to (replaces global)" .-> SINK + OLD -. "what we DON'T do" .-> STDLOG + + SINK --> STDJSON + SINK --> SIG + SINK --> STDHTTP + STDHTTP --> VBLIO + + VBLTASK -. drives periodic flush .-> SINK + VBLCFG --> XPAR + VBLTASK --> TASK + VBLIO --> ZIS + + VBLIO ==>|"HTTPS PUT · SigV4 signed -- crosses the network / trust boundary"| S3 + + classDef portable fill:#e6f2ff,stroke:#3399ff; + classDef vista fill:#fff0e6,stroke:#ff8c1a; + classDef internals fill:#eaffea,stroke:#33cc33; + classDef ext fill:#f3e6ff,stroke:#9933ff,stroke-dasharray: 5 3; + classDef producer fill:#f7f7f7,stroke:#888888; + classDef bad fill:#ffe6e6,stroke:#cc0000,stroke-dasharray: 3 3; + class STDLOG,STDJSON,SIG,STDHTTP portable; + class SINK,VBLIO,VBLCFG,VBLTASK vista; + class XPAR,TASK,ZIS internals; + class S3 ext; + class APPCODE producer; + class OLD bad; + + style PROD fill:#fcfcfc,stroke:#bbbbbb; + style L2 fill:#f3f9ff,stroke:#3399ff,stroke-width:2px; + style L3 fill:#fff8f2,stroke:#ff8c1a,stroke-width:2px; + style L4 fill:#f3fff3,stroke:#33cc33,stroke-width:2px; + style CLOUD fill:#faf3ff,stroke:#9933ff,stroke-width:3px,stroke-dasharray:8 5; +``` + +**What it proves.** Identical layering discipline as the FHIR example, exercised +outbound: every portable hop is `STD*` (engine-agnostic, YDB+IRIS), every VistA +hop is `VBL*`, and the only thing that knows about S3 is one `VBL*` sink plus +config — swap `VBLS3` for `VBLGCS`/`VBLAZ` and the same `STDLOG` calls ship to a +different cloud, unchanged. + +--- + +## 7. Part D — The VistA Integration Layer as an Independent Track + +### 7.1 Track identity + +`VBL`/vista-bridge-library is its **own track** with its own repo, version line, tracker, +and CI — a sibling of m-stdlib, not a branch of it. It depends on the **m-stdlib +base build** as a KIDS **Required Build** (not by copying `STD*` source in) and on +a VistA-bearing engine at **test time**. + +### 7.2 Managed to a VA VistA spec + +- **Namespace:** DBA-assigned (`VBL*`/`VWEB*` are working placeholders; `Z*` is + forbidden for distribution). +- **KIDS base build** (`VBLBLD`): BUILD #9.6 with `VBL*` routines, FileMan file + definitions (audit/config scratch files), XPAR parameter definitions, SECURITY + KEYs, and a **Required Build** dependency on the m-stdlib base — so `STD*` is + reused from its single shared install, never duplicated here. Apps like `VWEB` + add their own OPTIONs (listener startup, context options) and a mandatory + **environment-check routine** (engine type/version, TLS config presence, Kernel + patch level, IRIS-for-Health minimum), and **Require** the `VBL` base. +- **Patch identity** `VBL*1.0*n`; **DIBRG** (Deployment/Install/Back-out/Rollback + guide) authored per VA convention. +- **Back-out/rollback proven** as part of CI (install → verify → back-out → + verify-clean). + +### 7.3 Same quality toolchain as m-stdlib + +`VBL` adopts m-stdlib's `; doc:`-block → `make manifest`/`skill`/`doctest` +pipeline, `^STDASSERT` suites, **85%-per-module coverage**, fmt/lint, and +drift gates — so the integration layer is **as deterministic and gated** as the +portable library. + +### 7.4 Deliverable definition of done + +1. Every side-effecting `STD*` seam has an `VBL*` adapter passing its `*TST.m` + against a **live VistA** on **both** YDB and IRIS. +2. Every **pure** `STD*` module is proven to run unchanged under the VistA + transport (a conformance pass, not new code). +3. The `VWEB` end-to-end smoke test (§9) is green on both engines. +4. KIDS install + back-out verified; coverage + drift gates green; VA-spec + checklist signed off. + +--- + +## 8. Part E — End-to-End Testing Against VistA (No Mocking) + +### 8.1 The `VistaEngine` transport + +m-cli's runner already abstracts transports (`LocalEngine`, `DockerEngine`, +`SSHEngine`→vista-meta). The integration track adds a **`VistaEngine`**: a +**real VistA instance** (FOIA VistA loaded with FileMan + Kernel) on **YottaDB** +and a second on **IRIS for Health**, reachable by the runner. `VBL` `*TST.m` +suites run *inside* that instance — **no mocking, no spoofing, no FileMan +stubs.** Tests touch real FileMan files, real XPAR parameters, real `DUZ`/`^XUSEC` +context, and a real `^%ZIS` TLS socket. + +### 8.2 Determinism without mocks + +- **Scratch files / test namespace:** `VBL` tests file into dedicated test + FileMan files (or a sacrificial sub-range) and `STDFIX`/`STDSEED` set up and + **tear down** real records per test for isolation — the existing fixture + protocol, pointed at FileMan instead of locals. +- **Pinned fixtures:** a known test patient/user/parameter seed loaded via KIDS + pre-install or `STDSEED`, so assertions are deterministic against real data. +- **Dual-engine parity:** every suite runs on YDB **and** IRIS (extending the + `STDHARN` dual-engine work already on this branch); a result that differs + across engines is a failure, not a warning. +- **Gated:** `make ci` on the `VBL` repo requires the `VistaEngine` to be + reachable; coverage + KIDS install/back-out + drift gates must all pass. + +### 8.3 Why "no mocks" is feasible here + +The seams are few (storage, config, auth, socket, log) and each has a small, +well-documented VistA API. A live FOIA VistA is reproducible and scriptable, so +the cost of real-backend testing is bounded — and it is the only way to *prove* +the DBIA-conformant behavior the VA spec requires. + +--- + +## 9. Part F — Worked Example: Bidirectional HTTP Web Services (`VWEB`) as the Smoke Test + +The HTTPS stack in [`https-stack-spec.md`](https-stack-spec.md) is the **vertical +proof** that exercises every layer at once — the reason to build the seam. + +### 9.1 Inbound (server) path — exercises every layer + +1. **L1:** TLS socket accepts a connection (engine TLS config, named). +2. **L3 `VBLIO`/`VWEBL`:** Device Handler `^%ZIS` open + job-off worker + (mirrors the RPC Broker concurrency precedent, §3.7). +3. **L2 `STDHTTPMSG`:** parse request line / headers / body / chunked → `REQ` + (portable, identical on both engines). +4. **L3 `VWEBR` router + `VBLSEC` auth:** validate OAuth2 Bearer (introspection + via the outbound path), **bind principal → `DUZ`/#200**, enforce **scope → + context-option** (OPTION #19). +5. **L3 `VBLFS`:** handler reads/writes via FileMan DBS (`GETS^DIQ`/`UPDATE^DIE`) + — *native VistA data access, no shim*. +6. **L2 `STDJSON` + `STDHTTPMSG`:** serialize `RSP` → bytes. +7. **L1:** write response to the socket. **L3 `VBLLOG`:** audit record to a + FileMan file. + +### 9.2 Outbound (client) path — symmetric + +`STDHTTPMSG` builds the request, `VBLIO` opens a client TLS socket via `^%ZIS`, +`STDHTTPMSG` parses the response. Used by the introspection provider (VistA +validating an inbound token by acting as an HTTP client) — the dogfooding payoff. + +### 9.3 What the smoke test proves + +- The **portable/VistA line holds under load**: every byte-level transform is + `STD*`; every VistA binding is `VBL*`. +- **All m-stdlib functionality works against VistA**: JSON, HTTP codec, JWT, + base64url, URL, CSPRNG (pure) + FileMan storage, XPAR config, Kernel auth, + device TLS (adapted) — end-to-end, on a real VistA, both engines. +- VistA gains **pure-M, bidirectional, native web services** with **zero + non-M components** — closing the HWSC/VistaLink middleware gap (§3.8). + +--- + +## 10. Risks + +| # | Risk | Severity | Mitigation | +|---|---|---|---| +| R1 | **Socket hand-off + TLS differ across YDB vs IRIS** (the `^%ZIS`/job-off seam) — the single most engine-sensitive piece. | High | Confine to `VBLIO`; spike both engines early (mirrors [`https-stack-spec.md` D1](https-stack-spec.md)); dual-engine gate catches drift. | +| R2 | **No live `VistaEngine` available / hard to provision** for CI. | High | Stand up scriptable FOIA VistA on YDB + IRIS-for-Health; treat as required CI infra; without it, L3 is untestable per the no-mocks rule. | +| R3 | **DBA namespace not yet assigned** — `VBL*`/`VWEB*` are placeholders; routine/file/option/key names all carry it. | Medium | Request assignment before VA pilot; keep working names mechanical to rename (one sweep, as done for `ZHWS`→`VWEB`). | +| R4 | **FileMan-as-storage impedance** — `STDFS`'s path/byte model doesn't map cleanly onto FileMan's file/IENS/field model. | Medium | Don't force a POSIX shim; `VBLFS` is a *FileMan-native* storage adapter with its own contract; reserve `STDFS` HFS path for genuine host files. | +| R5 | **VistA TLS / OAuth introspection is undocumented in the gold corpus** (no native OAuth/SMART/in-M HTTP server found). | Medium | Fetch the IAM/OAuth + IRIS-TLS docs ([§12](#12-documentation-gaps--proposed-vdl-fetches)); until then, treat the introspection-AS contract as an open interface. | +| R6 | **Shared-base version skew** — a consumer needs an `STD*`/`VBL*` API newer than the base build installed on a site (the flip side of *not* vendoring). | Medium | KIDS **Required Build** version constraints + m-stdlib SemVer; a consumer declares the minimum base version it needs, and the environment-check routine fails fast if the installed base is too old. Solve version management with dependency metadata, **not** by copying routines in. | +| R7 | **Coverage on real backends is slower / flakier** than local. | Low–Med | Deterministic fixtures (§8.2), pinned seed data, per-test FileMan teardown; quarantine genuinely non-deterministic VistA behaviors explicitly. | +| R8 | **Scope creep — `VBL` accreting VistA app logic** (becoming a package, not an adapter). | Medium | Hard rule: `VBL*` contains only the VistA *binding*; anything reusable goes to `STD*`; anything app-specific goes to the consumer (`VWEB`). | +| R9 | **IRIS without VistA** (IRIS-for-Health sites whose VistA differs from FOIA) — adapter assumptions may not hold. | Medium | Pin to documented Kernel/FileMan DBIAs only; test against the IRIS-for-Health VistA build, not bare IRIS. | + +--- + +## 11. Open Questions + +| # | Question | Leaning | +|---|---|---| +| Q1 | **Name of the integration layer.** Chosen: repo `vista-bridge-library`, routine namespace `VBL*` ("VistA Bridge Library"). Alternatives considered: `MSV` (m-stdlib↔VistA), `VSL`/`VSTD` (VistA Std-Lib), `MVB` (m-stdlib VistA Bridge). | **`VBL` / `vista-bridge-library`** — `V`-prefix makes VistA-coupling visible per routine; distinct from `STD*`. **Namespace check (2026-06-06):** `VBL` is collision-free against both the 196-package KIDS registry (`vista-kids-packages.csv`) and the vdocs gold-docs corpus — no exact match, no prefix overlap; nearest neighbor `VBECS` diverges at char 3. Fully-clear fallbacks: `VBR`/`VLB`/`VHB`. **Still required:** DBA confirmation against the VA Approved Application Abbreviations registry (may include namespaces not in FOIA/VDL) before pilot. | +| Q2 | **One `VBL` package or several?** A single adapter package, or split (e.g. `VBLDATA`, `VBLAUTH`, `VBLNET`) so consumers install only what they need. | Single package v1; split only if install-footprint pressure appears. | +| Q3 | **FileMan storage contract.** Does `VBLFS` mimic `STDFS`'s signature, or expose a first-class `(file,iens,field)` FileMan API (and leave `STDFS` for HFS only)? | First-class FileMan API; do **not** fake POSIX over FileMan (R4). | +| Q4 | **Does m-stdlib gain a formal "VistA tier"** (like `optional`) so the manifest marks modules proven under `VistaEngine`, or is that purely an `VBL`-side conformance matrix? | `VBL`-side conformance matrix; keep m-stdlib VistA-unaware. | +| Q5 | **OAuth Authorization Server.** Which AS does `VWEB` introspect against at VA, and is mTLS or Bearer primary? (Depends on undocumented IAM specifics.) | Bearer primary, mTLS fallback (per spec D9); confirm AS once IAM docs fetched. | +| Q6 | **Test data seed.** Use FOIA VistA's built-in test patients/users, or ship a dedicated KIDS test-seed build? | Dedicated `VBL` test-seed build for determinism. | +| Q7 | **Identity map claim → #200.** Which NEW PERSON key (SECID, NPI, network username, dedicated map file)? (Mirrors [`https-stack-spec.md` D4](https-stack-spec.md).) | Dedicated mapping file keyed on a stable claim (SECID?). | +| Q8 | **Engine matrix scope.** YDB + IRIS-for-Health only, or also a GT.M exclusion note? (m-stdlib already excludes GT.M.) | YDB + IRIS only; GT.M out of scope, consistent with m-stdlib. | +| Q9 | **`STD*` VistA namespace + KIDS base build.** Install-once requires m-stdlib to have a *VistA KIDS form* with a DBA-assigned namespace. Can the prefix be `STD`, or must the VistA build remap to an assigned prefix? Does m-stdlib grow a `make kids` target, or does `VBL` own the m-stdlib base build? | Request `STD` (or assigned prefix) as a DBA namespace for the m-stdlib **base build**; m-stdlib repo gains a `make kids` target producing it; `VBL`/`VWEB` declare it as a Required Build. Confirm with DBA — this is the one place the otherwise-portable `STD*` acquires VistA namespace governance. | + +--- + +## 12. Documentation Gaps — Proposed VDL Fetches + +The gold corpus covered FileMan DBS, Kernel, XPAR, Device Handler, TaskMan, and +the RPC Broker well. The following were **not found** (or only indirectly) and +should be fetched from the VA VDL to make this architecture definitive: + +| Topic | Why needed | Proposed VDL fetch | +|---|---|---| +| **Standalone KIDS guide** | KIDS content is embedded in `XU/krn_8_0_sm` + `XU/krn_8_0_tm`; no dedicated KIDS Developer/SM guide is in the corpus, so build-API specifics (`XPD*`) are second-hand. | *Kernel 8.0 Systems Management — KIDS* and/or *KIDS Developer's Guide* (VDL → Infrastructure → Kernel). | +| **Device Handler dedicated guide** | `^%ZIS` socket/TLS device parameters are only summarized in `krn_8_0_sm`/`krn_8_0_tm`. | *Kernel 8.0 Systems Management — Device Handler* (VDL → Infrastructure → Kernel). | +| **TaskMan dedicated guide** | `^%ZTLOAD` startup-listener pattern is summarized only. | *Kernel 8.0 Systems Management — TaskMan* (VDL → Infrastructure → Kernel). | +| **VistA TLS / IRIS-for-Health socket-TLS enablement** | Engine-level TLS config (referenced by name from `^%ZIS`) — the cited `XU*8*787` TLS-enablement patch was not in the index. | *XU*8*787* (or current) TLS-enablement patch description; *IRIS for Health* TLS configuration guide. | +| **VistA IAM / OAuth 2.0 / SMART-on-FHIR** | No VistA-native OAuth/introspection doc exists in the corpus; `VWEB` auth depends on the VA AS contract. | VA *Identity & Access Management (IAM)* SSOi/STS docs; any *VistA FHIR / patient-facing API* OAuth guide. | +| **PARAMETER Tools full API reference** | Only `XT/ktk7_3p26sp` (a patch supplement) was found for XPAR; a consolidated API ref would firm up `VBLCFG`. | *Kernel Toolkit — Parameter Tools* full developer reference (VDL → Infrastructure → Kernel Toolkit). | +| **VIA architecture / design guide** | The only VIA doc in the corpus is a patch/back-out artifact (`VIAB*1*15`); the prior-art comparison (§3.9) infers VIA's middleware tier from the registry, not a design doc. | *VistA Integration Adapter (VIA)* technical/architecture or developer guide (VDL → Clinical → VistA Integration Adapter (VIA)). | + +--- + +## 13. References (VistA gold-docs) + +All citations are `is_latest=1` gold documents in `~/data/vdocs` (verified +present). Format: **`doc_key`** — Title — VDL URL. + +**FileMan (DI) — data access / DBS API (§3.2):** +- **`DI/fm22_2dg`** — FM 22.2 Developer's Guide — https://www.va.gov/vdl/documents/Infrastructure/Fileman/fm22_2dg.docx *(DBS API: `GETS^DIQ`, `$$GET1^DIQ`, `UPDATE^DIE`, `FILE^DIE`, `WP^DIE`, `VAL^DIE`/`CHK^DIE`, `FIND^DIC`/`LIST^DIC`, IENS, FDA, `DIERR`)* +- **`DI/fm22_2tm`** — FM 22.2 Technical Manual — https://www.va.gov/vdl/documents/Infrastructure/Fileman/fm22_2tm.docx +- **`DI/fm22_krn8_file_security`** — FM and Kernel File Access Security — https://www.va.gov/vdl/documents/Infrastructure/Fileman/fm22_krn8_file_security.docx + +**Kernel (XU) — KIDS, identity, TaskMan, Device Handler (§3.1, §3.4, §3.5, §3.6):** +- **`XU/krn_8_0_tm`** — Kernel 8.0 and Kernel Toolkit 7.3 Technical Manual — https://www.va.gov/vdl/documents/Infrastructure/Kernel/krn_8_0_tm.docx *(KIDS transport global, `DUZ`/#200, TaskMan, `^%ZIS`)* +- **`XU/krn_8_0_sm`** — Kernel 8.0 Systems Management: Main Directory — https://www.va.gov/vdl/documents/Infrastructure/Kernel/krn_8_0_sm.docx *(KIDS install phases, Device Handler, TaskMan)* +- **`XU/krn8_0st`** — Kernel 8.0 Security Tools Manual — https://www.va.gov/vdl/documents/Infrastructure/Kernel/krn8_0st.docx *(security keys, `^XUSEC`)* +- **`XU/xu_8_0_p775_dibrg`** — XU*8*775 Deployment, Installation, Back-Out, and Rollback Guide — https://www.va.gov/vdl/documents/Infrastructure/Kernel/xu_8_0_p775_dibrg.docx *(KIDS transport-global / DIBRG pattern)* + +**Kernel Toolkit (XT) — XPAR Parameter Tools (§3.3):** +- **`XT/ktk7_3p26sp`** — XT*7.3*26 Parameter Tools Supplement to Patch Description — https://www.va.gov/vdl/documents/Infrastructure/Kernel_Toolkit/ktk7_3p26sp.docx *(`GET^XPAR`/`EN^XPAR`, #8989.5/#8989.51, entity precedence)* + +**RPC Broker (XWB) — connectivity precedent (§3.7):** +- **`XWB/xwb_1_1_dg_r`** — XWB*1.1*73 Developer's Guide — https://www.va.gov/vdl/documents/Infrastructure/Remote_Proc_Call_Broker_(RPC)/xwb_1_1_dg_r.docx *(listener, REMOTE PROCEDURE #8994, `XWB CREATE CONTEXT`/`CRCONTXT^XWBSEC`, BSE)* +- **`XWB/xwb_1_1_tm_r`** — XWB*1.1*73 Technical Manual — https://www.va.gov/vdl/documents/Infrastructure/Remote_Proc_Call_Broker_(RPC)/xwb_1_1_tm_r.docx +- **`XWB/xwb_1_1_sm_r`** — XWB*1.1*73 Systems Management Guide — https://www.va.gov/vdl/documents/Infrastructure/Remote_Proc_Call_Broker_(RPC)/xwb_1_1_sm_r.docx *(listener startup / port 9200)* +- **`XWB/xwb1_1p34sp`** — M-to-M Broker XWB*1.1*34 Supplement to Patch Description — https://www.va.gov/vdl/documents/Infrastructure/M_to_M_Broker/xwb1_1p34sp.docx + +**Outbound web services today — HWSC / VistaLink (§3.8):** +- **`XOBW/xobw1_0dg`** — HealtheVet Web Services Client (HWSC) Version 1 Developer's Guide — https://www.va.gov/vdl/documents/VistA_GUI_Hybrids/HealtheVet_Web_Services_Client/xobw1_0dg.docx *(WEB SERVER #18.12; middleware-backed)* +- **`XOBW/xobw_1_0_p4_scg`** — XOBW*1*4 Security Configuration Guide — https://www.va.gov/vdl/documents/VistA_GUI_Hybrids/HealtheVet_Web_Services_Client/xobw_1_0_p4_scg.docx *(TLS/SSL config)* +- **`XOBV/vistalink_1_6_7_dg`** — VistaLink Version 1.6.7 Developer Guide — https://www.va.gov/vdl/documents/Infrastructure/VistALink/vistalink_1_6_7_dg.docx *(J2EE/WebLogic middleware — the non-pure-M baseline)* + +**Prior art — VistA Integration Adapter (§3.9):** +- **`VIAB/viab_1_15_installation_backout_rollback_plan_release_notes`** — VIAB*1*15 Installation, Back-Out, and Rollback Plan / Release Notes — https://www.va.gov/vdl/documents/Clinical/VistA_Integration_Adapter_(VIA)/viab_1_15_installation_backout_rollback_plan_release_notes.docx *(VIA = "VISTA INTEGRATION ADAPTOR"; KIDS-released `VIAB` BMS RPCs `LISTORDERS`/`LISTORDERACTIONS`; BMS migration off MDWS to VIA — the basis for the prior-art comparison. NB: the only VIA doc in the gold corpus is this patch artifact; a VIA architecture/design guide is a proposed VDL fetch — see §12.)* + +**m-stdlib internal cross-references:** +- [`https-stack-spec.md`](https-stack-spec.md) — VistA-native HTTPS stack (`VWEB`) — the §9 smoke test. +- [`future-modules-plan.md`](future-modules-plan.md) — `STDNET`/`STDHTTPMSG`/`STDJWT`/`STDVALID` proposals (the L2 pieces this architecture consumes). +- [`../../CLAUDE.md`](../../CLAUDE.md) — engine charset, module tiers, architectural priority rule. + +--- + +*End DRAFT v0.1. Sections are modular by design; each can be promoted to a +standalone spec as detail accretes. Naming set to **`VBL` / vista-bridge-library** +(Q1, pending an existing-namespace check). Next step: stand up the `VistaEngine` +transport (R2) so L3 can cross TDD-red against a live VistA.* diff --git a/docs/vista-de-facto-library-analysis.md b/docs/plans/vista-de-facto-library-analysis.md similarity index 100% rename from docs/vista-de-facto-library-analysis.md rename to docs/plans/vista-de-facto-library-analysis.md diff --git a/docs/tracking/changelog.md b/docs/tracking/changelog.md index 0274adb..4e658ac 100644 --- a/docs/tracking/changelog.md +++ b/docs/tracking/changelog.md @@ -21,7 +21,7 @@ here. See [`README.md`](README.md) § Bucket 4 for the rationale. ## [v0.5.0] — 2026-05-08 **Discoverability & tooling — Wave A.** First wave of the -[discoverability and tooling plan](../plans/discoverability-and-tooling-plan.md): +[discoverability and tooling plan](../plans/historical/discoverability-and-tooling-plan.md): structured-tag grammar, machine-readable manifest, CI manifest-drift gate. Doc + tooling only — no `src/STD*.m` runtime behaviour change. diff --git a/docs/tracking/discoverability-tracker.md b/docs/tracking/discoverability-tracker.md index d67c550..7224d8c 100644 --- a/docs/tracking/discoverability-tracker.md +++ b/docs/tracking/discoverability-tracker.md @@ -5,7 +5,7 @@ authority: this file is the canonical "what's done / in flight / queued" view for the discoverability & tooling plan. Any commit that touches a task below MUST update the row's Status and append to the task's narrative section in the same commit. -plan: docs/plans/discoverability-and-tooling-plan.md (the design doc; this +plan: docs/plans/historical/discoverability-and-tooling-plan.md (the design doc; this tracker drives the work items) sibling: docs/tracking/module-tracker.md (the module-level tracker; deferred decisions D1/D2/D3 there originated in the same plan) @@ -20,7 +20,7 @@ doc_type: [STATUS] # m-stdlib — discoverability & tooling implementation tracker This tracker drives the work in -[`docs/plans/discoverability-and-tooling-plan.md`](../plans/discoverability-and-tooling-plan.md). +[`docs/plans/historical/discoverability-and-tooling-plan.md`](../plans/historical/discoverability-and-tooling-plan.md). The plan is the **why** and **what**; this file is the **who/when/where**. Tasks are grouped into four waves (Wave A → D) matching the plan's § 8 @@ -82,7 +82,7 @@ the work without further orientation. | Concern | Path | |---|---| -| Plan / design doc | [`docs/plans/discoverability-and-tooling-plan.md`](../plans/discoverability-and-tooling-plan.md) | +| Plan / design doc | [`docs/plans/historical/discoverability-and-tooling-plan.md`](../plans/historical/discoverability-and-tooling-plan.md) | | Module tracker (sibling) | [`docs/tracking/module-tracker.md`](module-tracker.md) | | Toolchain findings | [`docs/tracking/TOOLCHAIN-FINDINGS.md`](TOOLCHAIN-FINDINGS.md) | | Released-module catalogue | [`docs/modules/index.md`](../modules/index.md) | diff --git a/docs/tracking/module-tracker.md b/docs/tracking/module-tracker.md index 875ab66..43e71fa 100644 --- a/docs/tracking/module-tracker.md +++ b/docs/tracking/module-tracker.md @@ -579,7 +579,7 @@ time, not development time: Decisions intentionally deferred. Each row records the deferred choice, the trigger condition that should re-open it, and the plan-doc location of the original analysis. Source: -[`../plans/discoverability-and-tooling-plan.md`](../plans/discoverability-and-tooling-plan.md) +[`../plans/historical/discoverability-and-tooling-plan.md`](../plans/historical/discoverability-and-tooling-plan.md) § 11. | ID | Deferred decision | Default in effect | Revisit when | Source |