From 86104334ed26615fcb3c5c0f1cc3e5a46196e5e7 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 07:20:00 +0200 Subject: [PATCH 01/21] docs(brief): add M1.0.4 milestone brief --- briefs/M1.0.4-scene-cook.md | 165 ++++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 briefs/M1.0.4-scene-cook.md diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md new file mode 100644 index 0000000..175039d --- /dev/null +++ b/briefs/M1.0.4-scene-cook.md @@ -0,0 +1,165 @@ +# M1.0.4 — Cooking `.scene.etch` → `.scene.bin` + +> **Status:** PLANNED +> **Phase:** 1 +> **Branch:** `phase-1/scene/scene-cook` +> **Tag:** `v0.10.4-scene-cook` +> **Dependencies:** M0.1 (Tier-0 ECS: `Archetype`, `ComponentSignature`, `Registry`, chunk layout), M0.2 (RTTI), M0.6 (asset registry + `.bin` runtime header convention), M0.8 (grammar v0.6 + scene/prefab descriptors, `compileTypeDecl`), M0.9 (cross-file `validateProject`), M1.0.3 (resource `string`/enum fields + persistent heap, `writeValueAsBytes`/`readBytesAsValue`) +> **Open date:** 2026-06-26 +> **Close date:** — + +--- + +# FROZEN SECTION + +*Authored by Claude.ai. Not modifiable by Claude Code outside a Claude.ai round-trip (see § Accepted deviations).* + +## Context + +First sub-milestone of the scene track (M1.0.4–6) in Phase 1. It delivers the **offline cook** `.scene.etch` → `.scene.bin` for scenes of **direct entities**, producing the zero-copy binary format the M1.0.5 runtime loader (blocker #1) will mmap. It validates the hypothesis that the M0.8 scene descriptors plus the Tier-0 RTTI/ECS infrastructure are sufficient to materialize a memcpy-ready archetype-SoA binary **without a live `World`**. The cook is World-free; runtime instantiation (memcpy into chunks, UUID→handle remap, `on_spawned` dispatch) is M1.0.5 and belongs there. + +**Verified capability boundary (governs the fixture and the negative tests).** The ECS runtime registry path that registers Etch-declared components (`compileTypeDecl` → `fieldKindFromTypeName`) maps only the scalar `FieldKind` set (`int_`/`float_`/`bool_`/`i32_`/`u32_`/`f32_`/`f64_`) plus the resource-only `string_`/`enum_`; any other field type yields `error.InvalidProgram`. A component field therefore holds only a numeric scalar/bool today. `Vec3`/`Entity`/`AssetHandle` type-check in the resolver but are rejected at registration. M1.0.4 introduces no 3D component, so the scalar vocabulary is exactly what the cook serializes; the fixture respects this (`Position { x: f32, y: f32, z: f32 }` as three `f32`, not a `Vec3`). + +This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`SceneEntityDesc`/`ComponentInstanceDesc`), `compileTypeDecl` (registration), `ecs_bridge.writeValueAsBytes`/`readBytesAsValue` (Value↔byte codec), `archetype.zig` (`ComponentSignature`, `sortComponentIds`, `signatureBytes`, `componentSize`), `persistent.zig` (resource `string` materialization). It reinvents none of them. The genuinely new surface is the `.scene.bin` binary codec (format + writer + zero-copy accessor) and the Etch-side cook driver that feeds it. + +## Scope + +**E1 — Etch-side cook driver (`src/etch/scene_cook.zig`).** +- Consume the M0.8 `Scene` descriptor (`src/etch/descriptor.zig`). Register the scene's `component`/`resource` declarations into a `Registry` via the existing `compileTypeDecl` registration path. +- For each entity component-instance field and each scene `resources` resource-instance field: resolve against `Registry.findField`, evaluate the value expression to a `Value` (const-eval), encode via `ecs_bridge.writeValueAsBytes(field.kind, slot, v)`. Resource `string` values are materialized into persistent bytes for the writer to copy; enum values are the declaration-order `u32` discriminant. +- Resolve each entity `parent` name to the parent entity's UUID (intra-scene reference). +- Group entities by `ComponentSignature` (`sortComponentIds` / `signatureBytes`); produce the **neutral writer input** (per-archetype raw component-byte columns + per-resource raw bytes + UUID/name/string entries + per-entity parent-UUID link). +- Reject `instance of` with a clear cook diagnostic (owned by M1.0.6 — see Out-of-scope). Surface the registration-time `error.InvalidProgram` on an unsupported component field type as a clear cook diagnostic, never a panic. +- Imports `weld_core.scene`. Inline tests for the driver's invariants where unit-testable without the full pipeline. + +**E2 — Tier-0 binary format + writer + zero-copy accessor (`src/core/scene/`).** +- `format.zig` — `extern struct SceneHeader`: `magic = "WSCN"`, `version: u16`, `content_version: u16`, `platform: u16`, `flags: u16`, `hash: u64`, `entity_count: u32`, `resource_count: u32`, `string_table_offset: u32`, `uuid_table_offset: u32`, `resources_offset: u32`, `archetypes_offset: u32`, plus `extensions_offset`/`crossrefs_offset` (reserved), cache-line-aligned (64 bytes). Format version/magic constants and the SoA column-layout rules (see Notes). **Shared verbatim with the M1.0.5 loader.** +- `writer.zig` — neutral writer input → `.scene.bin` bytes: deduplicated length-prefixed UTF-8 string table; 16-byte-each UUID table; resources block (`schema_id: u32`, `data_size: u32`, `data: [size]u8` per resource); archetype blocks (`component_mask` + `entity_count: u32` + SoA component columns + `entity_uuids: [N]u32` indices into UUID table + `entity_names: [N]u32` indices into string table). The Entity Extensions Table and Cross-references Table are written **empty** (reserved sections, zero entries). Computes `hash` over content-after-header and back-patches it. +- `accessor.zig` — zero-copy read view over `.scene.bin` bytes: typed getters for the header, a string by index, a UUID by index, a resource by index, archetype-block iteration, per-entity component-column slices. No allocation; validates `magic` + `version` on open. **Reused verbatim by the M1.0.5 loader** (which layers mmap + memcpy-into-chunks + UUID→handle remap + `on_spawned` on top — not duplicated in M1.0.5). +- `root.zig` — re-exports `format`/`writer`/`accessor`; test rooting per `engine-zig-conventions.md` §13. **Imports `weld_core` only — never `weld_etch`** (tier discipline, see Notes). + +**E3 — Resources block, determinism, CLI, CLAUDE.md.** +- Serialize the resources block (scalar + enum + materialized `string`), extend the round-trip to cover resources. +- Determinism: re-cooking the same source yields byte-identical output. +- CLI `tools/scene_cook/main.zig`: thin shim (arg parsing + file I/O via `std.Io`), in-process call to `weld_etch.scene_cook`. Form: `scene_cook --output `. Mirrors `tools/etch_cook` / `tools/asset_cook`. +- `CLAUDE.md` §3.4 update, applied on the milestone branch via `docs(claude-md): update for M1.0.4`: État courant table row for the scene track; Tags row entry `v0.10.4-scene-cook`; open-decisions update recording (a) the M1.0.4 scope boundary, (b) the dynamic-collections home correction (dynamic collections on resource fields — `string[]`, `[K:V]`, `Set` — are unscheduled-additive, folding into the first consuming milestone, **not** M1.0.4; the `M1.0.4` label in `briefs/M1.0.3-resource-nonpod-fields.md` Context is superseded), and (c) Last updated date. + +## Out-of-scope + +- **Prefab instance flattening** — resolving/expanding `instance of "Prefab" "Name" { overrides }`. The M0.8 descriptor parses it into `SceneInstanceDesc`, but cooking it is **owned by M1.0.6** per `engine-phase-1-plan.md`. The cook rejects `instance of` rather than emitting an un-flattened (incorrect) `.scene.bin`. +- **Entity→entity cross-references and the Extensions Table population** — **owned by M1.0.6** ("cross-references entité→entité + table d'extensions"). M1.0.4 writes both sections empty/reserved. +- **Runtime loading** — `.scene.bin` → ECS `World` (mmap, memcpy into chunks, UUID→runtime-handle remap, `on_spawned` dispatch) is **owned by M1.0.5**. M1.0.4 does not instantiate into a `World`; the round-trip uses the accessor. +- **`.prefab.bin` cooking** (standalone prefab templates). The codec could serve it, but M1.0.4 is scoped to `.scene.bin`; the prefab binary is part of the M1.0.6 prefab work. +- **3D / handle / entity field kinds** — `vec2`/`vec3`/`vec4`/`quat`/`mat3`/`mat4`/`color`/`asset_handle`/`entity` `FieldKind` variants, and the AssetHandle path→handle resolution they would enable. M1.0.4 introduces no component that uses them, so the current scalar vocabulary is the whole job; the kinds are introduced by the first milestone that adds a component needing them. +- **World partition** (`.cell.bin` / `.layer.bin` / `.manifest.bin`), HLOD, data layers, streaming. +- **Optional `.scene.bin` compression** (`engine-scene-serialization.md` §3). +- **Save/load** (`.sav`) and scene hot-reload. +- **`@range`/`@unit` enforcement at cook time** — the runtime component registry `FieldDesc` (for Etch-declared components) carries no range/unit metadata. The cook validates field kind/type only. + +## Specs to read first + +1. `engine-scene-serialization.md` — §4 (`.scene.bin` layout — PRIMARY), §1 (scene/prefab/save split), §2 (UUID + name). +2. `engine-ecs-internals.md` — §10 (ECS serialization) + archetype / chunk / component-bitset layout. +3. `engine-asset-pipeline.md` — §6.3 (scene cooking steps), cooking cache. +4. `etch-reference-part2.md` — §30 (`prefab` `of` / `extends`) + scene / resource constructs. +5. `etch-grammar.md` — `scene_decl` / `prefab` productions. +6. `engine-spec.md` — §19 (scene serialization), §3.5 (in-tree discipline), §4 (component POD/SoA invariant). +7. `engine-zig-conventions.md` — §13 (test rooting / lazy-analysis guard — mandatory so `tests/scene/` actually runs), Zig 0.16.x conventions. + +## Files to create or modify + +- `src/etch/scene_cook.zig` — **create** — Etch-side cook driver (E1). +- `src/core/scene/format.zig` — **create** — `extern SceneHeader` + format/magic/version constants + SoA layout rules. Shared with the M1.0.5 loader. +- `src/core/scene/writer.zig` — **create** — neutral input → `.scene.bin` bytes. +- `src/core/scene/accessor.zig` — **create** — zero-copy read view. Reused verbatim by M1.0.5. +- `src/core/scene/root.zig` — **create** — re-exports + test rooting. `weld_core` only. +- `tools/scene_cook/main.zig` — **create** — CLI shim, in-process call to `weld_etch.scene_cook`. +- `tests/scene/cook_roundtrip_test.zig` — **create** — round-trip + determinism (T1, T2). +- `tests/scene/cook_errors_test.zig` — **create** — negative cases (T3). +- `tests/fixtures/scene/arena_wave1.scene.etch` — **create** — round-trip fixture (≥2 archetypes + scalar/string/enum resources + parent). May be an inline source string in the test if preferred. +- `src/etch/root.zig` — **edit** — export `scene_cook` and ensure reachability. +- `build.zig` — **edit** — wire `src/core/scene/` into the `weld_core` module, the `scene_cook` tool artifact, the `tests/scene/` test target (rooted per §13), and an optional build-graph cook step for the observable behaviour. +- *(the `weld_core` module root must re-export `scene` — exact path to confirm in `build.zig`; mechanical wiring)* +- `CLAUDE.md` — **edit** — §3.4 (see Scope E3), via `docs(claude-md): update for M1.0.4` on the branch. + +## Acceptance criteria + +### Tests + +- `tests/scene/cook_roundtrip_test.zig` — `test "scene round-trips through cook and accessor"` — a `.scene.etch` (≥2 archetypes, entities with UUID + name + a `parent` link, a `resources` block with int + string + enum) cooks to `.scene.bin`; the accessor decodes it; every component field value, UUID, name, parent-UUID link, and resource value equals the source. +- `tests/scene/cook_roundtrip_test.zig` — `test "re-cook is byte-identical"` — cooking the same source twice yields identical bytes (determinism; mirrors the M0.6 `cached == cooked` invariant). +- `tests/scene/cook_errors_test.zig` — `test "instance of is rejected (M1.0.6 boundary)"` — a scene with `instance of` → cook error, no `.scene.bin` produced. +- `tests/scene/cook_errors_test.zig` — `test "unsupported component field kind is rejected"` — a component field typed `Vec3` → cook error (no panic). +- `tests/scene/cook_errors_test.zig` — `test "undeclared resource type is rejected"` — `resources { Bogus { … } }` → cook error. +- Inline tests in `src/core/scene/accessor.zig` — getters round-trip on a hand-built `.scene.bin` (magic/version check, string/UUID/resource/archetype access). + +### Benchmarks + +- None. The cook is offline; there is no perf target for M1.0.4. Determinism is covered by a test, not a benchmark. + +### Observable behaviour + +- `zig build scene-cook -- --output /tmp/arena_wave1.scene.bin tests/fixtures/scene/arena_wave1.scene.etch` (or the equivalent tool invocation) produces a `.scene.bin`; `xxd /tmp/arena_wave1.scene.bin | head` shows the `WSCN` magic at offset 0 and a 64-byte header. +- Re-running the same command produces a byte-identical file (`cmp` equal). + +### CI + +- `zig build` clean, zero warnings, on the configured matrix. +- `zig build test` green (debug + ReleaseSafe). +- `zig fmt --check` green. +- `zig build lint` green. +- `commit-msg` hook green on every commit of the branch. +- The new `tests/scene/` target is rooted (its tests appear in the test count) per `engine-zig-conventions.md` §13 — guard against the lazy-analysis silent skip. + +## Conventions + +- **Branch:** `phase-1/scene/scene-cook` +- **Final tag:** `v0.10.4-scene-cook` +- **PR title:** `Phase 1 / Scene / Cooking .scene.etch → .scene.bin` +- **Commit convention:** Conventional Commits (see `engine-development-workflow.md` §4.3) +- **Merge strategy:** squash-and-merge (see `engine-development-workflow.md` §4.6) + +## Notes + +- **Offline, World-free.** The cook produces `.scene.bin` from descriptors + the RTTI registry only. No `World` instantiation — that is M1.0.5. +- **Tier discipline (hard rule).** Tier-0 (`src/core/scene/`) must **never** import `weld_etch`. The Etch coupling (descriptors, `writeValueAsBytes`, const-eval) lives in `src/etch/scene_cook.zig`, which imports `weld_core.scene`. The writer input is raw bytes + tables, no Etch types. If you find yourself wanting to import Etch from `src/core/scene/`, **stop** (Case 2): the split is wrong. +- **The accessor is the read half of the codec**, reused verbatim by the M1.0.5 loader (which adds mmap + memcpy into chunks + UUID→handle remap + `on_spawned` dispatch). Do not duplicate it in M1.0.5. +- **SoA layout (correctness contract with M1.0.5).** Archetype-block component columns are flat N-element SoA arrays (chunk-agnostic); the M1.0.5 loader slices them across 16 KB chunks. Column order = sorted `component_mask` order (`sortComponentIds`); column stride = `Registry.componentSize(component_id)`; each column start aligned to the component alignment. `format.zig` is the **single source of truth** for these rules (shared with the loader). +- **Hash.** The header `hash` = `std.hash.XxHash64.hash(0, …)` over content-after-header — the in-tree convention (RTTI `computeSchemaHashFromParts`, the `zig_codegen` cache, the asset `runtime_bin.zig` header). `engine-scene-serialization.md` §4 says "xxHash3"; that is a wording divergence to reconcile KB-side ("xxHash3" → "XxHash64"). **Do not implement XXH3** — use `std.hash.XxHash64`. If the spec text reads "xxHash3", this note governs the implementation. +- **Format reserves future sections (design-at-day-1, not deferral).** The format reserves the Extensions and Cross-references sections now and the writer/accessor dispatch on `FieldKind`. Consequently, additive work introduced by later milestones — entity cross-refs and the extensions table (M1.0.6), or new 3D/handle field kinds (the first milestone adding such a component) — changes neither the on-disk format nor the loader: the writer/accessor gain columns, the reserved sections gain entries. This is the day-1 design that keeps that future work additive; it is **not** a deferral of M1.0.4 scope. + +--- + +# LIVING SECTION + +*Maintained by Claude Code during the milestone.* + +## Specs read + +- [ ] `engine-scene-serialization.md` (§4, §1, §2) — read +- [ ] `engine-ecs-internals.md` (§10) — read +- [ ] `engine-asset-pipeline.md` (§6.3) — read +- [ ] `etch-reference-part2.md` (§30) — read +- [ ] `etch-grammar.md` (`scene_decl` / `prefab`) — read +- [ ] `engine-spec.md` (§19, §3.5, §4) — read +- [ ] `engine-zig-conventions.md` (§13) — read + +## Execution log + +- — + +## Accepted deviations + +- — + +## Blockers encountered + +- — resolved by or + +## Closing notes + +- **What worked:** +- **What deviated from the original spec:** +- **What to flag explicitly in review:** +- **Final measurements:** +- **Residual risk / technical debt left deliberately:** From 8c5a68f55b75aa3ae9b9630ffeee37242b298e2e Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 07:22:00 +0200 Subject: [PATCH 02/21] docs(brief): confirm specs read for M1.0.4 --- briefs/M1.0.4-scene-cook.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index 175039d..c1a8427 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -136,13 +136,13 @@ This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`Scen ## Specs read -- [ ] `engine-scene-serialization.md` (§4, §1, §2) — read -- [ ] `engine-ecs-internals.md` (§10) — read -- [ ] `engine-asset-pipeline.md` (§6.3) — read -- [ ] `etch-reference-part2.md` (§30) — read -- [ ] `etch-grammar.md` (`scene_decl` / `prefab`) — read -- [ ] `engine-spec.md` (§19, §3.5, §4) — read -- [ ] `engine-zig-conventions.md` (§13) — read +- [x] `engine-scene-serialization.md` (§4, §1, §2) — read 2026-06-26 07:18 +- [x] `engine-ecs-internals.md` (§10) — read 2026-06-26 07:18 +- [x] `engine-asset-pipeline.md` (§6.3) — read 2026-06-26 07:19 +- [x] `etch-reference-part2.md` (§30) — read 2026-06-26 07:20 +- [x] `etch-grammar.md` (`scene_decl` / `prefab`) — read 2026-06-26 07:20 +- [x] `engine-spec.md` (§19, §3.5, §4) — read 2026-06-26 07:21 +- [x] `engine-zig-conventions.md` (§13) — read 2026-06-26 07:21 ## Execution log From 10e9c6814e6c572f46ee1ac42914a99c839bc68b Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 07:22:08 +0200 Subject: [PATCH 03/21] docs(brief): activate M1.0.4 --- briefs/M1.0.4-scene-cook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index c1a8427..32b81e9 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -1,6 +1,6 @@ # M1.0.4 — Cooking `.scene.etch` → `.scene.bin` -> **Status:** PLANNED +> **Status:** ACTIVE > **Phase:** 1 > **Branch:** `phase-1/scene/scene-cook` > **Tag:** `v0.10.4-scene-cook` From dbf8e1b0a4952670150a610abe713d17694c4ac2 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 08:43:28 +0200 Subject: [PATCH 04/21] docs(brief): record design decision on cook registration path --- briefs/M1.0.4-scene-cook.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index 32b81e9..167e63d 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -146,11 +146,11 @@ This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`Scen ## Execution log -- — +- 2026-06-26 07:30 — E1 start. Mapped every brick the milestone assembles (descriptor IR, `ecs_bridge` codec, ECS registry/archetype, `persistent` heap, `value`, `build.zig` wiring, hash convention, CLI shim) against source with file:line. Confirmed: descriptor field values are pre-rendered text (cook must work from the AST for `evalConst`/`compileTypeDecl`); `evalConst(ast, node)` is World-free; `compileTypeDecl` is private + `*World`-coupled in `interp.zig` but only uses `world.registry`. Resolved the registration design fork with Guy (see Accepted deviations). ## Accepted deviations -- — +- 2026-06-26 (verbal decision by Guy, in-session, Cas 3) — **Refactor the registration path in `src/etch/interp.zig`**, which is outside the brief's "Files to create or modify" list. `compileTypeDecl` (and its `compileComponent`/`compileResource` wrappers) currently take `*World` and are private; they only ever use `world.registry` (registration) — `compileResource` additionally seeds the resource store via `world.addResource`. To realize E1's "Register … into a `Registry` via the existing `compileTypeDecl` registration path" **while staying World-free** ("RTTI registry only. No World instantiation"), `compileTypeDecl` is changed to take a `*Registry` (instead of `*World`) and made `pub` so `src/etch/scene_cook.zig` reuses it verbatim against a standalone `Registry`. interp's wrappers pass `world.registry`; the resource-store seeding stays inside `compileResource`, unchanged in behaviour. Rationale: avoids duplicating ~120 lines of subtle layout/default/`FieldKind` logic (divergence risk flagged in the brick audit) and is the reading most faithful to the brief's "into a Registry / World-free" wording. Existing interp/interpreter tests must stay green (no behavioural change for the interp path). ## Blockers encountered From e41252405753593ea8b4f6687a94a48dc70c9357 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 09:23:28 +0200 Subject: [PATCH 05/21] refactor(etch): compileTypeDecl takes *Registry (World-free cook reuse) --- src/etch/interp.zig | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/src/etch/interp.zig b/src/etch/interp.zig index 810e1b2..750fcdc 100644 --- a/src/etch/interp.zig +++ b/src/etch/interp.zig @@ -3316,7 +3316,7 @@ fn compileComponent( literals: *std.ArrayListUnmanaged([*]u8), ) !void { const name = ast.strings.slice(decl.name); - _ = try compileTypeDecl(gpa, ast, world, bridge, name, decl.fields_start, decl.fields_len, .component, literals); + _ = try compileTypeDecl(gpa, ast, &world.registry, bridge, name, decl.fields_start, decl.fields_len, .component, literals); } fn compileResource( @@ -3332,19 +3332,33 @@ fn compileResource( // AND already lives in the resource store with its current value — adding // it again would reset it to defaults. Seed the store only on first compile. const pre_existing = world.registry.idOf(name) != null; - const id = try compileTypeDecl(gpa, ast, world, bridge, name, decl.fields_start, decl.fields_len, .resource, literals); + const id = try compileTypeDecl(gpa, ast, &world.registry, bridge, name, decl.fields_start, decl.fields_len, .resource, literals); if (!pre_existing) { const default_bytes = world.registry.componentDefaultBytes(id); try world.addResource(gpa, id, default_bytes); } } -const RegKind = enum { component, resource }; - -fn compileTypeDecl( +/// Registration origin threaded into `compileTypeDecl`: `.resource` unlocks the +/// resource-only `string`/`enum` field kinds (`.component` stays POD-strict). +/// `pub` so the scene cook can drive `compileTypeDecl` against its own registry. +pub const RegKind = enum { component, resource }; + +/// Register one Etch `component`/`resource` declaration into `registry`, +/// computing its byte layout (`FieldDesc` + size/alignment) and materializing +/// its compile-time default bytes (POD via `evalConst`, resource `string` via +/// an immortal persistent block, resource `enum` via the variant discriminant). +/// Returns the assigned `ComponentId` (or the existing one on a hot-reload +/// re-compile, idempotent). `bridge` records the name→id mapping. +/// +/// Operates on a bare `*Registry` — World-free by construction (it never touches +/// archetypes/entities). The interpreter passes `&world.registry`; the M1.0.4 +/// scene cook (`src/etch/scene_cook.zig`) reuses it verbatim against its own +/// standalone `Registry` so registration is shared, not duplicated. +pub fn compileTypeDecl( gpa: std.mem.Allocator, ast: *const AstArena, - world: *World, + registry: *Registry, bridge: *Bridge, name: []const u8, fields_start: u32, @@ -3445,7 +3459,7 @@ fn compileTypeDecl( // state (entities, component bytes, resource values) survives the swap. // The hot-reload contract is a rule-body edit with the declarations // UNCHANGED; a layout-changing reload (archetype migration) is Phase 2+. - if (world.registry.idOf(name)) |existing_id| { + if (registry.idOf(name)) |existing_id| { switch (reg_kind) { .component => try bridge.mapComponent(gpa, name, existing_id), .resource => try bridge.mapResource(gpa, name, existing_id), @@ -3453,7 +3467,7 @@ fn compileTypeDecl( return existing_id; } - const id = try world.registry.registerComponentRaw(gpa, .{ + const id = try registry.registerComponentRaw(gpa, .{ .name = name, .size = @intCast(size), .alignment = @intCast(max_align), From b1ffd5b2e98cdd12f0e5c17e9cae674f74f17207 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 09:23:29 +0200 Subject: [PATCH 06/21] feat(scene): tier-0 .scene.bin neutral cook model + format constants --- src/core/root.zig | 10 +++ src/core/scene/format.zig | 143 ++++++++++++++++++++++++++++++++++++++ src/core/scene/root.zig | 20 ++++++ 3 files changed, 173 insertions(+) create mode 100644 src/core/scene/format.zig create mode 100644 src/core/scene/root.zig diff --git a/src/core/root.zig b/src/core/root.zig index 9cd09b1..2c16379 100644 --- a/src/core/root.zig +++ b/src/core/root.zig @@ -115,6 +115,14 @@ pub const events = @import("events/root.zig"); /// Phase 3. pub const plugin_loader = @import("plugin_loader/root.zig"); +/// Scene namespace — Tier 0 `.scene.bin` format (M1.0.4). The neutral cook +/// model + on-disk format contract (`scene.format`), the byte writer +/// (`scene.writer`, E2) and the zero-copy accessor (`scene.accessor`, E2, +/// reused verbatim by the M1.0.5 loader). Single canonical entry point at +/// `src/core/scene/root.zig`. Imports `weld_core` internals only — never +/// `weld_etch` (tier discipline). +pub const scene = @import("scene/root.zig"); + comptime { // Force eager analysis of every IPC sub-file so inline tests are // picked up by `zig build test`. Zig 0.16's lazy semantic analysis @@ -182,6 +190,8 @@ comptime { _ = plugin_loader.desc; _ = plugin_loader.api; _ = plugin_loader.loader; + // M1.0.4 — pin the scene sub-files so their inline tests run. + _ = scene.format; // M0.3 — pin the new platform sub-files so their inline tests run. _ = platform.once; _ = platform.time; diff --git a/src/core/scene/format.zig b/src/core/scene/format.zig new file mode 100644 index 0000000..da8d923 --- /dev/null +++ b/src/core/scene/format.zig @@ -0,0 +1,143 @@ +//! `.scene.bin` format — Tier 0, single source of truth shared verbatim with +//! the M1.0.5 runtime loader (`engine-scene-serialization.md` §4). +//! +//! This file owns two things: +//! 1. The **on-disk format contract** — magic / version constants and the SoA +//! column-layout rules (the M1.0.5 loader memcpys archetype columns into +//! 16 KB ECS chunks, so the cook and the loader must agree on column order, +//! stride and alignment). The `SceneHeader` extern struct + the byte-level +//! writer/accessor land in E2. +//! 2. The **neutral cook model** (`CookModel`) — the in-memory representation +//! the M1.0.4 Etch cook driver (`src/etch/scene_cook.zig`) produces and the +//! E2 writer serializes. It is raw bytes + index tables + Tier-0 `FieldKind` +//! only — **no `weld_etch` types** (tier discipline: `src/core/scene/` never +//! imports `weld_etch`). +//! +//! SoA layout contract (correctness contract with the M1.0.5 loader): +//! * Component columns are flat N-element SoA arrays (chunk-agnostic); the +//! loader slices them across 16 KB chunks. +//! * Column order = sorted `component_mask` order (`archetype.sortComponentIds`). +//! * Column stride = `Registry.componentSize(component_id)`. +//! * Each column start is aligned to the component alignment +//! (`Registry.componentAlignment(component_id)`). +//! Components are POD-strict (validator-gated), so archetype columns are pure +//! byte-copyable scalars/enums — never the resource-only `string_` slot. + +const std = @import("std"); + +const registry_mod = @import("../ecs/registry.zig"); + +/// Re-exported so the cook model and the loader name component ids / field +/// kinds through `weld_core.scene.format` without reaching into `ecs.registry`. +pub const ComponentId = registry_mod.ComponentId; +/// Re-exported (see `ComponentId`). The writer/accessor dispatch on this to +/// handle the resource-only `string_` slot (a string-table reference on disk). +pub const FieldKind = registry_mod.FieldKind; + +// ── On-disk format constants ──────────────────────────────────────────────── + +/// File magic at offset 0. `[4]u8` (not a `u32`) so the on-disk byte order is +/// unambiguous regardless of host endianness (the `RuntimeHeader` precedent, +/// `src/modules/asset_pipeline/format/runtime_bin.zig`). +pub const magic = [4]u8{ 'W', 'S', 'C', 'N' }; + +/// `.scene.bin` binary format version (the codec/layout version — bumped on any +/// breaking layout change). Distinct from `content_version` (the authored +/// scene's `version:` field, opaque to the codec). +pub const format_version: u16 = 1; + +// ── Neutral cook model (E1 output → E2 writer input) ───────────────────────── +// +// All references below are indices into the `CookModel`'s own tables; the E2 +// writer resolves them to on-disk offsets. No `weld_etch` types appear here. + +/// `parent_uuid` sentinel: the entity has no parent (root entity). +pub const no_parent: u32 = std.math.maxInt(u32); + +/// One resource `string_` field. The field's 16-byte slot in `ResourceEntry.data` +/// is left zeroed; the writer fills it from `str` (the field's UTF-8 value lives +/// in `CookModel.strings`). Splitting the string out keeps `data` POD and lets +/// the writer dispatch the on-disk string-table reference on `FieldKind.string_`. +pub const StringFieldRef = struct { + /// Byte offset of the `string_` slot within `ResourceEntry.data`. + offset: u16, + /// Index into `CookModel.strings` — the field's resolved UTF-8 value. + str: u32, +}; + +/// One serialized resource (the scene's `resources { … }` block, one per +/// resource instance). `schema_id` is the registry `ComponentId` of the resource +/// type. `data` is `Registry.componentSize(schema_id)` bytes: POD scalar/enum +/// fields are encoded in place; each `string_` field's slot is zeroed and listed +/// in `string_fields`. +pub const ResourceEntry = struct { + schema_id: ComponentId, + data: []u8, + string_fields: []StringFieldRef, +}; + +/// Per-entity identity carried alongside its archetype's SoA columns. The entity +/// occupies slot `i` (its index in `ArchetypeBlock.entities`) of every column. +pub const EntityEntry = struct { + /// Index into `CookModel.strings` — the entity's name. + name: u32, + /// Index into `CookModel.uuids` — the entity's 16-byte UUID. + uuid: u32, + /// Index into `CookModel.uuids` — the parent entity's UUID, or `no_parent`. + parent_uuid: u32, +}; + +/// One archetype block: every entity sharing the same sorted component set, with +/// its components laid out as flat N-element SoA columns. +pub const ArchetypeBlock = struct { + /// Sorted-ascending component ids (`archetype.sortComponentIds`) — the + /// archetype identity / on-disk component mask. + component_ids: []ComponentId, + entity_count: u32, + /// One column per `component_ids` entry, same order. `columns[i]` is a flat + /// `entity_count * componentSize(component_ids[i])`-byte array; the entity at + /// slot `s` occupies `columns[i][s*sz .. (s+1)*sz]`. + columns: [][]u8, + /// Per-entity identity, `len == entity_count`, slot order matches `columns`. + entities: []EntityEntry, +}; + +/// The neutral, World-free model the cook produces. Owns every slice via an +/// internal arena; `deinit` frees the lot. The E2 writer reads it to emit +/// `.scene.bin`; the E1 cook test inspects it directly (no serialization). +pub const CookModel = struct { + /// Deduplicated UTF-8 strings: entity names + resource `string_` values. + strings: [][]const u8, + /// 16-byte UUIDs: entity UUIDs (indexed by `EntityEntry.uuid`/`parent_uuid`). + uuids: [][16]u8, + resources: []ResourceEntry, + archetypes: []ArchetypeBlock, + + /// Backing arena for every slice above — the cook builds into it, the model + /// owns it, `deinit` reclaims it in one shot. + arena: std.heap.ArenaAllocator, + + /// Free all model memory (the backing arena). The model is invalid after. + pub fn deinit(self: *CookModel) void { + self.arena.deinit(); + self.* = undefined; + } +}; + +test "CookModel arena round-trips an empty model" { + const gpa = std.testing.allocator; + var model: CookModel = .{ + .strings = &.{}, + .uuids = &.{}, + .resources = &.{}, + .archetypes = &.{}, + .arena = std.heap.ArenaAllocator.init(gpa), + }; + defer model.deinit(); + try std.testing.expectEqual(@as(usize, 0), model.archetypes.len); +} + +test "format magic + version constants are stable" { + try std.testing.expectEqualSlices(u8, "WSCN", &magic); + try std.testing.expectEqual(@as(u16, 1), format_version); +} diff --git a/src/core/scene/root.zig b/src/core/scene/root.zig new file mode 100644 index 0000000..11df07e --- /dev/null +++ b/src/core/scene/root.zig @@ -0,0 +1,20 @@ +//! Public surface of the Tier-0 `scene` submodule (re-exported from +//! `weld_core` as `weld_core.scene`). Owns the `.scene.bin` format: the neutral +//! cook model + on-disk format contract (`format.zig`), the byte writer +//! (`writer.zig`, E2) and the zero-copy accessor (`accessor.zig`, E2 — reused +//! verbatim by the M1.0.5 loader). +//! +//! **Imports `weld_core` internals only — never `weld_etch`** (tier discipline, +//! `engine-spec.md` §3.5 / the M1.0.4 brief Notes). The Etch coupling +//! (descriptors, const-eval, `writeValueAsBytes`) lives in +//! `src/etch/scene_cook.zig`, which consumes this surface. + +/// `.scene.bin` format contract + the neutral cook model (`CookModel`). +pub const format = @import("format.zig"); + +comptime { + // §13 lazy-analysis guard: pin every sub-file carrying inline `test` blocks + // so they run under the `core_tests` target (`engine-zig-conventions.md` + // §13). `writer`/`accessor` join here in E2. + _ = format; +} From acec13e8339084e42ce0ae22aa2d2564a35a852a Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 09:23:30 +0200 Subject: [PATCH 07/21] =?UTF-8?q?feat(etch):=20scene=20cook=20front-end=20?= =?UTF-8?q?=E2=86=92=20neutral=20model=20(M1.0.4=20E1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- src/etch/root.zig | 9 + src/etch/scene_cook.zig | 734 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 743 insertions(+) create mode 100644 src/etch/scene_cook.zig diff --git a/src/etch/root.zig b/src/etch/root.zig index 5e616ad..a99a0cb 100644 --- a/src/etch/root.zig +++ b/src/etch/root.zig @@ -47,8 +47,17 @@ comptime { _ = @import("value.zig"); _ = @import("ecs_bridge.zig"); _ = @import("persistent.zig"); + // M1.0.4 — pull the scene cook driver into the test import graph (§13). + _ = @import("scene_cook.zig"); } +/// M1.0.4 scene cook — `.scene.etch` source → the neutral Tier-0 scene model +/// (`weld_core.scene.format.CookModel`) the writer serializes to `.scene.bin`. +/// World-free: registers types into a standalone RTTI `Registry` and const-evals +/// the scene's values. Imports `weld_core.scene`; the Tier-0 side never imports +/// `weld_etch` (tier discipline). +pub const scene_cook = @import("scene_cook.zig"); + /// S5 Zig codegen surface — exposed at the module surface so /// `tools/etch_cook` and downstream consumers can drive the codegen /// without depending on the internal path layout. diff --git a/src/etch/scene_cook.zig b/src/etch/scene_cook.zig new file mode 100644 index 0000000..258a89e --- /dev/null +++ b/src/etch/scene_cook.zig @@ -0,0 +1,734 @@ +//! M1.0.4 — Etch-side `.scene.etch` cook driver (front-end). +//! +//! Consumes a parsed Etch program containing component/resource/enum +//! declarations plus exactly one `scene` construct, and produces the **neutral +//! cook model** (`weld_core.scene.format.CookModel`) the E2 writer serializes to +//! `.scene.bin`. World-free: it registers types into a standalone `Registry` +//! (RTTI only) and const-evaluates the scene's field values into raw component +//! bytes — it never instantiates a `World` (that is the M1.0.5 loader's job). +//! +//! Pipeline: +//! 1. Parse the source → AST. +//! 2. Register every `component`/`resource` declaration into a fresh +//! `Registry` via the shared `interp.compileTypeDecl` path (refactored to +//! take a `*Registry`, M1.0.4 deviation). Unsupported field types surface +//! `error.InvalidProgram` as a clear cook diagnostic. +//! 3. Locate the single `scene`; reject `instance of` (M1.0.6 boundary). +//! 4. For each entity component-instance field and each `resources` field: +//! resolve against `Registry.findField`, const-eval the value, encode via +//! `ecs_bridge.writeValueAsBytes`. Resource `string` values are interned +//! into the model's string table; enum values become the declaration-order +//! `u32` discriminant. +//! 5. Resolve each entity `parent` name to the parent's UUID. +//! 6. Group entities by `ComponentSignature` and transpose per-entity blobs +//! into per-archetype flat SoA columns. +//! +//! Tier discipline: this file (Etch side) imports `weld_core.scene`; the Tier-0 +//! `src/core/scene/` never imports `weld_etch`. The writer input is raw bytes + +//! index tables — no Etch types cross into the model. + +const std = @import("std"); + +const ast_mod = @import("ast.zig"); +const interp = @import("interp.zig"); +const bridge_mod = @import("ecs_bridge.zig"); +const persistent = @import("persistent.zig"); +const value_mod = @import("value.zig"); + +const weld_core = @import("weld_core"); +const Registry = weld_core.ecs.registry.Registry; +const ComponentId = weld_core.ecs.registry.ComponentId; +const FieldDesc = weld_core.ecs.registry.FieldDesc; +const FieldKind = weld_core.ecs.registry.FieldKind; +const archetype = weld_core.ecs.archetype; +const format = weld_core.scene.format; + +const AstArena = ast_mod.AstArena; +const StringId = ast_mod.StringId; +const NodeId = ast_mod.NodeId; +const Bridge = bridge_mod.Bridge; + +/// Cook failures. Each is a clear diagnostic, never a panic (the brief's +/// "surface `error.InvalidProgram` … as a clear cook diagnostic"). A companion +/// human message is written through the `diag_out` out-parameter of `cook`. +pub const CookError = error{ + /// The source did not parse cleanly. + ParseFailed, + /// No top-level `scene` construct in the program. + NoSceneConstruct, + /// More than one `scene` construct (a `.scene.etch` holds exactly one). + MultipleScenes, + /// `instance of "Prefab"` — prefab flattening is owned by M1.0.6. + InstanceOfUnsupported, + /// A component/resource field has a type the runtime registry rejects + /// (`error.InvalidProgram`: e.g. `Vec3`/`Entity`/`AssetHandle`, or `string` + /// on a component). + UnsupportedFieldKind, + /// A `component`/`resource` type name is declared twice. + DuplicateType, + /// A scene component/resource instance names a type that was never declared. + UndeclaredType, + /// A field name in an instance body is not a field of the resolved type. + UnknownField, + /// A `..spread` field appeared in a component/resource instance body. + SpreadUnsupported, + /// A field value expression is not constant-evaluable at cook time. + NonConstValue, + /// A value's type does not match the field's kind. + TypeMismatch, + /// A `uuid:`/`parent:` enum value referenced an unknown enum variant. + UnknownEnumVariant, + /// A `uuid:` string is not a valid canonical UUID. + BadUuid, + /// An entity `parent:` name does not match any entity in the scene. + ParentNotFound, + OutOfMemory, +}; + +/// The cook's output: the neutral model plus the `Registry` it was cooked +/// against. The registry owns the type metadata (names/sizes/field offsets + +/// kinds) needed to interpret the model's raw component bytes — the AST is +/// freed by `cook`, so the registry, not the AST, is the model's schema source. +/// Caller owns both; `deinit` frees the model arena then the registry. +pub const Cooked = struct { + model: format.CookModel, + registry: Registry, + + pub fn deinit(self: *Cooked, gpa: std.mem.Allocator) void { + self.model.deinit(); + self.registry.deinit(gpa); + self.* = undefined; + } +}; + +/// Cook an Etch source string into the neutral scene model + its registry. +/// On failure returns a `CookError` and, if `diag_out` is non-null, sets it to a +/// static human-readable message. No `.scene.bin` is produced on failure. +pub fn cook(gpa: std.mem.Allocator, source: []const u8, diag_out: ?*[]const u8) CookError!Cooked { + const parser = @import("parser.zig"); + var pr = parser.parse(gpa, source) catch return fail(diag_out, error.ParseFailed, "Etch parse failed (allocator error)"); + defer pr.deinit(gpa); + if (pr.diagnostics.len > 0) return fail(diag_out, error.ParseFailed, "Etch parse failed"); + const ast = &pr.ast; + + var registry = Registry.init(); + errdefer registry.deinit(gpa); + + var b = Builder.init(gpa, ast, ®istry); + defer b.deinitScratch(); + errdefer b.arena.deinit(); + + try b.registerDecls(diag_out); + const scene_decl = try b.findScene(diag_out); + const model = try b.build(scene_decl, diag_out); + + return .{ .model = model, .registry = registry }; +} + +fn fail(diag_out: ?*[]const u8, err: CookError, msg: []const u8) CookError { + if (diag_out) |d| d.* = msg; + return err; +} + +// ── Builder ────────────────────────────────────────────────────────────────── + +/// One in-progress entity, accumulated before archetype grouping. `comp_ids` is +/// sorted ascending and `comp_blobs[i]` is the `componentSize(comp_ids[i])`-byte +/// component blob for that entity (both in the model arena). +const EntityBuild = struct { + name_idx: u32, + uuid_idx: u32, + parent_name: []const u8, // "" if root; resolved to a uuid index after all entities seen + comp_ids: []ComponentId, + comp_blobs: [][]u8, +}; + +const Builder = struct { + gpa: std.mem.Allocator, + ast: *const AstArena, + registry: *Registry, + arena: std.heap.ArenaAllocator, + + // Scratch (gpa-owned, freed by `deinitScratch`). + bridge: Bridge, + literals: std.ArrayListUnmanaged([*]u8) = .empty, + string_map: std.StringHashMapUnmanaged(u32) = .empty, + uuid_map: std.AutoHashMapUnmanaged([16]u8, u32) = .empty, + name_to_uuid_idx: std.StringHashMapUnmanaged(u32) = .empty, + + // Model tables (arena-owned, become the CookModel slices). + strings: std.ArrayListUnmanaged([]const u8) = .empty, + uuids: std.ArrayListUnmanaged([16]u8) = .empty, + + fn init(gpa: std.mem.Allocator, ast: *const AstArena, registry: *Registry) Builder { + return .{ + .gpa = gpa, + .ast = ast, + .registry = registry, + .arena = std.heap.ArenaAllocator.init(gpa), + .bridge = Bridge.init(), + }; + } + + /// Free everything NOT owned by the produced model: the bridge, the immortal + /// persistent default blocks `compileTypeDecl` allocated, and the scratch + /// hashmaps. The model arena is transferred to the caller (not freed here). + fn deinitScratch(self: *Builder) void { + for (self.literals.items) |block| persistent.destroy(self.gpa, block); + self.literals.deinit(self.gpa); + self.bridge.deinit(self.gpa); + self.string_map.deinit(self.gpa); + self.uuid_map.deinit(self.gpa); + self.name_to_uuid_idx.deinit(self.gpa); + self.strings.deinit(self.gpa); + self.uuids.deinit(self.gpa); + } + + fn a(self: *Builder) std.mem.Allocator { + return self.arena.allocator(); + } + + /// Pass A: register every `component`/`resource` declaration into the + /// registry via the shared `interp.compileTypeDecl` path. + fn registerDecls(self: *Builder, diag_out: ?*[]const u8) CookError!void { + const kinds = self.ast.items.items(.kind); + const datas = self.ast.items.items(.data); + var i: usize = 0; + while (i < self.ast.items.len) : (i += 1) { + switch (kinds[i]) { + .component_decl => { + const decl = self.ast.component_decls.items[datas[i]]; + _ = self.registerOne(self.ast.strings.slice(decl.name), decl.fields_start, decl.fields_len, .component, diag_out) catch |e| return e; + }, + .resource_decl => { + const decl = self.ast.resource_decls.items[datas[i]]; + _ = self.registerOne(self.ast.strings.slice(decl.name), decl.fields_start, decl.fields_len, .resource, diag_out) catch |e| return e; + }, + else => {}, + } + } + } + + fn registerOne( + self: *Builder, + name: []const u8, + fields_start: u32, + fields_len: u32, + reg_kind: interp.RegKind, + diag_out: ?*[]const u8, + ) CookError!ComponentId { + return interp.compileTypeDecl(self.gpa, self.ast, self.registry, &self.bridge, name, fields_start, fields_len, reg_kind, &self.literals) catch |e| switch (e) { + error.InvalidProgram => fail(diag_out, error.UnsupportedFieldKind, "component/resource field has an unsupported type (only scalars, plus resource string/enum, are cookable)"), + error.DuplicateComponent => fail(diag_out, error.DuplicateType, "component/resource type declared more than once"), + else => error.OutOfMemory, + }; + } + + /// Locate the single `scene` construct, erroring if there are zero or many. + fn findScene(self: *Builder, diag_out: ?*[]const u8) CookError!ast_mod.SceneDecl { + const kinds = self.ast.items.items(.kind); + const datas = self.ast.items.items(.data); + var found: ?ast_mod.SceneDecl = null; + var i: usize = 0; + while (i < self.ast.items.len) : (i += 1) { + if (kinds[i] != .scene_decl) continue; + if (found != null) return fail(diag_out, error.MultipleScenes, "more than one scene construct in the source"); + found = self.ast.scene_decls.items[datas[i]]; + } + return found orelse fail(diag_out, error.NoSceneConstruct, "no scene construct in the source"); + } + + /// Build the full neutral model from the resolved scene. + fn build(self: *Builder, scene_decl: ast_mod.SceneDecl, diag_out: ?*[]const u8) CookError!format.CookModel { + // Reject `instance of` up front (M1.0.6 owns prefab flattening). + const children = self.ast.scene_children.items[scene_decl.children_start .. scene_decl.children_start + scene_decl.children_len]; + for (children) |child| { + if (child.kind == .instance) return fail(diag_out, error.InstanceOfUnsupported, "`instance of` (prefab instances) is not cooked by M1.0.4 (owned by M1.0.6)"); + } + + // Per-entity build, accumulating the name→uuid-index map for parent + // resolution in a second pass. + var entities: std.ArrayListUnmanaged(EntityBuild) = .empty; + defer entities.deinit(self.gpa); + for (children) |child| { + const e = self.ast.scene_entities.items[child.index]; + const eb = try self.buildEntity(e, diag_out); + try entities.append(self.gpa, eb); + } + + // Resolve `parent` names → parent uuid indices. + for (entities.items) |*eb| { + // (the parent index is stamped into the per-archetype EntityEntry below) + if (eb.parent_name.len != 0 and self.name_to_uuid_idx.get(eb.parent_name) == null) { + return fail(diag_out, error.ParentNotFound, "entity parent name does not match any entity in the scene"); + } + } + + const archetypes = try self.groupArchetypes(entities.items); + const resources = try self.buildResources(scene_decl, diag_out); + + return .{ + .strings = try self.a().dupe([]const u8, self.strings.items), + .uuids = try self.a().dupe([16]u8, self.uuids.items), + .resources = resources, + .archetypes = archetypes, + .arena = self.arena, + }; + } + + fn buildEntity(self: *Builder, e: ast_mod.SceneEntity, diag_out: ?*[]const u8) CookError!EntityBuild { + const name = self.ast.strings.slice(e.name); + const name_idx = try self.internString(name); + const uuid_bytes = try self.parseEntityUuid(e.uuid, diag_out); + const uuid_idx = try self.internUuid(uuid_bytes); + // Record name→uuid for parent resolution (last writer wins on dup names; + // names are unique within a scene per the spec). + try self.name_to_uuid_idx.put(self.gpa, name, uuid_idx); + + const parent_name: []const u8 = if (e.parent == 0) "" else self.ast.strings.slice(e.parent); + + // Build the per-component blobs, then sort components ascending so the + // entity's signature matches the archetype's canonical column order. + const instances = self.ast.component_instances.items[e.components_start .. e.components_start + e.components_len]; + var ids = try self.a().alloc(ComponentId, instances.len); + var blobs = try self.a().alloc([]u8, instances.len); + for (instances, 0..) |ci, k| { + const type_name = self.ast.strings.slice(ci.type_name); + const id = self.registry.idOf(type_name) orelse return fail(diag_out, error.UndeclaredType, "entity references an undeclared component type"); + ids[k] = id; + blobs[k] = try self.buildComponentBlob(id, ci, diag_out); + } + // Sort (ids, blobs) together by id ascending (insertion sort — component + // counts per entity are tiny). `archetype.sortComponentIds` sorts ids + // alone; we must keep blobs paired, so sort both here. + sortIdsBlobs(ids, blobs); + + return .{ + .name_idx = name_idx, + .uuid_idx = uuid_idx, + .parent_name = parent_name, + .comp_ids = ids, + .comp_blobs = blobs, + }; + } + + /// Build one component blob (`componentSize` bytes) from the type defaults + /// overridden by the instance's fields. Components are POD-strict, so every + /// field is a scalar — no `string_`/`enum_` slots to special-case here. + fn buildComponentBlob(self: *Builder, id: ComponentId, ci: ast_mod.ComponentInstance, diag_out: ?*[]const u8) CookError![]u8 { + const size = self.registry.componentSize(id); + const blob = try self.a().alloc(u8, size); + @memcpy(blob, self.registry.componentDefaultBytes(id)); + + const fields = self.ast.struct_lit_fields.items[ci.fields_start .. ci.fields_start + ci.fields_len]; + for (fields) |slf| { + if (slf.name == 0) return fail(diag_out, error.SpreadUnsupported, "`..spread` is not supported in a component instance"); + const fname = self.ast.strings.slice(slf.name); + const fd = self.registry.findField(id, fname) orelse return fail(diag_out, error.UnknownField, "component instance sets a field the component does not declare"); + try self.encodeScalar(blob, fd, slf.value, diag_out); + } + return blob; + } + + /// Encode a scalar field value into `blob` at `fd.offset`. Resource string / + /// enum fields take their own paths (see `buildResources`); this is for the + /// POD scalar kinds shared by components and resources. + fn encodeScalar(self: *Builder, blob: []u8, fd: FieldDesc, value: NodeId, diag_out: ?*[]const u8) CookError!void { + const slot = blob[fd.offset .. fd.offset + @as(u16, @intCast(fd.kind.sizeBytes()))]; + const v = interp.evalConst(self.ast, value) catch return fail(diag_out, error.NonConstValue, "field value is not constant at cook time"); + bridge_mod.writeValueAsBytes(fd.kind, slot, v) catch return fail(diag_out, error.TypeMismatch, "field value type does not match the field kind"); + } + + /// Group built entities by `ComponentSignature` (sorted ids) and transpose + /// each archetype's per-entity blobs into flat N-element SoA columns. + fn groupArchetypes(self: *Builder, entities: []const EntityBuild) CookError![]format.ArchetypeBlock { + // Map signatureBytes → index into the building archetype list. + var sig_to_idx: std.StringHashMapUnmanaged(usize) = .empty; + defer sig_to_idx.deinit(self.gpa); + // Per-archetype scratch: the list of entity indices that belong to it. + var groups: std.ArrayListUnmanaged(std.ArrayListUnmanaged(usize)) = .empty; + defer { + for (groups.items) |*g| g.deinit(self.gpa); + groups.deinit(self.gpa); + } + var sig_ids: std.ArrayListUnmanaged([]const ComponentId) = .empty; + defer sig_ids.deinit(self.gpa); + + for (entities, 0..) |eb, ei| { + const key = archetype.signatureBytes(eb.comp_ids); + const gop = try sig_to_idx.getOrPut(self.gpa, key); + if (!gop.found_existing) { + gop.value_ptr.* = groups.items.len; + try groups.append(self.gpa, .empty); + try sig_ids.append(self.gpa, eb.comp_ids); + } + try groups.items[gop.value_ptr.*].append(self.gpa, ei); + } + + const blocks = try self.a().alloc(format.ArchetypeBlock, groups.items.len); + for (groups.items, 0..) |group, gi| { + const ids = sig_ids.items[gi]; + const n: u32 = @intCast(group.items.len); + + // SoA columns: column c = concat over entities of blob-for-ids[c]. + const columns = try self.a().alloc([]u8, ids.len); + for (ids, 0..) |id, c| { + const stride = self.registry.componentSize(id); + const col = try self.a().alloc(u8, @as(usize, stride) * group.items.len); + for (group.items, 0..) |ei, slot| { + const eb = entities[ei]; + // The entity's blob for component `id` — find its position in + // the entity's (sorted) id list; same sort order as `ids`, so + // column index `c` maps to the entity's component index `c`. + @memcpy(col[slot * stride ..][0..stride], eb.comp_blobs[c]); + } + columns[c] = col; + } + + const ents = try self.a().alloc(format.EntityEntry, group.items.len); + for (group.items, 0..) |ei, slot| { + const eb = entities[ei]; + const parent_idx: u32 = if (eb.parent_name.len == 0) + format.no_parent + else + self.name_to_uuid_idx.get(eb.parent_name).?; + ents[slot] = .{ .name = eb.name_idx, .uuid = eb.uuid_idx, .parent_uuid = parent_idx }; + } + + blocks[gi] = .{ + .component_ids = try self.a().dupe(ComponentId, ids), + .entity_count = n, + .columns = columns, + .entities = ents, + }; + } + return blocks; + } + + /// Build the resource entries from the scene's `resources { … }` block. + fn buildResources(self: *Builder, scene_decl: ast_mod.SceneDecl, diag_out: ?*[]const u8) CookError![]format.ResourceEntry { + const insts = self.ast.component_instances.items[scene_decl.resources_start .. scene_decl.resources_start + scene_decl.resources_len]; + const out = try self.a().alloc(format.ResourceEntry, insts.len); + for (insts, 0..) |ci, ri| { + const type_name = self.ast.strings.slice(ci.type_name); + const id = self.registry.idOf(type_name) orelse return fail(diag_out, error.UndeclaredType, "resources block references an undeclared resource type"); + out[ri] = try self.buildResourceEntry(id, ci, diag_out); + } + return out; + } + + fn buildResourceEntry(self: *Builder, id: ComponentId, ci: ast_mod.ComponentInstance, diag_out: ?*[]const u8) CookError!format.ResourceEntry { + const size = self.registry.componentSize(id); + const blob = try self.a().alloc(u8, size); + @memcpy(blob, self.registry.componentDefaultBytes(id)); + + // Apply the instance's overrides. Scalars/enums are written into `blob`; + // string overrides are remembered (by field offset) so the string pass + // below picks them up. Reject unknown/spread fields. + const sl_fields = self.ast.struct_lit_fields.items[ci.fields_start .. ci.fields_start + ci.fields_len]; + for (sl_fields) |slf| { + if (slf.name == 0) return fail(diag_out, error.SpreadUnsupported, "`..spread` is not supported in a resources block"); + const fname = self.ast.strings.slice(slf.name); + const fd = self.registry.findField(id, fname) orelse return fail(diag_out, error.UnknownField, "resources block sets a field the resource does not declare"); + switch (fd.kind) { + .string_ => {}, // handled in the string pass below (needs the override expr) + .enum_ => try self.encodeEnum(blob, fd, slf.value, diag_out), + else => try self.encodeScalar(blob, fd, slf.value, diag_out), + } + } + + // String pass: every `string_` field of the type must be materialized + // (its default slot holds a process-local pointer, not serializable). The + // effective bytes are the instance override if present, else the default + // string the registration interned. Zero the slot and record the ref. + var string_fields: std.ArrayListUnmanaged(format.StringFieldRef) = .empty; + defer string_fields.deinit(self.gpa); + for (self.registry.componentFields(id)) |fd| { + if (fd.kind != .string_) continue; + const override_node = self.findFieldOverride(ci, fd.name); + const bytes = if (override_node) |node| + try self.stringValueBytes(node, diag_out) + else + self.defaultStringBytes(id, fd.offset); + const str_idx = try self.internString(bytes); + // Zero the 16-byte slot in `blob` (the default copied a live pointer). + @memset(blob[fd.offset .. fd.offset + @as(u16, @intCast(FieldKind.string_.sizeBytes()))], 0); + try string_fields.append(self.gpa, .{ .offset = fd.offset, .str = str_idx }); + } + + return .{ + .schema_id = id, + .data = blob, + .string_fields = try self.a().dupe(format.StringFieldRef, string_fields.items), + }; + } + + /// The instance's override expression for `field_name`, or null. + fn findFieldOverride(self: *Builder, ci: ast_mod.ComponentInstance, field_name: []const u8) ?NodeId { + const fields = self.ast.struct_lit_fields.items[ci.fields_start .. ci.fields_start + ci.fields_len]; + for (fields) |slf| { + if (slf.name == 0) continue; + if (std.mem.eql(u8, self.ast.strings.slice(slf.name), field_name)) return slf.value; + } + return null; + } + + /// The UTF-8 bytes of a `string_lit` value expression. + fn stringValueBytes(self: *Builder, node: NodeId, diag_out: ?*[]const u8) CookError![]const u8 { + if (self.ast.exprKind(node) != .string_lit) return fail(diag_out, error.NonConstValue, "resource string field value must be a string literal"); + return self.ast.strings.slice(self.ast.exprData(node)); + } + + /// The default string bytes interned at registration: read the immortal + /// `StringSlot` from the registry default bytes (ptr==0 ⇒ empty string). + fn defaultStringBytes(self: *Builder, id: ComponentId, offset: u16) []const u8 { + const defaults = self.registry.componentDefaultBytes(id); + var ss: persistent.StringSlot = undefined; + @memcpy(std.mem.asBytes(&ss), defaults[offset .. offset + @sizeOf(persistent.StringSlot)]); + if (ss.ptr == 0) return ""; + const p: [*]const u8 = @ptrFromInt(ss.ptr); + return p[0..ss.len]; + } + + /// Encode an enum field value (`.variant` tag_path) as its declaration-order + /// `u32` discriminant. + fn encodeEnum(self: *Builder, blob: []u8, fd: FieldDesc, value: NodeId, diag_out: ?*[]const u8) CookError!void { + if (self.ast.exprKind(value) != .tag_path) return fail(diag_out, error.NonConstValue, "resource enum field value must be an enum variant"); + const variant = self.ast.exprData(value); + const edecl = self.findEnumDecl(fd.enum_type_name_id) orelse return fail(diag_out, error.UnknownEnumVariant, "resource enum field references an unknown enum type"); + const idx = self.enumVariantIndex(edecl, variant) orelse return fail(diag_out, error.UnknownEnumVariant, "resource enum field references an unknown enum variant"); + const disc: u32 = idx; + @memcpy(blob[fd.offset .. fd.offset + @sizeOf(u32)], std.mem.asBytes(&disc)); + } + + // ── enum AST lookup (free-fn twins of interp's compile-pass helpers) ── + + fn findEnumDecl(self: *Builder, name: StringId) ?ast_mod.EnumDecl { + for (self.ast.enum_decls.items) |d| { + if (d.name == name) return d; + } + return null; + } + + fn enumVariantIndex(self: *Builder, edecl: ast_mod.EnumDecl, variant: StringId) ?u32 { + var i: u32 = 0; + while (i < edecl.variants_len) : (i += 1) { + if (self.ast.enum_variants.items[edecl.variants_start + i].name == variant) return i; + } + return null; + } + + // ── string / uuid interning (into the model arena) ── + + fn internString(self: *Builder, bytes: []const u8) CookError!u32 { + if (self.string_map.get(bytes)) |idx| return idx; + const owned = try self.a().dupe(u8, bytes); + const idx: u32 = @intCast(self.strings.items.len); + try self.strings.append(self.gpa, owned); + try self.string_map.put(self.gpa, owned, idx); + return idx; + } + + fn internUuid(self: *Builder, bytes: [16]u8) CookError!u32 { + if (self.uuid_map.get(bytes)) |idx| return idx; + const idx: u32 = @intCast(self.uuids.items.len); + try self.uuids.append(self.gpa, bytes); + try self.uuid_map.put(self.gpa, bytes, idx); + return idx; + } + + /// Parse an entity `uuid:` string to 16 bytes. Absent (`0`) ⇒ all-zero + /// (deterministic; the cook never auto-generates a random UUID — that would + /// break the re-cook byte-identity guarantee). + fn parseEntityUuid(self: *Builder, uuid_id: StringId, diag_out: ?*[]const u8) CookError![16]u8 { + if (uuid_id == 0) return std.mem.zeroes([16]u8); + return parseUuid(self.ast.strings.slice(uuid_id)) orelse fail(diag_out, error.BadUuid, "entity uuid is not a valid canonical UUID"); + } +}; + +/// Insertion-sort `(ids, blobs)` jointly by `ids` ascending. Component counts +/// per entity are tiny, so insertion sort is both simplest and fastest. +fn sortIdsBlobs(ids: []ComponentId, blobs: [][]u8) void { + var i: usize = 1; + while (i < ids.len) : (i += 1) { + const key_id = ids[i]; + const key_blob = blobs[i]; + var j = i; + while (j > 0 and ids[j - 1] > key_id) : (j -= 1) { + ids[j] = ids[j - 1]; + blobs[j] = blobs[j - 1]; + } + ids[j] = key_id; + blobs[j] = key_blob; + } +} + +/// Parse a canonical 36-char hyphenated UUID (`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`) +/// into 16 bytes, or null on any malformation. Hex pairs map to bytes in order. +fn parseUuid(s: []const u8) ?[16]u8 { + if (s.len != 36) return null; + var out: [16]u8 = undefined; + var oi: usize = 0; + var i: usize = 0; + while (i < s.len) { + if (i == 8 or i == 13 or i == 18 or i == 23) { + if (s[i] != '-') return null; + i += 1; + continue; + } + const hi = hexNibble(s[i]) orelse return null; + const lo = hexNibble(s[i + 1]) orelse return null; + out[oi] = (hi << 4) | lo; + oi += 1; + i += 2; + } + if (oi != 16) return null; + return out; +} + +fn hexNibble(c: u8) ?u8 { + return switch (c) { + '0'...'9' => c - '0', + 'a'...'f' => c - 'a' + 10, + 'A'...'F' => c - 'A' + 10, + else => null, + }; +} + +// ── tests ───────────────────────────────────────────────────────────────── + +test "parseUuid round-trips a canonical UUID" { + const u = parseUuid("7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e").?; + try std.testing.expectEqual(@as(u8, 0x7b), u[0]); + try std.testing.expectEqual(@as(u8, 0x4e), u[15]); + try std.testing.expect(parseUuid("not-a-uuid") == null); + try std.testing.expect(parseUuid("7b3e2f1a42a34f2b8c9da3f2b1c98d4e") == null); // no hyphens +} + +const e1_fixture = + \\component Position { x: f32 = 0.0, y: f32 = 0.0, z: f32 = 0.0 } + \\component Health { current: int = 100, max: int = 100 } + \\enum Weather { clear, rain, storm } + \\resource GameMode { max_players: int = 4, title: string = "arena", weather: Weather = .clear } + \\scene "ArenaWave1" { + \\ resources { + \\ GameMode { max_players: 8, title: "wave1", weather: .storm } + \\ } + \\ entity "Spawner" { + \\ uuid: "7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e" + \\ Position { x: 1.0, y: 2.0, z: 3.0 } + \\ } + \\ entity "Hero" { + \\ uuid: "9c4f3a2b-1e7d-4a5c-b8e9-f4d2c3a1b5e6" + \\ parent: "Spawner" + \\ Position { x: 10.0, y: 0.0, z: 0.0 } + \\ Health { current: 75, max: 100 } + \\ } + \\} +; + +/// Decode the field `field_name` of an entity at `slot` in `block` to a Value +/// (scalar kinds only — the test's components are POD scalars). +fn decodeColumn(reg: *const Registry, block: format.ArchetypeBlock, id: ComponentId, field_name: []const u8, slot: usize) value_mod.Value { + const col_idx = blk: { + for (block.component_ids, 0..) |cid, c| if (cid == id) break :blk c; + unreachable; + }; + const fd = reg.findField(id, field_name).?; + const stride = reg.componentSize(id); + const slot_bytes = block.columns[col_idx][slot * stride ..][0..stride]; + const fb = slot_bytes[fd.offset .. fd.offset + @as(u16, @intCast(fd.kind.sizeBytes()))]; + return bridge_mod.readBytesAsValue(fd.kind, fb); +} + +test "cook builds the neutral model from a scene (E1)" { + const gpa = std.testing.allocator; + var cooked = try cook(gpa, e1_fixture, null); + defer cooked.deinit(gpa); + + const model = cooked.model; + const reg = &cooked.registry; + const pos = reg.idOf("Position").?; + const health = reg.idOf("Health").?; + const game_mode = reg.idOf("GameMode").?; + + // Two archetypes: [Position] (Spawner) and [Position, Health] (Hero). + try std.testing.expectEqual(@as(usize, 2), model.archetypes.len); + + var solo: ?format.ArchetypeBlock = null; // [Position] + var pair: ?format.ArchetypeBlock = null; // [Position, Health] + for (model.archetypes) |blk| { + if (blk.component_ids.len == 1) solo = blk else pair = blk; + } + const s = solo.?; + const p = pair.?; + + // [Position] archetype — Spawner: one entity, x == 1.0, no parent. + try std.testing.expectEqual(@as(u32, 1), s.entity_count); + try std.testing.expectEqual(pos, s.component_ids[0]); + try std.testing.expectApproxEqAbs(@as(f64, 1.0), decodeColumn(reg, s, pos, "x", 0).float_, 1e-6); + try std.testing.expectEqualStrings("Spawner", model.strings[s.entities[0].name]); + try std.testing.expectEqual(format.no_parent, s.entities[0].parent_uuid); + try std.testing.expectEqual(@as(u8, 0x7b), model.uuids[s.entities[0].uuid][0]); + + // [Position, Health] archetype — Hero: sorted ids, x == 10.0, current == 75, + // parent UUID == Spawner's UUID. + try std.testing.expectEqual(@as(u32, 1), p.entity_count); + try std.testing.expect(p.component_ids[0] < p.component_ids[1]); // sorted ascending + try std.testing.expectApproxEqAbs(@as(f64, 10.0), decodeColumn(reg, p, pos, "x", 0).float_, 1e-6); + try std.testing.expectEqual(@as(i64, 75), decodeColumn(reg, p, health, "current", 0).int_); + try std.testing.expect(p.entities[0].parent_uuid != format.no_parent); + try std.testing.expectEqual(@as(u8, 0x7b), model.uuids[p.entities[0].parent_uuid][0]); // Spawner's uuid + + // Resource GameMode: int override 8, enum .storm == discriminant 2, string "wave1". + try std.testing.expectEqual(@as(usize, 1), model.resources.len); + const res = model.resources[0]; + try std.testing.expectEqual(game_mode, res.schema_id); + + const mp = reg.findField(game_mode, "max_players").?; + try std.testing.expectEqual(@as(i64, 8), bridge_mod.readBytesAsValue(.int_, res.data[mp.offset .. mp.offset + 8]).int_); + + const wf = reg.findField(game_mode, "weather").?; + var disc: u32 = 0; + @memcpy(std.mem.asBytes(&disc), res.data[wf.offset .. wf.offset + 4]); + try std.testing.expectEqual(@as(u32, 2), disc); // .storm + + const tf = reg.findField(game_mode, "title").?; + var title: ?[]const u8 = null; + for (res.string_fields) |sf| if (sf.offset == tf.offset) { + title = model.strings[sf.str]; + }; + try std.testing.expectEqualStrings("wave1", title.?); +} + +test "cook rejects instance of (M1.0.6 boundary)" { + const gpa = std.testing.allocator; + const src = + \\scene "S" { + \\ instance of "Torch" "T1" { } + \\} + ; + var msg: []const u8 = ""; + try std.testing.expectError(error.InstanceOfUnsupported, cook(gpa, src, &msg)); +} + +test "cook rejects an undeclared resource type" { + const gpa = std.testing.allocator; + const src = + \\scene "S" { + \\ resources { Bogus { x: 1 } } + \\} + ; + try std.testing.expectError(error.UndeclaredType, cook(gpa, src, null)); +} + +test "cook rejects an unsupported component field kind" { + const gpa = std.testing.allocator; + const src = + \\component Spin { axis: Vec3 = [0, 0, 0] } + \\scene "S" { + \\ entity "E" { uuid: "7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e" Spin { } } + \\} + ; + try std.testing.expectError(error.UnsupportedFieldKind, cook(gpa, src, null)); +} From 37f543745b72ffb05d5740a711bf7d6edcc19950 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 09:23:31 +0200 Subject: [PATCH 08/21] =?UTF-8?q?docs(brief):=20journal=20update=20?= =?UTF-8?q?=E2=80=94=20E1=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- briefs/M1.0.4-scene-cook.md | 1 + 1 file changed, 1 insertion(+) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index 167e63d..43e4082 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -147,6 +147,7 @@ This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`Scen ## Execution log - 2026-06-26 07:30 — E1 start. Mapped every brick the milestone assembles (descriptor IR, `ecs_bridge` codec, ECS registry/archetype, `persistent` heap, `value`, `build.zig` wiring, hash convention, CLI shim) against source with file:line. Confirmed: descriptor field values are pre-rendered text (cook must work from the AST for `evalConst`/`compileTypeDecl`); `evalConst(ast, node)` is World-free; `compileTypeDecl` is private + `*World`-coupled in `interp.zig` but only uses `world.registry`. Resolved the registration design fork with Guy (see Accepted deviations). +- 2026-06-26 07:55 — E1 done. (1) Refactored `interp.compileTypeDecl` → `*Registry` + `pub`, `RegKind` `pub` (callers pass `&world.registry`); interp suite stays green (no behaviour change). (2) Created Tier-0 `src/core/scene/{format,root}.zig`: the neutral `CookModel` (string/UUID tables, per-archetype sorted-id SoA columns, per-entity name/uuid/parent, resource blobs + `string_fields`) + `WSCN`/version constants; wired `weld_core.scene` re-export + §13 pin. (3) Created `src/etch/scene_cook.zig` — World-free driver: parse → register decls into a standalone `Registry` → reject `instance of` → const-eval entity/resource field values → resolve `parent` name→UUID → group by `ComponentSignature` → flat SoA columns. Returns `Cooked{model, registry}`. Resource `string` materialized into the model string table; enum → `u32` discriminant. (4) E1 inline tests (cook builds the model; rejects `instance of`/undeclared type/`Vec3` field) + format/parseUuid tests all green; `zig build test` EXIT 0, `zig fmt --check` + `zig build lint` clean. Design notes for the E1 STOP-gate review: `format.zig` (nominally E2 in the indicative découpage) is created in E1 because the cook's output is typed against it; resource `string` fields serialize as a string-table reference (the cook does the `FieldKind`-aware encoding, so the Tier-0 writer stays dumb and the accessor dispatches on `FieldKind` at read); the on-disk `component_mask` will be the sorted `ComponentId` list (the cook returns its `Registry` so the round-trip/loader interpret column bytes consistently — in-process for M1.0.4; cross-process name remap is M1.0.5); entity `uuid` absent → deterministic all-zero (the cook never random-generates, preserving re-cook byte-identity). ## Accepted deviations From b0f767faa000f37bc494dba533a3ba48c96caf09 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:12:10 +0200 Subject: [PATCH 09/21] fix(etch): require explicit entity uuid (error.MissingUuid) --- src/etch/scene_cook.zig | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/src/etch/scene_cook.zig b/src/etch/scene_cook.zig index 258a89e..1fd3e11 100644 --- a/src/etch/scene_cook.zig +++ b/src/etch/scene_cook.zig @@ -78,6 +78,9 @@ pub const CookError = error{ TypeMismatch, /// A `uuid:`/`parent:` enum value referenced an unknown enum variant. UnknownEnumVariant, + /// An entity declares no `uuid:`. Explicit, stable identity is required at + /// cook time — auto-generating a UUID is the editor's job, not the cook's. + MissingUuid, /// A `uuid:` string is not a valid canonical UUID. BadUuid, /// An entity `parent:` name does not match any entity in the scene. @@ -537,11 +540,12 @@ const Builder = struct { return idx; } - /// Parse an entity `uuid:` string to 16 bytes. Absent (`0`) ⇒ all-zero - /// (deterministic; the cook never auto-generates a random UUID — that would - /// break the re-cook byte-identity guarantee). + /// Parse an entity `uuid:` string to 16 bytes. An absent `uuid:` is a cook + /// error (`MissingUuid`): explicit identity is required, and the cook never + /// auto-generates one (a random UUID would break the re-cook byte-identity + /// guarantee; deterministic identity is the editor's responsibility). fn parseEntityUuid(self: *Builder, uuid_id: StringId, diag_out: ?*[]const u8) CookError![16]u8 { - if (uuid_id == 0) return std.mem.zeroes([16]u8); + if (uuid_id == 0) return fail(diag_out, error.MissingUuid, "entity requires an explicit uuid"); return parseUuid(self.ast.strings.slice(uuid_id)) orelse fail(diag_out, error.BadUuid, "entity uuid is not a valid canonical UUID"); } }; @@ -712,6 +716,17 @@ test "cook rejects instance of (M1.0.6 boundary)" { try std.testing.expectError(error.InstanceOfUnsupported, cook(gpa, src, &msg)); } +test "cook rejects an entity without uuid" { + const gpa = std.testing.allocator; + const src = + \\component Position { x: f32 = 0.0 } + \\scene "S" { + \\ entity "E" { Position { } } + \\} + ; + try std.testing.expectError(error.MissingUuid, cook(gpa, src, null)); +} + test "cook rejects an undeclared resource type" { const gpa = std.testing.allocator; const src = From ae95e1a6b6f1ae5f58b01a4e70b9e57fe70df888 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:12:11 +0200 Subject: [PATCH 10/21] docs(scene): on-disk identity is schema-registry-indexed not ComponentId --- src/core/scene/format.zig | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/src/core/scene/format.zig b/src/core/scene/format.zig index da8d923..deb4a03 100644 --- a/src/core/scene/format.zig +++ b/src/core/scene/format.zig @@ -16,12 +16,24 @@ //! SoA layout contract (correctness contract with the M1.0.5 loader): //! * Component columns are flat N-element SoA arrays (chunk-agnostic); the //! loader slices them across 16 KB chunks. -//! * Column order = sorted `component_mask` order (`archetype.sortComponentIds`). -//! * Column stride = `Registry.componentSize(component_id)`. +//! * Column order = ascending component order (`archetype.sortComponentIds`). +//! * Column stride = `Registry.componentSize(component)`. //! * Each column start is aligned to the component alignment -//! (`Registry.componentAlignment(component_id)`). +//! (`Registry.componentAlignment(component)`). //! Components are POD-strict (validator-gated), so archetype columns are pure //! byte-copyable scalars/enums — never the resource-only `string_` slot. +//! +//! Component identity (on-disk, `engine-ecs-internals.md` §10 — Schema Registry): +//! the `.scene.bin` carries a **Schema Registry** section; an archetype's +//! component mask and a resource's schema reference are encoded as **file-local +//! indices** into that table — never as runtime `ComponentId`s (which are not +//! stable across runs/processes). Phase-1 schema identity is the component +//! **name** (the runtime registry for Etch-declared components exposes only +//! `componentName`/`idOf` — there is no comptime `schema_hash` for them); the +//! M1.0.5 loader maps each schema name back to a runtime id via `idOf(name)`. +//! The in-memory `CookModel` below keeps its `ComponentId`s — the in-process +//! round-trip resolves them through the cook's own registry; only the E2 +//! on-disk encoding is schema-indexed. const std = @import("std"); @@ -66,10 +78,11 @@ pub const StringFieldRef = struct { }; /// One serialized resource (the scene's `resources { … }` block, one per -/// resource instance). `schema_id` is the registry `ComponentId` of the resource -/// type. `data` is `Registry.componentSize(schema_id)` bytes: POD scalar/enum -/// fields are encoded in place; each `string_` field's slot is zeroed and listed -/// in `string_fields`. +/// resource instance). `schema_id` is the cook's in-memory registry +/// `ComponentId` of the resource type (the E2 writer re-encodes it as a +/// file-local Schema Registry index on disk — see the file header). `data` is +/// `Registry.componentSize(schema_id)` bytes: POD scalar/enum fields are encoded +/// in place; each `string_` field's slot is zeroed and listed in `string_fields`. pub const ResourceEntry = struct { schema_id: ComponentId, data: []u8, @@ -91,7 +104,10 @@ pub const EntityEntry = struct { /// its components laid out as flat N-element SoA columns. pub const ArchetypeBlock = struct { /// Sorted-ascending component ids (`archetype.sortComponentIds`) — the - /// archetype identity / on-disk component mask. + /// in-memory archetype identity. These are the cook's runtime `ComponentId`s; + /// the E2 writer re-encodes them as file-local Schema Registry indices on + /// disk (the on-disk component mask is never raw `ComponentId`s — see the + /// component-identity note in the file header). component_ids: []ComponentId, entity_count: u32, /// One column per `component_ids` entry, same order. `columns[i]` is a flat From f6c0aea8ba70617afa11d56015ab903d5458c6b5 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:12:22 +0200 Subject: [PATCH 11/21] docs(brief): record pre-E2 corrections (uuid + schema identity) --- briefs/M1.0.4-scene-cook.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index 43e4082..0fa12a1 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -151,6 +151,8 @@ This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`Scen ## Accepted deviations +- 2026-06-26 (verbal decision by Guy, in-session, Cas 3) — **Entity `uuid` is mandatory at cook time.** Reverses E1 decision #4 (which mapped an absent `uuid:` to a deterministic all-zero UUID). `scene_cook.parseEntityUuid` now returns `error.MissingUuid` (message "entity requires an explicit uuid") when the entity declares no `uuid:`. Rationale: explicit, stable identity is required for cross-load references; auto-generating UUIDs is the editor's job, not the cook's (the cook stays deterministic without inventing identity). Realizes the brief's "entities with UUID" expectation (Fix #4 — recorded in the brief, skipped in E1). Adds the inline test "cook rejects an entity without uuid". +- 2026-06-26 (verbal decision by Guy, in-session, Cas 3) — **On-disk component identity follows `engine-ecs-internals` §10 (Schema Registry), not raw runtime `ComponentId`.** The `.scene.bin` (E2) emits a Schema Registry section per §10; `ArchetypeBlock.component_mask` and `ResourceEntry.schema_id` are encoded as **file-local indices** into that table, never as runtime `ComponentId`s (which are not stable across runs/processes). Phase-1 schema identity = the component **name** (the runtime registry for Etch-declared components exposes only `componentName`/`idOf`; there is no comptime `schema_hash` for them); at load, M1.0.5 does `idOf(name)` → runtime id + remap. The in-memory `CookModel` **keeps** its `ComponentId` (so the in-process round-trip via the cook's own registry stays valid); only the on-disk encoding becomes schema-indexed. E1 follow-up applied now: corrected the `format.zig` docstring that called `component_ids` an "on-disk component mask". - 2026-06-26 (verbal decision by Guy, in-session, Cas 3) — **Refactor the registration path in `src/etch/interp.zig`**, which is outside the brief's "Files to create or modify" list. `compileTypeDecl` (and its `compileComponent`/`compileResource` wrappers) currently take `*World` and are private; they only ever use `world.registry` (registration) — `compileResource` additionally seeds the resource store via `world.addResource`. To realize E1's "Register … into a `Registry` via the existing `compileTypeDecl` registration path" **while staying World-free** ("RTTI registry only. No World instantiation"), `compileTypeDecl` is changed to take a `*Registry` (instead of `*World`) and made `pub` so `src/etch/scene_cook.zig` reuses it verbatim against a standalone `Registry`. interp's wrappers pass `world.registry`; the resource-store seeding stays inside `compileResource`, unchanged in behaviour. Rationale: avoids duplicating ~120 lines of subtle layout/default/`FieldKind` logic (divergence risk flagged in the brick audit) and is the reading most faithful to the brief's "into a Registry / World-free" wording. Existing interp/interpreter tests must stay green (no behavioural change for the interp path). ## Blockers encountered From b13d938dcf8d27aedf1a75ff80951d951ff33ddd Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:32:02 +0200 Subject: [PATCH 12/21] feat(scene): add scene header, schema registry, column-offset helpers --- src/core/scene/format.zig | 181 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) diff --git a/src/core/scene/format.zig b/src/core/scene/format.zig index deb4a03..640cce8 100644 --- a/src/core/scene/format.zig +++ b/src/core/scene/format.zig @@ -58,6 +58,148 @@ pub const magic = [4]u8{ 'W', 'S', 'C', 'N' }; /// scene's `version:` field, opaque to the codec). pub const format_version: u16 = 1; +/// `SceneHeader` size — the fixed 64-byte (cache-line) prefix every file opens +/// with. All section offsets in the header are relative to the file start. +pub const header_size: usize = 64; + +/// Errors from `SceneHeader.read` / accessor open. +pub const ReadError = error{ + /// The byte slice is smaller than a `SceneHeader`. + TooShort, + /// The first four bytes are not `WSCN`. + BadMagic, + /// `version` is not a format version this build understands. + BadVersion, +}; + +/// `.scene.bin` header — 64 bytes, cache-line aligned, the prefix of every file. +/// Read/written field-by-field little-endian (the `RuntimeHeader` discipline, +/// `src/modules/asset_pipeline/format/runtime_bin.zig`) so the on-disk layout is +/// endianness-defined and unaligned-safe — never `@ptrCast`'d off an arbitrary +/// buffer. All Weld targets are little-endian, so this is also the native order. +/// +/// `hash` covers the content AFTER the header (all sections); a reader can +/// recompute `std.hash.XxHash64.hash(0, bytes[header_size..])` and compare. +/// Section offsets are file-relative. `schema_table_offset` / `schema_count` +/// locate the Schema Registry (`engine-ecs-internals.md` §10); `extensions` and +/// `crossrefs` are reserved (written empty in M1.0.4, populated by M1.0.6). +pub const SceneHeader = extern struct { + magic: [4]u8 = magic, // @0 + version: u16 = format_version, // @4 + content_version: u16 = 0, // @6 — authored scene `version:` (opaque to codec) + platform: u16 = 0, // @8 — reserved (0 = platform-agnostic) + flags: u16 = 0, // @10 — reserved + entity_count: u32 = 0, // @12 + resource_count: u32 = 0, // @16 + schema_count: u32 = 0, // @20 + string_table_offset: u32 = 0, // @24 + uuid_table_offset: u32 = 0, // @28 + schema_table_offset: u32 = 0, // @32 + resources_offset: u32 = 0, // @36 + archetypes_offset: u32 = 0, // @40 + extensions_offset: u32 = 0, // @44 — reserved (empty in M1.0.4) + crossrefs_offset: u32 = 0, // @48 — reserved (empty in M1.0.4) + _reserved: u32 = 0, // @52 — pads `hash` to the 8-aligned @56 + hash: u64 = 0, // @56 + + comptime { + std.debug.assert(@sizeOf(SceneHeader) == header_size); + std.debug.assert(@alignOf(SceneHeader) == 8); + std.debug.assert(@offsetOf(SceneHeader, "hash") == 56); + std.debug.assert(@offsetOf(SceneHeader, "entity_count") == 12); + } + + /// Serialize the header into `buf` little-endian, field by field. + pub fn writeTo(self: SceneHeader, buf: *[header_size]u8) void { + @memset(buf, 0); + @memcpy(buf[0..4], &self.magic); + std.mem.writeInt(u16, buf[4..6], self.version, .little); + std.mem.writeInt(u16, buf[6..8], self.content_version, .little); + std.mem.writeInt(u16, buf[8..10], self.platform, .little); + std.mem.writeInt(u16, buf[10..12], self.flags, .little); + std.mem.writeInt(u32, buf[12..16], self.entity_count, .little); + std.mem.writeInt(u32, buf[16..20], self.resource_count, .little); + std.mem.writeInt(u32, buf[20..24], self.schema_count, .little); + std.mem.writeInt(u32, buf[24..28], self.string_table_offset, .little); + std.mem.writeInt(u32, buf[28..32], self.uuid_table_offset, .little); + std.mem.writeInt(u32, buf[32..36], self.schema_table_offset, .little); + std.mem.writeInt(u32, buf[36..40], self.resources_offset, .little); + std.mem.writeInt(u32, buf[40..44], self.archetypes_offset, .little); + std.mem.writeInt(u32, buf[44..48], self.extensions_offset, .little); + std.mem.writeInt(u32, buf[48..52], self.crossrefs_offset, .little); + std.mem.writeInt(u64, buf[56..64], self.hash, .little); + } + + /// Parse + validate a header from the front of `bytes`. Checks length, + /// magic, and version; does NOT verify `hash` (the caller may). + pub fn read(bytes: []const u8) ReadError!SceneHeader { + if (bytes.len < header_size) return error.TooShort; + var h: SceneHeader = .{}; + @memcpy(&h.magic, bytes[0..4]); + if (!std.mem.eql(u8, &h.magic, &magic)) return error.BadMagic; + h.version = std.mem.readInt(u16, bytes[4..6], .little); + if (h.version != format_version) return error.BadVersion; + h.content_version = std.mem.readInt(u16, bytes[6..8], .little); + h.platform = std.mem.readInt(u16, bytes[8..10], .little); + h.flags = std.mem.readInt(u16, bytes[10..12], .little); + h.entity_count = std.mem.readInt(u32, bytes[12..16], .little); + h.resource_count = std.mem.readInt(u32, bytes[16..20], .little); + h.schema_count = std.mem.readInt(u32, bytes[20..24], .little); + h.string_table_offset = std.mem.readInt(u32, bytes[24..28], .little); + h.uuid_table_offset = std.mem.readInt(u32, bytes[28..32], .little); + h.schema_table_offset = std.mem.readInt(u32, bytes[32..36], .little); + h.resources_offset = std.mem.readInt(u32, bytes[36..40], .little); + h.archetypes_offset = std.mem.readInt(u32, bytes[40..44], .little); + h.extensions_offset = std.mem.readInt(u32, bytes[44..48], .little); + h.crossrefs_offset = std.mem.readInt(u32, bytes[48..52], .little); + h.hash = std.mem.readInt(u64, bytes[56..64], .little); + return h; + } +}; + +/// On-disk Schema Registry entry (`engine-ecs-internals.md` §10). One per +/// distinct component/resource type referenced by the scene. Phase-1 identity is +/// the component **name** (`name_ref` into the string table); `size`/`alignment` +/// (from `Registry.componentSize`/`componentAlignment`) make archetype columns +/// self-describing so the accessor slices them without a registry. The M1.0.5 +/// loader maps `name` → its runtime `ComponentId` via `idOf`. Field-level schema +/// (`engine-ecs-internals.md` §10 "champs", for migration) is deferred — Etch +/// components have no comptime schema hash; the name is the identity. +pub const SchemaEntry = extern struct { + /// String-table byte offset of the component's name. + name_ref: u32, + /// `Registry.componentSize` — the SoA column stride. + size: u16, + /// `Registry.componentAlignment` — the SoA column start alignment. + alignment: u16, +}; + +/// Byte offset (relative to the column region start `region_start`) of column +/// `i` within an archetype block, given each column's `sizes`/`aligns` in column +/// order and the block's `entity_count`. **Writer and accessor MUST call this** +/// so their offsets agree — it is the single source of truth for intra-block SoA +/// column placement (column start aligned to the component alignment). +pub fn columnOffset(region_start: usize, sizes: []const u16, aligns: []const u16, entity_count: u32, i: usize) usize { + var off = region_start; + var c: usize = 0; + while (c < i) : (c += 1) { + off = std.mem.alignForward(usize, off, aligns[c]); + off += @as(usize, sizes[c]) * entity_count; + } + return std.mem.alignForward(usize, off, aligns[i]); +} + +/// End offset of the whole column region (= start of whatever follows the +/// columns), given the same inputs as `columnOffset`. +pub fn columnsRegionEnd(region_start: usize, sizes: []const u16, aligns: []const u16, entity_count: u32) usize { + var off = region_start; + for (sizes, aligns) |sz, al| { + off = std.mem.alignForward(usize, off, al); + off += @as(usize, sz) * entity_count; + } + return off; +} + // ── Neutral cook model (E1 output → E2 writer input) ───────────────────────── // // All references below are indices into the `CookModel`'s own tables; the E2 @@ -157,3 +299,42 @@ test "format magic + version constants are stable" { try std.testing.expectEqualSlices(u8, "WSCN", &magic); try std.testing.expectEqual(@as(u16, 1), format_version); } + +test "SceneHeader writeTo/read round-trips little-endian" { + const h: SceneHeader = .{ + .entity_count = 7, + .resource_count = 2, + .schema_count = 3, + .string_table_offset = 64, + .archetypes_offset = 256, + .hash = 0xDEADBEEFCAFEF00D, + }; + var buf: [header_size]u8 = undefined; + h.writeTo(&buf); + try std.testing.expectEqualSlices(u8, "WSCN", buf[0..4]); + const back = try SceneHeader.read(&buf); + try std.testing.expectEqual(@as(u32, 7), back.entity_count); + try std.testing.expectEqual(@as(u32, 3), back.schema_count); + try std.testing.expectEqual(@as(u32, 256), back.archetypes_offset); + try std.testing.expectEqual(@as(u64, 0xDEADBEEFCAFEF00D), back.hash); +} + +test "SceneHeader.read rejects bad magic, short input, bad version" { + var buf: [header_size]u8 = undefined; + (SceneHeader{}).writeTo(&buf); + try std.testing.expectError(error.TooShort, SceneHeader.read(buf[0..10])); + buf[0] = 'X'; + try std.testing.expectError(error.BadMagic, SceneHeader.read(&buf)); + (SceneHeader{}).writeTo(&buf); + std.mem.writeInt(u16, buf[4..6], 999, .little); + try std.testing.expectError(error.BadVersion, SceneHeader.read(&buf)); +} + +test "columnOffset aligns each column to its component alignment" { + // Two columns: sz=8/al=8 then sz=1/al=1, 4 entities. region starts at 0. + const sizes = [_]u16{ 8, 1 }; + const aligns = [_]u16{ 8, 1 }; + try std.testing.expectEqual(@as(usize, 0), columnOffset(0, &sizes, &aligns, 4, 0)); + try std.testing.expectEqual(@as(usize, 32), columnOffset(0, &sizes, &aligns, 4, 1)); // after 8*4 + try std.testing.expectEqual(@as(usize, 36), columnsRegionEnd(0, &sizes, &aligns, 4)); // 32 + 1*4 +} From 01f12d28369f9e2ba34bdeb410311639e2e72b4b Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:32:16 +0200 Subject: [PATCH 13/21] feat(scene): writer serializes CookModel to .scene.bin bytes --- src/core/scene/writer.zig | 231 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 src/core/scene/writer.zig diff --git a/src/core/scene/writer.zig b/src/core/scene/writer.zig new file mode 100644 index 0000000..614b254 --- /dev/null +++ b/src/core/scene/writer.zig @@ -0,0 +1,231 @@ +//! `.scene.bin` writer — Tier 0. Serializes the neutral `format.CookModel` +//! (produced by the M1.0.4 Etch cook) into the on-disk byte image. The M1.0.5 +//! loader reads it back with `accessor.zig` (the read half of this codec). +//! +//! File layout (all section offsets are file-relative, recorded in the header): +//! ``` +//! [SceneHeader 64 B] +//! [String Table] length-prefixed UTF-8, deduplicated +//! [UUID Table] 16 B each +//! [Schema Registry] §10 — one SchemaEntry per distinct type +//! [Resources Block] per resource: schema-index + data + string refs +//! [Archetype Blocks] per archetype: schema mask + entity meta + SoA columns +//! [Entity Extensions Table] reserved — empty (M1.0.6) +//! [Cross-references Table] reserved — empty (M1.0.6) +//! ``` +//! `hash` covers everything after the header. Component identity on disk is the +//! Schema Registry index (never a runtime `ComponentId`); schema identity is the +//! component name (`engine-ecs-internals.md` §10, M1.0.4 brief deviation). +//! +//! Determinism (the E3 re-cook byte-identity guarantee): every ordering here is +//! deterministic — schemas sorted by ascending `ComponentId` (the cook assigns +//! ids in declaration order), tables emitted in model order, no hashing of +//! addresses. Same source → same registry ids → same bytes. + +const std = @import("std"); + +const format = @import("format.zig"); +const registry_mod = @import("../ecs/registry.zig"); + +const Registry = registry_mod.Registry; +const ComponentId = registry_mod.ComponentId; +const SceneHeader = format.SceneHeader; + +/// Writer failures: allocation, or a table/section larger than a `u32` offset +/// can address (`Overflow` — a scene far beyond any realistic size). +pub const WriteError = error{ OutOfMemory, Overflow }; + +/// Serialize `model` to `.scene.bin` bytes (caller owns the returned slice). +/// `registry` supplies each schema's name/size/alignment (the cook's own +/// registry — the same one `scene_cook.cook` returns alongside the model). +pub fn write(gpa: std.mem.Allocator, model: format.CookModel, registry: *const Registry) WriteError![]u8 { + var w: Writer = .{ .gpa = gpa, .model = model, .registry = registry }; + defer w.deinit(); + return try w.run(); +} + +const Writer = struct { + gpa: std.mem.Allocator, + model: format.CookModel, + registry: *const Registry, + + body: std.ArrayListUnmanaged(u8) = .empty, + // Distinct schema ids (sorted asc) and the inverse map id → file-local index. + schema_ids: std.ArrayListUnmanaged(ComponentId) = .empty, + id_to_index: std.AutoHashMapUnmanaged(ComponentId, u32) = .empty, + // model string ordinal → string-table byte offset (section-relative). + model_str_ref: []u32 = &.{}, + // file-local schema index → its name's string-table byte offset. + schema_name_ref: []u32 = &.{}, + + fn deinit(self: *Writer) void { + self.body.deinit(self.gpa); + self.schema_ids.deinit(self.gpa); + self.id_to_index.deinit(self.gpa); + if (self.model_str_ref.len != 0) self.gpa.free(self.model_str_ref); + if (self.schema_name_ref.len != 0) self.gpa.free(self.schema_name_ref); + } + + fn run(self: *Writer) WriteError![]u8 { + try self.collectSchemas(); + + var hdr: SceneHeader = .{ + .entity_count = try u32From(self.totalEntities()), + .resource_count = try u32From(self.model.resources.len), + .schema_count = try u32From(self.schema_ids.items.len), + }; + + hdr.string_table_offset = try self.sectionOffset(); + try self.writeStringTable(); + hdr.uuid_table_offset = try self.sectionOffset(); + try self.writeUuidTable(); + hdr.schema_table_offset = try self.sectionOffset(); + try self.writeSchemaTable(); + hdr.resources_offset = try self.sectionOffset(); + try self.writeResources(); + hdr.archetypes_offset = try self.sectionOffset(); + try self.writeArchetypes(); + hdr.extensions_offset = try self.sectionOffset(); + try self.appendU32(0); // reserved — empty (M1.0.6) + hdr.crossrefs_offset = try self.sectionOffset(); + try self.appendU32(0); // reserved — empty (M1.0.6) + + hdr.hash = std.hash.XxHash64.hash(0, self.body.items); + + const out = try self.gpa.alloc(u8, format.header_size + self.body.items.len); + var hbuf: [format.header_size]u8 = undefined; + hdr.writeTo(&hbuf); + @memcpy(out[0..format.header_size], &hbuf); + @memcpy(out[format.header_size..], self.body.items); + return out; + } + + // ── byte appenders (little-endian) ── + + fn appendU16(self: *Writer, v: u16) WriteError!void { + var b: [2]u8 = undefined; + std.mem.writeInt(u16, &b, v, .little); + try self.body.appendSlice(self.gpa, &b); + } + fn appendU32(self: *Writer, v: u32) WriteError!void { + var b: [4]u8 = undefined; + std.mem.writeInt(u32, &b, v, .little); + try self.body.appendSlice(self.gpa, &b); + } + fn appendBytes(self: *Writer, bytes: []const u8) WriteError!void { + try self.body.appendSlice(self.gpa, bytes); + } + /// Pad the body with zeros until `header_size + body.len` is `align`-aligned + /// (column starts are aligned in file-absolute space, = mmap memory space). + fn padToFileAlign(self: *Writer, alignment: usize) WriteError!void { + const cur = format.header_size + self.body.items.len; + const target = std.mem.alignForward(usize, cur, alignment); + var pad = target - cur; + while (pad > 0) : (pad -= 1) try self.body.append(self.gpa, 0); + } + /// Current section's file offset (where the next bytes will land). + fn sectionOffset(self: *Writer) WriteError!u32 { + return try u32From(format.header_size + self.body.items.len); + } + + fn totalEntities(self: *Writer) usize { + var n: usize = 0; + for (self.model.archetypes) |arch| n += arch.entity_count; + return n; + } + + // ── schema collection ── + + fn collectSchemas(self: *Writer) WriteError!void { + for (self.model.archetypes) |arch| { + for (arch.component_ids) |id| try self.noteSchema(id); + } + for (self.model.resources) |res| try self.noteSchema(res.schema_id); + // Deterministic order: ascending ComponentId. + std.mem.sort(ComponentId, self.schema_ids.items, {}, comptime std.sort.asc(ComponentId)); + for (self.schema_ids.items, 0..) |id, idx| try self.id_to_index.put(self.gpa, id, @intCast(idx)); + } + + fn noteSchema(self: *Writer, id: ComponentId) WriteError!void { + for (self.schema_ids.items) |existing| if (existing == id) return; + try self.schema_ids.append(self.gpa, id); + } + + // ── sections ── + + fn writeStringTable(self: *Writer) WriteError!void { + const section_start = self.body.items.len; + var dedup: std.StringHashMapUnmanaged(u32) = .empty; + defer dedup.deinit(self.gpa); + + self.model_str_ref = try self.gpa.alloc(u32, self.model.strings.len); + for (self.model.strings, 0..) |s, i| { + self.model_str_ref[i] = try self.internString(&dedup, section_start, s); + } + self.schema_name_ref = try self.gpa.alloc(u32, self.schema_ids.items.len); + for (self.schema_ids.items, 0..) |id, i| { + self.schema_name_ref[i] = try self.internString(&dedup, section_start, self.registry.componentName(id)); + } + } + + /// Intern `s` into the string table (dedup); returns its section-relative + /// byte offset. Each entry is `[u32 len][len bytes]`. + fn internString(self: *Writer, dedup: *std.StringHashMapUnmanaged(u32), section_start: usize, s: []const u8) WriteError!u32 { + if (dedup.get(s)) |off| return off; + const off = try u32From(self.body.items.len - section_start); + try self.appendU32(try u32From(s.len)); + try self.appendBytes(s); + try dedup.put(self.gpa, s, off); + return off; + } + + fn writeUuidTable(self: *Writer) WriteError!void { + for (self.model.uuids) |u| try self.appendBytes(&u); + } + + fn writeSchemaTable(self: *Writer) WriteError!void { + for (self.schema_ids.items, 0..) |id, i| { + try self.appendU32(self.schema_name_ref[i]); // name_ref + try self.appendU16(self.registry.componentSize(id)); // size + try self.appendU16(self.registry.componentAlignment(id)); // alignment + } + } + + fn writeResources(self: *Writer) WriteError!void { + for (self.model.resources) |res| { + try self.appendU32(self.id_to_index.get(res.schema_id).?); // schema index + try self.appendU32(try u32From(res.data.len)); // data_size + try self.appendBytes(res.data); // POD blob (string slots zeroed) + try self.appendU32(try u32From(res.string_fields.len)); // string_field_count + for (res.string_fields) |sf| { + try self.appendU32(sf.offset); // field byte offset in the blob + try self.appendU32(self.model_str_ref[sf.str]); // string-table ref + } + } + } + + fn writeArchetypes(self: *Writer) WriteError!void { + try self.appendU32(try u32From(self.model.archetypes.len)); + for (self.model.archetypes) |arch| { + const n = arch.entity_count; + // Per-column sizes/aligns in column (sorted-id) order. + try self.appendU32(try u32From(arch.component_ids.len)); // component_count + for (arch.component_ids) |id| try self.appendU32(self.id_to_index.get(id).?); // schema indices + try self.appendU32(n); // entity_count + for (arch.entities) |e| try self.appendU32(self.model_str_ref[e.name]); // name refs + for (arch.entities) |e| try self.appendU32(e.uuid); // uuid ordinals + for (arch.entities) |e| try self.appendU32(e.parent_uuid); // parent uuid ordinals / no_parent + + // SoA columns, each aligned (file-absolute) to its component alignment. + for (arch.component_ids, 0..) |id, c| { + try self.padToFileAlign(self.registry.componentAlignment(id)); + std.debug.assert(arch.columns[c].len == @as(usize, self.registry.componentSize(id)) * n); + try self.appendBytes(arch.columns[c]); + } + } + } +}; + +fn u32From(v: usize) WriteError!u32 { + return std.math.cast(u32, v) orelse error.Overflow; +} From 26df2e1be089887a2e053518923c225319083eef Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:32:21 +0200 Subject: [PATCH 14/21] feat(scene): zero-copy .scene.bin accessor + module wiring --- src/core/root.zig | 2 + src/core/scene/accessor.zig | 306 ++++++++++++++++++++++++++++++++++++ src/core/scene/root.zig | 9 +- 3 files changed, 315 insertions(+), 2 deletions(-) create mode 100644 src/core/scene/accessor.zig diff --git a/src/core/root.zig b/src/core/root.zig index 2c16379..dba6492 100644 --- a/src/core/root.zig +++ b/src/core/root.zig @@ -192,6 +192,8 @@ comptime { _ = plugin_loader.loader; // M1.0.4 — pin the scene sub-files so their inline tests run. _ = scene.format; + _ = scene.writer; + _ = scene.accessor; // M0.3 — pin the new platform sub-files so their inline tests run. _ = platform.once; _ = platform.time; diff --git a/src/core/scene/accessor.zig b/src/core/scene/accessor.zig new file mode 100644 index 0000000..b4c0a19 --- /dev/null +++ b/src/core/scene/accessor.zig @@ -0,0 +1,306 @@ +//! `.scene.bin` accessor — Tier 0, zero-copy read view over the bytes the +//! `writer.zig` produced. **Reused verbatim by the M1.0.5 runtime loader**, +//! which layers mmap + memcpy-into-chunks + UUID→handle remap + `on_spawned` on +//! top — none of that lives here. +//! +//! No allocation: every getter returns a slice/pointer into the borrowed bytes +//! (or a small value). `open` validates `magic` + `version`. Scalars are read +//! little-endian via `std.mem.readInt` (unaligned-safe — the byte buffer need +//! not be aligned); component column DATA is returned as raw byte slices for the +//! loader to memcpy. Self-describing: column strides/alignments come from the +//! Schema Registry, so no `Registry` is needed to slice archetype columns. +//! +//! Column placement is computed with `format.columnOffset` — the SAME routine +//! the writer used — so reader/writer offsets agree by construction. + +const std = @import("std"); + +const format = @import("format.zig"); + +const SceneHeader = format.SceneHeader; + +/// Max components per archetype the accessor will slice without heap (stack +/// scratch for the shared column-offset computation). Scene archetypes are +/// small; a malformed file exceeding this trips an assert, not silent corruption. +pub const max_components_per_archetype = 256; + +/// Zero-copy read view over a `.scene.bin` byte image. Construct with `open` +/// (validates the header); every getter borrows from `bytes` without allocating. +pub const Accessor = struct { + bytes: []const u8, + header: SceneHeader, + + /// Validate `magic` + `version` and parse the header. Does not verify the + /// content `hash` (call `verifyHash` if you want that). + pub fn open(bytes: []const u8) format.ReadError!Accessor { + const header = try SceneHeader.read(bytes); + return .{ .bytes = bytes, .header = header }; + } + + /// Recompute the content hash and compare it to the header's. + pub fn verifyHash(self: Accessor) bool { + if (self.bytes.len < format.header_size) return false; + return std.hash.XxHash64.hash(0, self.bytes[format.header_size..]) == self.header.hash; + } + + fn readU16(self: Accessor, off: usize) u16 { + return std.mem.readInt(u16, self.bytes[off..][0..2], .little); + } + fn readU32(self: Accessor, off: usize) u32 { + return std.mem.readInt(u32, self.bytes[off..][0..4], .little); + } + + // ── String / UUID tables ── + + /// The interned string at `ref` (a string-table-relative byte offset, as + /// stored in entity/schema/resource references). + pub fn stringAt(self: Accessor, ref: u32) []const u8 { + const base = self.header.string_table_offset + ref; + const len = self.readU32(base); + return self.bytes[base + 4 ..][0..len]; + } + + /// The 16-byte UUID at ordinal `idx`. + pub fn uuidAt(self: Accessor, idx: u32) *const [16]u8 { + const off = self.header.uuid_table_offset + @as(usize, idx) * 16; + return self.bytes[off..][0..16]; + } + + // ── Schema Registry (§10) ── + + pub const Schema = struct { name: []const u8, size: u16, alignment: u16 }; + + pub fn schemaCount(self: Accessor) u32 { + return self.header.schema_count; + } + + /// The schema at file-local index `idx` (name + column stride + alignment). + pub fn schema(self: Accessor, idx: u32) Schema { + const off = self.header.schema_table_offset + @as(usize, idx) * 8; // 8 = @sizeOf SchemaEntry + return .{ + .name = self.stringAt(self.readU32(off)), + .size = self.readU16(off + 4), + .alignment = self.readU16(off + 6), + }; + } + + // ── Resources ── + + pub fn resourceCount(self: Accessor) u32 { + return self.header.resource_count; + } + + /// A view over resource `idx` (walks the block sequentially — resource + /// counts are small). Fields: `schema_index`, the POD `data` blob, and the + /// string-field references. + pub fn resource(self: Accessor, idx: u32) Resource { + var off: usize = self.header.resources_offset; + var i: u32 = 0; + while (i < idx) : (i += 1) off = self.resourceEnd(off); + return self.resourceAt(off); + } + + fn resourceAt(self: Accessor, off: usize) Resource { + const schema_index = self.readU32(off); + const data_size = self.readU32(off + 4); + const data = self.bytes[off + 8 ..][0..data_size]; + const sf_off = off + 8 + data_size; + const sf_count = self.readU32(sf_off); + return .{ .acc = self, .schema_index = schema_index, .data = data, .sf_off = sf_off + 4, .sf_count = sf_count }; + } + + fn resourceEnd(self: Accessor, off: usize) usize { + const r = self.resourceAt(off); + return r.sf_off + @as(usize, r.sf_count) * 8; + } + + pub const Resource = struct { + acc: Accessor, + schema_index: u32, + data: []const u8, + sf_off: usize, // file offset of the first string-field pair + sf_count: u32, + + /// The materialized string value of the `string_` field at byte `offset` + /// within `data`, or null if no such field. `data`'s slot itself is zero + /// (the value lives in the string table — see the writer). + pub fn stringField(self: Resource, offset: u16) ?[]const u8 { + var i: u32 = 0; + while (i < self.sf_count) : (i += 1) { + const p = self.sf_off + @as(usize, i) * 8; + if (self.acc.readU32(p) == offset) return self.acc.stringAt(self.acc.readU32(p + 4)); + } + return null; + } + }; + + // ── Archetypes ── + + pub fn archetypeCount(self: Accessor) u32 { + return self.readU32(self.header.archetypes_offset); + } + + /// A view over archetype `idx` (walks blocks sequentially). + pub fn archetype(self: Accessor, idx: u32) Archetype { + var off: usize = self.header.archetypes_offset + 4; // skip the count + var i: u32 = 0; + while (i < idx) : (i += 1) off = self.archetypeEnd(off); + return self.archetypeAt(off); + } + + fn archetypeAt(self: Accessor, block_off: usize) Archetype { + const component_count = self.readU32(block_off); + std.debug.assert(component_count <= max_components_per_archetype); + const schema_indices_off = block_off + 4; + const entity_count_off = schema_indices_off + @as(usize, component_count) * 4; + const entity_count = self.readU32(entity_count_off); + const names_off = entity_count_off + 4; + const uuids_off = names_off + @as(usize, entity_count) * 4; + const parents_off = uuids_off + @as(usize, entity_count) * 4; + const columns_base = parents_off + @as(usize, entity_count) * 4; + return .{ + .acc = self, + .component_count = component_count, + .entity_count = entity_count, + .schema_indices_off = schema_indices_off, + .names_off = names_off, + .uuids_off = uuids_off, + .parents_off = parents_off, + .columns_base = columns_base, + }; + } + + fn archetypeEnd(self: Accessor, block_off: usize) usize { + const a = self.archetypeAt(block_off); + return a.columnsEnd(); + } + + pub const Archetype = struct { + acc: Accessor, + component_count: u32, + entity_count: u32, + schema_indices_off: usize, + names_off: usize, + uuids_off: usize, + parents_off: usize, + columns_base: usize, + + /// Schema Registry index of the `c`-th component (column order = the + /// on-disk component mask, sorted ascending by the cook). + pub fn schemaIndex(self: Archetype, c: usize) u32 { + return self.acc.readU32(self.schema_indices_off + c * 4); + } + /// Entity `slot`'s name. + pub fn entityName(self: Archetype, slot: usize) []const u8 { + return self.acc.stringAt(self.acc.readU32(self.names_off + slot * 4)); + } + /// Entity `slot`'s UUID. + pub fn entityUuid(self: Archetype, slot: usize) *const [16]u8 { + return self.acc.uuidAt(self.acc.readU32(self.uuids_off + slot * 4)); + } + /// Entity `slot`'s parent UUID ordinal, or `format.no_parent`. + pub fn entityParent(self: Archetype, slot: usize) u32 { + return self.acc.readU32(self.parents_off + slot * 4); + } + + fn columnByteOffset(self: Archetype, c: usize) usize { + var sizes: [max_components_per_archetype]u16 = undefined; + var aligns: [max_components_per_archetype]u16 = undefined; + var i: usize = 0; + while (i <= c) : (i += 1) { + const s = self.acc.schema(self.schemaIndex(i)); + sizes[i] = s.size; + aligns[i] = s.alignment; + } + return format.columnOffset(self.columns_base, sizes[0 .. c + 1], aligns[0 .. c + 1], self.entity_count, c); + } + + /// The full flat SoA column for the `c`-th component (`entity_count * + /// stride` bytes). + pub fn column(self: Archetype, c: usize) []const u8 { + const stride = self.acc.schema(self.schemaIndex(c)).size; + const start = self.columnByteOffset(c); + return self.acc.bytes[start..][0 .. @as(usize, stride) * self.entity_count]; + } + + /// Entity `slot`'s bytes for the `c`-th component. + pub fn componentSlot(self: Archetype, c: usize, slot: usize) []const u8 { + const stride = self.acc.schema(self.schemaIndex(c)).size; + return self.column(c)[slot * stride ..][0..stride]; + } + + fn columnsEnd(self: Archetype) usize { + var sizes: [max_components_per_archetype]u16 = undefined; + var aligns: [max_components_per_archetype]u16 = undefined; + var i: usize = 0; + while (i < self.component_count) : (i += 1) { + const s = self.acc.schema(self.schemaIndex(i)); + sizes[i] = s.size; + aligns[i] = s.alignment; + } + return format.columnsRegionEnd(self.columns_base, sizes[0..self.component_count], aligns[0..self.component_count], self.entity_count); + } + }; +}; + +// ── tests ───────────────────────────────────────────────────────────────── + +const registry_mod = @import("../ecs/registry.zig"); +const writer = @import("writer.zig"); + +test "accessor round-trips a hand-built model through the writer" { + const gpa = std.testing.allocator; + + // A registry with one component: Pos { x: f32, y: f32 } (size 8, align 4). + var reg = registry_mod.Registry.init(); + defer reg.deinit(gpa); + const fields = [_]registry_mod.FieldDesc{ + .{ .name = "x", .offset = 0, .kind = .f32_ }, + .{ .name = "y", .offset = 4, .kind = .f32_ }, + }; + const pos = try reg.registerComponentRaw(gpa, .{ .name = "Pos", .size = 8, .alignment = 4, .default_bytes = &[_]u8{0} ** 8, .fields = &fields }); + + // Build a tiny CookModel by hand: 1 archetype [Pos], 2 entities. + var arena = std.heap.ArenaAllocator.init(gpa); + const a = arena.allocator(); + const names = try a.dupe([]const u8, &.{ try a.dupe(u8, "A"), try a.dupe(u8, "B") }); + const uuids = try a.dupe([16]u8, &.{ [_]u8{1} ** 16, [_]u8{2} ** 16 }); + var col = try a.alloc(u8, 8 * 2); + std.mem.writeInt(u32, col[0..4], @bitCast(@as(f32, 1.5)), .little); // A.x + std.mem.writeInt(u32, col[4..8], @bitCast(@as(f32, 2.5)), .little); // A.y + std.mem.writeInt(u32, col[8..12], @bitCast(@as(f32, 9.0)), .little); // B.x + std.mem.writeInt(u32, col[12..16], @bitCast(@as(f32, 0.0)), .little); // B.y + const cols = try a.dupe([]u8, &.{col}); + const ents = try a.dupe(format.EntityEntry, &.{ + .{ .name = 0, .uuid = 0, .parent_uuid = format.no_parent }, + .{ .name = 1, .uuid = 1, .parent_uuid = 0 }, // B's parent = A + }); + const ids = try a.dupe(format.ComponentId, &.{pos}); + const blocks = try a.dupe(format.ArchetypeBlock, &.{.{ .component_ids = ids, .entity_count = 2, .columns = cols, .entities = ents }}); + var model: format.CookModel = .{ .strings = names, .uuids = uuids, .resources = &.{}, .archetypes = blocks, .arena = arena }; + defer model.deinit(); + + const bytes = try writer.write(gpa, model, ®); + defer gpa.free(bytes); + + var acc = try Accessor.open(bytes); + try std.testing.expect(acc.verifyHash()); + try std.testing.expectEqual(@as(u32, 1), acc.archetypeCount()); + try std.testing.expectEqual(@as(u32, 1), acc.schemaCount()); + try std.testing.expectEqualStrings("Pos", acc.schema(0).name); + + const arch = acc.archetype(0); + try std.testing.expectEqual(@as(u32, 2), arch.entity_count); + try std.testing.expectEqualStrings("A", arch.entityName(0)); + try std.testing.expectEqualStrings("B", arch.entityName(1)); + try std.testing.expectEqual(format.no_parent, arch.entityParent(0)); + try std.testing.expectEqual(@as(u8, 1), arch.entityUuid(0)[0]); + // B's parent ordinal resolves to A's uuid. + try std.testing.expectEqual(@as(u8, 1), arch.acc.uuidAt(arch.entityParent(1))[0]); + + // Decode A.x and B.x from the column. + const ax: f32 = @bitCast(std.mem.readInt(u32, arch.componentSlot(0, 0)[0..4], .little)); + const bx: f32 = @bitCast(std.mem.readInt(u32, arch.componentSlot(0, 1)[0..4], .little)); + try std.testing.expectApproxEqAbs(@as(f32, 1.5), ax, 1e-6); + try std.testing.expectApproxEqAbs(@as(f32, 9.0), bx, 1e-6); +} diff --git a/src/core/scene/root.zig b/src/core/scene/root.zig index 11df07e..bdc79c2 100644 --- a/src/core/scene/root.zig +++ b/src/core/scene/root.zig @@ -11,10 +11,15 @@ /// `.scene.bin` format contract + the neutral cook model (`CookModel`). pub const format = @import("format.zig"); +/// `.scene.bin` byte writer: `format.CookModel` → on-disk bytes. +pub const writer = @import("writer.zig"); +/// `.scene.bin` zero-copy accessor (read half; reused verbatim by M1.0.5). +pub const accessor = @import("accessor.zig"); comptime { // §13 lazy-analysis guard: pin every sub-file carrying inline `test` blocks - // so they run under the `core_tests` target (`engine-zig-conventions.md` - // §13). `writer`/`accessor` join here in E2. + // so they run under the `core_tests` target (`engine-zig-conventions.md` §13). _ = format; + _ = writer; + _ = accessor; } From 3cd4ffe8dff0c02de5d2fabfb79253b0735a904e Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:32:27 +0200 Subject: [PATCH 15/21] test(scene): cook to writer to accessor round-trip + build target --- build.zig | 10 ++ tests/fixtures/scene/arena_wave1.scene.etch | 31 ++++++ tests/scene/cook_roundtrip_test.zig | 111 ++++++++++++++++++++ 3 files changed, 152 insertions(+) create mode 100644 tests/fixtures/scene/arena_wave1.scene.etch create mode 100644 tests/scene/cook_roundtrip_test.zig diff --git a/build.zig b/build.zig index 2192c9c..0d65a8d 100644 --- a/build.zig +++ b/build.zig @@ -336,6 +336,10 @@ pub fn build(b: *std.Build) void { asset_pipeline: bool = false, /// M0.6 / E2 — when set, imports the `foundation` module (simd). foundation: bool = false, + /// M1.0.4 — when set, imports `weld_etch` (the scene cook driver). A + /// dedicated flag rather than `.etch` so `tests/scene/` does not pull in + /// the `corpus_facade` baggage `.etch` carries. + scene: bool = false, /// M0.4 stabilization — when set, create a dedicated `zig build /// ` step that runs ONLY this test. Used by the CI /// runtime-smoke-test job to gate strictly on the capture PSNR @@ -420,6 +424,9 @@ pub fn build(b: *std.Build) void { // matched-count observable. .{ .path = "tests/etch/v1/query_filters_test.zig", .etch = true }, .{ .path = "tests/etch_interp/corpus_test.zig", .etch_interp = true }, + // M1.0.4 / E2 — scene cook → writer → accessor round-trip (entities, + // archetypes, UUIDs, names, parent links). + .{ .path = "tests/scene/cook_roundtrip_test.zig", .scene = true }, // M0.3 — common platform layer tests. .{ .path = "tests/platform/fs_vfs_test.zig" }, .{ .path = "tests/platform/time_test.zig" }, @@ -539,6 +546,9 @@ pub fn build(b: *std.Build) void { if (spec.foundation) { t_mod.addImport("foundation", foundation_module); } + if (spec.scene) { + t_mod.addImport("weld_etch", etch_module); + } const t = b.addTest(.{ .root_module = t_mod }); const t_run = b.addRunArtifact(t); if (spec.needs_stub_plugins) { diff --git a/tests/fixtures/scene/arena_wave1.scene.etch b/tests/fixtures/scene/arena_wave1.scene.etch new file mode 100644 index 0000000..8251797 --- /dev/null +++ b/tests/fixtures/scene/arena_wave1.scene.etch @@ -0,0 +1,31 @@ +// M1.0.4 cook round-trip fixture. Two archetypes ([Position] and +// [Position, Health]) + a parent link + a resources block (int + string + enum). +// +// The component/resource/enum declarations precede the scene so the single-file +// cook resolves every referenced type: M1.0.4 cooks one self-contained program +// (cross-file `import` resolution at cook time is future work), so this fixture +// is a multi-construct `.etch` program rather than a strict §21 single-`scene` +// file. The cook driver parses it in general mode and rejects only `instance of` +// (owned by M1.0.6). +component Position { x: f32 = 0.0, y: f32 = 0.0, z: f32 = 0.0 } +component Health { current: int = 100, max: int = 100 } +enum Weather { clear, rain, storm } +resource GameMode { max_players: int = 4, title: string = "arena", weather: Weather = .clear } + +scene "ArenaWave1" { + resources { + GameMode { max_players: 8, title: "wave1", weather: .storm } + } + + entity "Spawner" { + uuid: "7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e" + Position { x: 1.0, y: 2.0, z: 3.0 } + } + + entity "Hero" { + uuid: "9c4f3a2b-1e7d-4a5c-b8e9-f4d2c3a1b5e6" + parent: "Spawner" + Position { x: 10.0, y: 0.0, z: 0.0 } + Health { current: 75, max: 100 } + } +} diff --git a/tests/scene/cook_roundtrip_test.zig b/tests/scene/cook_roundtrip_test.zig new file mode 100644 index 0000000..7dbfa52 --- /dev/null +++ b/tests/scene/cook_roundtrip_test.zig @@ -0,0 +1,111 @@ +//! M1.0.4 E2 — `.scene.etch` → cook → `.scene.bin` writer → zero-copy accessor +//! round-trip. Reads the committed fixture, cooks it (`weld_etch.scene_cook`), +//! serializes the model (`weld_core.scene.writer`), opens the bytes +//! (`weld_core.scene.accessor`), and asserts that entities, archetypes, UUIDs, +//! names, and parent links survive the round-trip byte-for-byte in meaning. +//! +//! Resource-block + determinism assertions are added in E3 (per the milestone +//! découpage); the writer already serializes resources, but this E2 gate covers +//! the entity/archetype/identity surface. + +const std = @import("std"); +const weld_core = @import("weld_core"); +const weld_etch = @import("weld_etch"); + +const scene = weld_core.scene; +const Registry = weld_core.ecs.registry.Registry; +const scene_cook = weld_etch.scene_cook; + +const fixture_path = "tests/fixtures/scene/arena_wave1.scene.etch"; + +fn readFixture(gpa: std.mem.Allocator, io: std.Io) ![]u8 { + const dir = std.Io.Dir.cwd(); + var file = try dir.openFile(io, fixture_path, .{}); + defer file.close(io); + const stat = try file.stat(io); + const buf = try gpa.alloc(u8, stat.size); + errdefer gpa.free(buf); + var read_buf: [16 * 1024]u8 = undefined; + var reader = file.reader(io, &read_buf); + var written: usize = 0; + while (written < buf.len) { + const n = try reader.interface.readSliceShort(buf[written..]); + if (n == 0) break; + written += n; + } + return buf[0..written]; +} + +/// Find, within `arch`, the column index whose schema name == `comp` (column +/// order is the cook's sorted-id mask, surfaced by name through the schema table). +fn columnOf(acc: scene.accessor.Accessor, arch: scene.accessor.Accessor.Archetype, comp: []const u8) ?usize { + var c: usize = 0; + while (c < arch.component_count) : (c += 1) { + if (std.mem.eql(u8, acc.schema(arch.schemaIndex(c)).name, comp)) return c; + } + return null; +} + +fn decodeF32(acc: scene.accessor.Accessor, reg: *const Registry, arch: scene.accessor.Accessor.Archetype, comp: []const u8, field: []const u8, slot: usize) f32 { + const c = columnOf(acc, arch, comp).?; + const fd = reg.findField(reg.idOf(comp).?, field).?; + const slot_bytes = arch.componentSlot(c, slot); + return @bitCast(std.mem.readInt(u32, slot_bytes[fd.offset..][0..4], .little)); +} + +fn decodeI64(acc: scene.accessor.Accessor, reg: *const Registry, arch: scene.accessor.Accessor.Archetype, comp: []const u8, field: []const u8, slot: usize) i64 { + const c = columnOf(acc, arch, comp).?; + const fd = reg.findField(reg.idOf(comp).?, field).?; + const slot_bytes = arch.componentSlot(c, slot); + return std.mem.readInt(i64, slot_bytes[fd.offset..][0..8], .little); +} + +test "scene round-trips through cook and accessor" { + const gpa = std.testing.allocator; + const io = std.testing.io; + + const src = try readFixture(gpa, io); + defer gpa.free(src); + + var cooked = try scene_cook.cook(gpa, src, null); + defer cooked.deinit(gpa); + + const bytes = try scene.writer.write(gpa, cooked.model, &cooked.registry); + defer gpa.free(bytes); + + var acc = try scene.accessor.Accessor.open(bytes); + try std.testing.expect(acc.verifyHash()); + try std.testing.expectEqual(@as(u32, 2), acc.archetypeCount()); + + // Spawner's UUID first byte, for the parent-link assertion. + const spawner_uuid0: u8 = 0x7b; + + var saw_solo = false; + var saw_pair = false; + var ai: u32 = 0; + while (ai < acc.archetypeCount()) : (ai += 1) { + const arch = acc.archetype(ai); + try std.testing.expectEqual(@as(u32, 1), arch.entity_count); + + if (arch.component_count == 1) { + // [Position] — Spawner (root). + saw_solo = true; + try std.testing.expectEqualStrings("Spawner", arch.entityName(0)); + try std.testing.expectEqual(scene.format.no_parent, arch.entityParent(0)); + try std.testing.expectEqual(spawner_uuid0, arch.entityUuid(0)[0]); + try std.testing.expectApproxEqAbs(@as(f32, 1.0), decodeF32(acc, &cooked.registry, arch, "Position", "x", 0), 1e-6); + } else { + // [Position, Health] — Hero (parented to Spawner). + saw_pair = true; + try std.testing.expectEqual(@as(u32, 2), arch.component_count); + try std.testing.expectEqualStrings("Hero", arch.entityName(0)); + try std.testing.expectApproxEqAbs(@as(f32, 10.0), decodeF32(acc, &cooked.registry, arch, "Position", "x", 0), 1e-6); + try std.testing.expectEqual(@as(i64, 75), decodeI64(acc, &cooked.registry, arch, "Health", "current", 0)); + // Parent link resolves to Spawner's UUID. + const parent = arch.entityParent(0); + try std.testing.expect(parent != scene.format.no_parent); + try std.testing.expectEqual(spawner_uuid0, acc.uuidAt(parent)[0]); + } + } + try std.testing.expect(saw_solo and saw_pair); +} From af2906617e6ae812881d29b1ff61b23d522c790d Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Fri, 26 Jun 2026 15:32:38 +0200 Subject: [PATCH 16/21] =?UTF-8?q?docs(brief):=20journal=20update=20?= =?UTF-8?q?=E2=80=94=20E2=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- briefs/M1.0.4-scene-cook.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index 0fa12a1..f154528 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -149,6 +149,8 @@ This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`Scen - 2026-06-26 07:30 — E1 start. Mapped every brick the milestone assembles (descriptor IR, `ecs_bridge` codec, ECS registry/archetype, `persistent` heap, `value`, `build.zig` wiring, hash convention, CLI shim) against source with file:line. Confirmed: descriptor field values are pre-rendered text (cook must work from the AST for `evalConst`/`compileTypeDecl`); `evalConst(ast, node)` is World-free; `compileTypeDecl` is private + `*World`-coupled in `interp.zig` but only uses `world.registry`. Resolved the registration design fork with Guy (see Accepted deviations). - 2026-06-26 07:55 — E1 done. (1) Refactored `interp.compileTypeDecl` → `*Registry` + `pub`, `RegKind` `pub` (callers pass `&world.registry`); interp suite stays green (no behaviour change). (2) Created Tier-0 `src/core/scene/{format,root}.zig`: the neutral `CookModel` (string/UUID tables, per-archetype sorted-id SoA columns, per-entity name/uuid/parent, resource blobs + `string_fields`) + `WSCN`/version constants; wired `weld_core.scene` re-export + §13 pin. (3) Created `src/etch/scene_cook.zig` — World-free driver: parse → register decls into a standalone `Registry` → reject `instance of` → const-eval entity/resource field values → resolve `parent` name→UUID → group by `ComponentSignature` → flat SoA columns. Returns `Cooked{model, registry}`. Resource `string` materialized into the model string table; enum → `u32` discriminant. (4) E1 inline tests (cook builds the model; rejects `instance of`/undeclared type/`Vec3` field) + format/parseUuid tests all green; `zig build test` EXIT 0, `zig fmt --check` + `zig build lint` clean. Design notes for the E1 STOP-gate review: `format.zig` (nominally E2 in the indicative découpage) is created in E1 because the cook's output is typed against it; resource `string` fields serialize as a string-table reference (the cook does the `FieldKind`-aware encoding, so the Tier-0 writer stays dumb and the accessor dispatches on `FieldKind` at read); the on-disk `component_mask` will be the sorted `ComponentId` list (the cook returns its `Registry` so the round-trip/loader interpret column bytes consistently — in-process for M1.0.4; cross-process name remap is M1.0.5); entity `uuid` absent → deterministic all-zero (the cook never random-generates, preserving re-cook byte-identity). +- 2026-06-26 09:10 — E2 done. `format.zig`: `SceneHeader` (64 B, explicit-LE `writeTo`/`read`, comptime offset asserts) + §10 Schema Registry (`SchemaEntry` = name_ref + size + align; name-based identity per the deviation) + shared `columnOffset`/`columnsRegionEnd` (writer + accessor compute identical SoA column placement). `writer.zig`: `CookModel` → `.scene.bin` (header + dedup length-prefixed string table + UUID table + schema registry + resources block + archetype blocks [schema-index mask + name/uuid/parent arrays + file-absolute-aligned SoA columns] + empty extensions/crossrefs; `XxHash64(0)` over content-after-header; deterministic — schemas sorted ascending by id). `accessor.zig`: zero-copy view (`open` validates magic+version; self-describing via the schema table, no registry needed to slice columns). Wired writer+accessor into `scene/root.zig` + core §13 pins; `tests/scene/` target wired in `build.zig` via a dedicated `scene` flag. `tests/scene/cook_roundtrip_test.zig` reads the committed fixture, cooks→writes→accessor, asserts entities/archetypes/uuids/names/parent; accessor + header + columnOffset inline tests green. `zig build test` EXIT 0, fmt+lint clean. E2-gate note: the writer already serializes the resources block (one cohesive codec) — E3 adds the resource round-trip assertions + determinism + CLI + `cook_errors_test.zig` + CLAUDE.md. + ## Accepted deviations - 2026-06-26 (verbal decision by Guy, in-session, Cas 3) — **Entity `uuid` is mandatory at cook time.** Reverses E1 decision #4 (which mapped an absent `uuid:` to a deterministic all-zero UUID). `scene_cook.parseEntityUuid` now returns `error.MissingUuid` (message "entity requires an explicit uuid") when the entity declares no `uuid:`. Rationale: explicit, stable identity is required for cross-load references; auto-generating UUIDs is the editor's job, not the cook's (the cook stays deterministic without inventing identity). Realizes the brief's "entities with UUID" expectation (Fix #4 — recorded in the brief, skipped in E1). Adds the inline test "cook rejects an entity without uuid". From 4f383169444b1f308bc43464a5e428f6553f6aff Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Sat, 27 Jun 2026 00:50:43 +0200 Subject: [PATCH 17/21] feat(scene): capture and propagate scene content_version --- src/core/scene/format.zig | 4 +++ src/core/scene/writer.zig | 1 + src/etch/scene_cook.zig | 60 ++++++++++++--------------------------- 3 files changed, 23 insertions(+), 42 deletions(-) diff --git a/src/core/scene/format.zig b/src/core/scene/format.zig index 640cce8..a1ab2a6 100644 --- a/src/core/scene/format.zig +++ b/src/core/scene/format.zig @@ -270,6 +270,10 @@ pub const CookModel = struct { uuids: [][16]u8, resources: []ResourceEntry, archetypes: []ArchetypeBlock, + /// The authored scene's `version:` field (0 if absent). Propagated to + /// `SceneHeader.content_version` — opaque to the codec, for the game's own + /// scene-versioning/migration. + content_version: u16 = 0, /// Backing arena for every slice above — the cook builds into it, the model /// owns it, `deinit` reclaims it in one shot. diff --git a/src/core/scene/writer.zig b/src/core/scene/writer.zig index 614b254..27406d9 100644 --- a/src/core/scene/writer.zig +++ b/src/core/scene/writer.zig @@ -70,6 +70,7 @@ const Writer = struct { try self.collectSchemas(); var hdr: SceneHeader = .{ + .content_version = self.model.content_version, .entity_count = try u32From(self.totalEntities()), .resource_count = try u32From(self.model.resources.len), .schema_count = try u32From(self.schema_ids.items.len), diff --git a/src/etch/scene_cook.zig b/src/etch/scene_cook.zig index 1fd3e11..9cc04b5 100644 --- a/src/etch/scene_cook.zig +++ b/src/etch/scene_cook.zig @@ -269,16 +269,31 @@ const Builder = struct { const archetypes = try self.groupArchetypes(entities.items); const resources = try self.buildResources(scene_decl, diag_out); + const content_version = try self.sceneContentVersion(scene_decl, diag_out); return .{ .strings = try self.a().dupe([]const u8, self.strings.items), .uuids = try self.a().dupe([16]u8, self.uuids.items), .resources = resources, .archetypes = archetypes, + .content_version = content_version, .arena = self.arena, }; } + /// Const-evaluate the scene's `version:` field to a `u16` (0 if absent). The + /// authored content version is otherwise silently lost (it rides through to + /// `SceneHeader.content_version`). + fn sceneContentVersion(self: *Builder, scene_decl: ast_mod.SceneDecl, diag_out: ?*[]const u8) CookError!u16 { + if (scene_decl.version.isNone()) return 0; + const v = interp.evalConst(self.ast, scene_decl.version) catch return fail(diag_out, error.NonConstValue, "scene version must be a constant int"); + const x: i64 = switch (v) { + .int_ => |n| n, + else => return fail(diag_out, error.NonConstValue, "scene version must be an int"), + }; + return std.math.cast(u16, x) orelse return fail(diag_out, error.NonConstValue, "scene version out of u16 range"); + } + fn buildEntity(self: *Builder, e: ast_mod.SceneEntity, diag_out: ?*[]const u8) CookError!EntityBuild { const name = self.ast.strings.slice(e.name); const name_idx = try self.internString(name); @@ -705,45 +720,6 @@ test "cook builds the neutral model from a scene (E1)" { try std.testing.expectEqualStrings("wave1", title.?); } -test "cook rejects instance of (M1.0.6 boundary)" { - const gpa = std.testing.allocator; - const src = - \\scene "S" { - \\ instance of "Torch" "T1" { } - \\} - ; - var msg: []const u8 = ""; - try std.testing.expectError(error.InstanceOfUnsupported, cook(gpa, src, &msg)); -} - -test "cook rejects an entity without uuid" { - const gpa = std.testing.allocator; - const src = - \\component Position { x: f32 = 0.0 } - \\scene "S" { - \\ entity "E" { Position { } } - \\} - ; - try std.testing.expectError(error.MissingUuid, cook(gpa, src, null)); -} - -test "cook rejects an undeclared resource type" { - const gpa = std.testing.allocator; - const src = - \\scene "S" { - \\ resources { Bogus { x: 1 } } - \\} - ; - try std.testing.expectError(error.UndeclaredType, cook(gpa, src, null)); -} - -test "cook rejects an unsupported component field kind" { - const gpa = std.testing.allocator; - const src = - \\component Spin { axis: Vec3 = [0, 0, 0] } - \\scene "S" { - \\ entity "E" { uuid: "7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e" Spin { } } - \\} - ; - try std.testing.expectError(error.UnsupportedFieldKind, cook(gpa, src, null)); -} +// The cook's negative cases (instance-of / unsupported field kind / undeclared +// type / missing uuid / unknown field / bad uuid) live in +// `tests/scene/cook_errors_test.zig`. From b59261e80e877cc71d5afd51d9afef445dc9cb8e Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Sat, 27 Jun 2026 00:50:48 +0200 Subject: [PATCH 18/21] feat(scene): scene_cook CLI + E3 tests (resources, determinism, errors) --- build.zig | 29 +++++- tests/fixtures/scene/arena_wave1.scene.etch | 2 + tests/scene/cook_errors_test.zig | 79 +++++++++++++++ tests/scene/cook_roundtrip_test.zig | 77 +++++++++++++++ tools/scene_cook/main.zig | 104 ++++++++++++++++++++ 5 files changed, 290 insertions(+), 1 deletion(-) create mode 100644 tests/scene/cook_errors_test.zig create mode 100644 tools/scene_cook/main.zig diff --git a/build.zig b/build.zig index 0d65a8d..6099302 100644 --- a/build.zig +++ b/build.zig @@ -425,8 +425,11 @@ pub fn build(b: *std.Build) void { .{ .path = "tests/etch/v1/query_filters_test.zig", .etch = true }, .{ .path = "tests/etch_interp/corpus_test.zig", .etch_interp = true }, // M1.0.4 / E2 — scene cook → writer → accessor round-trip (entities, - // archetypes, UUIDs, names, parent links). + // archetypes, UUIDs, names, parent links, content_version, resources, + // mixed-alignment columns, byte-identical determinism). .{ .path = "tests/scene/cook_roundtrip_test.zig", .scene = true }, + // M1.0.4 / E3 — scene cook negative cases (typed errors, no panic). + .{ .path = "tests/scene/cook_errors_test.zig", .scene = true }, // M0.3 — common platform layer tests. .{ .path = "tests/platform/fs_vfs_test.zig" }, .{ .path = "tests/platform/time_test.zig" }, @@ -931,6 +934,30 @@ pub fn build(b: *std.Build) void { ); asset_cook_step.dependOn(&asset_cook_run.step); + // M1.0.4 / E3 — `zig build scene-cook -- --output ` + // cooks one `.scene.etch` into the runtime `.scene.bin`, in-process via + // `weld_etch.scene_cook` + `weld_core.scene.writer`. Mirrors the asset_cook + // user-facing CLI step (built on the user target, args forwarded via `--`). + const scene_cook_module = b.createModule(.{ + .root_source_file = b.path("tools/scene_cook/main.zig"), + .target = target, + .optimize = optimize, + }); + scene_cook_module.addImport("weld_etch", etch_module); + scene_cook_module.addImport("weld_core", core_module); + const scene_cook_exe = b.addExecutable(.{ + .name = "scene_cook", + .root_module = scene_cook_module, + }); + b.installArtifact(scene_cook_exe); + const scene_cook_run = b.addRunArtifact(scene_cook_exe); + if (b.args) |args| scene_cook_run.addArgs(args); + const scene_cook_step = b.step( + "scene-cook", + "Cook a .scene.etch into a .scene.bin (scene_cook --output )", + ); + scene_cook_step.dependOn(&scene_cook_run.step); + // -------------------------------------------- Fixture facade (S4 demo) -- // `@embedFile` cannot escape the package root of the module that diff --git a/tests/fixtures/scene/arena_wave1.scene.etch b/tests/fixtures/scene/arena_wave1.scene.etch index 8251797..40c2e60 100644 --- a/tests/fixtures/scene/arena_wave1.scene.etch +++ b/tests/fixtures/scene/arena_wave1.scene.etch @@ -13,6 +13,8 @@ enum Weather { clear, rain, storm } resource GameMode { max_players: int = 4, title: string = "arena", weather: Weather = .clear } scene "ArenaWave1" { + version: 3 + resources { GameMode { max_players: 8, title: "wave1", weather: .storm } } diff --git a/tests/scene/cook_errors_test.zig b/tests/scene/cook_errors_test.zig new file mode 100644 index 0000000..6816910 --- /dev/null +++ b/tests/scene/cook_errors_test.zig @@ -0,0 +1,79 @@ +//! M1.0.4 — scene cook negative cases. Each ill-formed scene yields a typed +//! `CookError` (never a panic) and produces no `.scene.bin`. + +const std = @import("std"); +const weld_etch = @import("weld_etch"); + +const scene_cook = weld_etch.scene_cook; + +fn expectCookError(comptime want: anyerror, src: []const u8) !void { + const gpa = std.testing.allocator; + var msg: []const u8 = ""; + try std.testing.expectError(want, scene_cook.cook(gpa, src, &msg)); + // A clear diagnostic accompanies the error (the brief: "a clear cook + // diagnostic, never a panic"). + try std.testing.expect(msg.len > 0); +} + +test "instance of is rejected (M1.0.6 boundary)" { + try expectCookError(error.InstanceOfUnsupported, + \\scene "S" { + \\ instance of "Torch" "T1" { } + \\} + ); +} + +test "unsupported component field kind is rejected" { + // `Vec3` type-checks in the resolver but the runtime registry rejects it at + // registration (error.InvalidProgram → UnsupportedFieldKind), surfaced as a + // cook diagnostic rather than a panic. + try expectCookError(error.UnsupportedFieldKind, + \\component Spin { axis: Vec3 = [0, 0, 0] } + \\scene "S" { + \\ entity "E" { uuid: "7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e" Spin { } } + \\} + ); +} + +test "undeclared resource type is rejected" { + try expectCookError(error.UndeclaredType, + \\scene "S" { + \\ resources { Bogus { x: 1 } } + \\} + ); +} + +test "entity without uuid is rejected" { + try expectCookError(error.MissingUuid, + \\component Position { x: f32 = 0.0 } + \\scene "S" { + \\ entity "E" { Position { } } + \\} + ); +} + +test "undeclared component type on an entity is rejected" { + try expectCookError(error.UndeclaredType, + \\scene "S" { + \\ entity "E" { uuid: "7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e" Ghost { } } + \\} + ); +} + +test "unknown field on a declared component is rejected" { + try expectCookError(error.UnknownField, + \\component Position { x: f32 = 0.0 } + \\scene "S" { + \\ entity "E" { uuid: "7b3e2f1a-42a3-4f2b-8c9d-a3f2b1c98d4e" Position { z: 1.0 } } + \\} + ); +} + +test "malformed uuid is rejected" { + try expectCookError(error.BadUuid, + \\component Position { x: f32 = 0.0 } + \\scene "S" { + \\ entity "E" { uuid: "not-a-uuid" Position { } } + \\} + ); +} diff --git a/tests/scene/cook_roundtrip_test.zig b/tests/scene/cook_roundtrip_test.zig index 7dbfa52..7e76a1d 100644 --- a/tests/scene/cook_roundtrip_test.zig +++ b/tests/scene/cook_roundtrip_test.zig @@ -76,6 +76,8 @@ test "scene round-trips through cook and accessor" { var acc = try scene.accessor.Accessor.open(bytes); try std.testing.expect(acc.verifyHash()); try std.testing.expectEqual(@as(u32, 2), acc.archetypeCount()); + // Authored `version: 3` propagated to the header. + try std.testing.expectEqual(@as(u16, 3), acc.header.content_version); // Spawner's UUID first byte, for the parent-link assertion. const spawner_uuid0: u8 = 0x7b; @@ -108,4 +110,79 @@ test "scene round-trips through cook and accessor" { } } try std.testing.expect(saw_solo and saw_pair); + + // Resources block: GameMode { max_players: 8, title: "wave1", weather: .storm }. + try std.testing.expectEqual(@as(u32, 1), acc.resourceCount()); + const res = acc.resource(0); + try std.testing.expectEqualStrings("GameMode", acc.schema(res.schema_index).name); + const reg = &cooked.registry; + const gm = reg.idOf("GameMode").?; + + const mp = reg.findField(gm, "max_players").?; + try std.testing.expectEqual(@as(i64, 8), std.mem.readInt(i64, res.data[mp.offset..][0..8], .little)); + + const wf = reg.findField(gm, "weather").?; + try std.testing.expectEqual(@as(u32, 2), std.mem.readInt(u32, res.data[wf.offset..][0..4], .little)); // .storm + + const tf = reg.findField(gm, "title").?; + try std.testing.expectEqualStrings("wave1", res.stringField(tf.offset).?); +} + +test "scene round-trips a mixed-alignment archetype (column padding)" { + // A 1-byte (bool) component beside an 8-byte (f64) one on the same entity + // forces one SoA column to start on inter-column padding — locking the + // writer's `padToFileAlign` against the accessor's `columnOffset` on a real + // misalignment (the main fixture is all-4-byte, so its padding is a no-op). + const gpa = std.testing.allocator; + const src = + \\component Toggle { on: bool = false } + \\component Mass { value: f64 = 1.0 } + \\scene "Mix" { + \\ entity "G" { + \\ uuid: "00000000-0000-0000-0000-000000000001" + \\ Toggle { on: true } + \\ Mass { value: 2.5 } + \\ } + \\} + ; + var cooked = try scene_cook.cook(gpa, src, null); + defer cooked.deinit(gpa); + const bytes = try scene.writer.write(gpa, cooked.model, &cooked.registry); + defer gpa.free(bytes); + + var acc = try scene.accessor.Accessor.open(bytes); + try std.testing.expect(acc.verifyHash()); + try std.testing.expectEqual(@as(u32, 1), acc.archetypeCount()); + const arch = acc.archetype(0); + try std.testing.expectEqual(@as(u32, 2), arch.component_count); + + // Toggle.on (bool, 1 byte) — true. + const tc = columnOf(acc, arch, "Toggle").?; + const on_fd = cooked.registry.findField(cooked.registry.idOf("Toggle").?, "on").?; + try std.testing.expect(arch.componentSlot(tc, 0)[on_fd.offset] != 0); + + // Mass.value (f64, 8 bytes, column started on padding) — 2.5. + const mc = columnOf(acc, arch, "Mass").?; + const val_fd = cooked.registry.findField(cooked.registry.idOf("Mass").?, "value").?; + const raw = std.mem.readInt(u64, arch.componentSlot(mc, 0)[val_fd.offset..][0..8], .little); + try std.testing.expectApproxEqAbs(@as(f64, 2.5), @as(f64, @bitCast(raw)), 1e-12); +} + +test "re-cook is byte-identical" { + const gpa = std.testing.allocator; + const io = std.testing.io; + const src = try readFixture(gpa, io); + defer gpa.free(src); + + var c1 = try scene_cook.cook(gpa, src, null); + defer c1.deinit(gpa); + const b1 = try scene.writer.write(gpa, c1.model, &c1.registry); + defer gpa.free(b1); + + var c2 = try scene_cook.cook(gpa, src, null); + defer c2.deinit(gpa); + const b2 = try scene.writer.write(gpa, c2.model, &c2.registry); + defer gpa.free(b2); + + try std.testing.expectEqualSlices(u8, b1, b2); } diff --git a/tools/scene_cook/main.zig b/tools/scene_cook/main.zig new file mode 100644 index 0000000..19e4541 --- /dev/null +++ b/tools/scene_cook/main.zig @@ -0,0 +1,104 @@ +//! `scene_cook` — thin CLI shim around the M1.0.4 scene cook. Parses args + does +//! file I/O; all real work is `weld_etch.scene_cook.cook` + `weld_core.scene +//! .writer.write` in-process. Mirrors `tools/etch_cook` / `tools/asset_cook`. +//! +//! scene_cook --output + +const std = @import("std"); + +const weld_etch = @import("weld_etch"); +const weld_core = @import("weld_core"); + +const scene_cook = weld_etch.scene_cook; +const writer = weld_core.scene.writer; + +pub fn main(init: std.process.Init) !void { + const gpa = init.gpa; + const io = init.io; + const args = try init.minimal.args.toSlice(init.arena.allocator()); + + var output: ?[]const u8 = null; + var input: ?[]const u8 = null; + var i: usize = 1; + while (i < args.len) : (i += 1) { + const a = args[i]; + if (std.mem.eql(u8, a, "--output")) { + i += 1; + if (i >= args.len) return die(io, "missing path after --output"); + output = args[i]; + } else if (std.mem.startsWith(u8, a, "--")) { + return die(io, "unknown flag"); + } else { + if (input != null) return die(io, "more than one input file"); + input = a; + } + } + const out_path = output orelse return die(io, "missing --output "); + const in_path = input orelse return die(io, "missing "); + + const dir = std.Io.Dir.cwd(); + const source = readWholeFile(gpa, io, dir, in_path) catch |err| { + try printErr(io, "cannot read input: ", @errorName(err)); + return err; + }; + defer gpa.free(source); + + var diag: []const u8 = ""; + var cooked = scene_cook.cook(gpa, source, &diag) catch |err| { + try printErr(io, "cook failed: ", if (diag.len > 0) diag else @errorName(err)); + return err; + }; + defer cooked.deinit(gpa); + + const bytes = try writer.write(gpa, cooked.model, &cooked.registry); + defer gpa.free(bytes); + + try writeOutput(io, dir, out_path, bytes); +} + +fn die(io: std.Io, msg: []const u8) error{InvalidArgs} { + printErr(io, "", msg) catch {}; + return error.InvalidArgs; +} + +fn printErr(io: std.Io, prefix: []const u8, msg: []const u8) !void { + var b: [512]u8 = undefined; + var ew = std.Io.File.stderr().writer(io, &b); + const w = &ew.interface; + try w.print("scene_cook: {s}{s}\n", .{ prefix, msg }); + try w.flush(); +} + +fn readWholeFile(gpa: std.mem.Allocator, io: std.Io, dir: std.Io.Dir, path: []const u8) ![]u8 { + var file = try dir.openFile(io, path, .{}); + defer file.close(io); + const stat = try file.stat(io); + const buf = try gpa.alloc(u8, stat.size); + errdefer gpa.free(buf); + var read_buf: [16 * 1024]u8 = undefined; + var reader = file.reader(io, &read_buf); + var written: usize = 0; + while (written < buf.len) { + const n = try reader.interface.readSliceShort(buf[written..]); + if (n == 0) break; + written += n; + } + return buf[0..written]; +} + +fn writeOutput(io: std.Io, dir: std.Io.Dir, path: []const u8, bytes: []const u8) !void { + if (std.fs.path.dirname(path)) |sub| { + if (sub.len > 0) { + dir.createDirPath(io, sub) catch |err| switch (err) { + error.PathAlreadyExists, error.NotDir => {}, + else => return err, + }; + } + } + var file = try dir.createFile(io, path, .{}); + defer file.close(io); + var write_buf: [16 * 1024]u8 = undefined; + var w = file.writer(io, &write_buf); + try w.interface.writeAll(bytes); + try w.interface.flush(); +} From dd9c39ad281fa51620bd5b6ed2eba7a114bfb3c8 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Sat, 27 Jun 2026 00:50:56 +0200 Subject: [PATCH 19/21] docs(claude-md): update for M1.0.4 --- CLAUDE.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 89801eb..e1fd5b4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,9 +11,9 @@ knowledge base — see § Quick links spec. |---|---| | Phase | 1 (Etch ↔ ECS) | | Current milestone | (none — between milestones) | -| Last released tag | `v0.10.3-resource-nonpod-fields` | +| Last released tag | `v0.10.4-scene-cook` | | Active branch | `main` | -| Next planned milestone | (next M1.0.x — to be scoped) | +| Next planned milestone | M1.0.5 — runtime `.scene.bin` loader (to be scoped) | ## Tags @@ -41,6 +41,7 @@ knowledge base — see § Quick links spec. | `v0.10.1-changed-detection` | 2026-06-22 | M1.0.1 — Change detection + scheduler data-race fix | `entity has T changed` exposed to the interpreter; `Scheduler.shutdown` data-race fixed (`std.atomic.Value(bool)`); permanent test watchdog added. | | `v0.10.2-etch-events-observers` | 2026-06-23 | M1.0.2 — Events + structural observers | `emit`/`@on_event` + five lifecycle-annotation observers (`@on_added`/`@on_removed`/`@on_replaced`/`@on_spawned`/`@on_despawned`), annotation-routed; Tier 0 observer registry completed (`on_replaced`, ctx, old/new). Diagnostics E1208/E1209/E1215. Non-POD (`string`) event-field fix folded in via fix-as-you-go. | | `v0.10.3-resource-nonpod-fields` | 2026-06-24 | M1.0.3 — String and enum resource fields | Resource fields reach spec parity for the two scalar non-POD cases; founds the Phase 1 persistent heap (system allocator + atomic refcount + drop-by-`type_id` + immortal-interned sentinel). Resource-only `FieldKind.string_`/`.enum_`; components stay POD-strict (validator-gated). The Option A alignment (the former deferred "tranche 7"). | +| `v0.10.4-scene-cook` | 2026-06-27 | M1.0.4 — Cooking `.scene.etch` → `.scene.bin` | Offline, World-free cook of direct-entity scenes. Tier-0 `src/core/scene/` codec — `SceneHeader` (64 B) + §10 Schema Registry + `writer` + zero-copy `accessor` (the read half, reused verbatim by the M1.0.5 loader). Etch driver `src/etch/scene_cook.zig` reuses `compileTypeDecl` (refactored `*World`→`*Registry`) + `evalConst`; groups entities by `ComponentSignature` into flat SoA columns. On-disk component identity is the Schema-Registry index → component **name** (no raw `ComponentId`). Resource `string`/enum materialized; `parent` name→UUID; `instance of` + unsupported field kinds rejected with clear diagnostics. `scene_cook` CLI + re-cook byte-identical determinism. | ## Hypotheses validated by spikes @@ -61,6 +62,9 @@ knowledge base — see § Quick links spec. - **`spec/` directory in the repo**: out of scope for Phase −1. Spec lives in the claude.ai knowledge base; re-evaluated during Phase 0 if the absence creates friction. - **Phase 0.6 IPC debts (SCM_RIGHTS fd-passing + editor Windows path)**: resolved in M0.7 (`v0.7.0-M0.7-ipc`). SCM_RIGHTS is the primary POSIX shm attach (create fd over AF_UNIX, runtime `mmap`s the received fd, sidestepping the macOS BSD shm quirk); the editor's Windows `CreateProcessW` + named-pipe path is wired (`src/editor/main.zig` no longer returns `error.Unimplemented`). - **`sendWithHandles` Windows (Phase 3)**: `transport_windows.zig:sendWithHandles` returns `error.Unimplemented`. The `DuplicateHandle`-based equivalent lands with the GPU shared framebuffer when an exportable Vulkan semaphore appears upstream (cf. `engine-ipc.md` §4.7). +- **M1.0.4 scope boundary (scene cook)**: the cook is offline + **World-free** (no entity instantiation). Runtime `.scene.bin` → ECS `World` (mmap + memcpy-into-chunks + UUID→handle remap + `on_spawned`) is **M1.0.5** — it reuses `src/core/scene/accessor.zig` verbatim. Prefab `instance of` flattening + entity→entity cross-refs + the Entity Extensions Table are **M1.0.6** (M1.0.4 rejects `instance of` and writes both reserved sections empty). On-disk component identity is the §10 Schema-Registry index → component name (Phase-1 identity = name; no comptime schema-hash for Etch components), so M1.0.5 remaps via `idOf(name)`. Registration deviation: `interp.compileTypeDecl` was refactored `*World`→`*Registry` (+ `pub`) so the cook reuses it World-free (traced in `briefs/M1.0.4-scene-cook.md` Accepted deviations). +- **Dynamic collections on resource fields (`string[]`, `[K:V]`, `Set`)**: unscheduled-**additive** — they fold into the first consuming milestone, **not** M1.0.4. The `M1.0.4` label in `briefs/M1.0.3-resource-nonpod-fields.md` Context is **superseded** (M1.0.4 is the scene cook, not dynamic collections). The persistent heap's `type_id`→drop dispatch + open `TypeId` set already accommodate them. +- **Additive future format sections (design-at-day-1, not deferral)**: M1.0.4's `.scene.bin` reserves the Extensions + Cross-references sections (written empty) and the writer/accessor dispatch on `FieldKind` + name-based schema identity — so M1.0.6 (cross-refs/extensions) and the first milestone adding a 3D/handle `FieldKind` add columns/entries **without** changing the on-disk format or the loader. ## Non-negotiable rules @@ -194,4 +198,4 @@ The `briefs/` directory is the source of truth for milestone state. The brief's --- -Last updated: 2026-06-24 +Last updated: 2026-06-27 From 5d13fb84ad7d327005428857977262456a890002 Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Sat, 27 Jun 2026 00:51:00 +0200 Subject: [PATCH 20/21] docs(brief): journal + closing notes for M1.0.4 E3 --- briefs/M1.0.4-scene-cook.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index f154528..7eccdff 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -151,6 +151,8 @@ This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`Scen - 2026-06-26 09:10 — E2 done. `format.zig`: `SceneHeader` (64 B, explicit-LE `writeTo`/`read`, comptime offset asserts) + §10 Schema Registry (`SchemaEntry` = name_ref + size + align; name-based identity per the deviation) + shared `columnOffset`/`columnsRegionEnd` (writer + accessor compute identical SoA column placement). `writer.zig`: `CookModel` → `.scene.bin` (header + dedup length-prefixed string table + UUID table + schema registry + resources block + archetype blocks [schema-index mask + name/uuid/parent arrays + file-absolute-aligned SoA columns] + empty extensions/crossrefs; `XxHash64(0)` over content-after-header; deterministic — schemas sorted ascending by id). `accessor.zig`: zero-copy view (`open` validates magic+version; self-describing via the schema table, no registry needed to slice columns). Wired writer+accessor into `scene/root.zig` + core §13 pins; `tests/scene/` target wired in `build.zig` via a dedicated `scene` flag. `tests/scene/cook_roundtrip_test.zig` reads the committed fixture, cooks→writes→accessor, asserts entities/archetypes/uuids/names/parent; accessor + header + columnOffset inline tests green. `zig build test` EXIT 0, fmt+lint clean. E2-gate note: the writer already serializes the resources block (one cohesive codec) — E3 adds the resource round-trip assertions + determinism + CLI + `cook_errors_test.zig` + CLAUDE.md. +- 2026-06-27 10:30 — E3 done. (1) Resource round-trip: extended `cook_roundtrip_test` to assert `GameMode` int/string/enum (max_players=8, title="wave1", weather=.storm). (2) `content_version`: capture the scene's `version:` (`evalConst`→u16) into `CookModel` + propagate to `SceneHeader.content_version`; fixture now `version: 3`, asserted in the round-trip (Guy E3 add #2). (3) Mixed-alignment case: a `bool`+`f64` archetype forces a real inter-column pad — decodes correctly, locking `padToFileAlign`/`columnOffset` (Guy E3 add #1). (4) Determinism: `re-cook is byte-identical` test + CLI `cmp` both green. (5) CLI `tools/scene_cook/main.zig` + `zig build scene-cook` step (mirrors asset_cook); observable verified — `WSCN` magic, content_version=3, 2 entities/1 resource/3 schemas, 344 B, byte-identical re-cook. (6) `tests/scene/cook_errors_test.zig` (instance-of / unsupported-field-kind / undeclared-type / missing-uuid / unknown-field / bad-uuid) — moved the rejections out of `scene_cook.zig` inline. (7) `CLAUDE.md` §3.4 folded (État courant + Tags `v0.10.4-scene-cook` + open-decisions incl. dynamic-collections-home correction + additive-future-sections + Last updated 2026-06-27). `zig build test` EXIT 0, fmt+lint clean. Status left ACTIVE — CLOSED is Guy's call after review (per standing feedback). + ## Accepted deviations - 2026-06-26 (verbal decision by Guy, in-session, Cas 3) — **Entity `uuid` is mandatory at cook time.** Reverses E1 decision #4 (which mapped an absent `uuid:` to a deterministic all-zero UUID). `scene_cook.parseEntityUuid` now returns `error.MissingUuid` (message "entity requires an explicit uuid") when the entity declares no `uuid:`. Rationale: explicit, stable identity is required for cross-load references; auto-generating UUIDs is the editor's job, not the cook's (the cook stays deterministic without inventing identity). Realizes the brief's "entities with UUID" expectation (Fix #4 — recorded in the brief, skipped in E1). Adds the inline test "cook rejects an entity without uuid". @@ -164,7 +166,22 @@ This milestone **assembles** existing bricks — `descriptor.zig` (`Scene`/`Scen ## Closing notes - **What worked:** + - Maximal reuse, as the brief intended: the cook reuses `interp.compileTypeDecl` (refactored to `*Registry`) + `evalConst` + `ecs_bridge.writeValueAsBytes`; the only genuinely new surface is the `.scene.bin` codec. Up-front parallel brick-mapping (signatures + file:line for every reused symbol) meant near-zero signature surprises during implementation. + - The `format.columnOffset`/`columnsRegionEnd` shared helper made writer and accessor agree on SoA column placement *by construction*. Guy's E3 mixed `bool`+`f64` archetype case exercised real inter-column padding (not a no-op) and passed — locking that agreement. + - Determinism fell out of deterministic ordering (schemas sorted by ascending id, tables in model order, `XxHash64(0)` over content-after-header) — re-cook byte-identity confirmed by both the test and the CLI `cmp`. - **What deviated from the original spec:** + - `interp.compileTypeDecl` refactored `*World`→`*Registry` + made `pub` (in `interp.zig`, outside the brief file list) — approved deviation, World-free registration reuse. + - On-disk component identity = §10 Schema-Registry index → component **name** (correction #2), not raw `ComponentId`; `SceneHeader` gained `schema_count` + `schema_table_offset`. + - Entity `uuid` is mandatory (`error.MissingUuid`, correction #1) — reverses E1's zero-fill. + - `format.zig` (nominally E2) was created in E1 (the cook model is typed against it); the writer serializes the full file incl. the resources block in E2 (one cohesive codec), with the resource round-trip *assertions* added in E3. + - Hash is `std.hash.XxHash64` per the brief Note (overrides the spec's "xxHash3"). - **What to flag explicitly in review:** + - The fixture is a **multi-construct `.etch`** (component/resource/enum decls + one `scene`), not a strict §21 single-`scene` file: M1.0.4 cooks one self-contained program; cross-file `import` resolution at cook time is future work. The cook parses in general mode and rejects only `instance of`. + - Name-based on-disk identity ⇒ the M1.0.5 loader remaps via `idOf(name)`; the M1.0.4 round-trip is in-process (the cook returns its own `Registry`). + - Reserved-empty Extensions/Cross-refs sections + `FieldKind`-dispatch keep M1.0.6 (cross-refs/extensions) and any new 3D/handle `FieldKind` additive (no on-disk/loader change). - **Final measurements:** + - No perf gate (offline cook). Determinism: re-cook byte-identical (round-trip test + CLI `cmp`). Cooked `arena_wave1.scene.bin` = 344 bytes, `WSCN` magic @0, 64-byte header. `zig build test` EXIT 0 (debug + ReleaseSafe via pre-push), `zig fmt --check` + `zig build lint` clean. - **Residual risk / technical debt left deliberately:** + - The accessor caps components-per-archetype at 256 (stack scratch for `columnOffset`); a malformed file beyond that trips an assert rather than a graceful error — fine for cook-produced files; the M1.0.5 loader may add input validation. + - Resource field-level schema (§10 "champs") is not serialized — Phase-1 identity is the name; resource-layout migration/validation is future. + - No cross-file `import` resolution at cook time (single self-contained program per cook). From 74045de3915c2c3a5bc8a3f1e4c41c829e39445e Mon Sep 17 00:00:00 2001 From: Guy Senpai Date: Sat, 27 Jun 2026 01:07:22 +0200 Subject: [PATCH 21/21] docs(brief): close M1.0.4 --- briefs/M1.0.4-scene-cook.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/briefs/M1.0.4-scene-cook.md b/briefs/M1.0.4-scene-cook.md index 7eccdff..76861bb 100644 --- a/briefs/M1.0.4-scene-cook.md +++ b/briefs/M1.0.4-scene-cook.md @@ -1,12 +1,12 @@ # M1.0.4 — Cooking `.scene.etch` → `.scene.bin` -> **Status:** ACTIVE +> **Status:** CLOSED > **Phase:** 1 > **Branch:** `phase-1/scene/scene-cook` > **Tag:** `v0.10.4-scene-cook` > **Dependencies:** M0.1 (Tier-0 ECS: `Archetype`, `ComponentSignature`, `Registry`, chunk layout), M0.2 (RTTI), M0.6 (asset registry + `.bin` runtime header convention), M0.8 (grammar v0.6 + scene/prefab descriptors, `compileTypeDecl`), M0.9 (cross-file `validateProject`), M1.0.3 (resource `string`/enum fields + persistent heap, `writeValueAsBytes`/`readBytesAsValue`) > **Open date:** 2026-06-26 -> **Close date:** — +> **Close date:** 2026-06-27 ---