WIP: initialize v1.11 Gemma 4 E2B milestone#49
Conversation
gabewillen
left a comment
There was a problem hiding this comment.
Summary
Planning-only PR that rescopes active .planning/ docs from the Liquid v1.9 slice to a narrow Gemma 4 E2B text-generation milestone (v1.11, Phases 38–42). Scope honesty (text-only, no mmproj/tools, explicit gemma4 path, reference-pin risk) is generally strong and metadata claims look coherent. The main merge risks are internal planning contradictions (especially REF-01 scheduled after the runtime phase while pitfalls say never do that), under-specified model-family placement (model/data.* only, no model/gemma4 contract surface), and phase/requirement traceability gaps that would steer implementation the wrong way if followed literally.
Issue counts by severity
- bugs: 2
- suggestions: 6
- nits: 2
| **Goal**: The reference lane is Gemma 4-capable, the exact maintained text slice is proven | ||
| correct against the reference, and the existing maintained anchors stay green. | ||
| **Depends on**: Phase 40 | ||
| **Requirements**: REF-01, PAR-03, VER-03 |
There was a problem hiding this comment.
[bug] REF-01 (Gemma 4-capable pinned llama.cpp reference lane) is required only in Phase 41, which depends on Phase 40 runtime bring-up. The same PR’s pitfalls research marks “Ignore the pinned reference commit until parity phase” as never acceptable (.planning/research/PITFALLS.md:107) and Architecture Pattern 4 / Anti-Pattern 3 warn that discovering a non-Gemma4 pin after runtime work is a late dead-end. Following the roadmap as written recreates the documented failure mode: full EMEL runtime work can complete before anyone learns parity/bench cannot load the fixture.
Suggestion: Promote reference-pin audit/upgrade to Phase 38 or a dedicated early phase (or make Phase 40 depend on a completed REF-01 gate). Keep PAR-03/VER-03 in the later proof phase, but do not defer the pin readiness check until after runtime.
| | RUN-07 | Phase 39 | Pending | | ||
| | RUN-08 | Phase 40 | Pending | | ||
| | RUN-09 | Phase 39 | Pending | | ||
| | REF-01 | Phase 41 | Pending | |
There was a problem hiding this comment.
[bug] Traceability maps REF-01 -> Phase 41 only, so requirement tracking encodes the same late-ref ordering bug as the roadmap. FEATURES dependency graph correctly shows parity requires reference-lane readiness, but does not force that readiness before runtime bring-up; REQUIREMENTS/ROADMAP therefore diverge from the pitfall mitigation the research itself recommends.
Suggestion: Remap REF-01 to an early phase (38 or 39), or split into REF-01a (pin audit/upgrade before runtime) and REF-01b (parity/bench consumers use the upgraded pin). Update ROADMAP success criteria and STATE “roadmap coverage” bullets to match.
| Liquid should be added here additively, not via a new harness. | ||
| | `tests/models/README.md` + tool constants | Pin one official Gemma 4 truth anchor | Stable path, checksum, download URL, and slug for one maintained file | | ||
| | `tools/generation_formatter_contract.hpp` | Resolve one canonical Gemma 4 text contract | Match one supported template subset with no media and no tools | | ||
| | `src/emel/model/data.*` | Accept and validate `gemma4` metadata/tensor contracts | Explicit architecture gate plus Gemma 4-specific topology handling | |
There was a problem hiding this comment.
[suggestion] Architecture guidance places the Gemma 4 model contract primarily in src/emel/model/data.* and execution in src/emel/generator/..., with no dedicated src/emel/model/gemma4/** family surface. Repo engineering rules keep model-family contracts under model/<family> (not bulk-only data.*, not a top-level gemma4 domain). STATE already claims maintained lfm2 support exists, so a family-local module pattern should be the research default; otherwise implementers are steered toward growing data.* / generator blobs and risk domain-boundary or “runtime choice in detail” violations later.
Suggestion: Explicitly prescribe src/emel/model/gemma4/** (or equivalent family-local contract modules) for topology/metadata validation, with model/data.* only as the architecture gate/dispatch into that contract. Mirror the same wording in STACK.md and SUMMARY.md “Major components.”
| | `tools/paritychecker/parity_runner.cpp` | Add one official Liquid maintained fixture path/slug and `lfm2` architecture validation | This keeps parity truthful on one explicit asset instead of broadening to generic Liquid support. | | ||
| | `tools/bench/generation_bench.cpp` and [`tools/bench/bench_cases.hpp`](/Users/gabrielwillen/.superset/worktrees/emel.cpp/feat/liquid-ai/tools/bench/bench_cases.hpp) | Add one Liquid benchmark case family using the same maintained fixture | This preserves the existing compare/docs workflow with one new case set instead of a new toolchain. | | ||
| | [`tests/models/README.md`](/Users/gabrielwillen/.superset/worktrees/emel.cpp/feat/liquid-ai/tests/models/README.md) | Record official model provenance for the maintained Liquid fixture | The milestone should have the same provenance discipline as Qwen and Llama. | | ||
| | `src/emel/model/data.*` | Add `gemma4` to supported execution architectures and define a Gemma 4 text contract | Current code explicitly supports maintained families such as `llama`, `qwen3`, and `lfm2`; the new fixture reports `architecture=gemma4`. | |
There was a problem hiding this comment.
[suggestion] Same placement gap as Architecture: “Required Additions In EMEL” lists src/emel/model/data.* for the Gemma 4 text contract and src/emel/generator/... for runtime path, but never names a family-owned model module. That is incomplete scaffolding relative to AGENTS domain rules and will matter as soon as Phase 39 implementation starts from this research.
Suggestion: Add a row for src/emel/model/gemma4/** (metadata, tensor/topology contract, text-only validation helpers) and describe data.* as the architecture allow-list / binding site only.
| produces bounded text generation on ARM. | ||
| 2. Runtime execution for the maintained Gemma 4 slice uses the explicit `gemma4` path and the | ||
| maintained text-only acceptance surface rather than broad multimodal or family claims. | ||
| 3. Maintained runtime evidence publishes a truthful quantized-path contract for the official |
There was a problem hiding this comment.
[suggestion] Phase 40 success criterion 3 requires publishing “a truthful quantized-path contract for the official Q8_0 fixture only,” but Phase 40 requirements list only RUN-08 (initialize/generate). There is no RUN/META requirement that owns quantized-path evidence, so closeout can claim criterion 3 without a mapped requirement, or implementers can skip the contract proof while still checking RUN-08.
Suggestion: Either add a Phase 40 requirement (e.g. RUN-10) for the Q8_0 quantized-path/publication contract, or drop success criterion 3 and fold any needed quant-boundary language into RUN-08/RUN-09 text.
| **Phase Numbering:** | ||
| - Integer phases continue across milestones. | ||
| - v1.9 starts at Phase 33 because v1.7 ended at Phase 32. | ||
| - `v1.11` starts at Phase 38 because the current visible roadmap ceiling in `.planning/phases/` is |
There was a problem hiding this comment.
[suggestion] Milestone phase numbering restarts at Phase 38 because “the current visible roadmap ceiling in .planning/phases/ is Phase 37,” while PROJECT/STATE simultaneously treat adjacent v1.8 size-benchmark work as real concurrent context. In this repo’s planning history, Phase 38+ is already used by adjacent milestone closeout bookkeeping. Reusing 38–42 while those streams remain “adjacent” creates phase-ID collisions when branches merge or when .planning/phases/ directories are reconciled.
Suggestion: Pick the next free global phase ID after all adjacent in-flight milestones (including closeout phases), or namespace phase dirs by milestone (v1.11-phases/38-... style) and document the collision policy in ROADMAP phase-numbering notes.
| | FIX-03 | Phase 38 | Pending | | ||
| | META-02 | Phase 38 | Pending | | ||
| | COND-04 | Phase 38 | Pending | | ||
| | COND-05 | Phase 39 | Pending | |
There was a problem hiding this comment.
[suggestion] COND-05 (explicit reject of mmproj/image/audio/video/tool-call shapes) is deferred to Phase 39, while Phase 38 only locks the positive text-only contract (COND-04). PROJECT constraints say the maintained path must reject unsupported media shapes explicitly, and Pitfall 2 maps false multimodal claims to Phases 38 and 39. As written, Phase 38 can complete without any rejection success criterion, leaving a window where docs claim text-only honesty without a reject surface.
Suggestion: Move at least a minimal reject-contract requirement into Phase 38 (document/test that unsupported media/tool request shapes are out of contract), or add a Phase 38 success criterion that the maintained formatter/conditioning contract defines explicit rejection outcomes even if model-gate wiring lands in Phase 39.
| execute the maintained Liquid slice truthfully. | ||
| 4. `tools/paritychecker` and `tools/bench`: prove and publish only the same fixture and contract | ||
| after runtime support is real. | ||
| canonical Gemma 4 text contract; `src/emel/model/data.*` and the generator runtime own explicit |
There was a problem hiding this comment.
[suggestion] Summary still states that src/emel/model/data.* and “the generator runtime” own explicit gemma4 acceptance and topology handling, and the phase write-up puts reference readiness only in Phase 41 after Phases 39–40 model/runtime work. That repeats the Architecture placement gap and the late-REF ordering bug in the document most people will skim before planning phases.
Suggestion: Rewrite the architecture/phase paragraphs to (1) name model/gemma4 family ownership, (2) schedule ref-pin readiness before runtime/parity, and (3) restore a short “gaps / confidence” section so known unknowns (exact rope/shared-KV runtime mapping, pin upgrade strategy) are not dropped relative to the Liquid research summary.
|
|
||
| **Why it happens:** | ||
| Benchmark rows are visible and easy to demo, while parity/runtime bring-up is slower. | ||
| Benchmark rows are visible and easy to demo, while reference/runtimе parity is slower. |
There was a problem hiding this comment.
[nit] Typo uses a Cyrillic “е” in runtimе (reference/runtimе parity), which can break search and look like mojibake in review tooling.
Suggestion: Replace with ASCII runtime.
| | `mmproj-gemma-4-e2b-it-f16.gguf` in `v1.11` | It implies image/media execution that the repo cannot currently prove | Keep `v1.11` text only | | ||
| | Broad "Gemma 4 support" language | It sounds stronger but hides the multimodal boundary | Name one exact maintained text fixture | | ||
| | Tool use or media placeholders on the maintained path | The official template allows them, but EMEL does not yet have maintained support | Reject those request shapes explicitly | | ||
| | Assuming the pinned `llama.cpp` reference lane already supports Gemma 4 | The current pinned commit appears not to contain `gemma4` | Make ref-pin readiness explicit in the milestone | |
There was a problem hiding this comment.
[nit] Soft claim language “appears not to contain gemma4” is repeated across PROJECT/STATE/research while the pin SHA is known (ecbcb7ea9d3303097519723b264a8b5f1e977028) and a pin-specific raw source URL is already listed. For a milestone that treats ref readiness as first-class, the research should state a hard verified result (present/absent symbols) rather than hedging.
Suggestion: Record an explicit verification result for that commit (e.g. no LLM_ARCH_GEMMA4 / gemma4 architecture path in pin tree) and only then keep the upgrade requirement.
Summary
v1.11 Gemma 4 E2B Text Generation SliceIncluded
.planning/PROJECT.md.planning/REQUIREMENTS.md.planning/ROADMAP.md.planning/STATE.md.planning/research/*Milestone Shape
gemma-4-e2b-it-Q8_0.ggufmmproj, image/audio/video inputs, and tool-use surfacesllama.cppreference lane before Gemma 4 parity/bench claimsNext
$gsd-discuss-phase 38Notes
scripts/quality_gates.shwas started during planning validation but not carried through to a completed result in this PR context