diff --git a/.gitignore b/.gitignore index a4e323dc..a6a10357 100644 --- a/.gitignore +++ b/.gitignore @@ -50,3 +50,4 @@ website/build/ website/yarn.lock website/node_modules website/i18n/* +*.tgz diff --git a/.nvmrc b/.nvmrc new file mode 100644 index 00000000..2bd5a0a9 --- /dev/null +++ b/.nvmrc @@ -0,0 +1 @@ +22 diff --git a/CLAY-BUILD.md b/CLAY-BUILD.md new file mode 100644 index 00000000..234bdb94 --- /dev/null +++ b/CLAY-BUILD.md @@ -0,0 +1,789 @@ +# clay build — New Asset Pipeline + +> This document covers the **`clay build`** command introduced in claycli 5.1. It explains what changed from the legacy `clay compile` command, why, how they compare, and how to run both side-by-side. + +--- + +## Table of Contents + +1. [Why We Changed It](#1-why-we-changed-it) +2. [Commands At a Glance](#2-commands-at-a-glance) +3. [Architecture: Old vs New](#3-architecture-old-vs-new) +4. [Pipeline Comparison Diagram](#4-pipeline-comparison-diagram) +5. [Feature-by-Feature Comparison](#5-feature-by-feature-comparison) +6. [Configuration](#6-configuration) +7. [Running Both Side-by-Side](#7-running-both-side-by-side) +8. [Code References](#8-code-references) +9. [Performance](#9-performance) +10. [Learning Curve](#10-learning-curve) +11. [For Product Managers](#11-for-product-managers) +12. [Tests](#12-tests) +13. [Migration Guide](#13-migration-guide) + +--- + +## 1. Why We Changed It + +The legacy `clay compile` pipeline was built on **Browserify + Gulp**, tools designed for the 2014–2018 JavaScript ecosystem. Over time these became pain points: + +| Problem | Impact | +|---|---| +| Browserify megabundle (all components in one file per alpha-bucket) | Any change = full rebuild of all component JS, slow watch mode | +| Gulp orchestration with 20+ plugins | Complex dependency chain, hard to debug, slow npm install | +| Sequential compilation steps | CSS, JS, templates all ran in series — total time = sum of all steps | +| No shared chunk extraction | If two components shared a dependency, each dragged it in separately via the Browserify registry | +| No tree shaking | Browserify bundled entire CJS modules regardless of how much was used; no support for ESM dependency tree shaking | +| No source maps | Build errors in production pointed to minified line numbers, not source | +| No content-hashed filenames | Static filenames (`article.client.js`) forced full cache invalidation on every deploy | +| Babelify transpilation overhead | Slow even on small changes | +| `_registry.json` + `_ids.json` numeric module graph | Opaque, hard to inspect or extend | +| `_prelude.js` / `_postlude.js` custom runtime | Browserify's own module system loaded on every page, adding baseline overhead | +| `browserify-cache.json` stale cache risk | Corrupted/out-of-sync cache produced builds where old module code was silently served | +| 20+ npm dependencies just for bundling | Large attack surface, slow installs, difficult version management | + +The new `clay build` pipeline replaces Browserify/Gulp with **esbuild + PostCSS 8**: + +- esbuild bundles JS/Vue in **milliseconds** (not seconds) with native code-splitting and tree shaking for ESM dependencies +- PostCSS 8's programmatic API replaces Gulp's stream-based CSS pipeline +- All build steps (JS, CSS, fonts, templates, vendor, media) run **in parallel** +- A human-readable `_manifest.json` replaces the numeric `_registry.json`/`_ids.json` pair +- Watch mode starts instantly — no initial build, only rebuilds what changed +- **Source maps** generated automatically — errors point to exact source file, line, and column +- **Content-hashed filenames** (`article/client-A1B2C3.js`) — browsers and CDNs cache files forever; only changed files get new URLs on deploy +- **Native ESM** output — no custom `window.require()` runtime, browsers handle imports natively +- **Build-time `process.env.NODE_ENV`** — dead branches like `if (process.env.NODE_ENV !== 'production')` are eliminated at compile time, not runtime +- Dependency footprint reduced from 20+ bundler packages to a handful + +--- + +## 2. Commands At a Glance + +Both commands co-exist. You choose which pipeline to use. + +### Legacy pipeline (Browserify + Gulp) + +```bash +# One-shot compile +clay compile + +# Watch mode +clay compile --watch +``` + +### New pipeline (esbuild + PostCSS 8) + +```bash +# One-shot build +clay build + +# Aliases (backward-compatible) +clay b +clay pn # ← kept so existing Makefiles don't break +clay pack-next # ← kept for the same reason + +# Watch mode +clay build --watch + +# Minified production build +clay build --minify +``` + +Both commands read **`claycli.config.js`** in the root of your Clay instance, but they look at **different config keys** so they never conflict (see [Configuration](#6-configuration)). + +--- + +## 3. Architecture: Old vs New + +### Old: `clay compile` (Browserify + Gulp) + +``` +clay compile +│ +├── scripts.js ← Browserify megabundler +│ ├── Each component client.js → {name}.client.js (individual file) +│ ├── Each component model.js → {name}.model.js + _models-{a-d}.js (bucket in minified mode) +│ ├── Each component kiln.js → {name}.kiln.js + _kiln-{a-d}.js (bucket in minified mode) +│ ├── Shared deps → {number}.js + _deps-{a-d}.js (bucket in minified mode) +│ ├── _prelude.js / _postlude.js ← Browserify custom module runtime (window.require, window.modules) +│ ├── _registry.json ← numeric module ID graph (e.g. { "12": ["4","7"] }) +│ ├── _ids.json ← module ID to filename map +│ └── _client-init.js ← runtime that calls window.require() on each .client module +│ +├── styles.js ← Gulp + PostCSS 7 +│ └── styleguides/**/*.css → public/css/{component}.{styleguide}.css +│ +├── templates.js← Gulp + Handlebars precompile +│ └── components/**/template.hbs → public/js/*.template.js +│ +├── fonts.js ← Gulp copy + CSS concat +│ └── styleguides/*/fonts/* → public/fonts/ + public/css/_linked-fonts.*.css +│ +└── media.js ← Gulp copy + └── components/**/media/* → public/media/ +``` + +**Key runtime behaviour:** `getDependencies()` in view mode walks `_registry.json` for only the components amphora placed on the page — it is page-specific. `_client-init.js` then calls `window.require(key)` for every `.client` key in `window.modules`, which is populated only by the scripts that were served. The subtle issue is that it mounts every loaded `.client` module regardless of whether that component's DOM element is actually present on the page. + +--- + +### New: `clay build` (esbuild + PostCSS 8) + +``` +clay build +│ +├── build.js ← esbuild (JS + Vue SFCs, code-split) +│ ├── Entry points: every components/**/client.js, model.js, kiln.js +│ ├── Code-split chunks: shared dependencies extracted automatically +│ ├── _manifest.json ← human-readable entry→file+chunks map +│ └── .clay/_view-init.js ← generated bootstrap (mounts components, sticky events) +│ +├── styles.js ← PostCSS 8 programmatic API (parallel, p-limit 20) +│ └── styleguides/**/*.css → public/css/{component}.{styleguide}.css +│ +├── templates.js← Handlebars precompile (sequential, progress-tracked) +│ └── components/**/template.hbs → public/js/*.template.js +│ +├── fonts.js ← fs-extra copy + CSS concat +│ └── styleguides/*/fonts/* → public/fonts/ + public/css/_linked-fonts.*.css +│ +├── vendor.js ← fs-extra copy +│ └── clay-kiln/dist/*.js → public/js/ +│ +└── media.js ← fs-extra copy + └── components/**/media/* → public/media/ +``` + +**Key runtime behaviour:** `_view-init.js` loads a component's `client.js` **only when that component's element exists in the DOM**. A built-in sticky-event shim ensures `auth:init` and similar events are received even by late subscribers. + +--- + +## 4. Pipeline Comparison Diagram + +Both pipelines share the same source files and produce the same `public/` output. The difference is in *how* the steps are wired together. + +```mermaid +flowchart LR + SRC(["📁 Source Files"]):::src + + subgraph LEGACY[ ] + direction TB + LH["🕐 clay compile · Browserify + Gulp · ~90s"]:::hdr + L1["📦 JS Bundle\nBrowserify + Babel\n30–60 s"]:::slow + L2["🎨 CSS\nGulp + PostCSS 7\n15–30 s"]:::slow + L3["📄 Templates\nGulp + Handlebars\n10–20 s"]:::med + L4["🔤 Fonts + 🖼 Media\nGulp copy · 2–5 s"]:::fast + LH ~~~ L1 -->|"waits"| L2 -->|"waits"| L3 -->|"waits"| L4 + end + + subgraph MODERN[ ] + direction TB + NH["⚡ clay build · esbuild + PostCSS 8 · ~33s"]:::hdr + N0["🖼 Media\nfs-extra · ~0.7 s"]:::fast + N1["📦 JS + Vue\nesbuild · ~3 s"]:::vfast + N2["🎨 CSS\nPostCSS 8 · ~32 s"]:::slow + N3["📄 Templates\nHandlebars · ~16 s"]:::med + N4["🔤 Fonts + 📚 Vendor\nfs-extra · ~1 s"]:::fast + NH ~~~ N0 -->|"all at once"| N1 & N2 & N3 & N4 + end + + OUT(["📂 public/"]):::out + + SRC --> LEGACY --> OUT + SRC --> MODERN --> OUT + + classDef src fill:#1e293b,color:#94a3b8,stroke:#334155 + classDef out fill:#1e293b,color:#94a3b8,stroke:#334155 + classDef hdr fill:#1e3a5f,color:#93c5fd,stroke:#1d4ed8,font-weight:bold + classDef slow fill:#7f1d1d,color:#fca5a5,stroke:#991b1b + classDef med fill:#78350f,color:#fcd34d,stroke:#92400e + classDef fast fill:#14532d,color:#86efac,stroke:#166534 + classDef vfast fill:#052e16,color:#4ade80,stroke:#166534 +``` + +**Color guide:** 🔴 slow (>15s) · 🟡 medium (10–20s) · 🟢 fast (<5s) · 🌿 very fast (<3s) + +| | `clay compile` | `clay build` | Δ | +|---|---|---|---| +| **Total time** | ~60–120s | ~33s | **~2–3× faster** | +| **Execution** | Sequential — each step waits for the one before it | Parallel — all steps run simultaneously after media | ⚠️ Different shape; same end result | +| **JS tool** | Browserify + Babel (megabundles) | esbuild (code-split per component) | 🔄 Replaced; esbuild is ~10–20× faster than Browserify | +| **CSS tool** | Gulp + PostCSS 7 | PostCSS 8 programmatic API | 🔄 Replaced; same PostCSS plugin ecosystem, newer API | +| **Module graph** | `_registry.json` + `_ids.json` | `_manifest.json` (human-readable) | ⚠️ Different format; same purpose (maps components → files) | +| **Component loader** | `_client-init.js` — mounts every loaded `.client` module, even if its DOM element is absent | `.clay/_view-init.js` — mounts only components whose DOM element is present | ✅ Better; avoids executing component code when the component isn't on the page | +| **JS output** | Per-component files + individual dep files, page-scoped via registry walk | Per-component files + `chunks/` (shared deps extracted once) | ✅ Better; shared deps are downloaded once even when multiple components use them | +--- + +## 5. Feature-by-Feature Comparison + +### JavaScript Bundling + +| Aspect | `clay compile` (Browserify) | `clay build` (esbuild) | +|---|---|---| +| **Bundler** | Browserify 17 + babelify | esbuild | +| **Transpilation** | Babel (preset-env) | esbuild native (ES2017 target) | +| **Vue SFCs** | `@nymag/vueify` Browserify transform | Custom esbuild plugin (`plugins/vue2.js`) using same underlying `vue-template-compiler` | +| **Bundle strategy** | Per-component files + alpha-bucket dep bundles (`_deps-a-d.js`) | Per-component files + auto-extracted shared `chunks/` | +| **Output filenames** | Static: `article.client.js` | Content-hashed: `components/article/client-A1B2C3.js` | +| **Module runtime** | `_prelude.js` + `_postlude.js` (custom `window.require`) | Native ESM — no runtime overhead | +| **Module graph** | `_registry.json` (numeric IDs) + `_ids.json` | `_manifest.json` (human-readable keys) | +| **Component loader** | `_client-init.js` mounts every `.client` module in `window.modules` (page-scoped, but not DOM-presence-checked) | `_view-init.js` mounts a component only when its DOM element exists | +| **Tree shaking** | None — CJS modules bundled whole; no ESM analysis | For ESM dependencies (packages that ship an ESM build): unused exports eliminated. CJS dependencies (e.g. classic `lodash`) are still bundled whole in both pipelines. | +| **Source maps** | Not generated | Yes — `*.js.map` alongside every output file | +| **Dead code elimination** | `process.env.NODE_ENV` set at runtime; dead branches survive minification | Set at build time via `define` — `if (dev) { ... }` blocks removed in production builds | +| **Full rebuild time** | ~30–60s | ~3–4s | +| **Watch rebuild** | Full rebuild on any change | Incremental: only changed module + its dependents | + +> **Same result:** In both cases, the browser receives compiled, browser-compatible JavaScript. Component `client.js` logic runs when the component is on the page. + +> **Key difference:** With Browserify, top-level side-effects in a `client.js` (e.g. `new Vue(...)`) run at page load for every component whose scripts were served, regardless of whether that component's DOM element is present. With esbuild + `_view-init.js`, component code runs only when the element is found in the DOM. + +--- + +### CSS Compilation + +| Aspect | `clay compile` (Gulp + PostCSS 7) | `clay build` (PostCSS 8) | +|---|---|---| +| **API** | Gulp stream pipeline | PostCSS programmatic API | +| **Concurrency** | Sequential per-file | Parallel with `p-limit(20)` | +| **PostCSS plugins** | autoprefixer, postcss-import, postcss-mixins, postcss-simple-vars, postcss-nested | Same plugins | +| **Minification** | cssnano (when `CLAYCLI_COMPILE_MINIFIED` set) | cssnano (same flag) | +| **Error handling** | Stream error halts the entire pipeline | Per-file error logged; remaining files continue compiling | +| **Output format** | `public/css/{component}.{styleguide}.css` | **Identical** | +| **Watch: CSS variation rebuild** | Recompiles changed file only | Recompiles all variations of the same component name (e.g. `article.css` change rebuilds `article_amp.css` too) | + +> **Same result:** Output CSS files are byte-for-byte identical between pipelines (same PostCSS plugins, same naming convention). + +> **Key difference:** In watch mode, `clay compile` ran the full CSS glob on every change and used `gulp-changed` (ctime comparison) to skip files whose output was already up-to-date — it had no awareness of component variants. `clay build` explicitly derives the component prefix from the changed filename (e.g. `text-list_amp.css` → prefix `text-list`) and rebuilds every matching variant (`text-list.css`, `text-list_amp.css`, etc.) across all styleguides in one pass. + +--- + +### Template Compilation + +| Aspect | `clay compile` (Gulp + clayhandlebars) | `clay build` (Node + clayhandlebars) | +|---|---|---| +| **API** | Gulp stream | Direct `fs.readFile` / `hbs.precompile` | +| **Output** | `public/js/{name}.template.js` | **Identical** | +| **Minified output** | `_templates-{a-d}.js` (bucketed) | **Identical** | +| **Error handling** | Stream error calls `process.exit(1)` — crashes the entire build on a single bad template | Per-template error logged; remaining templates continue compiling | +| **Missing `{{{ read }}}` file** | `process.exit(1)` — build crashes immediately | Error logged; template compiles with token unreplaced so the missing asset is visible in browser | +| **Progress tracking** | None | `onProgress(done, total)` callback → live % display | + +> **Same result:** The `window.kiln.componentTemplates['name'] = ...` assignment format is identical. + +--- + +### Fonts + +| Aspect | `clay compile` | `clay build` | +|---|---|---| +| **Binary fonts** | Gulp copy to `public/fonts/{sg}/` | fs-extra copy, same dest | +| **Font CSS** | Concatenated to `_linked-fonts.{sg}.css` | **Identical** | +| **Asset host substitution** | `$asset-host` / `$asset-path` variables | **Identical** | + +> **Same result:** Font CSS and binary output is identical. + +--- + +### Module / Script Resolution + +| Aspect | `clay compile` | `clay build` | +|---|---|---| +| **How scripts are resolved** | `getDependencies(scripts, assetPath)` reads `_registry.json`, walks numeric dep graph | `getDependenciesNextForComponents(names, assetPath, globalKeys)` reads `_manifest.json`, walks `imports` array | +| **Edit mode scripts** | All `_deps-*.js` + `_models-*.js` + `_kiln-*.js` + templates | `getEditScripts()` returns equivalent set from manifest | +| **View mode scripts** | Numeric IDs resolved to file paths | Human-readable component keys resolved to hashed file paths | + +> **Same result:** Both pipelines return a list of `