🍕 clay build — esbuild + PostCSS 8 asset pipeline, full backward compatibility, tests, and docs#239
Open
🍕 clay build — esbuild + PostCSS 8 asset pipeline, full backward compatibility, tests, and docs#239
Conversation
## Why this change exists
The legacy `clay compile` command uses Browserify (via megabundler) to produce
browser bundles. Browserify generates a runtime `window.modules` registry and
synchronous `window.require()` function. This architecture has two problems:
1. The output is CommonJS/IIFE — not native ES Modules — so browsers cannot use
<script type="module"> loading, import() splitting, or tree-shaking.
2. Several dependencies (`event-stream`, `glob@7`, `kew`) are unmaintained and
incompatible with Node 20.
This PR adds a new `clay pack-next` command backed by esbuild and updates the
`clay compile` sub-commands to be Node 20-compatible.
---
## New: `cli/pack-next.js` + `lib/cmd/pack-next/`
### `build.js` — esbuild configuration and entry-point discovery
- Discovers all `components/*/client.js`, `components/*/model.js`,
`layouts/*/client.js`, and any path returned by `packNextConfig.extraEntries`.
- Sets `format: 'esm'`, `splitting: true`, `chunkNames: 'chunks/chunk-[hash]'`
so every entry gets a hashed ES module with shared code-split into chunks.
- Writes a `public/js/_manifest.json` that maps logical entry keys
(e.g. `components/article/client`) to their hashed output path plus the
list of transitively-imported chunks. `resolveMedia.js` consumes this to
inject exactly the right `<script type="module">` tags per page.
- **Kiln edit-mode aggregator**: before building, `generateKilnEditEntry()`
scans the repo for every `model.js` and `kiln.js` file and writes
`.clay/_kiln-edit-init.js`. This generated file imports all of them and
explicitly assigns the exports to `window.kiln.componentModels`,
`window.kiln.componentKilnjs`, and calls the site's `services/kiln/index.js`
plugin initialiser. This completely replaces the Browserify `window.modules`
registry for edit-mode without changing any component source files.
- Reads a `packNextConfig(config)` hook from `claycli.config.js` in the
project root, allowing per-project esbuild overrides (aliases, defines,
plugins, inject files, etc.).
### `get-script-dependencies.js` — manifest-based script lookup
- `getDependenciesNextForComponents(components, assetPath, globalKeys)`:
given a list of Clay component instance names (from `locals._components`)
returns the deduplicated set of hashed JS URLs (entry + chunks) needed to
run those components on a page — plus any always-loaded global entries.
- `getEditScripts(assetPath)`: returns the hashed URLs for the kiln edit-mode
aggregator bundle (`_kiln-edit-init`) and all its shared chunks.
### `plugins/service-rewrite.js` — server→client service redirect
An esbuild `onResolve` plugin that mirrors the Browserify
`rewriteServiceRequire` transform: any import whose path contains
`services/server` is redirected to the matching `services/client` path.
This prevents server-only code (database drivers, Redis clients, etc.)
from ever being bundled into browser output.
### `plugins/vue2.js` — Vue 2 SFC compiler plugin
Compiles `.vue` files (Vue 2 Single-File Components) at bundle time:
- Parses the SFC with `@vue/component-compiler-utils` +
`vue-template-compiler`.
- Extracts the `<script>` block; strips `export default` / `module.exports =`
and reassigns to a local `__sfc__` variable so render functions can be
injected.
- Compiles the `<template>` block to `{ render, staticRenderFns }` and
attaches them to `__sfc__`.
- **CJS/ESM interop fix**: if the original `<script>` block used
`module.exports = {}` (CommonJS style), the plugin emits
`module.exports = __sfc__; module.exports.default = __sfc__`
so that `require('./foo.vue')` returns the component object directly.
If the original used `export default {}` (ESM style), the plugin emits
`export default __sfc__`. Without this distinction esbuild's interop
wraps the component in `{ default: component }` and Vue's runtime-only
build reports "template or render function not defined" because `render`
is not at the top level of the object.
- Injects scoped-style `data-v-XXXX` attributes when `<style scoped>` is
present.
- Inlines `<style>` blocks as `document.createElement('style')` side-effects.
---
## `clay compile` Node 20 compatibility fixes
All five compile sub-commands (`fonts`, `media`, `scripts`, `styles`,
`templates`) used `event-stream` for stream manipulation. `event-stream` is
abandoned, has a known supply-chain vulnerability history, and emits Node 20
deprecation warnings. It has been replaced with `through2` (object-mode
transform streams) and `merge-stream` (stream merging), both actively
maintained.
Additional Node 20 fixes:
- `new Buffer(...)` → `Buffer.from(..., 'utf8')` throughout (the `Buffer()`
constructor has been deprecated since Node 10 and removed in Node 22).
- `glob` upgraded from v7 to v10; synchronous calls updated to use
the named `globSync` export instead of `glob.sync`.
- `kew` (unmaintained Promise library) removed; usages replaced with native
Promises.
### `gulp-plugins/gulp-newer/index.js`
The bundled `gulp-newer` plugin was rewritten from a prototype-chain class to
an ES6 `class extends Transform`. The key behaviour fix: the original code
called `fs.promises.stat()` on the destination file in the constructor and
let the resulting Promise float. On Node 20, unhandled promise rejections from
`ENOENT` (destination file does not exist yet on a clean build) become fatal
errors. The rewrite catches `ENOENT` in the stat call and resolves to `null`
so the plugin correctly falls through to copying the file.
---
## Dependency changes (`package.json`)
Added:
- `esbuild` ^0.27 — the bundler
- `@vue/component-compiler-utils` ^3.3 — Vue 2 SFC parsing/compilation
- `through2` ^4 — replaces event-stream for object-mode stream transforms
- `merge-stream` — replaces es.merge() for fanning-in multiple gulp streams
- `glob` upgraded to ^10 (breaking: use globSync named export)
- `autoprefixer` upgraded from ^9 to ^10 (PostCSS 8 compatible)
Removed:
- `event-stream` — abandoned, replaced by through2/merge-stream
- `kew` — abandoned Promise library
- `isomorphic-fetch` — replaced by native fetch (Node 18+)
Dev dependencies:
- `jest` upgraded from ^24 to ^29 (Node 20 compatible)
- `jest-fetch-mock`, `jest-mock-console` updated to match
Made-with: Cursor
… steps Introduces five new build steps that bring the full legacy `clay compile` asset pipeline into `clay pack-next`: - **styles.js** — PostCSS 8 pipeline: `postcss-import` resolves `@import` chains across styleguide directories, `autoprefixer` adds vendor prefixes, and `cssnano` minifies in production. Writes per-styleguide CSS files to `public/css/`. - **fonts.js** — copies `global/fonts/**` and styleguide font files to `public/fonts/`, and writes `_linked-fonts.<styleguide>.css` files. - **templates.js** — compiles every `*.template.handlebars` file to a JS module that writes `window.kiln.componentTemplates[name]`, bucketed into `_templates-a-d.js` etc. in production. - **vendor.js** — copies `global/js/vendor/**` to `public/js/vendor/`. - **media.js** — copies `global/media/**` to `public/media/`. Adds `cssnano` and `p-limit` to dependencies; `p-limit` bounds the PostCSS concurrency to avoid memory pressure when processing large styleguide sets. Made-with: Cursor
The vue2 esbuild plugin now accumulates raw CSS from every `<style>` block it
processes across all `.vue` SFCs and writes them into
`public/css/_kiln-plugins.css` via an `onEnd` hook.
Previously the plugin injected styles as `document.createElement('style')`
runtime side-effects, which caused a Flash Of Unstyled Content in Kiln edit
mode and made the CSS invisible to server-side rendering. Writing a single
concatenated CSS file lets the page load it as a normal `<link>` tag.
Made-with: Cursor
… noise esbuild reports warnings for every file it touches on each incremental rebuild, not just the file that changed. In `make watch` this floods the terminal with hundreds of irrelevant lint-style messages on every keystroke. Watch mode now surfaces only errors. Full warning output remains visible during `clay pack-next` (one-shot build / `make compile`). Made-with: Cursor
…ate _view-init.js
build.js — orchestration and watch mode
- `buildAll()` now calls all asset steps (styles, fonts, templates, vendor,
media) in parallel alongside the esbuild JS step.
- `watch()` uses chokidar (polling, for Docker + macOS host volumes) to watch
every file type individually. Each watcher debounces rebuilds and logs which
file changed (`Changed:`) and what was rebuilt (`Rebuilt:`) in distinct
colours.
- CSS watch rebuilds all variation files sharing a component basename
(e.g. editing `text-list.css` also rebuilds `text-list_amp.css`).
- `global/js/*.js` added to ENTRY_GLOBS so global scripts (ads, cid, facebook,
aaa-module-mounting) compile as independent manifest entries — no hand-rolled
init file required to pull them in.
- Non-existent `extraEntries` (e.g. the webpack-era `components/init.js` still
referenced in older `claycli.config.js` files) are silently skipped.
generateViewInitEntry() — _view-init.js generator
- Before every build and on every JS file add/remove in watch mode, claycli
writes `.clay/_view-init.js` containing:
1. A sticky custom-event shim: if `auth:init` (or any listed event) has
already fired when a late `window.addEventListener` subscriber registers,
the handler receives an immediate replay in the next microtask. This
restores the Browserify synchronous-bundle guarantee without touching
`auth.js` or any `client.js` file.
2. An explicit component module map (all `components/**/client.js` entries)
that esbuild resolves to content-hashed paths at build time.
3. Per-element component mounting that supports both patterns:
- Function-export: `mod(element)` is called directly.
- Dollar-Slice controller: `DS.get(name, element)` instantiates the
controller after `aaa-module-mounting.js` has registered globals.
- The consuming repo no longer needs to maintain `components/init.js`.
get-script-dependencies.js — auto-inject _view-init
- `getDependenciesNextForComponents()` now automatically prepends the generated
`_view-init` scripts before any caller-supplied GLOBAL_KEYS. The consuming
`resolveMedia.js` does not need to know the generated key.
Made-with: Cursor
Removes the need for pack-next-inject.js (a webpack ProvidePlugin clone)
from consuming repos entirely.
Previously, consuming repos had to ship a pack-next-inject.js file and wire
it up via config.inject in packNextConfig. This was a webpack-era pattern
that has no place in the esbuild pipeline.
Two cleaner alternatives used instead:
**DS / Eventify / Fingerprint2**
aaa-module-mounting.js already sets window.DS, window.Eventify, and
window.Fingerprint2 synchronously before any component module code runs.
esbuild define entries now map the free variable references (DS, Eventify,
Fingerprint2) to their window-registered equivalents at compile time.
No runtime behaviour change; the inject file is no longer needed.
**process.*
esbuild define entries cover every common pattern:
process.env.NODE_ENV → respects NODE_ENV env var (defaults to 'development')
process.env → { NODE_ENV: ... }
process.browser → true
process.version → ""
process.versions → {}
global, __filename, __dirname, and mainFields are also promoted to claycli
defaults so consuming repos don't need to repeat them in packNextConfig.
Made-with: Cursor
…cosystem browser compat
Moves all Clay/Node.js compatibility stubs out of sites' claycli.config.js
and into claycli itself so no consuming repo needs to maintain this boilerplate:
browserCompatPlugin (new):
- Stubs clay-log and services/universal/log.js (Node-only transports)
- Stubs Clay server-only packages (amphora-search, amphora-storage-postgres, etc.)
- Stubs all Node.js built-in modules (fs, crypto, net, path, etc.)
- Provides rich stubs for events, stream, util, buffer (needed for util.inherits chains)
- Provides http/https stubs with agent-base patch flag pre-set
serviceRewritePlugin (enhanced):
- Case 1 (existing): rewrites imports whose raw string contains 'services/server'
- Case 2 (new): catches relative '../server/' imports from services/universal/
whose raw path doesn't contain 'services/server' but whose resolved absolute
path lands inside services/server/ — e.g. require('../server/db') in utils.js
Made-with: Cursor
…m pack-next to build
File/directory renames:
- lib/cmd/pack-next/ → lib/cmd/build/
- cli/pack-next.js → cli/build.js
CLI changes:
- Command is now 'clay build' (exports.command = 'build')
- 'clay pack-next' and 'pn' kept as backward-compat aliases so existing
Makefiles don't break before they are updated
- New short alias 'b' added (clay b)
- cli/index.js routing updated: b/pn/pack-next all resolve to cli/build.js
Internal changes:
- browserCompatPlugin and serviceRewritePlugin registered in getEsbuildConfig()
- Config hook key renamed from packNextConfig → esbuildConfig in claycli.config.js
- All user-facing error messages and comments updated from 'clay pack-next' to 'clay build'
- Auto-generated _view-init.js header updated accordingly
Made-with: Cursor
The >=22 constraint was unnecessarily strict. The only API that could have justified it is fetch(), which has been stable and unflagged since Node 18. No Node 22-specific APIs are used anywhere in claycli. Sites running Node 20 LTS do not need to upgrade their runtime just to use clay build. Made-with: Cursor
- buildAll() now prints a permanent done line for each step as it finishes, with an animated single-line spinner showing in-progress steps and their live completion percentage (e.g. ⠸ [styles 45%] [templates 67%] (18s)) - buildStyles and buildTemplates accept an onProgress callback so the progress bar shows real done/total counts rather than a spinner alone - buildStyles accepts an onError callback so CSS compile errors are routed through the display manager instead of stderr, preventing them from corrupting the progress line - cli/build.js suppresses verbose per-warning logs in build mode; shows a single summary count instead (e.g. "21 esbuild warning(s)") - Add *.tgz to .gitignore Made-with: Cursor
Tests (39 passing): - manifest.test.js: writeManifest — entry keys, chunk skipping, public URL mapping - styles.test.js: buildStyles — changedFiles, onProgress, onError routing - templates.test.js: buildTemplates — HBS precompile, progress, watch-mode error resilience, minified buckets - media.test.js: copyMedia — component + layout media copy - get-script-dependencies.test.js: hasManifest, getDependenciesNextForComponents — dedup, _view-init ordering Documentation (CLAY-BUILD.md): - Full pipeline comparison: Browserify+Gulp vs esbuild+PostCSS 8 - ASCII architecture diagram and step-by-step pipeline comparison table - Per-feature comparison (JS, CSS, templates, fonts, script resolution) - Performance numbers: full build 2-4x faster, watch JS rebuild 60-200x faster - Audience-specific sections: developer, SRE, product manager - Configuration guide, migration guide, code references with file links Also: update jest collectCoverageFrom to include lib/cmd/build/ modules Made-with: Cursor
Two color-coded flowcharts (one per pipeline) replacing the hard-to-read side-by-side ASCII box. Nodes are color-coded by speed (red=slow, green=fast, amber=medium). Followed by clean tables for shared outputs and key differences. Made-with: Cursor
Replace two separate top-down flowcharts with one left-to-right diagram containing both pipelines as labeled subgraphs. Source and output nodes flank both lanes, making the parallel-vs-sequential contrast immediately visible. Adds a summary table directly below the diagram. Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
…ll glob + ctime filter, not single-file rebuild Made-with: Cursor
…ion table Made-with: Cursor
…b vitals/SEO/analytics - Correct ~0.5s CSS watch claim to a realistic ~1–3s in both the performance table and the PM section - Expand PM section: code splitting payload reduction, Core Web Vitals (LCP, INP) explanation, SEO ranking signal context, and analytics impact (bounce rate, session depth, conversion) Made-with: Cursor
…mponents on every page clay compile getDependencies() in view mode was page-specific — it walked _registry.json for only the components amphora placed on the page. Correct all instances that claimed otherwise and accurately describe the real differences: no shared chunk deduplication, and _client-init.js mounting loaded modules without DOM-presence checks. Made-with: Cursor
…tions Errors fixed: - _modules-a-d.js → actual output is per-component files + _deps-a-d.js buckets - @nymag/vueify esbuild plugin → custom plugin using vue-template-compiler directly - Vue top-level side-effects claim scoped correctly - Globals section: standard Clay globals already handled in claycli defaults New content added: - Source maps (new generates them, old did not) in Why, JS table, SRE, Dev sections - Content-hashed filenames and CDN cache efficiency throughout - Tree shaking (esbuild) vs no tree shaking (Browserify) - Build-time process.env.NODE_ENV → dead code elimination in minified builds - _prelude.js/_postlude.js custom runtime overhead called out explicitly - browserify-cache.json stale-build risk and rollback safety - Template process.exit(1) vs graceful error continuation - npm dependency reduction table (~20 removed, ~5 added) - CI compute cost reduction (~63%) in PM and SRE sections - CDN cache efficiency and operational confidence in PM section - Rollback safety (manifest atomicity) in SRE section Made-with: Cursor
jjpaulino
changed the title
jjpaulino
changed the title
jjpaulino
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR adds
A new
clay buildcommand that replaces the legacy Browserify + Gulp asset pipeline with esbuild + PostCSS 8, while keepingclay compilefully intact and available. Both commands coexist — teams opt in toclay buildon their own timeline.See
CLAY-BUILD.mdfor full documentation, architecture diagrams, feature-by-feature comparisons, performance benchmarks, and migration guide.Why
The legacy
clay compilepipeline was built on tools designed for 2014–2018:article.client.js)_prelude.js/_postlude.jscustom runtimewindow.requireoverhead loaded on every pagebrowserify-cache.jsonNew command:
clay buildKey improvements over
clay compilePerformance
Developer experience
Changed:/Rebuilt:output per file typeBrowser / runtime
<script type="module">) — nowindow.requireruntime overheadcomponents/article/client-A1B2C3.js) — CDN caches files forever; only changed files get new URLs on deployprocess.env.NODE_ENV— dead branches removed during minification, not shipped to users_view-init.js— components mount only when their DOM element is present, not at page load; includes a sticky custom-event shim soauth:initand similar events are never missedInfrastructure
clay buildfails, the previous_manifest.jsonis untouched;hasManifest()serves the last good buildArchitecture
All steps except media run in parallel — total build time is
max(slowest step)≈ 33s, not their sum.Backward compatibility
clay compileis not removed or modified — it works exactly as beforeclaycli.config.jsusing separate keys (babelTargetsvsesbuildConfig) so they never conflictresolveMedia.jsand a Makefile targetTests
39 tests across 5 new files covering all new build modules:
lib/cmd/build/manifest.test.jswriteManifest— entry key derivation, chunk handling, public URL mappinglib/cmd/build/styles.test.jsbuildStyles— compilation, incremental mode,onProgress,onErrorlib/cmd/build/templates.test.jsbuildTemplates— HBS precompile,onProgress, error resilience, minified bucketslib/cmd/build/media.test.jscopyMedia— component + layout media copylib/cmd/build/get-script-dependencies.test.jshasManifest,getDependenciesNextForComponents— chunk dedup, ordering, missing componentsFile map
cli/build.jslib/cmd/build/build.js_kiln-edit-init.js+_view-init.jsgenerationlib/cmd/build/get-script-dependencies.jslib/cmd/build/styles.jslib/cmd/build/fonts.js_linked-fonts.cssgenerationlib/cmd/build/templates.jslib/cmd/build/vendor.jslib/cmd/build/media.jslib/cmd/build/manifest.js_manifest.jsonfrom esbuild metafilelib/cmd/build/plugins/browser-compat.jslib/cmd/build/plugins/service-rewrite.jsservices/server/*→services/client/*at bundle timelib/cmd/build/plugins/vue2.js_kiln-plugins.cssCLAY-BUILD.mdRelated
clay buildin Docker build +resolveMedia.js