From d0dde92b49b30aff537c6062558b18dd2efe5f8b Mon Sep 17 00:00:00 2001 From: Miguel Angel Simon Sierra Date: Tue, 7 Jul 2026 20:23:04 -0400 Subject: [PATCH 01/11] =?UTF-8?q?feat(media-use):=20color=20grading=20?= =?UTF-8?q?=E2=80=94=20grade/lut=20resolve,=20smart-grade,=20grade-compare?= =?UTF-8?q?=20CLI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add color grading to media-use as first-class resolve types plus a faithful comparison command. All local, offline, deterministic — no model, no GPU. - resolve -t grade / -t lut: produce a data-color-grading block (or a frozen .cube). Look cascade: core preset (no file) -> bundled .cube library -> parametric buildCube. Emitted .cube is Rec.709 and validated against core's colorLuts constraints (LUT_3D_SIZE <= 64) before it is frozen. - smart grade (grade --for ): ffmpeg signalstats -> adjust suggestion (exposure / contrast / white balance), surfaced with the measured evidence on stderr as a starting point; never auto-applied. - hyperframes grade-compare: renders N candidate grades onto a reference frame through the real runtime shader into one labeled comparison PNG, so an agent picks a look without opening Studio. Prepends an "original" baseline cell by default (--no-baseline to omit). Shares the headless-capture pipeline with snapshot via capture/captureCompositionFrame. - media-use SKILL: proactive "media opportunity pass" guidance (grounded signal -> offer, ask once, surface don't mutate). Verified: media-use 116/116, grade-compare 7/7, snapshot 9/9, lint + format clean, full build green, comparison renders end to end. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv --- CLAUDE.md | 2 +- README.md | 20 +- docs/guides/skills.mdx | 2 +- .../capture/captureCompositionFrame.test.ts | 39 + .../src/capture/captureCompositionFrame.ts | 145 ++++ packages/cli/src/cli.ts | 1 + .../cli/src/commands/grade-compare.test.ts | 159 ++++ packages/cli/src/commands/grade-compare.ts | 624 +++++++++++++++ packages/cli/src/commands/snapshot.ts | 130 +--- packages/cli/src/help.ts | 4 + skills-manifest.json | 8 +- skills/hyperframes-cli/SKILL.md | 4 +- skills/hyperframes/SKILL.md | 2 +- skills/media-use/SKILL.md | 123 ++- skills/media-use/luts/bleach-bypass.cube | 733 ++++++++++++++++++ skills/media-use/luts/film-fade.cube | 733 ++++++++++++++++++ skills/media-use/luts/index.json | 26 + .../luts/teal-orange-blockbuster.cube | 733 ++++++++++++++++++ skills/media-use/scripts/lib/candidates.mjs | 3 +- skills/media-use/scripts/lib/cube-build.mjs | 180 +++++ .../media-use/scripts/lib/cube-build.test.mjs | 80 ++ .../media-use/scripts/lib/cube-validate.mjs | 198 +++++ .../scripts/lib/cube-validate.test.mjs | 125 +++ .../media-use/scripts/lib/grade-analyzer.mjs | 162 ++++ .../scripts/lib/grade-analyzer.test.mjs | 137 ++++ .../scripts/lib/lut-preset-provider.mjs | 158 ++++ .../scripts/lib/lut-preset-provider.test.mjs | 72 ++ skills/media-use/scripts/lib/manifest.mjs | 2 + .../media-use/scripts/lib/manifest.test.mjs | 12 + skills/media-use/scripts/lib/registry.mjs | 9 + .../media-use/scripts/lib/registry.test.mjs | 4 +- skills/media-use/scripts/resolve.mjs | 264 ++++++- skills/media-use/scripts/resolve.test.mjs | 222 +++++- 33 files changed, 4958 insertions(+), 158 deletions(-) create mode 100644 packages/cli/src/capture/captureCompositionFrame.test.ts create mode 100644 packages/cli/src/capture/captureCompositionFrame.ts create mode 100644 packages/cli/src/commands/grade-compare.test.ts create mode 100644 packages/cli/src/commands/grade-compare.ts create mode 100644 skills/media-use/luts/bleach-bypass.cube create mode 100644 skills/media-use/luts/film-fade.cube create mode 100644 skills/media-use/luts/index.json create mode 100644 skills/media-use/luts/teal-orange-blockbuster.cube create mode 100644 skills/media-use/scripts/lib/cube-build.mjs create mode 100644 skills/media-use/scripts/lib/cube-build.test.mjs create mode 100644 skills/media-use/scripts/lib/cube-validate.mjs create mode 100644 skills/media-use/scripts/lib/cube-validate.test.mjs create mode 100644 skills/media-use/scripts/lib/grade-analyzer.mjs create mode 100644 skills/media-use/scripts/lib/grade-analyzer.test.mjs create mode 100644 skills/media-use/scripts/lib/lut-preset-provider.mjs create mode 100644 skills/media-use/scripts/lib/lut-preset-provider.test.mjs diff --git a/CLAUDE.md b/CLAUDE.md index d4778bef55..7c895db16d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ Atomic capabilities the creation workflows compose against — pull one when you - `/hyperframes-animation` — all animation knowledge: atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP default, plus Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). - `/hyperframes-keyframes` — seek-safe keyframe authoring across runtimes: GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, text trails, 3D depth; plus `hyperframes keyframes` diagnostics for surfacing and verifying rendered motion. - `/hyperframes-creative` — non-animation creative direction: `frame.md` / `design.md` handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. -- `/media-use` — the media OS: resolve any media need (BGM, SFX, image, icon, logo, voice) into a frozen local file + ledger record; generate via TTS / music / image models when the catalog misses; transcribe, caption, remove backgrounds, and reuse assets across projects. One shared `scripts/audio.mjs` engine + manifest tracking; keeps search noise on disk. +- `/media-use` — the media OS: resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record; generate via TTS / music / image models when the catalog misses; transcribe, caption, remove backgrounds, and reuse assets across projects. One shared `scripts/audio.mjs` engine + manifest tracking; keeps search noise on disk. - `/hyperframes-cli` — CLI dev loop: `init`, `add`, `lint`, `validate`, `inspect`, `preview`, `render`, `publish`, `doctor`, `lambda` (AWS Lambda cloud rendering). - `/hyperframes-registry` — install and wire registry blocks and components into compositions via `hyperframes add`. Covers authoring a new block or component to contribute upstream. - `/figma` — import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition. diff --git a/README.md b/README.md index 4a224041f3..adcac1ef18 100644 --- a/README.md +++ b/README.md @@ -83,16 +83,16 @@ Installs stay lean after that: `npx hyperframes init` keeps the **core set** fre Atomic capabilities the creation workflows compose against — pull one when you need that specific layer. -| Skill | Covers | -| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `/hyperframes-core` | The composition contract — `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, determinism rules. | -| `/hyperframes-animation` | All animation knowledge — atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). | -| `/hyperframes-keyframes` | Seek-safe keyframe authoring across runtimes — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth — plus `hyperframes keyframes` diagnostics for rendered motion. | -| `/hyperframes-creative` | Non-animation creative direction — `frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. | -| `/media-use` | The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice) into a frozen local file + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. | -| `/hyperframes-cli` | CLI dev loop — `init`, `lint`, `validate`, `inspect`, `preview`, `render`, `publish`, `doctor`, plus AWS Lambda cloud rendering (`lambda deploy / render / progress`). | -| `/hyperframes-registry` | Install and wire registry blocks and components into compositions via `hyperframes add`. Authoring a new block or component to contribute upstream. | -| `/figma` | Import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition. | +| Skill | Covers | +| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `/hyperframes-core` | The composition contract — `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, determinism rules. | +| `/hyperframes-animation` | All animation knowledge — atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). | +| `/hyperframes-keyframes` | Seek-safe keyframe authoring across runtimes — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth — plus `hyperframes keyframes` diagnostics for rendered motion. | +| `/hyperframes-creative` | Non-animation creative direction — `frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. | +| `/media-use` | The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. | +| `/hyperframes-cli` | CLI dev loop — `init`, `lint`, `validate`, `inspect`, `preview`, `render`, `publish`, `doctor`, plus AWS Lambda cloud rendering (`lambda deploy / render / progress`). | +| `/hyperframes-registry` | Install and wire registry blocks and components into compositions via `hyperframes add`. Authoring a new block or component to contribute upstream. | +| `/figma` | Import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition. | For visual design handoff workflows, see the [Claude Design guide](https://hyperframes.heygen.com/guides/claude-design) and [Open Design guide](https://hyperframes.heygen.com/guides/open-design). diff --git a/docs/guides/skills.mdx b/docs/guides/skills.mdx index 394a572f9a..9579f7df62 100644 --- a/docs/guides/skills.mdx +++ b/docs/guides/skills.mdx @@ -98,7 +98,7 @@ Atomic capabilities the creation workflows compose against — pull one when you | `/hyperframes-animation` | All animation knowledge — atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). | | `/hyperframes-keyframes` | Seek-safe keyframe authoring across runtimes — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth — plus `hyperframes keyframes` diagnostics for rendered motion. | | `/hyperframes-creative` | Non-animation creative direction — `frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. | -| `/media-use` | The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice) into a frozen local file + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. | +| `/media-use` | The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. | | `/hyperframes-cli` | CLI dev loop — `init`, `lint`, `validate`, `inspect`, `preview`, `render`, `publish`, `doctor`, plus AWS Lambda cloud rendering (`lambda deploy / render / progress / destroy / policies`). | | `/hyperframes-registry` | Install and wire registry blocks and components into compositions via `hyperframes add`. Authoring a new block or component to contribute upstream. | | `/figma` | Import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition. | diff --git a/packages/cli/src/capture/captureCompositionFrame.test.ts b/packages/cli/src/capture/captureCompositionFrame.test.ts new file mode 100644 index 0000000000..cf942565c9 --- /dev/null +++ b/packages/cli/src/capture/captureCompositionFrame.test.ts @@ -0,0 +1,39 @@ +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { runFfmpegOnce } from "./captureCompositionFrame.js"; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "hf-capture-frame-test-")); +} + +describe("runFfmpegOnce", () => { + it("returns the process exit code and collected stderr", async () => { + const dir = tempDir(); + try { + const script = join(dir, "fail.cjs"); + writeFileSync(script, 'process.stderr.write("ffmpeg failed"); process.exit(3);\n'); + + const result = await runFfmpegOnce(process.execPath, [script], 1000); + + expect(result).toEqual({ code: 3, stderr: "ffmpeg failed", timedOut: false }); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("terminates the process when the timeout elapses", async () => { + const dir = tempDir(); + try { + const script = join(dir, "hang.cjs"); + writeFileSync(script, "setTimeout(() => {}, 10000);\n"); + + const result = await runFfmpegOnce(process.execPath, [script], 50); + + expect(result.timedOut).toBe(true); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/cli/src/capture/captureCompositionFrame.ts b/packages/cli/src/capture/captureCompositionFrame.ts new file mode 100644 index 0000000000..8b12b49eb8 --- /dev/null +++ b/packages/cli/src/capture/captureCompositionFrame.ts @@ -0,0 +1,145 @@ +import { spawn } from "node:child_process"; +import type { Browser, Page } from "puppeteer-core"; +import { c } from "../ui/colors.js"; +import { resolveCompositionViewportFromHtml } from "../utils/compositionViewport.js"; + +const CHROME_LAUNCH_ARGS = [ + "--no-sandbox", + "--disable-gpu", + "--disable-dev-shm-usage", + "--enable-webgl", + "--use-gl=angle", + "--use-angle=swiftshader", +]; + +const SHADER_TRANSITIONS_TIMEOUT_MS = 90_000; +const CAPTURE_SETTLE_MS = 1500; + +export interface SettledCompositionPage { + browser: Browser; + page: Page; +} + +export interface OpenSettledCompositionPageOptions { + renderReadyTimeoutMs: number; + renderReadyWarningSuffix: string; +} + +export interface FfmpegRunResult { + code: number | null; + stderr: string; + timedOut: boolean; +} + +function compositionRuntimeReadyInBrowser(): boolean { + return Boolean(Reflect.get(window, "__renderReady")); +} + +function shaderTransitionsReadyInBrowser(): boolean { + function shaderTransitionRegistryReady(): boolean | undefined { + const hf = Reflect.get(window, "__hf"); + if (typeof hf !== "object" || hf === null) return undefined; + + const shaderTransitions = Reflect.get(hf, "shaderTransitions"); + if (typeof shaderTransitions !== "object" || shaderTransitions === null) return undefined; + + for (const key of Object.keys(shaderTransitions)) { + const entry = Reflect.get(shaderTransitions, key); + if (typeof entry !== "object" || entry === null) return false; + if (Reflect.get(entry, "ready") !== true) return false; + } + return true; + } + + function shaderLoadingOverlayReady(): boolean { + const overlay = document.querySelector("[data-hyper-shader-loading]"); + if (!overlay) return true; + if (!(overlay instanceof HTMLElement)) return true; + return window.getComputedStyle(overlay).display === "none"; + } + + return shaderTransitionRegistryReady() ?? shaderLoadingOverlayReady(); +} + +async function waitForCompositionSettle( + page: Page, + options: OpenSettledCompositionPageOptions, +): Promise { + const runtimeReady = await page + .waitForFunction(compositionRuntimeReadyInBrowser, { timeout: options.renderReadyTimeoutMs }) + .then(() => true) + .catch(() => false); + + if (!runtimeReady) { + console.warn( + `\n ${c.warn("⚠")} Runtime did not become render-ready within ${options.renderReadyTimeoutMs}ms — ${options.renderReadyWarningSuffix}`, + ); + } + + await page + .waitForFunction(shaderTransitionsReadyInBrowser, { + timeout: SHADER_TRANSITIONS_TIMEOUT_MS, + }) + .catch(() => { + console.warn(` ${c.warn("⚠")} Shader transitions did not finish pre-rendering`); + }); + + await page.evaluate(() => document.fonts.ready).catch(() => {}); + await new Promise((resolveSettle) => setTimeout(resolveSettle, CAPTURE_SETTLE_MS)); +} + +export async function openSettledCompositionPage( + html: string, + url: string, + options: OpenSettledCompositionPageOptions, +): Promise { + const { ensureBrowser } = await import("../browser/manager.js"); + const browser = await ensureBrowser(); + const puppeteer = await import("puppeteer-core"); + + let chromeBrowser: Browser | undefined; + try { + chromeBrowser = await puppeteer.default.launch({ + headless: true, + executablePath: browser.executablePath, + args: CHROME_LAUNCH_ARGS, + }); + + const page = await chromeBrowser.newPage(); + await page.setViewport(resolveCompositionViewportFromHtml(html)); + await page.goto(url, { waitUntil: "domcontentloaded", timeout: 10000 }); + await waitForCompositionSettle(page, options); + return { browser: chromeBrowser, page }; + } catch (err) { + await chromeBrowser?.close().catch(() => {}); + throw err; + } +} + +export async function runFfmpegOnce( + ffmpegPath: string, + args: readonly string[], + timeoutMs: number, +): Promise { + return await new Promise((resolvePromise) => { + const ff = spawn(ffmpegPath, args); + let stderr = ""; + let timedOut = false; + const timer = setTimeout(() => { + timedOut = true; + ff.kill("SIGTERM"); + }, timeoutMs); + + ff.stderr.on("data", (data: Buffer) => { + stderr += data.toString(); + }); + ff.on("close", (code) => { + clearTimeout(timer); + resolvePromise({ code, stderr, timedOut }); + }); + ff.on("error", () => { + clearTimeout(timer); + resolvePromise({ code: null, stderr: "ffmpeg spawn failed", timedOut }); + }); + }); +} diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 2c2726e79a..6671964db3 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -139,6 +139,7 @@ const commandLoaders = { events: () => import("./commands/events.js").then((m) => m.default), validate: () => import("./commands/validate.js").then((m) => m.default), snapshot: () => import("./commands/snapshot.js").then((m) => m.default), + "grade-compare": () => import("./commands/grade-compare.js").then((m) => m.default), capture: () => import("./commands/capture.js").then((m) => m.default), lambda: () => import("./commands/lambda.js").then((m) => m.default), cloudrun: () => import("./commands/cloudrun.js").then((m) => m.default), diff --git a/packages/cli/src/commands/grade-compare.test.ts b/packages/cli/src/commands/grade-compare.test.ts new file mode 100644 index 0000000000..382be87c68 --- /dev/null +++ b/packages/cli/src/commands/grade-compare.test.ts @@ -0,0 +1,159 @@ +import { readFileSync, writeFileSync } from "node:fs"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { basename, join } from "node:path"; +import { HF_COLOR_GRADING_ATTR, serializeHfColorGrading } from "@hyperframes/core"; +import { describe, expect, it } from "vitest"; +import { + buildGradeCompareHtml, + parseGradeCompareArgs, + parseGradesFile, + prepareGradeCompareTempProject, + prependBaselineCell, + resolveLutCells, +} from "./grade-compare.js"; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "hf-grade-compare-test-")); +} + +describe("parseGradesFile", () => { + it("parses a valid grades array into labeled cells", () => { + const dir = tempDir(); + try { + const file = join(dir, "grades.json"); + writeFileSync( + file, + JSON.stringify([ + { label: "warm", grading: { preset: "warm-daylight" } }, + { label: "punch", grading: { adjust: { exposure: 0.5, contrast: 0.4 } } }, + ]), + ); + + expect(parseGradesFile(file)).toEqual([ + { label: "warm", grading: { preset: "warm-daylight" } }, + { label: "punch", grading: { adjust: { exposure: 0.5, contrast: 0.4 } } }, + ]); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("errors clearly for missing or invalid grades files", () => { + const dir = tempDir(); + try { + expect(() => parseGradesFile(join(dir, "missing.json"))).toThrow( + /Grades file not found: .*missing\.json/, + ); + + const invalid = join(dir, "invalid.json"); + writeFileSync(invalid, JSON.stringify({ label: "not an array" })); + expect(() => parseGradesFile(invalid)).toThrow( + "Grades file must be a JSON array of { label, grading } objects", + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe("prependBaselineCell", () => { + it("prepends an ungraded 'original' reference cell", () => { + const candidates = resolveLutCells("/tmp/luts/film.cube"); + const withBaseline = prependBaselineCell(candidates); + expect(withBaseline).toHaveLength(candidates.length + 1); + expect(withBaseline[0].label).toBe("original"); + // empty grading (no preset / no lut) → normalizes to inactive → renders the + // source frame untouched, giving a reference to judge candidates against + expect(withBaseline[0].grading.preset ?? null).toBeNull(); + expect(withBaseline[0].grading.lut ?? null).toBeNull(); + expect(withBaseline.slice(1)).toEqual(candidates); + }); +}); + +describe("resolveLutCells", () => { + it("expands comma-separated LUT paths into labeled grading cells", () => { + expect(resolveLutCells("/tmp/luts/film.cube,./cool.look.cube")).toEqual([ + { label: "film", grading: { lut: { src: "/tmp/luts/film.cube" } } }, + { label: "cool.look", grading: { lut: { src: "./cool.look.cube" } } }, + ]); + }); +}); + +describe("parseGradeCompareArgs", () => { + it("requires exactly one grade source", () => { + expect(() => parseGradeCompareArgs({ for: "frame.png" })).toThrow( + "Exactly one of --grades or --luts is required", + ); + expect(() => + parseGradeCompareArgs({ for: "frame.png", grades: "grades.json", luts: "a.cube" }), + ).toThrow("Exactly one of --grades or --luts is required"); + }); +}); + +describe("buildGradeCompareHtml", () => { + it("renders one labeled color-graded image per cell with composition metadata", () => { + const cells = [ + { label: "warm ", grading: { preset: "warm-daylight" } }, + { label: "cool", grading: { adjust: { temperature: -0.8 } } }, + { label: "punchy", grading: { adjust: { exposure: 0.5, contrast: 0.4 } } }, + ]; + + const html = buildGradeCompareHtml({ + cells, + frameSrc: "frame.png", + frameWidth: 640, + frameHeight: 360, + }); + + const escapedAttr = HF_COLOR_GRADING_ATTR.replace(/[-/\\^$*+?.()|[\]{}]/g, "\\$&"); + const attrPattern = new RegExp(`]+${escapedAttr}=`, "g"); + expect(html.match(attrPattern)).toHaveLength(cells.length); + expect(html).toContain('data-composition-id="grade-compare"'); + expect(html).toContain('data-width="1168"'); + expect(html).toContain('data-height="742"'); + expect(html).toContain('data-duration="1"'); + expect(html).toContain("warm <daylight>"); + for (const cell of cells) { + expect(html).toContain(`${HF_COLOR_GRADING_ATTR}='${serializeHfColorGrading(cell.grading)}'`); + } + }); +}); + +describe("prepareGradeCompareTempProject", () => { + it("copies the frame and LUTs into a temp project and rewrites LUT src values", async () => { + const dir = tempDir(); + try { + const framePath = join(dir, "frame.png"); + const lutPath = join(dir, "look.cube"); + writeFileSync(framePath, "fake-png"); + writeFileSync(lutPath, "LUT_3D_SIZE 2\n"); + + const prepared = await prepareGradeCompareTempProject({ + projectDir: dir, + framePath, + frameBuffer: Buffer.from("fake-png"), + cells: [ + { label: "film", grading: { lut: { src: basename(lutPath), intensity: 0.7 } } }, + { label: "warm", grading: { preset: "warm-daylight" } }, + ], + frameWidth: 640, + frameHeight: 360, + }); + + try { + expect(readFileSync(join(prepared.tempDir, "frame.png"), "utf-8")).toBe("fake-png"); + expect(readFileSync(join(prepared.tempDir, "lut-0.cube"), "utf-8")).toBe("LUT_3D_SIZE 2\n"); + expect(readFileSync(join(prepared.tempDir, "index.html"), "utf-8")).toContain( + `${HF_COLOR_GRADING_ATTR}='${serializeHfColorGrading({ + lut: { src: "lut-0.cube", intensity: 0.7 }, + })}'`, + ); + } finally { + rmSync(prepared.tempDir, { recursive: true, force: true }); + } + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/cli/src/commands/grade-compare.ts b/packages/cli/src/commands/grade-compare.ts new file mode 100644 index 0000000000..7a8fc68e53 --- /dev/null +++ b/packages/cli/src/commands/grade-compare.ts @@ -0,0 +1,624 @@ +import { + copyFileSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { basename, dirname, extname, isAbsolute, join, relative, resolve } from "node:path"; +import { + HF_COLOR_GRADING_ATTR, + normalizeHfColorGrading, + serializeHfColorGrading, +} from "@hyperframes/core"; +import { defineCommand } from "citty"; +import sharp from "sharp"; +import type { Example } from "./_examples.js"; +import { openSettledCompositionPage, runFfmpegOnce } from "../capture/captureCompositionFrame.js"; +import { findFFmpeg } from "../browser/ffmpeg.js"; +import { c } from "../ui/colors.js"; +import { normalizeErrorMessage } from "../utils/errorMessage.js"; +import { serveStaticProjectHtml } from "../utils/staticProjectServer.js"; +import { withMeta } from "../utils/updateCheck.js"; + +const COMPOSITION_ID = "grade-compare"; +const COMPOSITION_DURATION = "1"; +const DEFAULT_CELL_WIDTH = 560; +const MAX_COLUMNS = 4; +const GRID_PADDING = 16; +const LABEL_HEIGHT = 32; +const FFMPEG_EXTRACT_TIMEOUT_MS = 30_000; + +export interface GradeCompareCell { + label: string; + grading: unknown; +} + +interface ParsedGradeCompareArgs { + framePath: string; + projectDir: string; + outPath: string; + source: { kind: "grades"; path: string } | { kind: "luts"; value: string }; + json: boolean; +} + +interface PreparedGradeCompareProject { + tempDir: string; + html: string; + cells: GradeCompareCell[]; +} + +interface GradeCompareHtmlOptions { + cells: readonly GradeCompareCell[]; + frameSrc: string; + frameWidth: number; + frameHeight: number; +} + +interface ReferenceFrame { + buffer: Buffer; + width: number; + height: number; + stagedName: string; +} + +export const examples: Example[] = [ + [ + "Compare grade presets on one reference frame", + "hyperframes grade-compare --for frame.png --grades grades.json", + ], + [ + "Compare LUT files and print agent-friendly JSON", + "hyperframes grade-compare --for frame.png --luts looks/a.cube,looks/b.cube --json", + ], +]; + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function hasOwn(record: Record, key: string): boolean { + return Object.prototype.hasOwnProperty.call(record, key); +} + +function cloneRecord(record: Record): Record { + const next: Record = {}; + for (const [key, value] of Object.entries(record)) { + next[key] = value; + } + return next; +} + +function readOptionalString(value: unknown): string | undefined { + if (typeof value !== "string") return undefined; + const trimmed = value.trim(); + return trimmed ? trimmed : undefined; +} + +function resolveFromBase(baseDir: string, input: string): string { + return isAbsolute(input) ? input : resolve(baseDir, input); +} + +function displayPath(projectDir: string, filePath: string): string { + const rel = relative(projectDir, filePath); + if (rel && !rel.startsWith("..") && !isAbsolute(rel)) return rel; + return filePath; +} + +function escapeXml(value: string): string { + return value + .replace(/&/g, "&") + .replace(//g, ">") + .replace(/"/g, """) + .replace(/'/g, "'"); +} + +function escapeSingleQuotedAttr(value: string): string { + return value + .replace(/&/g, "&") + .replace(//g, ">") + .replace(/'/g, "'"); +} + +function validateCell(label: string, grading: unknown): GradeCompareCell { + if (!normalizeHfColorGrading(grading)) { + throw new Error(`Invalid color grading for cell "${label}"`); + } + return { label, grading }; +} + +function serializedGradingForCell(cell: GradeCompareCell): string { + const normalized = normalizeHfColorGrading(cell.grading); + if (!normalized) { + throw new Error(`Invalid color grading for cell "${cell.label}"`); + } + return serializeHfColorGrading(normalized); +} + +function lutSrcFromGrading(grading: unknown): string | null { + if (!isRecord(grading) || !hasOwn(grading, "lut")) return null; + const lut = grading.lut; + if (typeof lut === "string" && lut.trim()) return lut.trim(); + if (!isRecord(lut)) return null; + const src = lut.src; + return typeof src === "string" && src.trim() ? src.trim() : null; +} + +function rewriteGradingLutSrc(grading: unknown, src: string): unknown { + if (!isRecord(grading) || !hasOwn(grading, "lut")) return grading; + const lut = grading.lut; + const next = cloneRecord(grading); + if (typeof lut === "string") { + next.lut = { src }; + return next; + } + if (isRecord(lut)) { + const nextLut = cloneRecord(lut); + nextLut.src = src; + next.lut = nextLut; + } + return next; +} + +export function parseGradesFile(filePath: string): GradeCompareCell[] { + if (!existsSync(filePath)) { + throw new Error(`Grades file not found: ${filePath}`); + } + + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(filePath, "utf-8")); + } catch (err) { + throw new Error(`Could not parse grades JSON: ${normalizeErrorMessage(err)}`); + } + + if (!Array.isArray(parsed)) { + throw new Error("Grades file must be a JSON array of { label, grading } objects"); + } + + return parsed.map((entry, index) => { + if (!isRecord(entry)) { + throw new Error(`Grade entry ${index + 1} must be an object with label and grading`); + } + const label = entry.label; + if (typeof label !== "string" || !label.trim()) { + throw new Error(`Grade entry ${index + 1} must have a non-empty string label`); + } + if (!hasOwn(entry, "grading")) { + throw new Error(`Grade entry "${label}" must include a grading value`); + } + return validateCell(label, entry.grading); + }); +} + +export function resolveLutCells(luts: string): GradeCompareCell[] { + const paths = luts + .split(",") + .map((part) => part.trim()) + .filter(Boolean); + if (paths.length === 0) { + throw new Error("--luts must include at least one LUT path"); + } + return paths.map((lutPath) => + validateCell(basename(lutPath, extname(lutPath)), { lut: { src: lutPath } }), + ); +} + +// The ungraded frame as a leading reference cell. An empty grading object +// normalizes to inactive, so the runtime renders the source image untouched — +// giving the agent a baseline to judge every candidate look against. +export function prependBaselineCell(cells: GradeCompareCell[]): GradeCompareCell[] { + return [validateCell("original", {}), ...cells]; +} + +export function parseGradeCompareArgs(args: { + for?: unknown; + grades?: unknown; + luts?: unknown; + project?: unknown; + out?: unknown; + json?: unknown; +}): ParsedGradeCompareArgs { + const frameArg = readOptionalString(args.for); + if (!frameArg) throw new Error("--for is required"); + + const gradesArg = readOptionalString(args.grades); + const lutsArg = readOptionalString(args.luts); + if (Boolean(gradesArg) === Boolean(lutsArg)) { + throw new Error("Exactly one of --grades or --luts is required"); + } + + const projectDir = resolve(readOptionalString(args.project) ?? process.cwd()); + const framePath = resolveFromBase(projectDir, frameArg); + const outPath = resolveFromBase(projectDir, readOptionalString(args.out) ?? "grade-compare.png"); + + return { + framePath, + projectDir, + outPath, + source: gradesArg + ? { kind: "grades", path: resolveFromBase(projectDir, gradesArg) } + : { kind: "luts", value: lutsArg ?? "" }, + json: args.json === true, + }; +} + +function gridMetrics( + cellCount: number, + frameWidth: number, + frameHeight: number, +): { + columns: number; + rows: number; + cellImageWidth: number; + cellImageHeight: number; + width: number; + height: number; +} { + const columns = Math.max(1, Math.min(MAX_COLUMNS, Math.ceil(Math.sqrt(cellCount)))); + const rows = Math.ceil(cellCount / columns); + const cellImageWidth = DEFAULT_CELL_WIDTH; + const aspect = frameHeight > 0 && frameWidth > 0 ? frameHeight / frameWidth : 9 / 16; + const cellImageHeight = Math.max(1, Math.round(cellImageWidth * aspect)); + return { + columns, + rows, + cellImageWidth, + cellImageHeight, + width: columns * cellImageWidth + (columns + 1) * GRID_PADDING, + height: rows * (cellImageHeight + LABEL_HEIGHT) + (rows + 1) * GRID_PADDING, + }; +} + +export function buildGradeCompareHtml(options: GradeCompareHtmlOptions): string { + if (options.cells.length === 0) { + throw new Error("At least one grade cell is required"); + } + const metrics = gridMetrics(options.cells.length, options.frameWidth, options.frameHeight); + + const cellHtml = options.cells + .map((cell, index) => { + const serialized = escapeSingleQuotedAttr(serializedGradingForCell(cell)); + const label = escapeXml(cell.label); + const row = Math.floor(index / metrics.columns); + const col = index % metrics.columns; + const left = GRID_PADDING + col * (metrics.cellImageWidth + GRID_PADDING); + const top = GRID_PADDING + row * (metrics.cellImageHeight + LABEL_HEIGHT + GRID_PADDING); + return `
+
${label}
+ ${label} +
`; + }) + .join("\n"); + + return ` + + + + + HyperFrames Grade Compare + + + + +
+${cellHtml} + +
+ + +`; +} + +function frameFileNameForPath(framePath: string): string { + const ext = extname(framePath).toLowerCase(); + if (ext === ".jpg" || ext === ".jpeg") return `frame${ext}`; + return "frame.png"; +} + +export async function prepareGradeCompareTempProject(opts: { + projectDir: string; + framePath: string; + frameBuffer: Buffer; + cells: readonly GradeCompareCell[]; + frameWidth: number; + frameHeight: number; + frameFileName?: string; +}): Promise { + const tempDir = mkdtempSync(join(tmpdir(), "hf-grade-compare-")); + try { + const frameFileName = opts.frameFileName ?? frameFileNameForPath(opts.framePath); + writeFileSync(join(tempDir, frameFileName), opts.frameBuffer); + + let lutIndex = 0; + const stagedCells = opts.cells.map((cell) => { + const lutSrc = lutSrcFromGrading(cell.grading); + if (!lutSrc) return cell; + + const sourcePath = resolveFromBase(opts.projectDir, lutSrc); + if (!existsSync(sourcePath)) { + throw new Error(`LUT file not found for "${cell.label}": ${sourcePath}`); + } + const lutExt = extname(sourcePath) || ".cube"; + const stagedName = `lut-${lutIndex}${lutExt}`; + lutIndex += 1; + copyFileSync(sourcePath, join(tempDir, stagedName)); + return { + label: cell.label, + grading: rewriteGradingLutSrc(cell.grading, stagedName), + }; + }); + + const html = buildGradeCompareHtml({ + cells: stagedCells, + frameSrc: frameFileName, + frameWidth: opts.frameWidth, + frameHeight: opts.frameHeight, + }); + writeFileSync(join(tempDir, "index.html"), html); + return { tempDir, html, cells: stagedCells }; + } catch (err) { + rmSync(tempDir, { recursive: true, force: true }); + throw err; + } +} + +function isVideoPath(filePath: string): boolean { + const ext = extname(filePath).toLowerCase(); + return [".mp4", ".mov", ".m4v", ".webm", ".mkv", ".avi", ".mpeg", ".mpg", ".ogv"].includes(ext); +} + +async function extractVideoFrameToBuffer(videoPath: string): Promise { + const tmp = mkdtempSync(join(tmpdir(), "hf-grade-compare-frame-")); + const outPath = join(tmp, "frame.png"); + try { + const ffmpegPath = findFFmpeg(); + if (!ffmpegPath) return null; + const args = [ + "-hide_banner", + "-loglevel", + "error", + "-ss", + "0", + "-i", + videoPath, + "-frames:v", + "1", + "-q:v", + "2", + "-y", + outPath, + ]; + const result = await runFfmpegOnce(ffmpegPath, args, FFMPEG_EXTRACT_TIMEOUT_MS); + if (result.timedOut) { + throw new Error(`ffmpeg timed out extracting first frame from ${videoPath}`); + } + if (result.code !== 0 || !existsSync(outPath)) { + const detail = result.stderr.trim() ? `: ${result.stderr.trim()}` : ""; + throw new Error(`ffmpeg could not extract first frame from ${videoPath}${detail}`); + } + return readFileSync(outPath); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +} + +async function loadReferenceFrame(framePath: string): Promise { + if (!existsSync(framePath)) { + throw new Error(`Reference frame not found: ${framePath}`); + } + + const buffer = isVideoPath(framePath) + ? await extractVideoFrameToBuffer(framePath) + : readFileSync(framePath); + if (!buffer) { + throw new Error(`Could not extract a frame from video: ${framePath}`); + } + + const metadata = await sharp(buffer).metadata(); + if (!metadata.width || !metadata.height) { + throw new Error(`Could not read reference frame dimensions: ${framePath}`); + } + + return { + buffer, + width: metadata.width, + height: metadata.height, + stagedName: isVideoPath(framePath) ? "frame.png" : frameFileNameForPath(framePath), + }; +} + +async function captureGradeCompareSheet(projectDir: string, timeoutMs: number): Promise { + const { bundleToSingleHtml } = await import("@hyperframes/core/compiler"); + + const html = await bundleToSingleHtml(projectDir); + const server = await serveStaticProjectHtml(projectDir, html); + const sheetPath = join(projectDir, "grade-compare.png"); + + try { + const { browser: chromeBrowser, page } = await openSettledCompositionPage(html, server.url, { + renderReadyTimeoutMs: timeoutMs, + renderReadyWarningSuffix: "grade comparison may be inaccurate", + }); + + try { + await page.screenshot({ path: sheetPath, type: "png" }); + return sheetPath; + } finally { + await chromeBrowser.close(); + } + } finally { + await server.close(); + } +} + +function printJson(payload: object): void { + console.log(JSON.stringify(withMeta(payload), null, 2)); +} + +export default defineCommand({ + meta: { + name: "grade-compare", + description: "Render candidate color grades onto a reference frame as one comparison PNG", + }, + args: { + for: { + type: "string", + description: "Reference image path, or a video path to sample at t=0", + required: true, + }, + grades: { + type: "string", + description: "JSON array of { label, grading } candidate grades", + }, + luts: { + type: "string", + description: "Comma-separated .cube LUT paths to compare", + }, + project: { + type: "string", + description: "Base directory for relative --for, --grades, and LUT paths (default: cwd)", + }, + out: { + type: "string", + description: "Output PNG path (default: /grade-compare.png)", + }, + json: { + type: "boolean", + description: "Output result as JSON", + default: false, + }, + baseline: { + type: "boolean", + description: + "Prepend the ungraded frame as an 'original' reference cell (--no-baseline to omit)", + default: true, + }, + }, + async run({ args }) { + const jsonRequested = args.json === true; + let preparedDir: string | null = null; + try { + const parsed = parseGradeCompareArgs({ + for: args.for, + grades: args.grades, + luts: args.luts, + project: args.project, + out: args.out, + json: args.json, + }); + let cells = + parsed.source.kind === "grades" + ? parseGradesFile(parsed.source.path) + : resolveLutCells(parsed.source.value); + if (cells.length === 0) { + throw new Error("At least one grade candidate is required"); + } + if (args.baseline !== false) { + cells = prependBaselineCell(cells); + } + + if (!parsed.json) { + console.log( + `${c.accent("◆")} Rendering ${cells.length} grade candidates from ${c.accent(basename(parsed.framePath))}`, + ); + } + + const frame = await loadReferenceFrame(parsed.framePath); + const prepared = await prepareGradeCompareTempProject({ + projectDir: parsed.projectDir, + framePath: parsed.framePath, + frameBuffer: frame.buffer, + frameWidth: frame.width, + frameHeight: frame.height, + frameFileName: frame.stagedName, + cells, + }); + preparedDir = prepared.tempDir; + + const tempSheet = await captureGradeCompareSheet(prepared.tempDir, 5000); + mkdirSync(dirname(parsed.outPath), { recursive: true }); + copyFileSync(tempSheet, parsed.outPath); + + const sheet = displayPath(parsed.projectDir, parsed.outPath); + if (parsed.json) { + printJson({ ok: true, sheet, cells: prepared.cells.length }); + } else { + console.log(`\n${c.success("◇")} Grade comparison saved to ${sheet}`); + } + } catch (err) { + const message = normalizeErrorMessage(err); + if (jsonRequested) { + printJson({ ok: false, error: message }); + } else { + console.error(`\n${c.error("✗")} Grade compare failed: ${message}`); + } + process.exit(1); + } finally { + if (preparedDir) { + rmSync(preparedDir, { recursive: true, force: true }); + } + } + }, +}); diff --git a/packages/cli/src/commands/snapshot.ts b/packages/cli/src/commands/snapshot.ts index 42d408659f..d5d7b46b3f 100644 --- a/packages/cli/src/commands/snapshot.ts +++ b/packages/cli/src/commands/snapshot.ts @@ -1,12 +1,11 @@ // fallow-ignore-file complexity -import { spawn } from "node:child_process"; import { defineCommand } from "citty"; import { existsSync, mkdtempSync, readFileSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { resolve, join, relative, isAbsolute, basename } from "node:path"; +import { openSettledCompositionPage, runFfmpegOnce } from "../capture/captureCompositionFrame.js"; import { resolveProject } from "../utils/project.js"; import { normalizeErrorMessage } from "../utils/errorMessage.js"; -import { resolveCompositionViewportFromHtml } from "../utils/compositionViewport.js"; import { serveStaticProjectHtml } from "../utils/staticProjectServer.js"; import { c } from "../ui/colors.js"; import { findFFmpeg } from "../browser/ffmpeg.js"; @@ -67,46 +66,25 @@ async function extractVideoFrameToBuffer( try { const ffmpegPath = findFFmpeg(); if (!ffmpegPath) return null; - const result = await new Promise<{ code: number | null; stderr: string; timedOut: boolean }>( - (resolvePromise) => { - // `-ss` before `-i` performs a fast keyframe seek; adequate for snapshot accuracy - // (±1 frame) and orders of magnitude faster than the decode-and-scan alternative. - const args = ["-hide_banner", "-loglevel", "error"]; - if (useVp9AlphaDecoder) { - args.push("-c:v", "libvpx-vp9"); - } - args.push( - "-ss", - String(Math.max(0, timeSeconds)), - "-i", - videoPath, - "-frames:v", - "1", - "-q:v", - "2", - "-y", - outPath, - ); - const ff = spawn(ffmpegPath, args); - let stderr = ""; - let timedOut = false; - const timer = setTimeout(() => { - timedOut = true; - ff.kill("SIGTERM"); - }, FFMPEG_EXTRACT_TIMEOUT_MS); - ff.stderr.on("data", (d: Buffer) => { - stderr += d.toString(); - }); - ff.on("close", (code) => { - clearTimeout(timer); - resolvePromise({ code, stderr, timedOut }); - }); - ff.on("error", () => { - clearTimeout(timer); - resolvePromise({ code: null, stderr: "ffmpeg spawn failed", timedOut }); - }); - }, + // `-ss` before `-i` performs a fast keyframe seek; adequate for snapshot accuracy + // (±1 frame) and orders of magnitude faster than the decode-and-scan alternative. + const args = ["-hide_banner", "-loglevel", "error"]; + if (useVp9AlphaDecoder) { + args.push("-c:v", "libvpx-vp9"); + } + args.push( + "-ss", + String(Math.max(0, timeSeconds)), + "-i", + videoPath, + "-frames:v", + "1", + "-q:v", + "2", + "-y", + outPath, ); + const result = await runFfmpegOnce(ffmpegPath, args, FFMPEG_EXTRACT_TIMEOUT_MS); if (result.code !== 0 || result.timedOut || !existsSync(outPath)) return null; return readFileSync(outPath); } finally { @@ -193,7 +171,6 @@ async function captureSnapshots( }, ): Promise { const { bundleToSingleHtml } = await import("@hyperframes/core/compiler"); - const { ensureBrowser } = await import("../browser/manager.js"); const numFrames = opts.frames ?? 5; @@ -203,75 +180,12 @@ async function captureSnapshots( const savedPaths: string[] = []; try { - const browser = await ensureBrowser(); - const puppeteer = await import("puppeteer-core"); - const chromeBrowser = await puppeteer.default.launch({ - headless: true, - executablePath: browser.executablePath, - args: [ - "--no-sandbox", - "--disable-gpu", - "--disable-dev-shm-usage", - "--enable-webgl", - "--use-gl=angle", - "--use-angle=swiftshader", - ], + const { browser: chromeBrowser, page } = await openSettledCompositionPage(html, server.url, { + renderReadyTimeoutMs: opts.timeout ?? 5000, + renderReadyWarningSuffix: "snapshots may be inaccurate", }); try { - const page = await chromeBrowser.newPage(); - await page.setViewport(resolveCompositionViewportFromHtml(html)); - - await page.goto(server.url, { - waitUntil: "domcontentloaded", - timeout: 10000, - }); - - // __renderReady is set after the player is constructed AND the root - // timeline is bound — waiting for it guarantees renderSeek will work. - const timeoutMs = opts.timeout ?? 5000; - const runtimeReady = await page - .waitForFunction(() => !!(window as any).__renderReady, { timeout: timeoutMs }) - .then(() => true) - .catch(() => false); - - if (!runtimeReady) { - console.warn( - `\n ${c.warn("⚠")} Runtime did not become render-ready within ${timeoutMs}ms — snapshots may be inaccurate`, - ); - } - - // Wait for shader transition pre-rendering (HyperShader IndexedDB hydration). - // Uses the ready state flag as primary signal, with the loading overlay - // display:none as a fallback for older builds. - await page - .waitForFunction( - () => { - const win = window as unknown as { - __hf?: { shaderTransitions?: Record }; - }; - const shaderTransitions = win.__hf?.shaderTransitions; - if (shaderTransitions !== undefined) { - return Object.values(shaderTransitions).every((s) => s.ready === true); - } - const overlay = document.querySelector( - "[data-hyper-shader-loading]", - ) as HTMLElement | null; - if (!overlay) return true; - return window.getComputedStyle(overlay).display === "none"; - }, - { timeout: 90_000 }, - ) - .catch(() => { - console.warn(` ${c.warn("⚠")} Shader transitions did not finish pre-rendering`); - }); - - // Wait for fonts to finish loading before capturing - await page.evaluate(() => document.fonts.ready).catch(() => {}); - - // Extra settle time for media and animations to initialize - await new Promise((r) => setTimeout(r, 1500)); - // Font verification — split into loaded / errored / unused. Only status // "error" is a real failure; a face still "unloaded"/"loading" after // document.fonts.ready + the settle wait was simply never requested by any diff --git a/packages/cli/src/help.ts b/packages/cli/src/help.ts index 41f22a6acf..d0512ab4ad 100644 --- a/packages/cli/src/help.ts +++ b/packages/cli/src/help.ts @@ -41,6 +41,10 @@ const GROUPS: Group[] = [ ["inspect", "Inspect rendered visual layout across the timeline"], ["keyframes", "Inspect keyframes and render onion-shot diagnostics"], ["snapshot", "Capture key frames as PNG screenshots for visual verification"], + [ + "grade-compare", + "Render candidate color grades onto a reference frame as one labeled comparison PNG", + ], ["info", "Print project metadata"], ["compositions", "List all compositions in a project"], ["docs", "View inline documentation in the terminal"], diff --git a/skills-manifest.json b/skills-manifest.json index e1e6ae374a..ca3c2d2393 100644 --- a/skills-manifest.json +++ b/skills-manifest.json @@ -18,7 +18,7 @@ "files": 1 }, "hyperframes": { - "hash": "24ec436b52957fcc", + "hash": "43717f4e64cae9c1", "files": 1 }, "hyperframes-animation": { @@ -26,7 +26,7 @@ "files": 116 }, "hyperframes-cli": { - "hash": "493e59cafaab3d10", + "hash": "23d4f4d24f4df747", "files": 7 }, "hyperframes-core": { @@ -46,8 +46,8 @@ "files": 10 }, "media-use": { - "hash": "22d8da57c4af6ddd", - "files": 103 + "hash": "243277506847ddf3", + "files": 115 }, "motion-graphics": { "hash": "96ed2f7d8051b009", diff --git a/skills/hyperframes-cli/SKILL.md b/skills/hyperframes-cli/SKILL.md index 5cd24cfced..9acbe2f49c 100644 --- a/skills/hyperframes-cli/SKILL.md +++ b/skills/hyperframes-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: hyperframes-cli -description: HyperFrames CLI dev loop. Use when running npx hyperframes init, add, catalog, capture, lint, validate, inspect, layout, snapshot, preview, play, render, publish, feedback, lambda, doctor, browser, info, upgrade, skills, compositions, docs, benchmark, telemetry, transcribe, tts, or remove-background, or when troubleshooting the HyperFrames build/render environment. Entry point for AWS Lambda cloud rendering (`hyperframes lambda deploy / render / progress / destroy / policies / sites`). +description: HyperFrames CLI dev loop. Use when running npx hyperframes init, add, catalog, capture, lint, validate, inspect, layout, snapshot, grade-compare, preview, play, render, publish, feedback, lambda, doctor, browser, info, upgrade, skills, compositions, docs, benchmark, telemetry, transcribe, tts, or remove-background, or when troubleshooting the HyperFrames build/render environment. Entry point for AWS Lambda cloud rendering (`hyperframes lambda deploy / render / progress / destroy / policies / sites`). --- # HyperFrames CLI @@ -24,7 +24,7 @@ Everything runs through `npx hyperframes` unless project instructions specify a Run lint, validate, and inspect before preview. `lint` catches missing `data-composition-id`, overlapping tracks, and unregistered timelines. `validate` loads the composition in headless Chrome and reports runtime console errors plus WCAG contrast issues. `inspect` seeks through the timeline and reports text spilling out of bubbles/containers or off the canvas — and, when a `*.motion.json` sidecar is present, verifies motion intent (entrances firing under seek, stagger order, in-frame, liveness) against that same seeked timeline. -For motion-heavy work, prefer snapshot-driven iteration and a `*.motion.json` sidecar — see `references/lint-validate-inspect.md` for the discipline and motion-verification spec. +For motion-heavy work, prefer snapshot-driven iteration and a `*.motion.json` sidecar — see `references/lint-validate-inspect.md` for the discipline and motion-verification spec. For color-grade selection, use `npx hyperframes grade-compare --for --grades grades.json` (or `--luts a.cube,b.cube`) to render every candidate through the real WebGL grading runtime into one labeled PNG before choosing the winner. ## Agent Conventions diff --git a/skills/hyperframes/SKILL.md b/skills/hyperframes/SKILL.md index 5a674a6539..b0cf4e210f 100644 --- a/skills/hyperframes/SKILL.md +++ b/skills/hyperframes/SKILL.md @@ -31,7 +31,7 @@ Atomic capabilities you load **on demand** — not full workflows; they never ow | **Animate** — atomic motion, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU) | `/hyperframes-animation` | | **Author seek-safe keyframes** — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth, plus `hyperframes keyframes` diagnostics | `/hyperframes-keyframes` | | **Creative direction** — `frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive | `/hyperframes-creative` | -| **Media** — resolve/generate BGM, SFX, image, icon, brand logo, voice; TTS voiceover, transcription, background removal, captions; cross-project reuse | `/media-use` | +| **Media** — resolve/generate BGM, SFX, image, icon, brand logo, voice, color grade, LUT; TTS voiceover, transcription, background removal, captions; cross-project reuse | `/media-use` | | **CLI dev loop** — init, lint, validate, inspect, preview, render, publish, doctor | `/hyperframes-cli` | | **Install registry blocks / components** (`hyperframes add`) | `/hyperframes-registry` | | **Import Figma content** — assets, tokens, components, storyboards→reconstructed motion (REST/CLI); Motion (MCP), shaders (MCP source / native export) | `/figma` | diff --git a/skills/media-use/SKILL.md b/skills/media-use/SKILL.md index 40e94fcc74..351eb21f00 100644 --- a/skills/media-use/SKILL.md +++ b/skills/media-use/SKILL.md @@ -1,6 +1,6 @@ --- name: media-use -description: Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, or voice into a frozen local file + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Keeps search noise on disk, hands the agent a path. Use for any audio, image, icon, voiceover, caption, or media-asset need. +description: Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Keeps search noise on disk, hands the agent one path or block. Use for any audio, image, icon, logo, voiceover, caption, color-grading, or media-asset need. --- # media-use @@ -21,13 +21,36 @@ HyperFrames owns media _playback_; media-use owns everything else. Each row is e | No transcript-driven cutting | `scripts/transcript-cut.mjs` compiles word-timestamp edits into cut lists | | No auto-duck / publish loudness | `scripts/audio-duck.mjs` + `references/operations.md` loudnorm/sidechain recipes | | No cross-project memory | global content-addressed cache + auto-promote (`~/.media`) | +| No color-grade authoring | `resolve --type grade` emits a paste-ready `data-color-grading` block; `resolve --type lut` freezes validated `.cube` files | | No image generation | RAM-graded local mflux (FLUX) via `scripts/lib/mflux-provider.mjs`, codex `image_gen` upsell (`scripts/lib/codex-provider.mjs`) | | No video generation | spec-gated local LTX (`videogen` in `scripts/lib/local-models.mjs`); `heygen video create` avatar upsell | | Weak local-model defaults | free-usage HeyGen first (TTS, bg-removal) via the `heygen` CLI; local open-source only as an opt-out fallback (`scripts/lib/local-run.mjs`) | ## When to use -Call `resolve` whenever a composition needs media: background music, sound effects, images, icons, brand logos, or voice. For voiceover / TTS, music, SFX, and caption timing, use the **audio engine** (below); background removal is delegated to the `hyperframes` CLI; transcription defaults to Parakeet (better than whisper.cpp: 6.05% vs 7.44% WER, 5-10x faster) via `scripts/transcribe.mjs`, with whisper.cpp auto-fallback (see `references/operations.md`). For cutting / reframing / transforming existing media, see `references/operations.md`. media-use searches the HeyGen catalog first, freezes the best match locally, registers it in a manifest, and hands the agent one line; all search noise stays on disk. +Call `resolve` whenever a composition needs media: background music, sound effects, images, icons, brand logos, voice, a color grade, or a LUT. For voiceover / TTS, music, SFX, and caption timing, use the **audio engine** (below); background removal is delegated to the `hyperframes` CLI; transcription defaults to Parakeet (better than whisper.cpp: 6.05% vs 7.44% WER, 5-10x faster) via `scripts/transcribe.mjs`, with whisper.cpp auto-fallback (see `references/operations.md`). For cutting / reframing / transforming existing media, see `references/operations.md`. media-use searches the HeyGen catalog first for media files, resolves official logos through the logo cascade, uses local deterministic color grading for `grade`/`lut`, freezes the best match locally when a file is needed, registers it in a manifest, and hands the agent one line; all search noise stays on disk. + +## Be proactive — run a media opportunity pass + +The human usually can't tell which media would lift the piece. You can. When you build or review a composition, do **one** grounded scan and then **ask once** — don't silently add, and don't nag per asset. + +Surface an opportunity only when a concrete signal is present: + +| Signal detected | Offer | +| ------------------------------------------------------ | ------------------------------------------------------------------------------------------- | +| On-screen text / a script with no voiceover | TTS voiceover (audio engine) | +| Emoji or a `
` styled as an icon | resolve real `icon`s | +| Image that is a placeholder, tiny, or upscaled-looking | a better `image` (and/or upscale — see `references/operations.md`) | +| Hard scene cuts / transitions with no sound | transition `sfx` | +| A piece over ~10s with no music bed | `bgm` | +| Footage that reads under/over-exposed or color-cast | a corrective `grade` (analyze with `grade --for`, preview with `hyperframes grade-compare`) | + +Rules that keep this a help, not nagware: + +- **Grounded, not generic.** No signal → no suggestion. Never open with "want better images?". +- **Opinionated + concrete.** Propose the specific fix ("add a VO from your script, swap 3 emoji for real icons, replace the 400×400 hero, whooshes on the 4 cuts"), with defaults chosen — the human just approves **all / some / none**. +- **Once per project.** One consolidated ask, top few highest-value items. Respect "leave it" and don't re-raise. +- **Surface, never silently mutate.** Color grades especially: propose and preview, never auto-apply — a gray-world "correction" ruins an intentional sunset or neon look. ## Resolve @@ -39,14 +62,16 @@ Returns one line: `resolved → (, )` ### Types -| Type | What it finds | Provider | -| ------- | -------------------- | -------------------------------------------------------- | -| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) | -| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog | -| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) | -| `icon` | Icons, symbols | HeyGen asset search (type=icon) | -| `logo` | Official brand marks | svgl → simple-icons → GitHub org avatar → domain favicon | -| `voice` | TTS voiceover | Local Kokoro (free); HeyGen TTS upsell | +| Type | What it finds | Provider / cascade | +| ------- | -------------------------------- | ---------------------------------------------- | +| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) | +| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog | +| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) | +| `icon` | Icons, symbols | HeyGen asset search (type=icon) | +| `logo` | Official brand marks | svgl → simple-icons → GitHub org avatar → domain favicon | +| `voice` | TTS voiceover | Local Kokoro (free); HeyGen TTS upsell | +| `grade` | HyperFrames color-grading blocks | Core preset → bundled LUT → deterministic cube | +| `lut` | Reusable `.cube` LUT files | Bundled LUT → deterministic cube | ### Examples @@ -70,19 +95,28 @@ node /scripts/resolve.mjs --type icon --intent "rocket" --project . # Brand logo (official mark — never redrawn by hand) node /scripts/resolve.mjs --type logo --entity linkedin --intent "LinkedIn logo" --project . # → resolved logo_001 → .media/images/logo_001.svg (logo, official mark) + +# Color grade block +node /scripts/resolve.mjs --type grade --intent "warm daylight" --project . --json +# → {"ok":true,"preset":"warm-daylight","grading":{"preset":"warm-daylight","intensity":1},...} + +# LUT file +node /scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project . +# → resolved lut_001 → .media/luts/lut_001.cube (lut) ``` ### Flags | Flag | Description | | --------------- | ------------------------------------------------------------------------------------ | -| `--type, -t` | Media type: bgm, sfx, image, icon, voice | +| `--type, -t` | Media type: bgm, sfx, image, icon, logo, voice, grade, lut | | `--intent, -i` | What you need (natural language) | | `--entity, -e` | Entity name for cache matching (optional) | | `--project, -p` | Project directory (default: .) | | `--candidates` | List reusable assets (project + global cache) for `--type`; no download, no mutation | | `--reuse ` | Import a specific global-cache asset (by content sha/prefix, from `--candidates`) | | `--from` | Freeze a local file or direct public URL (ingest) | +| `--for` | Analyze a local image/video and add measured adjust suggestions (`grade` only) | | `--local-only` | Offline: skip every network provider (cache + local only) | | `--provider` | Force one generator (e.g. `codex`, `mflux`, `kokoro`, `heygen`) | | `--adopt` | Bulk-import existing assets/ into manifest | @@ -90,7 +124,7 @@ node /scripts/resolve.mjs --type logo --entity linkedin --intent "Lin ## Reuse before you resolve -Before resolving bgm/sfx/image/icon, **check what already exists and reuse it when it fits.** media-use does not semantically match for you — you are the judge. It surfaces candidates; you decide. +Before resolving bgm/sfx/image/icon/logo/grade/lut, **check what already exists and reuse it when it fits.** media-use does not semantically match for you — you are the judge. It surfaces candidates; you decide. ```bash node /scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --candidates --project . @@ -110,6 +144,66 @@ Read the list and judge semantic fit yourself — "upbeat tech launch" ≈ "ener The deterministic floor still runs automatically: an identical (case/whitespace-insensitive) repeat auto-reuses with no `--candidates` step. `--candidates` is only for the semantic layer above that floor — and a fuzzy match is **never** auto-applied; reuse is always your explicit call. On a resolve that misses the floor and is about to fetch, media-use prints a one-line stderr hint when similar cached assets exist, pointing you back here. +## Color grading + +Use `grade` when you need the actual HyperFrames `data-color-grading` value to paste onto an `` or `