Client-side media: GIF-to-video via plugin#76964
Open
adamsilverstein wants to merge 18 commits into
Open
Conversation
This was referenced Apr 1, 2026
adamsilverstein
added a commit
that referenced
this pull request
Apr 2, 2026
Add client-side JPEG XL image processing using a canonical plugin (wp-vips-jxl) that ships the vips-jxl.wasm module on demand, avoiding a 3.1 MB increase to the Gutenberg bundle. The implementation follows the same pattern as FFmpeg WASM (#76964): - Plugin sets window.__vipsJxlConfig on editor pages - Auto-installs via REST API when user uploads JXL and has capabilities - Falls back to server-side processing when plugin unavailable Key changes: - vips: Add setJxlWasmUrl() for runtime JXL WASM URL injection with lazy re-initialization of the vips instance - upload-media: Add jxl-plugin.ts for plugin detection, installation, and config fetching - upload-media: Add JXL to supported MIME types and image formats - upload-media: Add JXL plugin availability check in prepareItem() with graceful degradation Closes #76981
7 tasks
Introduce client-side animated GIF to MP4/WebM conversion using FFmpeg WASM, following the architecture proven in swissspidy/media-experiments. New @wordpress/ffmpeg package: - FFmpeg WASM wrapper with inlined base64 WASM (same approach as @wordpress/vips) - Web Worker architecture via Blob URL pattern - Lazy loaded only when an animated GIF upload is detected Upload pipeline integration: - New TranscodeGif operation type in the upload queue - isAnimatedGif() binary analysis utility for detecting multi-frame GIFs - Concurrency limited to 1 simultaneous video transcode - Graceful fallback when crossOriginIsolated is unavailable Settings: - gifConvert: enable/disable conversion (default: true) - videoOutputFormat: output format preference (default: video/mp4) Closes #76942
Address CodeRabbit review findings: - Add serialization lock to prevent concurrent operations from corrupting shared FFmpeg MEMFS state - Use unique filenames per operation (keyed by item ID) - Add cancellation checks after async boundaries (lock wait, core init) - Fix output.buffer to use proper slice to avoid including bytes outside the Uint8Array view range
Move the heavy FFmpeg WASM binary (~31MB) out of Gutenberg into a separate canonical WordPress plugin (wp-ffmpeg-wasm). This eliminates the ~33MB base64-inlined WASM from the Gutenberg bundle. Architecture: - The wp-ffmpeg-wasm plugin ships the WASM binary as a static asset and exposes URLs via window.__ffmpegWasmConfig and a REST endpoint - Gutenberg's @wordpress/ffmpeg package becomes a thin wrapper that loads WASM from the plugin's URLs instead of inlining - On first animated GIF upload, Gutenberg silently installs the plugin via the REST API (same pattern as Connectors plugin installation) - Graceful fallback: if plugin can't be installed, GIF uploads as-is Key changes: - Remove @ffmpeg/core dependency from packages/ffmpeg - FFmpeg index.ts accepts coreUrl/wasmUrl params instead of inlining - New ffmpeg-plugin.ts utility handles plugin detection and installation - Remove gutenberg_enqueue_ffmpeg_loader from client-assets.php - Add @wordpress/core-data dependency to upload-media for plugin install The canonical plugin (wp-ffmpeg-wasm) is created separately and will be hosted on WordPress.org.
…re core.reset() reliability - Add .gitignore for auto-generated worker-code.ts (matching vips pattern) - Update README to reflect canonical plugin architecture instead of base64 inlining - Consolidate duplicate FFmpegConfig type: re-export from @wordpress/ffmpeg - Add @wordpress/ffmpeg as dependency of @wordpress/upload-media - Move core.reset() and file cleanup into finally block for reliability - Add ffmpeg reference to upload-media tsconfig
…ation Tests cover all degradation paths: plugin already active, user cannot install, successful install with REST config fetch, install failure, and REST fetch failure. Also verifies session-level caching behavior.
adamsilverstein
force-pushed
the
gif-to-video-canonical-plugin
branch
from
c0b5ea7 to
950ce66
Compare
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
adamsilverstein
added
[Status] In Progress
…ation - Clear cached ffmpegPromise on initialization failure so subsequent calls can retry instead of returning the rejected promise permanently - Deduplicate concurrent ensureFFmpegAvailable() calls with an in-flight promise to prevent parallel plugin installation attempts
5 tasks
Replace import of store constant from core-data with inline store name string 'core'. This avoids the circular reference chain: upload-media → core-data → block-editor → upload-media.
adamsilverstein
requested review from
ajitbohra,
fabiankaegy and
juanmaguitar
as code owners
…e and ArrayBuffer types - Define FFmpegConfig interface locally instead of re-exporting from @wordpress/ffmpeg to avoid build failures when ffmpeg has TS errors - Replace WorkerGlobalScope type with inline type assertion - Add ArrayBuffer type assertion for Uint8Array.buffer.slice()
…tring store name When using a string store name instead of the store descriptor object, TypeScript returns 'unknown'. Add inline type assertions to fix.
|
Size Change: +2.75 kB (+0.03%) Total Size: 7.97 MB 📦 View Changed
ℹ️ View Unchanged
|
|
Flaky tests detected in d6903bd. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/25953628061
|
Remove the separate @wordpress/ffmpeg package and move FFmpeg conversion logic into upload-media as an inline classic worker. The worker loads FFmpeg WASM from the wp-ffmpeg-wasm plugin's assets URL via importScripts() at runtime, keeping the heavy binary out of Gutenberg. Wire up the full GIF-to-video pipeline: animated GIF detection via isAnimatedGif(), on-demand plugin installation via ensureFFmpegAvailable(), and conversion via ffmpegConvertGifToVideo() in a web worker with concurrency limited to 1 operation.
adamsilverstein
changed the title
This was referenced Apr 20, 2026
…al-plugin # Conflicts: # packages/upload-media/src/store/private-actions.ts
adamsilverstein
requested review from
andrewserong and
swissspidy
and removed request for
ajitbohra,
fabiankaegy and
juanmaguitar
Contributor
|
Just adding a similar comment to #76990 (comment): overall I like the idea of figuring out how to exclude the large wasm from the Gutenberg plugin / bundle itself, but there are some potential drawbacks as it effectively means only sites that can install plugins or admin users are able to use the feature. Are there any ways to mitigate those limitations? My main concern is about the idea of adding features that sound like they should just work, but wind up only being accessible to a limited number of sites or users. Apologies if I'm overthinking this! |
Member
Author
|
@andrewserong - I replied to the other ticket about possible approaches. It looks like ffmpeg adds 14MB which is more than vips, so we should be a bit MORE wary of bundling it - see #76946 (comment) |
The plugin-served FFmpeg core (@ffmpeg/core 0.12.x) ignores a user-supplied locateFile: during init it overwrites Module.locateFile with its own implementation that reads the WASM URL from a base64 JSON fragment on mainScriptUrlOrBlob. The worker previously called createFFmpegCore() with neither, so the core fell back to fetching a bare 'ffmpeg-core.wasm' which fails inside the Blob-URL worker (no base URL). Thread the plugin's wasmUrl through to the worker and advertise it via the mainScriptUrlOrBlob fragment so conversion actually works. Also harden the worker: - Copy output via new Uint8Array(output).slice().buffer. FS.readFile() views MEMFS, which under crossOriginIsolated FFmpeg builds can be a SharedArrayBuffer that cannot be transferred via postMessage. Slicing the typed array always yields a standalone, transferable ArrayBuffer. - Wrap exec/read in try/finally so a failed run still unlinks MEMFS files and calls reset(), preventing stale state from corrupting the next message that reuses the cached module. - Drop the dead maxDimensions parameter and its scaling branch; nothing in the upload pipeline ever supplied a value.
The catch block in transcodeGifItem replaces the underlying error with a generic user-facing message, leaving the actual failure (worker load, FFmpeg core init, codec setup) invisible. Log the cause so it can be diagnosed from the browser console without changing the notification.
Adds an editor.BlockEdit filter that watches a core/image block's attachment record. When client-side animated GIF to video conversion (@wordpress/upload-media) replaces the uploaded GIF with an MP4/WebM, the attachment mime_type becomes video/*. This swaps the core/image block for a core/video block configured to mirror the original GIF behavior (muted, looping, autoplaying, inline, no controls), so the end-to-end feature produces a usable block instead of a video URL stranded in an image block.
The wp-ffmpeg-wasm plugin ships a self-contained core with the WASM inlined as a data: URI in wasmBinaryFile. The core's isDataURI guard loads that directly and never calls locateFile, so the mainScriptUrlOrBlob fragment is never consulted on that path — unconditionally passing it (with an empty wasmURL) was inert but the accompanying comment misdescribed why it was there. Make the fragment conditional on a non-empty wasmUrl so it is only advertised when the plugin actually serves a separate WASM binary (@ffmpeg/core 0.12.x ignores locateFile and reads the URL from that fragment in that case), and correct the comment to match reality.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Alternative to #76946 that moves the heavy FFmpeg WASM binary (~31MB) out of Gutenberg into a separate canonical WordPress plugin (
wp-ffmpeg-wasm).Why a separate plugin? FFmpeg core WASM is ~25MB (~33MB as base64). Inlining it in Gutenberg increases the plugin size for all users, even though most never upload animated GIFs. A canonical plugin keeps the WASM only on sites that need it.
How it works
isAnimatedGif()binary analysiswindow.__ffmpegWasmConfigexists (set by the plugin on page load)wp-ffmpeg-wasmvia REST API (saveEntityRecord— same pattern as Connectors)/wp-ffmpeg-wasm/v1/config)The canonical plugin (
wp-ffmpeg-wasm)A minimal WordPress plugin (~50 lines PHP) that:
ffmpeg-core.wasm(31MB) andffmpeg-core.jsas static assetswindow.__ffmpegWasmConfigon editor pages viawp_add_inline_scriptGraceful degradation
Relationship to #76946
This PR builds on #76946 (inlined approach) and refactors it:
@ffmpeg/corenpm dependency from@wordpress/ffmpegffmpeg-plugin.tshandles plugin detection and on-demand installationgutenberg_enqueue_ffmpeg_loaderfrom PHP (plugin handles it)Closes #76942
Test plan
wp-ffmpeg-wasmplugin manually, verifywindow.__ffmpegWasmConfigis setinstall_plugins→ verify GIF uploads as-is