Client-side media: Add animated GIF to video conversion (MP4/WebM)#76946
Closed
adamsilverstein wants to merge 18 commits into
Closed
Client-side media: Add animated GIF to video conversion (MP4/WebM)#76946adamsilverstein wants to merge 18 commits into
adamsilverstein wants to merge 18 commits into
Conversation
Introduce client-side animated GIF to MP4/WebM conversion using FFmpeg WASM, following the architecture proven in swissspidy/media-experiments. New @wordpress/ffmpeg package: - FFmpeg WASM wrapper with inlined base64 WASM (same approach as @wordpress/vips) - Web Worker architecture via Blob URL pattern - Lazy loaded only when an animated GIF upload is detected Upload pipeline integration: - New TranscodeGif operation type in the upload queue - isAnimatedGif() binary analysis utility for detecting multi-frame GIFs - Concurrency limited to 1 simultaneous video transcode - Graceful fallback when crossOriginIsolated is unavailable Settings: - gifConvert: enable/disable conversion (default: true) - videoOutputFormat: output format preference (default: video/mp4) Closes #76942
Member
Author
WASM Loading Strategy DiscussionThe current implementation inlines the FFmpeg WASM as base64 (matching the Option 1: Inlined base64 (current implementation)The WASM binary is inlined as a base64 data URL at build time, bundled into the worker code string. Pros:
Cons:
Option 2: CDN loading (e.g., jsdelivr, unpkg)Load the WASM from a public CDN at runtime, only when needed. Pros:
Cons:
Option 3: Canonical WordPress plugin with on-demand installationShip FFmpeg WASM as a separate canonical (WordPress.org-hosted) plugin. On first animated GIF upload, upon user confirmation install/activate it in the background, adding FFmpeg capability to the site. Pros:
Cons:
RecommendationFor the initial implementation, the inlined approach (Option 1) is simplest and matches existing patterns. However, given the ~33MB size, Option 3 (canonical plugin) is worth exploring as a follow-up — it keeps everything within the WordPress ecosystem while avoiding the bundle size impact. The upload-media pipeline is already designed for graceful fallback, so the detection/installation flow could be added without changing the core architecture. |
Address CodeRabbit review findings: - Add serialization lock to prevent concurrent operations from corrupting shared FFmpeg MEMFS state - Use unique filenames per operation (keyed by item ID) - Add cancellation checks after async boundaries (lock wait, core init) - Fix output.buffer to use proper slice to avoid including bytes outside the Uint8Array view range
Move the heavy FFmpeg WASM binary (~31MB) out of Gutenberg into a separate canonical WordPress plugin (wp-ffmpeg-wasm). This eliminates the ~33MB base64-inlined WASM from the Gutenberg bundle. Architecture: - The wp-ffmpeg-wasm plugin ships the WASM binary as a static asset and exposes URLs via window.__ffmpegWasmConfig and a REST endpoint - Gutenberg's @wordpress/ffmpeg package becomes a thin wrapper that loads WASM from the plugin's URLs instead of inlining - On first animated GIF upload, Gutenberg silently installs the plugin via the REST API (same pattern as Connectors plugin installation) - Graceful fallback: if plugin can't be installed, GIF uploads as-is Key changes: - Remove @ffmpeg/core dependency from packages/ffmpeg - FFmpeg index.ts accepts coreUrl/wasmUrl params instead of inlining - New ffmpeg-plugin.ts utility handles plugin detection and installation - Remove gutenberg_enqueue_ffmpeg_loader from client-assets.php - Add @wordpress/core-data dependency to upload-media for plugin install The canonical plugin (wp-ffmpeg-wasm) is created separately and will be hosted on WordPress.org.
This reverts commit 7a3db7f.
5 tasks
Member
Author
|
Trying the canonical plugin approach in this PR: #76964 |
Two build issues: - wasmInlinePlugin regex matches paths ending in .wasm, but '@ffmpeg/core/wasm' is a package exports alias that doesn't match. Use the direct path '@ffmpeg/core/dist/umd/ffmpeg-core.wasm' instead. - @ffmpeg/core ESM entry uses import.meta.url which breaks in Blob URL workers. Add wpWorkers resolve mapping to redirect esm/ffmpeg-core.js to umd/ffmpeg-core.js (same pattern as vips).
The @ffmpeg/core package uses an exports map where './wasm' resolves to the .wasm file. The wasmInlinePlugin filter only matched paths ending in '.wasm', missing the '/wasm' alias. Direct paths like '@ffmpeg/core/dist/umd/ffmpeg-core.wasm' were also blocked by esbuild's exports map enforcement. Fix by broadening the wasmInlinePlugin regex to also match '/wasm' suffixes, with a guard that verifies the resolved path actually ends in '.wasm'. This supports both direct .wasm imports (wasm-vips pattern) and package exports aliases (@ffmpeg/core pattern). Also add wpWorkers resolve mapping to redirect the @ffmpeg/core ESM entry (which uses import.meta.url) to the UMD entry.
|
Size Change: +14.1 MB (+177.54%) 🆘 Total Size: 22.1 MB 📦 View Changed
ℹ️ View Unchanged
|
|
Flaky tests detected in c3af01d. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/25953530963
|
7 tasks
Resolve conflicts in: - package-lock.json (regenerated with npm install) - packages/upload-media/src/store/private-actions.ts (combine HEIC handling with GIF-to-video) - packages/upload-media/src/store/types.ts (adopt new mediaFinalize signature alongside gifConvert settings)
adamsilverstein
requested review from
ajitbohra,
fabiankaegy,
juanmaguitar and
spacedmonkey
as code owners
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
adamsilverstein
requested review from
andrewserong and
swissspidy
and removed request for
ajitbohra,
fabiankaegy,
juanmaguitar and
spacedmonkey
adamsilverstein
added
[Status] In Progress
Cover the wrapper at packages/upload-media/src/store/utils/ffmpeg.ts (output file naming for mp4/webm, argument forwarding, module-cache no-ops before load) with a jest moduleNameMapper stub for @wordpress/ffmpeg/worker that matches the existing @wordpress/vips/worker pattern. Extend packages/upload-media/src/store/test/private-actions.js with tests for transcodeGifItem (success dispatches CacheBlobUrl + finishOperation, failure dispatches cancelItem with a GIF_TRANSCODING_ERROR UploadError, early return when item missing) and for the prepareItem animated-GIF branch (mp4 default, webm override, additionalData preservation, gifConvert=false bypass, non-isolated bypass, static-GIF fallthrough). Add selector tests for getActiveVideoProcessingCount and getPendingVideoProcessing covering mixed-queue scenarios.
Member
Author
|
As expected, this PR adds significant weight to the plugin bundle: #76946 (comment) (14 MB currently) |
Member
Author
|
I will work on the backport PR once the Client Side Media feature is re-introduced to core. |
Resolves package-lock.json conflict by regenerating against trunk while preserving the new packages/ffmpeg workspace and its dependencies.
Three review fixes: - Tighten Settings.videoOutputFormat to 'video/mp4' | 'video/webm' so callers can't silently fall through to MP4 by passing 'webm' without the video/ prefix. - Wrap the FFmpeg exec/read block in try/finally so an empty-output throw still unlinks MEMFS files and calls core.reset(), preventing stale state from leaking into the next operation on the shared core. - Drop the unused ffmpegConvertGifToVideo / ffmpegCancelOperations re-exports from the worker-side module; worker.ts already imports the unprefixed names and no other consumer references them.
The catch block in transcodeGifItem replaces the underlying error with a generic user-facing message, leaving the actual failure (worker module load, FFmpeg core init, codec setup) invisible. Log the cause so it can be diagnosed from the browser console without changing the user-visible notification.
The new @wordpress/ffmpeg package was missing the per-package .gitignore and .npmrc that @wordpress/vips has. Without the .gitignore, the build-generated src/worker-code.ts (placeholder or 40MB+ inlined-WASM bundle) could be accidentally committed. .npmrc disables a stray per-package lockfile, matching vips.
@ffmpeg/core 0.12.x does not honor a user-supplied locateFile: during init it unconditionally overwrites Module.locateFile with its own implementation that resolves the WASM URL from a base64-encoded JSON fragment on Module.mainScriptUrlOrBlob. Our locateFile callback was therefore silently discarded, and the core fell back to fetching the bare 'ffmpeg-core.wasm' filename, which fails inside the Blob-URL worker (no base URL) with 'Failed to parse URL from ffmpeg-core.wasm'. Decode the inlined base64 WASM data URL into a Blob URL once and pass it through the mainScriptUrlOrBlob fragment, which is the mechanism the core actually reads.
Member
Author
TestingI tested uploading two animated gif images: svgator-bird-alligator-cta.mp4and rotating-globe.mp4 |
6 tasks
Member
Author
|
I resolved some remaining issues and this is testing well for me now. |
The transcodeGifItem error path intentionally logs the failure cause via console.error for debuggability. jest-console treats any unexpected console.error as a test failure, so assert the expected error explicitly.
adamsilverstein
added
the
No Core Sync Required
Member
|
@adamsilverstein One thing I wanted to look into for media-experiments is to replace ffmpeg with Mediabunny. I haven't really checked it out closely though. |
Contributor
Wow, looks worth looking into given its relatively small bundle size. |
Member
Author
Oh neat! I hadn't seen this library. Seems worth checking out if we can achieve the same results - especially if the package is lighter weight. Looks like it offers great performance. What was appealing to you about it? |
Member
Author
I gave mediabunny a try and as expected it is much lighter weight: #78410 |
Member
It uses native browser APIs, the examples seemed straightforward and has backing from big projects such as Remotion. And of course the smaller bundle size and reduced complexity due to not having to use wasm etc :-) |
7 tasks
Member
Author
|
Closing in favor of #78410, which implements the same animated GIF → video feature using mediabunny + WebCodecs instead of FFmpeg WASM: ~775 KB shipped JS (vs ~14 MB inlined WASM here) and hardware-accelerated encoding. #78410 also now consolidates the render-time swap layer that was stacked on this PR as #78369, so the whole feature lives in one PR. Thanks for the groundwork here — continuing in #78410. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds client-side animated GIF to MP4/WebM conversion using FFmpeg WASM during upload, following the architecture proven in swissspidy/media-experiments.
Why: Animated GIFs are 5-10x larger than equivalent MP4/WebM videos. Converting at upload time dramatically reduces file sizes, improves page performance, and uses hardware-accelerated video playback instead of CPU-bound GIF decoding — all transparently to the user.
This is the base of a 2-PR stack that together deliver the complete, user-testable feature: upload an animated GIF and get an autoplaying, looping video block.
Stack
What's included (this PR)
New
@wordpress/ffmpegpackage — FFmpeg WASM wrapper mirroring@wordpress/vipsarchitecture:@ffmpeg/coreviamainScriptUrlOrBlob(the mechanism the core actually honors; it ignores a userlocateFile)Upload pipeline integration (
@wordpress/upload-media):TranscodeGifoperation type with concurrency limit of 1isAnimatedGif()binary analysis utility (checks GIF magic bytes + frame count)transcodeGifItem()handler following thetranscodeImageItem()patterngifConvert(enable/disable) andvideoOutputFormat(video/mp4orvideo/webm, defaultvideo/mp4)crossOriginIsolatedcontext (SharedArrayBuffer) — graceful fallback to the normal image pipeline otherwisePHP enqueue — Registers
@wordpress/ffmpeg/loaderin the import mapCompleted by the stacked PR (#78369)
core/imageblock is swapped for an autoplaying/loopingcore/videoblock once the converted attachment is availableOut of scope (future work)
core/videoCloses #76942
Testing the complete feature
To test the full end-to-end experience, apply both PRs in this stack (checking out
gif-to-video-block-transformincludes everything inworktree-gif-to-video).npm install && npm run build -- --skip-types(full--skip-typesis required while unrelated trunk TypeScript errors exist; runtime bundles still build).The
@wordpress/ffmpegworker bundle is build-generated and gitignored, so a fresh build is required after pulling —git pullalone is not enough.npm run wp-env start.SharedArrayBufferis available — required for FFmpeg WASM. Without it, GIFs fall back to the normal image pipeline.core/videoblock that autoplays, loops, is muted, and has no controls — matching the original GIF behavior.Screencast
gif.to.video.mp4
Test plan
isAnimatedGif()— animated GIFs, static GIFs, non-GIF files, edge casesnpm run build -- --skip-types, confirm WASM is inlined as a data URL inbuild/modules/ffmpeg/worker.min.js)