Client-side media: Add animated GIF to video conversion via mediabunny#78410
Client-side media: Add animated GIF to video conversion via mediabunny#78410adamsilverstein wants to merge 44 commits into
Conversation
Standalone alternative to PR #76946 (FFmpeg WASM). mediabunny + WebCodecs avoids the ~14MB inlined WASM and the crossOriginIsolated requirement, so conversion runs on installs that are not cross-origin isolated.
Client-side media already requires Document Isolation Policy, so avoiding SharedArrayBuffer is not a differentiator and Firefox is out of scope. The mediabunny win is bundle size and hardware-accelerated encode performance.
Adds worker plumbing for @wordpress/mediabunny following the @wordpress/vips pattern: worker.ts exposes the API via expose(), mediabunny-worker.ts lazily creates a Blob URL Worker and wraps it for main-thread RPC, and loader.ts registers the dynamic module import for the WordPress import map.
Add TranscodeGif operation type and gifConvert/videoOutputFormat settings to the upload-media store. Implement getActiveVideoProcessingCount and getPendingVideoProcessing private selectors (mirroring image-processing equivalents). Create lazy dynamic-import wrapper for @wordpress/mediabunny/worker following the vips util pattern. Wire tsconfig project reference to mediabunny.
Add gutenberg_enqueue_mediabunny_loader() to client-assets.php (parallel to the existing vips loader) so @wordpress/mediabunny/loader is registered in the block editor import map. Add @wordpress/mediabunny entry to docs/manifest.json. Map @wordpress/mediabunny/worker to a jest stub so upload-media unit tests can run without the auto-generated worker-code.ts.
Add gif-to-video.spec.js which uploads the existing 100x80 animated GIF fixture via the image block and asserts the resulting attachment has a video/mp4 or video/webm MIME type after the mediabunny pipeline runs. Mirrors the client-side-media-processing.spec.js harness conventions. Also add packages/mediabunny/src/worker-code.ts to the ESLint global ignores, mirroring the existing vips exclusion - it is auto-generated by the build process and must not be linted.
adamsilverstein
requested review from
ajitbohra,
fabiankaegy,
juanmaguitar and
spacedmonkey
as code owners
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
The crossOriginIsolated check in skipIfGifConversionInactive is a conservative test-environment safety check, not a runtime capability differentiator. Add a comment explaining this so reviewers do not mistake it for feature-gating logic.
adamsilverstein
requested review from
andrewserong and
swissspidy
and removed request for
ajitbohra,
fabiankaegy,
juanmaguitar and
spacedmonkey
adamsilverstein
added
[Type] Enhancement
adamsilverstein
added
the
[Status] In Progress
Swapping every <img> with a companion video at render time broke layouts when the image was nested: galleries (crop-to-fit, lightbox, captions), Media & Text and Cover. Gate the swap by block context: a render_block_core/image filter marks only eligible images - standalone Image blocks whose author has not opted out (preserveAnimatedGif) and that are not inside a Gallery (galleryId context). Media & Text and Cover render their media as the block's own <img>, not a core/image block, so they never reach the filter. wp_content_img_tag swaps only marked images, prefers the static poster companion, and the controller learns the 'animated-video-poster' role. Adds PHP tests for the gating, swap and cleanup.
adamsilverstein
added
Needs PHP backport
|
Updated the labels: removed No Core Sync Required and added Needs PHP backport, since this PR ships real server-side behavior in PHP that needs to exist in Core for the feature to work:
None of this can be delivered through the JS packages alone, so a Core backport is required. Note on timing: the client-side media feature was reverted from Core (WordPress/wordpress-develop#11309), so the backport is currently blocked. I'll work on the Core backport once the feature is restored in Core. |
What about the idea of using a GIF variation of the video block that @swissspidy mentioned on the other PR. So, like you'd explored previously, when a user uploads a GIF image (and we confirm it isn't transparent) we perform the transformation and the user gets a "GIF" block in the editor. I quite liked that idea. It also unlocks:
So we wouldn't filter on the (As an aside, I'll be AFK for a little bit shortly, so apologies if my replies take a little longer than usual!) Thanks again for all the back and forth on this! |
Sure, I can give that a try. I was already thinking about exploring that further. I'll just adjust the current pr, we can always revert back to the current state at fa4526d if we decide this is a better approach. |
Register a 'GIF' variation of core/video (alongside an explicit 'Video' variation) identified purely by its attribute combination: no controls, looping, autoplaying, muted and inline. No new block attribute is added. The GIF variation is the editor representation of an animated GIF that client-side media has converted to a video, so it is scoped to block and transform only (not the inserter). The editor preview plays it like a GIF, and a toolbar control restores it to the original GIF image.
When client-side media converts an opaque animated GIF, the GIF stays the image attachment and the converted video and poster are sideloaded as companion files. Watch for that companion and, for a standalone Image block the author has not opted out of, replace it with the Video block's GIF variation playing the companion video. This moves the GIF-to-video swap into the editor (block switching) instead of rewriting the markup at render time. Gallery images are left as images, the fetch is gated on a .gif URL so other images are unaffected, and a client-id guard keeps undo from immediately re-converting. The existing 'Display as original GIF' toggle now controls this editor conversion.
The GIF-to-video swap now happens in the editor (the Video block's GIF variation), so the converted block already serializes a <video> and renders natively. Drop the render_block_core/image marking filter and the wp_content_img_tag <img>-to-<video> swap; PHP no longer rewrites the markup. What remains is removing the sideloaded companion video and poster when their attachment is deleted, which core does not handle. Tests are narrowed to the companion path resolution (including traversal safety) and that cleanup.
Unit-test isGifVariation and that exactly one variation is active for any attributes (and that the GIF variation is not in the inserter). Rework the e2e spec to assert an uploaded GIF is switched to the Video block's GIF variation playing the converted companion, with the attachment still the original GIF image.
|
Thanks @andrewserong - I went ahead and implemented this. The render-time PHP swap is gone; an uploaded GIF now becomes a "GIF" variation of the Video block in the editor, and a "Display as GIF" toolbar control restores the original image. Largely follows the
Restore is a toolbar control rather than a generic block transform, since recovering the original GIF URL needs the attachment record (async), which a synchronous transform can't do. No rush on a re-review - enjoy your time AFK! |
ScreencastsUploaded GIF converted to GIF video block, then transformed to regular video block: covert.to.gif.block.mp4Transformed video, restore to show as GIF instead: convert.to.gif.mp4 |
The comment still described the removed render-time swap. The feature now swaps to a video in the editor (the converted block is a real core/video), so the PHP only cleans up sideloaded companions on attachment delete.
The companion video and poster are no longer swapped into the markup at render time; the editor switches the uploaded image block to the Video block's GIF variation, which serializes a native <video>. Update the comments in the REST sideload handler and the upload-media pipeline to describe the editor block switch instead of the removed render-time swap. The companion files and their delete_attachment cleanup are unchanged.
The upload/processing spinner is absolutely centered on the .wp-block-image figure, which spans the full content width. When the preview is narrower than that (e.g. a small animated GIF held in the uploading state for several seconds while it transcodes to a video), the spinner drifted to the right of the image. Shrink the figure to the preview with width:fit-content while .is-transient, so the centered spinner lands on the image. Wide and full alignments are excluded so they keep their own width.
andrewserong
left a comment
andrewserong
left a comment
There was a problem hiding this comment.
Great progress here @adamsilverstein, this is pretty close to how I was hoping it'd work, and feels like the best of both worlds to me! Also nice that MediaBunny works so well and has such a relatively small footprint 👍
A few observations while testing, some of it is nitpicky design stuff:
For this toggle, I'd go with a button instead as we're swapping out the block when a user toggles it. I'd usually expect a toggle to still be present after clicking:
It also might make it consistent with the transform to variation button on the Video block:
Speaking of which, I really like this! It's actually a lovely improvement outside of the GIF conversion stuff IMO as there's a way to really quickly make any video behave like a looping GIF 🎉
Also, I like the Display as GIF button in the toolbar, feels nice and easy to find if a user wants to switch to using the real GIF:
I haven't had a detailed look at the code just yet, but I had a question about the useEffect in AnimatedGifConverter. Should that also use __unstableMarkNextChangeAsNotPersistent()? I wasn't sure if this should or shouldn't be covered in the Undo stack. Currently you can CMD+Z directly after upload to restore to an image block, and I assume we probably don't want to allow that? I.e. if a user goes from an empty Image block to uploading a GIF, the first undo step should probably return to an empty Image block.
Also, I imagine we probably need a useEffect but I was wondering if there's a way to more closely tie it to the upload action, just in case there's any situation where it can accidentally fire on page load? I don't think there is, but just something to keep in mind as we've run into tricky bugs with useEffects in blocks before. Not sure how feasible (or worth it) it'd be, but if there's some way to tie this to "an upload just occurred" that might help harden things. (I imagine it's not possible for this to be an explicit conversion that occurs once an upload completes) 🤔
Finally, I found that some .gif files I downloaded from GIPHY didn't automatically convert to MP4 even though (at least visually) they don't seem to have transparency. I've attached one below:
Should that one be being converted, or is it already small enough that it isn't? (Also don't take the content of that GIF as a signal that I'm disappearing from this PR thread 😄)
Anyway, the UX is feeling really good to me overall so far! It might be worth getting a couple extra pairs of eyes on it when you feel it's ready for a final review to see if other folks agree, but IMO this is feeling really nice. Thanks again for all the back and forth 👍
Good idea!
Hmm, good point. I was wondering about the correct action to take when a user hits undo and agree it should take them back to the state before the attempted upload (typically an empty image block).
Interesting, I will investigate.
Hmm, let me review that specifically to see what is happening with this one. |
It could probably be done within image block's gutenberg/packages/block-library/src/image/edit.js Lines 215 to 233 in 220a59c |
…pload undo Two review-feedback fixes to the editor-side conversion: - Gate the auto-swap on the upload-media queue: an animated GIF only becomes a video block when its attachment id passed through the queue during this browser session. Existing image blocks loaded from saved post content are no longer silently swapped on page load. - Wrap the swap with __unstableMarkNextChangeAsNotPersistent so the block change folds into the upload's undo step. A single Cmd+Z from the converted video block goes back to the pre-upload state instead of stopping at an intermediate image-with-GIF block.
…esence hasTransparency previously returned true for any image with an alpha channel. That was a false positive for two common shapes: - Animated GIFs declare a transparent color index for their disposal- method frame compositing, so vips reports hasAlpha() === true even when no pixel ever renders transparently. This caused GIPHY-style animated GIFs to skip the GIF-to-video conversion path. - PNG encoders frequently retain the alpha channel of an RGBA source even when every pixel is opaque, so PNG -> JPEG transcoding was skipped unnecessarily. Sample the alpha band's minimum on the first frame instead. For frame 0 any transparent pixel is visible (there is no previous frame to inherit from), and min() over a single channel is cheap.
|
Thanks again @andrewserong - all three points from the review are now addressed: Toggle → buttonThe inspector toggle is gone; the image block now has a "Display as video" toolbar button that mirrors the video block's "Display as GIF" toolbar control. Same icon language, same direction of motion - no more disappearing toggle. Undo +
|
| Check | Result |
|---|---|
isAnimatedGif() byte scan |
true (28 frames) |
vips.Image.hasAlpha() |
true (bands=4) |
| Actual alpha-min on frame 0 (uchar) | 255 - fully opaque |
| ImageMagick frame inspection | frames 1-27 have mean-alpha=1.0 |
The alpha channel in frames 1-27 isn't visible transparency - it's the GIF disposal-method sentinel for "this pixel keeps the previous frame's value". hasAlpha() just checks channel presence and can't tell the difference. This pattern is extremely common for GIPHY-style animated GIFs, so the old gate was potentially wrong for the majority of real-world animated GIFs. Nice catch!
Fixed in e487510: hasTransparency() now extractBand(bands-1).min() on frame 0 and compares against the format-appropriate opaque value (255 for uchar, 65535 for ushort). For frame 0 of an animated GIF any transparent pixel is by definition visible - there's no previous frame to inherit from - so this is both correct and cheap (one min() over a single channel). 7 new tests in packages/vips/src/test/has-transparency.ts cover the opaque-with-alpha-channel case, partial/full transparency, both bit depths, and the alpha-band selection.
This also incidentally fixes an analogous false positive in the PNG → JPEG transcoding path: PNG encoders often retain an alpha channel from an RGBA source even when every pixel is opaque, and those PNGs were skipping JPEG transcoding for no reason.
If you re-test with the homer gif, you should now see the video conversion.
Per swissspidy's suggestion, fold the GIF-to-video swap into the image block's onSelectImage callback. When the selected media is a GIF whose sideloaded animated_video companion is present, build the video block and replaceBlock() instead of setAttributes(). Tying the swap to the upload's onChange (rather than watching the attachment record from a separate component) means: - Already-saved image blocks are left alone on page load - no watcher fires when the post loads, so the explicit toolbar button is the only path to convert an existing GIF block. This removes the session-upload latch the watcher needed. - The conversion happens once, deterministically, on the upload event itself, sidestepping the useEffect timing concerns Andrew flagged. Drops the now-unused pieces: - packages/block-library/src/image/animated-gif-converter.js - The image block's preserveAnimatedGif attribute (no opt-out flag is needed when there is no auto-watcher to opt out of) - The preserveAnimatedGif: true that gif-restore-control set on restore
|
Great call @swissspidy - done in 62248a4.
Net result is ~140 lines lighter and a much simpler mental model. Thanks for steering this in a better direction! |
3 participants
What
This PR adds client-side animated GIF to video conversion to the block editor.
When an animated GIF is uploaded, the new pipeline converts it to an MP4 (or WebM) client-side using the browser's native WebCodecs
ImageDecoderfor frame extraction and the mediabunny library for encoding. The converted video is stored as a companion file of the original GIF attachment, and the editor switches the block to a "GIF" variation of the Video block that plays just like the original animated GIF.This is a standalone alternative to #76946 (which used inlined FFmpeg WASM); see Why and Supersedes.
Fixes #76942
Why
Bundle size and hardware-accelerated encode performance.
Since WebCodecs encoding is hardware-accelerated on supported platforms, conversion is also significantly faster than software-only FFmpeg WASM.
How
Conversion pipeline (off the main thread)
@wordpress/video-conversionpackage (new): mirrors the@wordpress/vipsworker-threads pattern. A web worker is bundled with the mediabunny library and exposed via a Comlink-style proxy. The main thread callsconvertGifToVideo(file, mimeType), which runs fully off the main thread.src/index.ts):ImageDecoderdecodes each GIF frame, honoring real per-framedelayvalues. Each decodedVideoFrameis fed to mediabunny'sVideoSampleSource, then encoded viaVideoEncoder(AVC/H.264 for MP4, VP9 for WebM).@wordpress/upload-mediaintegration: a newOperationType.TranscodeGifoperation, anisAnimatedGif()utility (GIF89a Graphic Control Extension detection), and atranscodeGifItem()action wired intoprepareItem(). Conversion is gated on WebCodecs availability atprepareItemtime, runs with a concurrency limit of 1 (memory-intensive encode), and falls back gracefully: anUnsupportederror from the worker leaves the original GIF in the queue.Storage model (companion files)
An uploaded opaque animated GIF stays a single
core/imageattachment - it remains one item in the media library. The converted video and a static first-frame poster are sideloaded as companion files of that same attachment (like the HEIC original), recorded inmedia_details.animated_videoandanimated_video_poster. They are never separate attachments. Transparent GIFs are not converted (a<video>cannot reproduce GIF transparency), so they have no companion and upload as a normal image.Editor block switch
The GIF→video swap happens in the editor as a block switch, not at render time.
core/videogains two variations, "Video" and "GIF", identified purely by their attribute combination (!controls && loop && autoplay && muted && playsInline) - no new block attribute. The GIF variation is scoped toblock+transform(not the inserter), since it is the editor representation of a converted GIF. Its editor preview plays like a GIF (muted, looping, autoplaying).core/imageblock). A.gif-URL gate keeps non-GIF images from triggering an attachment fetch, and a client-id guard keeps undo from immediately re-converting.core/image(and opts it out of re-conversion). The Image block's "Display as original GIF" toggle (preserveAnimatedGif) controls this editor conversion, so the round-trip is reversible.core/video, it serializes a native<video autoplay loop muted playsinline poster>and renders on the front end with no filtering.PHP (minimal)
@wordpress/video-conversion/loaderscript module in the editor (lib/client-assets.php).class-gutenberg-rest-attachments-controller.php).wp_delete_attachment_files()does not know about them (lib/media/animated-gif-to-video.php).Scope & browser support
Client-side media already requires Document Isolation Policy globally (for
SharedArrayBuffer/crossOriginIsolated). This PR does not change that requirement, and Firefox (which lacksVideoEncoder) is out of scope for both approaches. The conversion path is gated ontypeof ImageDecoder !== 'undefined' && typeof VideoEncoder !== 'undefined'and falls back gracefully to uploading the original GIF when WebCodecs is unavailable.Screencasts
Uploaded GIF converted to GIF video block, then transformed to regular video block:
covert.to.gif.block.mp4
Transformed video, restore to show as GIF instead:
convert.to.gif.mp4
Testing
Verified manually end-to-end (see screencasts above): uploading an animated GIF converts it to the Video block's GIF variation, the front end renders a native looping
<video>, and the round-trip back to the original GIF works in both directions.Automated coverage:
isAnimatedGif, store selectors, andprepareItemwiring: 21 suites / 214 tests, all passing.core/videoGIF variation (isGifVariationdetection; exactly one active variation; not in the inserter).test/e2e/specs/editor/various/gif-to-video.spec.js: uploads an animated GIF via the Image block and asserts the block is switched to the Video block's GIF variation playing the converted companion, while the attachment remains the originalimage/gifwith the companion recorded inmedia_details.animated_video. The spec skips automatically if WebCodecs or cross-origin isolation is not active. PHP/E2E run in CI (wp-env was unavailable locally).Changes from earlier revisions
Following review feedback (h/t @andrewserong and @swissspidy), the GIF→video swap moved from a render-time PHP filter to an editor block switch (the "GIF variation of the Video block" idea from the review thread). This removed most of the feature's PHP: the
render_block_core/imagemarking filter and thewp_content_img_tag<img>→<video>swap are gone (~208 lines of PHP removed), leaving only companion cleanup on delete. The upload/conversion pipeline and the companion-file storage model are unchanged; only where the swap happens moved.Supersedes
This PR consolidates the full GIF→video feature on the mediabunny/WebCodecs engine and replaces the FFmpeg-WASM stack:
References