Client-side media: Transform image block to video block after GIF conversion#78369
Client-side media: Transform image block to video block after GIF conversion#78369adamsilverstein wants to merge 12 commits into
Conversation
Adds an editor.BlockEdit filter that watches a core/image block's attachment record. When client-side animated GIF to video conversion (@wordpress/upload-media) replaces the uploaded GIF with an MP4/WebM, the attachment mime_type becomes video/*. This swaps the core/image block for a core/video block configured to mirror the original GIF behavior (muted, looping, autoplaying, inline, no controls). Stacked on the GIF-to-video conversion PR so the end-to-end feature (upload animated GIF, get an autoplaying looping video block) can be tested as a whole.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
The transcodeGifItem error path intentionally logs the failure cause via console.error for debuggability. jest-console treats any unexpected console.error as a test failure, so assert the expected error explicitly.
Remove the maxDimensions parameter (and the now-unused padToEven helper) from convertGifToVideo and its wrappers. Nothing in the upload pipeline ever supplied a value, so the scaling branch was unreachable dead code. Copy FFmpeg output via new Uint8Array(output).slice().buffer instead of casting output.buffer.slice() to ArrayBuffer. Under crossOriginIsolated FFmpeg builds the MEMFS view can be backed by a SharedArrayBuffer, so slicing .buffer would yield a SharedArrayBuffer the cast misrepresented. Slicing the typed array always allocates a standalone ArrayBuffer.
|
Size Change: +80 B (0%) Total Size: 22.1 MB 📦 View Changed
ℹ️ View Unchanged
|
adamsilverstein
added
[Type] Feature
|
Nice! 👍 Looks similar to what I've built in media-experiments before: swissspidy/media-experiments#242 One thing I struggled to build when doing this is to attach the generated video poster image to the newly converted video block. See swissspidy/media-experiments#163 The original upload references the old image block clientId, but this reference is lost upon conversion. Solving this might affect how the transformation is done in the first place, so I'd love your thoughts on it. |
|
This is an interesting one in terms of UX. A couple of thoughts while testing this out:
It's a fairly subtle issue to deal with, but I'm wondering what the common user expectation is here? And how can we preserve GIFs as GIFs when that's what the user intended 🤔 Just a couple of ideas, but what if:
What do you think? Am I overthinking this, or could we use some design or UX input? |
For the vast majority of users, they are thinking "I want this (video|meme|animated gif|animation) on my web page" - they don't really know what the format it or what the different formats are. If they upload a gif, they likely want a looping animation or video, because thats what gifs do. If they upload a video, they probably want a video player with some controls to play and pause the video. WordPress should endeavor to deliver that experience the user wants, without making them worry about the technical implementation details.
I generally think this makes sense. The video is set to autoplay and has no controls, to it behaves exactly like the gif would on the page, so the user gets what they (most likely want). I do agree that fact that it is a video block is slightly surprising, and might confuse some users. A notification might help, but it still feels odd.
That also makes sense, although arguably the video would give them the same experience, it would be even more confusing to have an image block you just inserted converted to a video block.
Design input would be good especially if want to add some sort of UI to convert a GIF to a video, or to switch back to the GIF (which we should preserve). I have a new idea I'm going to try - what about if we leave the GIF as an image block in the editor, and swap it our for a video at run-time. That is, the user uploads a GIF, which stays a GIF and is also converted to a video. Then, while generating the HTML for the front end, if we find we have GIF images with matching videos, we swap in the video instead of the GIF. This results in the user getting the exact behavior they expect, without any unexpected block conversions. I still think your idea of a UI for converting GIF to video is useful for existing GIF uploads, something to explore in a separate issue. |
The editor-side block transform replaced a just-inserted core/image
with a core/video, which is surprising to users and loses the
upload's clientId reference needed to attach a poster.
Instead, keep the uploaded GIF as a normal image attachment (the
block stays a valid core/image) and upload a companion video
attachment generated from the same GIF. Both uploads carry a shared
animated_gif_pair_token so the two are linked server-side via
_animated_video_id post meta. On the front end, wp_content_img_tag
(inside wp_filter_content_tags()) swaps any GIF <img> that has a
linked video for a GIF-behaving <video>, covering every block that
emits a wp-image-{id} image (Image, Gallery, Media & Text, Cover).
Remove the editor BlockEdit transform hook in favor of this
render-time approach.
|
I've reworked this to not change the block. Pushed in 47044fc. Instead of converting This directly addresses both pieces of feedback:
PR description and test plan updated to match. Feedback welcome on the linking mechanism (shared |
Lets developers keep specific GIFs as GIFs based on the attachment, the linked video, or the rendering context.
|
Reworking this a bit then will re-open for review. |
Uploading a GIF created two media library items: the GIF and the converted video as its own attachment. The video also drove a stray top-level upload (the off-center spinner was that, not the image block). Treat the video like the HEIC original: transcode it in a child sideload of the GIF item and record only its filename in the GIF attachment metadata (animated_video). No second attachment, no second top-level queue item. The render-time swap reads the metadata; a delete_attachment hook removes the file, rebuilding the path from the attachment's own directory + recorded basename and confirming it resolves inside the uploads dir before deleting. Removes the pair-token attachment-linking mechanism.
|
Thanks for the updates, this sure is a tricky one to settle on a good experience!
While clever, I'm not sold on this approach for several reasons:
Here's a screenshot of the site frontend on a site where a user might have downloaded some small transparent animated GIFs e.g. from https://slackmojis.com/ and have placed them within block content (or even in a template) where background colors and global styles rules are present. Note that in the top block (where the GIF was uploaded on trunk) the animated gif's transparency is preserved, and the global styles rules for border are applied. Whereas in the converted MP4 there is no transparency and the In my mind this means that implicitly converting and swapping out images is unfortunately going to be a risky flow as it's making too many assumptions about user intent. My preference would be:
Overall, I still think the conversion from GIF -> MP4 is a really cool feature, I'm just not sure it's a feature that the majority of users will want as a default, and I think it's easy for us to accidentally break things if we try to convert or swap things out implicitly. But, again, this is just my take! It could be worth getting some more opinions, too 🙂 |
We could add some transparency detection and not convert the GIF in that case. Transparent videos are kinda possible on the web, but hacky, see https://jakearchibald.com/2024/video-with-transparency/.
I would want to avoid adding any sort of UI for this wherever possible. I don't see GIF -> MP4 as a user-facing feature, but as an implementation detail. Adam has already nicely put it: "WordPress should endeavor to deliver that experience the user wants, without making them worry about the technical implementation details." The frontend-only swapping sounds intriguing, but I agree that this might be worse for theme compatibility. By the way, in the media-experiments plugin I originally added "GIF" as a video block variant, so that when you upload a GIF, it creates a video block but it says "GIF" in the sidebar: Screen.Recording.2023-10-11.at.20.31.20.movI think from a UX that's quite neat :) Especially since you could then choose GIF from the inserter, drop your GIF file, and under the hood it will be a video.
Transparency aside, what would be a reason for a user to really want the GIF image format? 🤔 Again, if we do it right, the user shouldn't see a difference, except that the video will load much faster. |
While the presence of transparency is a useful heuristic, I'm not sure the complexity of doing that would be worth it. It's tricky, but I'm generally cautious of us adding too many conditions that could be hard to reason about or untangle further down the track.
I think for me it's that WordPress is so widely used, I don't feel confident making an assumption on why a user might want to use a GIF. It's that up until now folks can upload and use a GIF that they've designed, and we shouldn't prevent them from being able to do that if that's what they want to do. For me it comes down to the experience that whenever we go to change things about how the Image block works, inevitably we'll discover a use case after the fact that we haven't thought of. The use case I'm thinking of is: a designer has carefully crafted a GIF that they want to use just as they've created it. WordPress should allow them to do that, as it does currently.
A counterpoint to this is that if WordPress unexpectedly converts a file into an unexpected format or changes the output on the site frontend in a way they didn't intend, then we expose the technical implementation details to them. All that said, I also really like this feature! And I appreciate what you say about making things transparent to the user so we don't end up with complexity in the UI. It's a tough balance for sure. My main concern is that this feature is quite clever but it's also opinionated in a way that I'm used to seeing more in plugins than in core. For me, I'm wondering if there's a sweet spot somewhere where if a user drags to the desktop (like in your video) but hasn't explicitly added an Image block, then we do the convert to video and add a video block thing. But as for making the Image block automatically convert to a video, I think it's overall risky. As a user, if WordPress is doing conversions, I'd expect to be able to control the output, like how we allow users to select the image size. Whether that's converting or switching between Video and Image blocks (or a gif video block variation like in your cool example), I think a blocker overall to me is that a user must be able to output a raw GIF if they really want to, or we'll create a regression for someone in a flow they're used to. |
|
Coming in late. This is a very cool feature. I know a lot of platforms perform the conversion for performance reasons. I think that's a big win, especially for the single Image block with opaque GIF case.
On this note, how will we handle galleries for example? I found the conversion to MP4 breaks a few things here:
Similar issues with the Media and Text block. Cover block looks okay so far but I didn't do much testing. Maybe layout could be improved with more specific selectors (e.g., All this makes me guess that any 3rd party blocks with Image as an InnerBlock could be affected. Would some themes have style rules that we'd break too? It looks like we're raising not one, but entire classes of regressions. On the broader "users don't care about the format, just the experience", I'd agree with this, but if the consequences of the swap are user-facing, even when the format isn't, then that falls over. From what I've read, could there be a stack of mitigations?
Apologies if they sound naive, I'm just catching up on the comments. Footnotes
|
How about offering an opt-out? Instead of just simply converting the images or asking the user "do you want to use the "We have automatically optimized this GIF for optimal user experience. If you're seeing issues, you can restore the original. [Restore]" Where "Restore" would mean turning the GIF video block back into an image.
FWIW it's not complex at all. We already do this to prevent transparent PNG -> JPEG conversion: gutenberg/packages/vips/src/index.ts Lines 639 to 655 in 30bb0b0 gutenberg/packages/upload-media/src/store/private-actions.ts Lines 615 to 631 in 30bb0b0 |
|
Thanks for the feedback @ramonjd, @andrewserong and @swissspidy!
Oh neat, I like the way they changes to video when you turned on the controls in your screencast.
FWIW we do a similar thing when you upload a HEIC. We convert it to a JPEG so it will work for most site visitors (unlike an heic). Our assumption is "the user wants an image on their website" not "the user wants an HEIC on their website". GIFs are similar, most users don't know they want a GIF, they know they want an animated image.
For the rare GIF aficionado, I'm sure the community will provide a "no gif to video" plugin.
The interface for adjusting image sizes is terrible, users shouldn't be able to control that at all. My opinion anyway :)
I'm open to the idea of adding UI for the conversion, curious to hear some design feedback (ping @jasmussen @karmatosed). I prefer auto-conversion, but we might still need UI and I'm really not sure how that should work.
This feels like an edge case we could account for with a filter and 'co-video-to-gif' plugin.
Fair point. As we see in the examples, changing from an image tag to a video tag has surprising cascading effects. Dang CSS!
Need more feedback on how the UI for this would work. I'm guessing we would need the ability to toggle display between video and gif.
I like this suggestion, especially coupled with skipping auto conversion for transparent gifs and gifs in inner blocks. @swissspidy do you still favor transforming the block to a video block immediately on upload? I overall prefer the approach of runtime conversion of the tag, but am open to persuasion. The video block likely provides a better editor wysiwyg preview and by changing its name to "GIF" as in your screencast the transformation become less unexpected. |
adamsilverstein
requested review from
ramonjd
and removed request for
spacedmonkey
A <video> cannot reproduce GIF transparency, so converting a transparent animated GIF would visibly change the image (e.g. small decorative or emoji-like GIFs over a colored background). Detect transparency at upload time with vipsHasTransparency() - reusing the PNG to JPEG check pattern - and keep transparent GIFs as a normal image instead of generating a companion video. Errs toward keeping the GIF if the check fails.
Swapping every <img> with a companion video at render time broke layouts when the image was nested: galleries (crop-to-fit, lightbox, captions), Media & Text and Cover. Gate the swap by block context. A new render_block_core/image filter marks only eligible images - standalone Image blocks whose author has not opted out (preserveAnimatedGif) and that are not inside a Gallery (galleryId context). Media & Text and Cover render their media as the block's own <img>, not a core/image block, so they never reach the filter. wp_content_img_tag then swaps only marked images.
Using the GIF itself as the <video> poster makes the browser download the full animated GIF just to paint the poster, undercutting the conversion's performance win. Sideload a lightweight static first-frame poster as a second companion of the GIF attachment: a vips image transcode (vips decodes only the first GIF frame) under image_size 'animated-video-poster', recorded in metadata as 'animated_video_poster'. The render swap prefers it for the <video poster> and the delete_attachment hook removes it with the video.
Some authors deliberately want the original GIF (a carefully crafted animation, or one a theme styles as an image). Add a 'Display as original GIF' toggle to the Image block's Media settings, shown only when the attachment has a converted video companion (media_details.animated_video). It sets a new preserveAnimatedGif attribute that the render-time swap honors, leaving that image as a GIF on the front end.
adamsilverstein
requested review from
ajitbohra,
fabiankaegy and
juanmaguitar
as code owners
Covers the render-time gating and swap: top-level Image blocks with a companion video are marked and swapped to <video> (carrying the accessible name and using the static poster); gallery inner images (galleryId context), author opt-outs (preserveAnimatedGif) and images without a companion are left as <img>; and the delete_attachment hook removes both companion files. Note: pending a wp-env run.
|
Closing as superseded by #78410. #78410 switches the conversion engine from FFmpeg WASM to mediabunny + WebCodecs and already includes this PR's render-time swap / companion-file architecture. It also folds in the review feedback from this thread:
Thanks @swissspidy, @andrewserong and @ramonjd for the feedback — it's carried over to #78410. |
|
Thanks for engaging will all these ideas, folks! IMO so long as there's an opt-out and we don't introduce any bugs (i.e. the top-level Image block idea you mention rather than applying to nested Image blocks), that services the "why did you break this?" customer feedback I'm concerned about. For context, I worked for a few years on WordPress.com where a lot of users don't have access to install plugins, and customer feedback often goes wildly against how we think things should work technically. Also, glad to hear that the transparency check is easy to do! With the above mitigations I reckon we'll have covered the vast majority of cases/concerns 👍 |
Thanks for the ping. I'll extend it a bit to @WordPress/gutenberg-design though also specifically to @fcoveram. He has worked on Openverse, and media in general, and has had a lot of thoughts around the media library and those bits. In general that's here my instinct goes, that instead of thinking in atomic flows attached to blocks or editors, we think in terms of tooling and how to access it. I.e. if there's an "edit" button on an image that we build for the media library, an edit button from inside the editor could invoke the same interface. Reasonably there could be a similar parallel if we engage with video compression/converesion tooling. |
5 participants
Summary
Stacked on #76946 (base branch:
worktree-gif-to-video).#76946 adds client-side animated GIF → MP4/WebM conversion during upload, but per its scope the originating
core/imageblock stays an image block.This PR completes the feature without changing the block and without adding a second media library item. An uploaded animated GIF stays a single, normal image attachment (the editor is unchanged). The GIF is also transcoded to a video which is sideloaded as a companion file of that same attachment — exactly like the existing HEIC original — and recorded in the attachment metadata under
animated_video. On the front end the GIF<img>is swapped for a GIF-behaving<video>while the HTML is generated.This avoids the surprising block-type change (raised by @andrewserong) and the lost clientId/poster reference (raised by @swissspidy), and keeps the media library clean (one item per GIF).
How it works
Producer (
@wordpress/upload-media)Upload → ThumbnailGeneration → Finalize); the original is stashed on the queue item (animatedGifFile).generateThumbnails, a child sideload item of the GIF transcodes it (TranscodeGif, preserving the existing FFmpeg-WASM concurrency limit) and sideloads the result withimage_size: 'animated-video'. No separate attachment, no second top-level upload item.Server (CSM attachments controller)
sideload/finalizelearn theanimated-videorole: the video filename is written towp_get_attachment_metadata()['animated_video'](mirrorsoriginal-heic).Render swap + cleanup (
lib/media/animated-gif-to-video.php)wp_content_img_tag(insidewp_filter_content_tags()→ post content, widgets, excerpts): if the attachment has ananimated_video, reads the<img>attributes withWP_HTML_Tag_Processorand returns<video autoplay loop muted playsinline poster=…>. Covers Image, Gallery, Media & Text, Cover, etc.gutenberg_swap_animated_gif_for_videofilter lets developers keep specific GIFs as GIFs per-image.delete_attachment: deletes the companion video. The path is rebuilt from the attachment's own directory + the recorded basename only, then confirmed viarealpath()to be a regular file strictly inside the uploads directory before deletion — it can only ever remove the sideloaded companion.Removed: the editor
BlockEditblock-transform hook and the earlier separate-attachment + pair-token linking mechanism.Test plan
With client-side media processing enabled in a cross-origin-isolated browser:
wp_get_attachment_metadata()for the GIF has ananimated_videoentry; the file exists next to the GIF<video>(no controls)<img>)A block-level "convert GIF ↔ video" affordance for existing uploads, and a generated poster image, are intentionally out of scope (separate follow-ups).