Media: Add HEIC support using platform capabilities#76731
Merged
Conversation
When users upload HEIC images, try server processing first. If the server lacks HEIC support, fall back to the browser's native createImageBitmap() + OffscreenCanvas to decode and convert to JPEG. The original HEIC is sideloaded as the original, the JPEG as the scaled version, and sub-sizes are generated from the JPEG via the existing vips resize pipeline. PHP: bypass MIME type check for HEIC/HEIF uploads so files can be stored even when the server can't process them. JS: add HEIC detection in prepareItem(), canvas conversion utility, and HEIC-aware generateThumbnails() fallback.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
|
Size Change: +3.66 kB (+0.05%) Total Size: 7.75 MB 📦 View Changed
ℹ️ View Unchanged
|
adamsilverstein
added
[Type] Enhancement
|
Flaky tests detected in 84cdd06. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/24679521744
|
Links core PR wordpress-develop#11323 to Gutenberg PR #76731.
Three issues prevented the HEIC canvas fallback from working: 1. generate_sub_sizes was set to true for HEIC uploads, causing the server to attempt (and silently fail) sub-size generation instead of letting the client handle it. 2. The server can't read HEIC files, so it always returned an empty missing_image_sizes array. The client now derives missing sizes from registered image sizes in settings. 3. createImageBitmap() doesn't support HEIC in Chrome. Added a fallback using <img> element + HTMLCanvasElement, which works on macOS because Chrome exposes OS-level HEIC decoding through the img rendering pipeline.
Chrome doesn't expose macOS's native HEIC decoder through image APIs (createImageBitmap, ImageDecoder, <img>). However, Chrome 107+ supports HEVC video decoding via the WebCodecs VideoDecoder API using macOS VideoToolbox. This adds a new decoding strategy that bridges the gap: 1. Parse the HEIC/ISOBMFF container in pure JS to extract the HEVC decoder configuration (hvcC) and compressed image data 2. Feed the HEVC frames through VideoDecoder (hardware-accelerated) 3. Composite decoded tiles onto a canvas and convert to JPEG Handles both single-image and grid/tiled HEIC files (the common iPhone format where photos are split into multiple HEVC tiles). No WASM decoder or patented codec is shipped — only the standard container format is parsed, and decoding uses Chrome's already-licensed platform decoder. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Register HEIC/HEIF as allowed upload MIME types so macOS file picker doesn't silently convert them to JPEG - Handle idat construction method in HEIC container parser for grid descriptor data stored inside the meta box - Relax grid tile count validation to allow extra iref entries (alpha planes, thumbnails) - Disable server-side sub-size generation and format conversion for HEIC uploads (client handles these) - Add debug logging throughout HEIC upload pipeline
The initial upload already saves the HEIC file on the server. The scaled sideload handler records it as original_image in metadata, so a separate original sideload is unnecessary and creates a duplicate file.
Don't update blocks with HEIC attachment URLs since browsers can't display them. The scaled JPEG sideload will provide a usable URL once client-side conversion completes. Also removes all HEIC debug logging added during development.
Tests cover: - Bit reversal for HEVC codec string construction - Single-image HEIC parsing with synthetic binary fixtures - Codec string generation (e.g. hvc1.1.6.L93.B0) - HEVCDecoderConfigurationRecord extraction - Error cases: empty buffer, missing meta/pitm/iloc boxes
Validate grid descriptor data length before accessing fields to avoid out-of-bounds reads on malformed files.
Safari can decode HEIC natively via createImageBitmap() but lacks SharedArrayBuffer for VIPS. Split the PHP gate so HEIC infrastructure (MIME types, REST controller, sideload endpoint) loads independently of VIPS requirements. Add a HEIC-only mode that converts HEIC to JPEG client-side and delegates sub-size generation to the server.
adamsilverstein
requested review from
ajitbohra,
ellatrix and
fabiankaegy
as code owners
github-actions
Bot
added
[Package] Editor
Contributor
|
HEIC got quickly processed in Chrome and in Safari. Thank you @adamsilverstein! |
adamziel
approved these changes
Apr 7, 2026
Contributor
adamziel
left a comment
adamziel
left a comment
There was a problem hiding this comment.
I am not familiar with the code and not comfortable to review the code quality, but I've tested a HEIC image functionally on Mac and it is an improvement. This should still get a code review before merging.
Member
Author
For this PR i decided at some point to keep the extracted JPEG as the original. I tried storing the heic directly originally and was getting a duplicate image. I agree that ideally, the original heic would be stored instead of the jpeg version.
@adamziel This may be fixed by #76707. Are you able to reproduce it with that PR active?
Other than HEIC, subsize handling defaults to client side.
Thanks for testing! |
Keep the original HEIC in sourceFile instead of replacing it with the converted JPEG. After upload, sideload the HEIC as the "original" so WordPress stores it in attachment metadata (original_image). The uploaded JPEG remains the main display file and is used for all thumbnail generation. This addresses reviewer feedback that the original HEIC appeared lost after upload since both file and sourceFile were replaced with JPEG. Now the media library preserves the HEIC as the original image while all display and sub-size files remain as JPEG. Also skip vips rotation for HEIC sources since rotation is already handled during the canvas conversion step.
Member
Author
|
I adjusted this slightly to try to correctly preserve the uploaded heic original file in 2f41440. I'm going to test this out a bit further before merging. |
Resolve conflicts in packages/upload-media/src/store/private-actions.ts and its tests by keeping the HEIC canvas fallback changes from this branch.
This reverts commit 2f41440.
Keep the converted JPEG in both file and sourceFile so all downstream
paths (vips, thumbnails, retries) see an editor-supported image and
never leak HEIC into the main /wp/v2/media create endpoint.
Store the original HEIC on item.originalHeicFile instead. In
generateThumbnails, dispatch a single addSideloadItem with parentId
set, which guarantees processItem routes it to sideloadItem and the
/wp/v2/media/{id}/sideload endpoint. The server accepts HEIC there
because the mime type is registered via upload_mimes and the sideload
permission check does not apply wp_prevent_unsupported_mime_type_uploads.
When a HEIC sideload overwrites an existing original_image value (set by the scaled-sideload flow to point at the un-scaled JPEG), delete that JPEG file first. wp_delete_attachment_files() only consults the current value of original_image, so without this the pre-scaled JPEG would remain on disk forever when the attachment is deleted.
The HEIC sideload previously used image_size 'original', which set $metadata['original_image']. The scaled-JPEG sideload runs after and overwrites that same key with the un-scaled JPEG basename, losing the HEIC reference. The HEIC then lingered on disk after attachment delete, because wp_delete_attachment_files() only consults original_image. Use a dedicated 'original-heic' image_size that writes to a separate 'original_image_heic' meta key, so the two sideloads no longer collide. A delete_attachment hook removes the companion HEIC file when the attachment is deleted.
This cleanup was added to handle HEIC overwriting an existing
original_image, but Option A (dedicated original_image_heic key)
means HEIC no longer touches the 'original' branch. The remaining
non-HEIC callers ('original' for rotated images) only fire once per
upload, so there is nothing to clean up.
Member
Author
|
@adamziel - I added storage of the original heic upload. I used a separate meta key to keep logic changes to a minimum, only adding the original sideload and deletion when the parent is deleted. Leaving 'original_image' pointing to the extracted jpeg will let existing references or uses of that image work as expected. |
Contributor
Oh, this is an interesting problem. So Just thinking out loud, but I imagine we might have future uses for this kind of concept of a source file that needs transcoding in order to be presented on the site frontend. If so, is it worth figuring out an API shape for this that isn't specifically coupled to HEIC? I.e. this is beyond the scope of current work, but what if our source file was a PSD or a camera RAW file, for example? What about something like A (much) more verbose example of this kind of relationship is how core handles the poster image for MP3 files, which wind up being added as a separate attachment altogether. I imagine we want to avoid that level of complexity, as it'd look like duplicated items in the media library? Still, just linking to it in case it sparks any ideas: In general, though, I like the idea that we have both the full converted JPEG and a reference to the HEIC. I don't mind the approach here, my main question was really about naming, and how we might handle UI concerns for folks to retrieve the original/source file again if they need to (not a blocker for this PR, but curious how we might handle it). |
Member
Author
|
Right, we need I like |
Member
|
If it helps, in media-experiments I sideload the original HEIC image and store it in _attachment_metadata with the key |
Member
Author
Oh, that is good to know. Might as well follow your lead and. use |
Per PR discussion, align the meta key with the convention used in media-experiments (swissspidy/media-experiments) so a future switch between the two is seamless. 'original_image' continues to point at the web-viewable JPEG derivative — existing consumers rely on that.
Member
Author
|
Updated naming in 6ff5a4c |
Resolve conflicts in lib/media/class-gutenberg-rest-attachments-controller.php and packages/upload-media/src/store/private-actions.ts by keeping the PR's HEIC-handling approach. Trunk's #75888 sideload refactor introduced an incompatible sub-size accumulation pattern; reconciling with the PR's 'original-heic' branch and scaled-sideload flow is deferred as follow-up.
Member
Author
|
This needs some refactoring after #75888 was merged, working on that now. |
Adopt the sub_size_data accumulation pattern introduced in #75888 on trunk and fold the HEIC companion handling into it: PHP (class-gutenberg-rest-attachments-controller.php) - finalize route: add sub_sizes schema argument - finalize_item(): apply accumulated sub_sizes to $metadata, including a new 'original-heic' branch that writes to $metadata['original'] - sideload_item(): return a $sub_size_data shape; the 'original-heic' case returns file=wp_basename($path) for finalize to pick up TS (upload-media/src/store/private-actions.ts) - Remove shouldPauseForSideload and resumeItemByPostId; trunk gates concurrency via getActiveUploadCount / maxConcurrentUploads - sideloadItem(): replace onFileChange with onSuccess(subSize) that dispatches AccumulateSubSize against item.parentId - generateThumbnails(): drop onChange callbacks from thumbnail and scaled addSideloadItem dispatches; sideloads no longer return a full attachment - finalizeItem(): pass item.subSizes to mediaFinalize
Member
Author
|
This is ready for final testing and merge. |
Member
Author
|
I am going ahead and merging this, it is testing well for me in Safari and Chrome (on Mac OS). It should also work on Edge in windows as long Microsoft's HEVC Video Extension is installed (which is common since its also required to play H.265/HEVC video files). |
adamsilverstein
changed the title
1 task
Member
Author
|
I realized after merging this that I left off a fix to use the 'jpg' extension vs 'jpeg' for the exported file names. I opened a follow up PR to fix this - #77506 |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds client-side HEIC image handling with browser-native decoding. When users upload HEIC images:
irotbox) is parsed from the HEIC container and applied during conversion, ensuring portrait images display correctlySafari support
Safari lacks
SharedArrayBuffer/Document-Isolation-Policysupport, so full client-side VIPS processing is unavailable. However, Safari can natively decode HEIC viacreateImageBitmap(). This PR adds a "HEIC-only" mode that:window.__heicUploadSupportflag for browsers that can decode HEIC via canvascreateImageBitmap, sideloads the JPEG withgenerate_sub_sizes: trueso the server generates all thumbnails from itClient-side decoding strategies
The HEIC-to-JPEG conversion tries three strategies in order:
createImageBitmap()+OffscreenCanvas— Works natively in Safari (all platforms). The simplest path.ImageDecoderAPI — Uses platform codecs exposed via the WebCodecs image pipeline. May work in future Chrome versions if HEIC is added.VideoDecoder— For Chrome 107+ on macOS. Parses the HEIC/ISOBMFF container in pure JS to extract the HEVC bitstream and decoder configuration (hvcC), then feeds the HEVC frames through Chrome's hardware-acceleratedVideoDecoder(backed by macOS VideoToolbox). Handles both single-image and grid/tiled HEIC files (the common iPhone format). Parses theirot(image rotation) property box and applies the rotation to the output, so portrait photos are correctly oriented.Upload flow
Server has HEIC support: Upload → server processes everything → done.
Browser can decode, VIPS available (Chromium): Upload HEIC → VideoDecoder decodes HEIC → JPEG → sideload as scaled → generate sub-sizes client-side via VIPS → sideload each → finalize.
Browser can decode, no VIPS (Safari): Upload HEIC →
createImageBitmapdecodes HEIC → JPEG → sideload as scaled withgenerate_sub_sizes: true→ server generates all sub-sizes from the JPEG → finalize.Neither supports HEIC: Upload → all client-side decode strategies fail → error shown to user.
Browser support matrix
createImageBitmapFixes #76732
Screencast
screencast.2026-03-20.22-50-56.mp4
Testing tips:
Test instructions
createImageBitmap, sideload JPEG, and server generates all sub-sizesirotrotation handling.original_imagein attachment metadataTechnical diagram
flowchart TD A["User selects HEIC file"] --> B{"Server has<br/>HEIC support?"} B -->|Yes| C["Upload HEIC"] C --> D["Server generates<br/>all sub-sizes"] D --> E["Done ✓"] B -->|No| F{"Browser can<br/>decode HEIC?"} F -->|No| G["Error: cannot<br/>process HEIC ✗"] F -->|Yes| H["Upload original HEIC<br/>(generate_sub_sizes: false)"] H --> I["Decode HEIC → JPEG<br/>(browser-native)"] I --> J{"VIPS available?<br/>(SharedArrayBuffer)"} J -->|"Yes (Chromium)"| K["Sideload JPEG<br/>"] K --> L["Generate each sub-size<br/>client-side via VIPS"] L --> M["Sideload each<br/>sub-size to server"] M --> N["Finalize ✓"] J -->|"No (Safari)"| O["Sideload JPEG <br/>(generate_sub_sizes: true)"] O --> P["Server generates<br/>all sub-sizes from JPEG"] P --> Q["Finalize ✓"] subgraph decode ["Decoding strategies (tried in order)"] direction TB S1["1. createImageBitmap()"] -->|fails| S2["2. ImageDecoder API"] S2 -->|fails| S3["3. HEIC parser +<br/>VideoDecoder"] S3 -->|fails| S4["Error: cannot\nprocess HEIC ✗"] end I -.->|"uses"| decode style E fill:#2d6,stroke:#1a4,color:#fff style N fill:#2d6,stroke:#1a4,color:#fff style Q fill:#2d6,stroke:#1a4,color:#fff style G fill:#d33,stroke:#a22,color:#fff style S4 fill:#d33,stroke:#a22,color:#fff