Media: Add UltraHDR (ISO 21496-1) gain map support#74873
Media: Add UltraHDR (ISO 21496-1) gain map support#74873adamsilverstein wants to merge 34 commits into
Conversation
adamsilverstein
requested review from
ajitbohra,
gziolo,
nerrad,
ntwb and
ryanwelcher
as code owners
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Unlinked AccountsThe following contributors have not linked their GitHub and WordPress.org accounts: @kleisauke, @gregbenz. Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases. If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
adamsilverstein
added
[Type] Enhancement
github-project-automation
Bot
moved this to 🔎 Needs Review
in WordPress 7.0 Editor Tasks
|
Flaky tests detected in b608550. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/26533550660
|
|
Size Change: +228 kB (+2.78%) Total Size: 8.41 MB 📦 View Changed
ℹ️ View Unchanged
|
andrewserong
left a comment
andrewserong
left a comment
There was a problem hiding this comment.
Nice, this is testing well for me! I was going to ask how heavy the upstream library is, and then I noticed that it's from your own repo, so I assume you're quite comfortable with it 😄
The main question I have is to do with the idea that if most uploaded JPEGs are not UltraHDR, how expensive is the operation to detect it? If it's very cheap, then this seems like a great addition. I've left a comment, but before I looked at the code, my assumption is that this kind of operation would flag if it succeeds / does something, but wouldn't warn if it can't find an UltraHDR data.
What do you think?
This is all still behind an experiment of course, so don't let my questions here stop you from progressing! (Also I'll be AFK early next week, so apologies if my reply is delayed until mid-week)
| } catch ( error ) { | ||
| // If UltraHDR detection fails, continue with regular upload | ||
| // eslint-disable-next-line no-console | ||
| console.warn( 'UltraHDR detection failed:', error ); | ||
| } |
There was a problem hiding this comment.
I'm getting this warning for uploading normal (non UltraHDR) jpegs from my desktop. Given that the majority of jpegs won't be ultra HDR, should we just fail silently here?
There was a problem hiding this comment.
yes, this is mainly for debugging purposes and I will remove before merging to trunk.
I am comfortable with it working based on the tests, although I also plan to ask for help doing manual testing to verify it properly handles real world images.
That is a great question and worth investigating more. I will review the detection logic to see if it can be optimized. Also, I'm still not certain about what we should output. My assumption is that UltraHDR JPEGs are decoder compatible with existing JPEGs so we would want to output UltraHDR for uploaded UltraHDR. Then again, some users may prefer to use the SDR version for their website exclusively because they want smaller file sizes and don't care about HDR. We might need a new way for developers to implement that since the mime type is still image/jpeg and out=r
Hopefully for something fun! |
|
@adamsilverstein libultrahdr includes a probe() method to validate that required components are present without decoding. I haven't tested it for performance, but it may serve as inspiration. Gain maps require an auxiliary image. You could check the header and reject if that's missing. If it exists, you could then further check for a supported gain map (ie a more robust check, but you've already quickly skipped most images that aren't relevant). There are two kinds of map encoding supported by libultra: (a) ISO standard gain maps and (b) gain maps encoded with the Android XMP spec. The ISO encoding is widely used for new images now and has been for a while in most encoders. Many now dual encode (as the auxiliary image is the same and you're just writing redundant data - binary in the aux image for ISO and standard XMP for the Android spec). The only thing you'd miss by skipping XMP would be older images captured on Android or exported with old versions of Adobe software (Apple has a proprietary method they used pre-ISO and it is not supported by libultra). I believe skipping XMP would generally be ok to do, as it is a very limited edge case (and the impact would be that transcoding falls back to SDR rather than total failure). |
|
@adamsilverstein If the source includes a supported gain map, the output should preserve it by default. SharpJS is working towards a keepGainMap() method which should do this, however, I'm not sure how much guidance libvips needs. Coordinating with SharpJS may make sense here, as both efforts are likely chasing the same goals for default transcoding (ie ability to do basic crop/resize/compress while retaining an output which shows high fidelity to the original). Questions for transcoding will inevitably come up around tuning (compression of the base image / map). Testing to make sure the final result is ok based on the compression applied to both the base and map will be an important step to validate quality vs size objectives. Existing approaches for the base image probably translate well, and this is probably mostly a question of how to compress the map. Aside from that, most of the options chosen for encoding should remain as they were in the source. The metadata should be unchanged (HDR capacity, offsets, ranges, etc). The map scaling (1:1 vs 1:2 or 1:4) and number of channels (1 for luminosity only map vs 3 for full color) should remain the same. It should not alter sub-sampling (ie keep 444 if that's how it was encoded as the map is not a color image and the assumptions behind sub-sampling are not applicable in this domain, and can cause artifacts if altered). An affordance for aggressive compression of the image (including loss of HDR) may be useful for some users / workflows, but is not required and could get complicated. There are also other ways beyond stripping the map which can compress the image without stripping HDR, but they involve quality tradeoffs that would need careful assessment. For example, you could downgrade the map from 1:1 resolution in full color to a 1:2 map with luminosity only. That will cause loss of high frequency detail in the HDR rendition, as well as color error (which may be a small or significant issue depending on the relationship of the SDR to the HDR rendition). Due to the effort, complexity, and risk of confusion here, it may be best that initial support just preserve the gain map and not use more complex options to compress the image further. That can be managed now via the encoding of the file uploaded to WordPress (upload SDR if you only want SDR, phone captures are already low resolution luminosity maps, etc). Advanced transcoding for compression might best be left for 3rd-party plugins to address. |
great, thanks for the tip - i will take a look at that.
👍🏼
Right, be default we will always try to use the uploaded format for the output (so uploaded UltraHDR should output UltraHDR). My pondering was more about how would enable developers to choose to extract only the SDR image for output/front end if thats what they wanted. The current output mapping won't work, so we may need a more explicit filter to choose HDR or SDR encoding when an UltraHDR is uploaded. |
I created an issue to add a probe mode to lib-open-ultrahdr: |
Thanks again for the tip @gregbenz - I added the probe feature in adamsilverstein/lib-open-ultrahdr#6 based on the libultrahdr approach |
Thanks again for the question @andrewserong - it turns out libultrahdr has a |
adamsilverstein
removed request for
ajitbohra,
gziolo,
nerrad,
ntwb and
ryanwelcher
adamsilverstein
added
the
[Status] In Progress
# Conflicts: # packages/vips/CHANGELOG.md
Cover the new UltraHDR pipeline at three levels.
Vips unit (packages/vips/src/test/ultrahdr.ts): mock wasm-vips and
verify getUltraHdrInfo returns dims + log2-stop capacity for valid
inputs, returns null on missing gain map / decode failure, and falls
back to zero capacity when the metadata field is absent. Verify
resizeImage with isUltraHdr=true routes through uhdrloadBuffer +
thumbnailImage + uhdrsaveBuffer instead of newFromBuffer +
thumbnailBuffer + writeToBuffer.
Upload Media unit (packages/upload-media/src/store/test/
private-actions.js): mock vipsGetUltraHdrInfo and exercise the
detectUltraHdr thunk. Cover: positive detection writes
attachment.meta.ultrahdr + hdr_capacity, no metadata change when not
UltraHDR, graceful pass-through when probing throws, normalization
when REST returns meta as [] (rather than {}), and early return
when the queue item is missing.
E2E (test/e2e/specs/editor/various/client-side-media-processing.spec
.js): upload a synthesized 1024x768 UltraHDR JPEG fixture, fetch the
medium sub-size from REST, and re-probe it through wasm-vips to
confirm the gain map survived the resize. The fixture is generated
deterministically by .gen-ultrahdr-fixture.mjs (committed alongside
so it can be regenerated without vendoring a third-party binary).
| smartCrop = false, | ||
| quality = 0.82 | ||
| quality = 0.82, | ||
| isUltraHdr = false |
There was a problem hiding this comment.
You don't need this. Since uhdrload*() has a higher priority than jpegload*(), you can continue using vips.Image.newFromBuffer() (or vips.Image.thumbnailBuffer()). Likewise, jpegsave*() automatically delegates to uhdrsave* when the image contains a gainmap or uses the scRGB color space.
Here's an example to help you get started:
https://gist.github.com/kleisauke/f90c05b839ef871cbb97b294e9435bf7
There was a problem hiding this comment.
Fantastic, thank you!
There was a problem hiding this comment.
This also needs to crop the gainmap, if present. An example is available here:
https://www.libvips.org/API/current/uhdr.html#a-la-carte-processing
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
Address kleisauke's review on PR #74873: - Drop the explicit isUltraHdr branching in resizeImage. uhdrload* has higher priority than jpegload* in libvips, so newFromBuffer and thumbnailBuffer auto-detect UltraHDR JPEGs and decode the gain map alongside the base image. jpegsave* delegates to uhdrsave* on output when a gain map is attached, so writeToBuffer handles the save side too. Removes the parameter from vipsResizeImage at every layer. - Crop the gain map alongside the main image on positional crop. Scale crop coordinates to the gain map's resolution, since the gain map can be smaller than the main image. Follows libvips's a-la-carte processing example: https://www.libvips.org/API/current/uhdr.html#a-la-carte-processing - Clean up the keep:'icc|gainmap' save option formatting that was applied as a raw suggestion in the previous commit. Extend the ForeignKeep type to include 'gainmap' and pipe-combined values. Update tests to cover the simplified path, the new gain-map crop branch, and the keep:'icc|gainmap' save option.
|
Thanks again for the review @kleisauke — addressed all three points in f17ac14:
Tests in |
Apply prettier formatting and add eslint-disable directives for the legitimate console output and wasm-vips import (the script is a manually-run fixture generator, not part of CI).
The cache pins decoded image references, which prevents libheif from releasing its AVIF decoder state across iterations. On a 3000x2000 AVIF input, the WASM heap exhausts after a few runs and the AVIF encode fails with 'heif: Memory allocation error'. Calling Cache.max(0) once after init resolves it; the cache offered no value here since each iteration uses a fresh thumbnail call.
Resolve conflicts: - packages/upload-media/CHANGELOG.md: keep UltraHDR Enhancement under Unreleased, with the new 0.31.0 scaled-suffix bug fix entry below. - packages/vips/CHANGELOG.md: keep UltraHDR New Features under Unreleased, followed by the 1.6.0 release header. - packages/upload-media/src/store/private-actions.ts: keep UltraHDR detection step at the top of prepareItem(), and adopt trunk's removal of the client-side bigImageSizeThreshold ResizeCrop. Threshold scaling now happens as a sideload in generateThumbnails(), so the original is uploaded unchanged and the un-suffixed basename feeds sub-size naming.
|
@gregbenz this should be ready for testing and should also be testable using Playground as we resolved the incompatibility there. Try https://playground.wordpress.net/?gutenberg-pr=74873 |
|
@adamsilverstein Thank you! I am seeing that the basic functionality works. I drag and drop HDR JPGs and am able to pick smaller than full sizes and see HDR. I see a couple of issues:
My quick test in the Playground: https://playground.wordpress.net/scope:kind-quiet-country/2026/05/15/greg-hdr-test/ |
|
Thanks for testing @gregbenz
This should default to 82 quality and is adjustable using a code filter, there is no UI built into WordPress to adjust, but you can likely install a plugin to add that. eg install https://wordpress.org/plugins/image-quality/ on the playground instance. Separately while testing this I noticed a bug where the srcset isn't being generated correctly, even though I see all of the sub-sized images created on the backend. I am investigating this separately as it affects trunk and must have been introduced recently.
I haven't worked on the cropping functionality at all, I know there is active work on that at the moment. It might be worth testing this in the new experimental media editor, accessible under the Gutenberg->Experiments menu: 
|
|
@adamsilverstein I am seeing that the editor does not match the rendered page. When editing, there is no srcset and the lowest res image is used. It is rather low quality. When viewing the published page, a srcset is used and what I see shows a vastly better image. It looks the same size on screen, but is appears to be pulling from a higher res image and downsampling in the browser. Shouldn't the editor use srcset to render a preview closer to the live page? Here is the rendered image element:
|
|
@adamsilverstein I'm not seeing that image-quality plugin impact results. If I install it and set it to 1% or 100% quality and then upload an image to a post, I see the same results. Perhaps the way the quality setting is passed normally does not flow through here? Or I may not be using it properly in this context. However, I think the quality issue I see is something else. When I test my own implementation of wasm-vips (https://gregbenzphotography.com/test/wasm-vips-hdr-demo/), I get a much higher quality result for the 150x150 thumbnail at 82%. My image is 25K vs 13K when WordPress beta does the resizing. The quality factor applied to the gain map does not appear consistent what is being applied to the base image (my relevant JS for transcoding via wasm-vips can be seen with a link at the bottom of that page). That said, I'm not getting down to 13KB without vastly lower quality settings, so something is not apples to apples here. Try using the "load P3 sample" on my site. That same image resized in the WordPress playground shows soft green text near the top. The quality from the WP beta may ultimately be worth it given the size (the loss of quality here is much less for image content than this part of the test pattern), but I think it would be good to understand the discrepancy. Separately for testing efficiently: In the playground, I'm not sure how to download the resized asset (the way the URL shows on page is not standard and the browser shows an error when saving: "file wasn't available on site", and the media library only shows the full-sized asset). I found a very convoluted solution via JS, but if you have any tips to pull the transcoded image, that would be helpful. The colon in the image URL in the playground seems to be an issue. |
Hmm, maybe the quality setting isn't being applied to the gain map? I'll have to look into what options there are.
can you give me a link to the image or upload it here?
You should be able to use the urls in the html srcset to download each image, except your urls do look odd, eg I can try to set up a regular test environment for you with this PR active, it may work differently than in playground. |
|
@adamsilverstein See gainmapQuality in gregbenzphotography.com/test/wasm-vips-hdr-demo/wasm-vips-hdr-transcode.js for how I'm controlling it independently of the base quality (I set same as default but my demo page allows independent control). The "scope:" in the image URL shows up with standard JPGs, I'm assuming this is the nature of the playground environment. |
|
hmmm. i thought we already returned the filtered with that in place, we can:
|
# Conflicts: # packages/upload-media/src/store/private-actions.ts
Resolve conflicts: - packages/vips/src/index.ts: keep getUltraHdrInfo; drop batchResizeImage (removed in trunk's 2.0.0 breaking change, #77247). - packages/vips/CHANGELOG.md: keep UltraHDR entries under Unreleased; place trunk's 2.0.0 (2026-05-27) breaking change section below. - packages/upload-media/CHANGELOG.md: keep UltraHDR enhancement under Unreleased; place trunk's 0.32.0 (2026-05-27) bug fix section below.
|
Thanks for the careful reviews, @kleisauke and @andrewserong! Just rebased on @kleisauke —
|
getVips() calls vipsInstance.Cache.max(0) to disable libvips's operation cache. Without it in the wasm-vips mock, the first getUltraHdrInfo test threw inside getVips, never invoking uhdrloadBuffer. Because the queued mockReturnValueOnce values were not consumed and jest.clearAllMocks() does not drain them, the next tests then read stale mock returns and produced unexpected results.
kleisauke
left a comment
kleisauke
left a comment
There was a problem hiding this comment.
I'm not a WordPress expert, but the wasm-vips part and its usage w.r.t. UltraHDR processing looks good to me.
|
@gregbenz this is ready for additional testing |
|
Thanks for the updates @adamsilverstein! Just double-checking but is the intention that generated sub-size images retain their gain maps? In manual testing (with one Greg's lovely photos: https://gregbenzphotography.com/hdr-gain-map-gallery/) it appears that the uploaded file has the gain map, but if I go to view the generated subsize jpegs on my phone (the only HDR display I have unfortunately!), they appear to be rendering in SDR and the highlights are no longer super bright. Is it working for you? I tried running (Code-wise this is looking nice, good work simplifying things!) |
|
|
||
| ### Enhancement | ||
|
|
||
| - UltraHDR (ISO 21496-1 gain map) JPEGs are now detected and resized via libvips's native `uhdrload`/`uhdrsave` pipeline. The standalone `open-ultrahdr` and `open-ultrahdr-wasm` dependencies have been removed; gain maps are preserved automatically through the existing resize step ([#74873](https://github.com/WordPress/gutenberg/pull/74873)). |
There was a problem hiding this comment.
The standalone
open-ultrahdrandopen-ultrahdr-wasmdependencies have been removed;
Tiniest of nits for this changelog entry, mightn't even be worth updating, but we never landed those did we? We can likely remove that sentence.
| const { promisify } = require( 'util' ); | ||
| const { confirm } = require( '@inquirer/prompts' ); | ||
| const checkSync = require( 'check-node-version' ); | ||
| const tools = require( 'check-node-version/tools' ); | ||
| const { promisify } = require( 'util' ); |
There was a problem hiding this comment.
Probably not an issue at all, but this change seems unintentional / a stray leftover from when we needed to build the separate libraries?
There was a problem hiding this comment.
right, will restore.
| /** | ||
| * Tracks parent item IDs whose source file is an UltraHDR JPEG so that | ||
| * sub-size resize operations can route through libvips's uhdrload/uhdrsave | ||
| * to preserve the gain map. Entries are cleared when the parent finalizes. | ||
| */ | ||
| const ultraHdrItems = new Set< QueueItemId >(); | ||
|
|
There was a problem hiding this comment.
This is probably so minor it's not really worth mentioning, but I noticed with a Claude-assisted review that the items in this Set get cleared when the processing is successful, but not when it's cancelled. It doesn't seem that having the set increase over time will cause any issues, but just thought I'd mention it in case it's worth clearing when we cancel an item. Though that appears to happen in actions.ts rather than private-actions.ts so would involve sharing the set across multiple modules, which might not be ideal.
In any case, probably not much of an issue, but thought I'd flag it just in case 🙂
4 participants
Summary
@wordpress/vipsworker — no extra WASM module is bundledCloses #74874
Similar to adamsilverstein/client-side-media-experiments#15
Approach
UltraHDR support comes from
wasm-vips(libvips compiled to WebAssembly), which gained native UltraHDR (uhdrload/uhdrsave) in0.0.17. This was unblocked bygoogle/libultrahdr#386, which dual-licensed libultrahdr underApache-2.0 OR MITand made it compatible with WordPress's GPLv2-or-later codebase.@wordpress/vipsis bumped towasm-vips@^0.0.17and gains:getUltraHdrInfo(buffer)helper that probes a JPEG for an embedded gain map and reports HDR headroomisUltraHdrflag onresizeImagethat routes throughuhdrloadBuffer+uhdrsaveBufferso the gain map is downsized in lockstep with the base image and re-embedded on save@wordpress/upload-mediauses these helpers in its existingDetectUltraHdroperation and tags sub-size resizes withisUltraHdrso the resize step transparently preserves the gain map. Thanks to @kleisauke for landing native UltraHDR support inwasm-vips.Removed
open-ultrahdrandopen-ultrahdr-wasmdependencies are no longer needed.EncodeUltraHdrqueue operation type is gone — there's no separate re-encode step; resize handles it inline.encodeUltraHdrItem) is removed.Follow-up
wasm-vips@0.0.18will land a docs-only update toTHIRD-PARTY-NOTICES.mdto make the dual-licensing explicit. We can bump to^0.0.18when it's published.Test plan