Add support for lossy HTJ2K compressor by palemieux · Pull Request #2425 · AcademySoftwareFoundation/openexr

palemieux · 2026-05-17T21:07:05Z

add support for a separate lossy HTJ2K compressor (using a log transfer function as opposed to DWA's log-gamma transfer function)
adds log MSE computation to exrmetrics (to measure the distortion introduced by lossy compressors)

See https://academysoftwarefdn.slack.com/archives/CMLRW4N73/p1777317753088359

cary-ilm · 2026-05-27T17:26:34Z

@Vertexwahn, any suggestion to fix the Bazel failure here? I see two errors in the logs:

src/lib/OpenEXRCore/internal_ht.cpp:10:10: fatal error: openjph/ojph_arch.h: No such file or directory
   10 | #include <openjph/ojph_arch.h>

and:

In file included from src/lib/OpenEXRCore/base.c:8:
src/lib/OpenEXRCore/openexr_version.h:11: warning: "OPENEXR_VERSION_MAJOR" redefined
11 | # define OPENEXR_VERSION_MAJOR 4

cary-ilm · 2026-05-27T17:27:10Z

And @palemieux, can you amend the commits with -s to add the required DCO?

Vertexwahn · 2026-05-28T08:20:26Z

Short time solution: Change in MODULE.bazel the line bazel_dep(name = "openjph", version = "0.27.3") to bazel_dep(name = "openjph", version = "0.27.0")

Vertexwahn · 2026-05-28T09:02:29Z

Once bazelbuild/bazel-central-registry#9032 is merge you can change bazel_dep(name = "openjph", version = "0.27... to bazel_dep(name = "openjph", version = "0.27.3.bcr.1") (bcr.1 is a patch that contains the missing include path)

peterhillman · 2026-05-28T21:32:05Z

It might be better to put the exrmetrics changes into a separate PR (apart from minimal support for modifying the amount of compression). The MSE changes aren't directly related to lossy HTJ2K

cary-ilm

Just to confirm, are the changes to the vendored OpenJPH simply from vendoring in 0.27.3? Are there any changes we need to make sure don't get stomped on?

palemieux · 2026-05-29T03:58:49Z

Just to confirm, are the changes to the vendored OpenJPH simply from vendoring in 0.27.3? Are there any changes we need to make sure don't get stomped on?

Stock 27.3. No additional changes. Thanks for making sure.

palemieux · 2026-05-29T04:00:16Z

It might be better to put the exrmetrics changes into a separate PR (apart from minimal support for modifying the amount of compression). The MSE changes aren't directly related to lossy HTJ2K

Ok. Will split it off before the lossy J2K branch is merged.

cary-ilm · 2026-05-30T02:52:12Z

@peterhillman and @palemieux, since this PR is going into the lossy-htj2k feature branch, the exrmetrics changes can stay here and also go into a separate PR for main. That will highlight them more prominently in the commit history and release notes.

palemieux · 2026-05-30T02:58:10Z

Ok. I can create a PR against main.

cary-ilm · 2026-05-31T17:42:44Z

@palemieux, I think this is only waiting on @michaeldsmith's comment suggestions?

palemieux · 2026-06-01T03:05:11Z

See #2448 for a standalone PR adding MSE to exrmetrics.

cary-ilm

A few minor suggestions, otherwise it looks ready to go to me.

Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com>

Signed-off-by: Pierre-Anthony Lemieux <pal@palemieux.com>

Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com>

* Improve the handling of non-finite samples Signed-off-by: Pierre-Anthony Lemieux <pal@palemieux.com>

Co-authored-by: Cary Phillips <cary@ilm.com> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com>

Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com>

* Fix install manifest for linux build 13 (#2466) The hard-coded 4_0 release number in the new install manifest should have been converted to a symbolic value. Signed-off-by: Cary Phillips <cary@ilm.com> * Bump sigstore/gh-action-sigstore-python from 3.3.0 to 3.4.0 (#2467) Bumps [sigstore/gh-action-sigstore-python](https://github.com/sigstore/gh-action-sigstore-python) from 3.3.0 to 3.4.0. - [Release notes](https://github.com/sigstore/gh-action-sigstore-python/releases) - [Changelog](https://github.com/sigstore/gh-action-sigstore-python/blob/main/CHANGELOG.md) - [Commits](sigstore/gh-action-sigstore-python@04cffa1...5b79a39) --- updated-dependencies: - dependency-name: sigstore/gh-action-sigstore-python dependency-version: 3.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add section on CVE Assignment to SECURITY.md (#2471) * Bump msys2/setup-msys2 from 2.31.1 to 2.32.0 (#2474) Bumps [msys2/setup-msys2](https://github.com/msys2/setup-msys2) from 2.31.1 to 2.32.0. - [Release notes](https://github.com/msys2/setup-msys2/releases) - [Changelog](https://github.com/msys2/setup-msys2/blob/main/CHANGELOG.md) - [Commits](msys2/setup-msys2@e989830...66cd2cc) --- updated-dependencies: - dependency-name: msys2/setup-msys2 dependency-version: 2.32.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix infinite loop in SampleCountChannel roundListSizeUp() (#2468) Reject sample counts above 0x80000000 before rounding up to the next power of two, since larger values cause the unsigned shift loop to wrap and never terminate. Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Reject NULL bytes type_hint and data in exr_attr_bytes_create() (#2470) Validate non-null pointers when hint length or data size is non-zero, and apply the same checks in the in-place exr_attr_set_bytes() path. Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Drop support for Python 3.8 (#2477) 3.8 was never actually included in any vfx reference platform year, and the platform has been at Python 3.9+ since CY2022. And cibuildwheel 4.0 drops support for 3.8. This will take effect in OpenEXR v3.5. Signed-off-by: Cary Phillips <cary@ilm.com> * Fix the DWAA size checks (#2476) * Bump pypa/cibuildwheel from 3.4.1 to 4.1.0 (#2475) Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 3.4.1 to 4.1.0. - [Release notes](https://github.com/pypa/cibuildwheel/releases) - [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md) - [Commits](pypa/cibuildwheel@8d2b08b...2947353) --- updated-dependencies: - dependency-name: pypa/cibuildwheel dependency-version: 4.1.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cary Phillips <cary@ilm.com> * Unlock write context in exr_get_chunk_table_offset() return paths (#2472) EXR_LOCK_WRITE_AND_DEFINE_PART() acquires the mutex on write contexts but both success and error returns leaked the lock, deadlocking later API calls such as exr_get_count(). Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Split macOS wheel arches across CI runners (#2479) Both macOS matrix jobs previously built x86_64, arm64, and universal2, producing duplicate wheel filenames. Prior to #2433, the workflow downloaded each built wheel in succession, which would cause the duplicate wheel to overwrite the original, but it avoided an error. The new merge-multiple in the publish job corrupt the files, causing TestPyPI uploads to fail with "Mis-matched data size". This change builds x86_64 and universal2 on macos-15-intel and arm64 on macos-latest so merged artifacts have unique names. This also adds a validation step to the python-wheels.yml workflow to ensure the wheels have the expected structure. Signed-off-by: Cary Phillips <cary@ilm.com> * Fix CI-old workflow failure due to Node 24 on old VFX platform containers (#2483) GitHub Actions deprecated Node 20 and now forces Node 24 by default, which requires glibc >= 2.25. The VFX 2021/2022 containers are CentOS 7 (glibc 2.17) and can't run Node 24, breaking the checkout step. Set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true to allow the runner to use the Node 20 glibc-2.17 build installed in /node20217, which is the existing workaround for these old containers. Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Reject negative pixelDataSize in rawPixelDataToBuffer() (#2490) A negative int cast to uint64_t bypassed the buffer size check and allowed exr_read_chunk() to write past a caller buffer. Addresses GHSA-gjpj-qv64-vwhf Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Fix OpenEXRUtil row() OOB read with non-zero data window origin (#2488) The row() method in TypedDeepImageChannel, TypedFlatImageChannel, and SampleCountChannel is documented as 0-based within the data window, but used the _base offset meant for absolute (x, y) access. Index the compact backing arrays instead so row(0) aliases the first pixel when dataWindow.min is non-zero. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-hwmv-39v6-739m Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Fix deep pixelmode heap buffer overflow in exrmetrics (#2487) When --pixelmode float (or --bench) is used, outHeader channels are rewritten from the input type (e.g. HALF) to FLOAT. The DeepSlice framebuffer entries were already built from outHeader, requesting 4-byte FLOAT writes. However, the sample data buffers were allocated using the *input* header's channel types (2 bytes for HALF), so readPixels() wrote 4-byte values into 2-byte buffers — a heap buffer overflow. Fix: allocate sampleData using outHeader.channels() (the output/requested type) so that buffer sizes match the per-sample write size. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-6jj8-cxcr-j8hm Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Bump vmactions/freebsd-vm from 1.4.6 to 1.4.8 (#2495) Bumps [vmactions/freebsd-vm](https://github.com/vmactions/freebsd-vm) from 1.4.6 to 1.4.8. - [Release notes](https://github.com/vmactions/freebsd-vm/releases) - [Commits](vmactions/freebsd-vm@a6de934...b84ab55) --- updated-dependencies: - dependency-name: vmactions/freebsd-vm dependency-version: 1.4.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Reject oversized DWAA buffer allocations on ILP32 (#2494) Check packed AC/DC and RLE buffer sizes against SIZE_MAX before calling alloc_fn(size_t). Without this, uint64_t sizes above 4 GiB truncate on 32-bit builds while later DWAA decode logic still uses the full size, leading to heap corruption during AC inflate. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-6662-fq6f-93mp Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Fix ILP32 unpacked buffer size truncation in RLE decode (#2493) In update_pack_unpack_ptrs(), decode->chunk.unpacked_size is uint64_t but was passed directly to internal_decode_alloc_buffer() which takes size_t. On ILP32 builds a crafted chunk with unpacked_size > 4 GiB silently truncates, allocating a tiny buffer while the decompressor and unpacker iterate over the full declared channel width, causing an out-of-bounds read. Add a truncation check before the allocation and return EXR_ERR_OUT_OF_MEMORY if unpacked_size exceeds SIZE_MAX. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-v6v5-344m-64vm Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix signed integer overflow in unpack_sample_table() (#2484) * Fix signed integer overflow in unpack_sample_table() Declare w and h as int64_t instead of int32_t so that index arithmetic y*w and w*h in unpack_sample_table() does not overflow when tile dimensions are large (e.g. 46342x46342). Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-rqp5-pmwm-wj6x Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Cary Phillips <cary@ilm.com> * declare y as int64_t Signed-off-by: Cary Phillips <cary@ilm.com> --------- Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Bump actions/checkout from 6.0.3 to 7.0.0 (#2481) Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.3 to 7.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@df4cb1c...9c091bb) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cary Phillips <cary@ilm.com> * Fix ILP32 deep sample-count table size overflow in decoding (#2492) In update_pack_unpack_ptrs(), sampsize was computed as size_t, so on ILP32 builds a tile with width*height >= 2^30 wraps to zero (or four with the INDIVIDUAL flag), causing the sample-count table to be allocated too small and unpack_sample_table() to read far past it. Compute the size in uint64_t, check for truncation before converting to size_t, and return EXR_ERR_OUT_OF_MEMORY if the required size exceeds SIZE_MAX. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-p42q-g5c9-mh9w Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix empty multiView crash in viewFromChannelName() (#2496) Dotless channel names used multiView[0] without checking for an empty vector. Use defaultViewName() so malformed files return "" instead of OOB access. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-g8f2-r72m-48vx Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Avoid memory allocation in idmanifest parsing (#2498) * Avoid memory allocation in idmanifest parsing Signed-off-by: Peter Hillman <peterh@wetafx.co.nz> * check for negative string count, fix table size count Signed-off-by: Peter Hillman <peterh@wetafx.co.nz> --------- Signed-off-by: Peter Hillman <peterh@wetafx.co.nz> Co-authored-by: Cary Phillips <cary@ilm.com> * Fix integer overflow in Array2D::resizeErase() on ILP32/LLP64 builds (#2486) * Fix integer overflow in Array2D::resizeErase() on ILP32/LLP64 builds sizeX * sizeY used long arithmetic, which is 32 bits on ILP32 (32-bit Linux) and on Windows (MSVC LLP64). A tile size of 65537x65537 overflows to a small positive value, causing new[] to under-allocate and subsequent tile decoding to write past the buffer. Cast both operands to size_t before multiplying so the allocation is always correct regardless of the width of long. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-f667-c4wm-c8gq Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Cary Phillips <cary@ilm.com> * validate non-negative sizes Signed-off-by: Cary Phillips <cary@ilm.com> --------- Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Reject NULL preview rgba data in exr_attr_set_preview() (#2469) Validate the source rgba pointer before memcpy in exr_attr_preview_create() and the same-dimensions update path in exr_attr_set_preview(), preventing a NULL pointer dereference when nonzero preview dimensions are supplied without pixel data. Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Reject NULL channel list entries in exr_attr_chlist_duplicate() (#2473) Validate num_channels > 0 with entries == NULL before pointer arithmetic, covering exr_set_channels() and exr_attr_set_channels(). Also, refactor exr_attr_set_channels to call exr_attr_chlist_duplicate() since it does the same work. Addresses https://github.com/AcademySoftwareFoundation/openexr/security/advisories/GHSA-mqmw-xv8w-5jh4 Signed-off-by: Cary Phillips <cary@ilm.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Name::operator= to null-terminate long strings (#2500) Name::operator= leaves the string without a terminating null if the string is longer than MAX_LENGTH. Write a null to ensure there is one. Signed-off-by: Cary Phillips <cary@ilm.com> * Add support for lossy HTJ2K compressor (#2425) * Add lossy HTJ2K Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Add mean-square error (MSE) to exrmetrics Signed-off-by: Pierre-Anthony Lemieux <pal@palemieux.com> * Improve float metric Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Fix exrmetrics bug Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * * Remove MSE computation for UINT samples * Improve the handling of non-finite samples Signed-off-by: Pierre-Anthony Lemieux <pal@palemieux.com> * Update src/lib/OpenEXRCore/openexr_part.h Co-authored-by: Cary Phillips <cary@ilm.com> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Update src/test/OpenEXRTest/compareHTJ2KL256.cpp Co-authored-by: Cary Phillips <cary@ilm.com> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Update src/lib/OpenEXRCore/internal_ht.cpp Co-authored-by: Cary Phillips <cary@ilm.com> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Update src/lib/OpenEXRCore/openexr_part.h Co-authored-by: Cary Phillips <cary@ilm.com> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Update src/lib/OpenEXRCore/openexr_base.h Co-authored-by: Cary Phillips <cary@ilm.com> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Update src/lib/OpenEXRCore/openexr_base.h Co-authored-by: Cary Phillips <cary@ilm.com> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> * Replaced MSE with the more generic "distortion" Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> --------- Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> Signed-off-by: Pierre-Anthony Lemieux <pal@palemieux.com> Co-authored-by: Cary Phillips <cary@ilm.com> --------- Signed-off-by: Cary Phillips <cary@ilm.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Peter Hillman <peterh@wetafx.co.nz> Signed-off-by: Pierre-Anthony Lemieux <pal@sandflow.com> Signed-off-by: Pierre-Anthony Lemieux <pal@palemieux.com> Co-authored-by: Cary Phillips <cary@ilm.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: vlazar-ilm <vlazar@ilm.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: peterhillman <peterh@wetafx.co.nz>

palemieux requested a review from cary-ilm May 17, 2026 21:07

palemieux force-pushed the feature/lossy-htj2k branch 3 times, most recently from b87f94b to 5011ecd Compare May 28, 2026 18:22

michaeldsmith reviewed May 28, 2026

View reviewed changes

Comment thread src/bin/exrmetrics/exrmetrics.cpp Outdated

Comment thread src/bin/exrmetrics/exrmetrics.cpp Outdated

cary-ilm reviewed May 29, 2026

View reviewed changes

Comment thread cmake/OpenEXRSetup.cmake

cary-ilm reviewed May 29, 2026

View reviewed changes

Comment thread src/lib/OpenEXR/ImfHeader.h

cary-ilm reviewed May 29, 2026

View reviewed changes

TodicaIonut reviewed May 29, 2026

View reviewed changes

Comment thread src/lib/OpenEXR/ImfCompressor.cpp