Skip to content

LDM3D rgblike_to_depthmap truncates the 16-bit depth map back to the 8-bit input dtype #14206

Description

@chuenchen309

Describe the bug

VaeImageProcessorLDM3D.rgblike_to_depthmap combines two 8-bit channels into a 16-bit depth value and then casts the result back to the 8-bit input dtype, truncating it. postprocess(output_type="pil") then feeds that into numpy_to_depth, which builds a mode="I;16" image and fails.

Reproduction

import numpy as np
from diffusers.image_processor import VaeImageProcessorLDM3D

p = VaeImageProcessorLDM3D()
img = np.zeros((1, 8, 8, 6), dtype=np.uint8)
img[..., 1] = 200   # high byte
img[..., 2] = 100   # low byte
p.postprocess(img, output_type="pil")

Logs

ValueError: buffer is not large enough

Cause

image_processor.py:1048-1075. Both branches widen for the arithmetic and then narrow again:

original_dtype = image.dtype
image_safe = image.to(torch.int32)
depth_map = image_safe[:, :, 1] * 256 + image_safe[:, :, 2]
return depth_map.to(original_dtype)      # 16-bit result -> 8-bit dtype

The function's whole purpose is to produce a 16-bit depth map from 8-bit RGB, so casting back to the input dtype is never correct. The commented-out # depth_map = depth_map.to(torch.uint16) lines right there suggest this was known but left undone.

Introduced by #12546, which fixed a real NumPy 2.0 uint8 * 256 overflow and replaced it with this truncation. tests/others/test_image_processor.py has no VaeImageProcessorLDM3D coverage, which is why it shipped.

Proposed fix

Return the wider integer type instead of narrowing: numpy → uint16 (matching the mode="I;16" consumer), torch → the int32 combination.

Coordination note

I should have opened this before the PR rather than after — the AI-agent guidelines ask for an explicit maintainer acknowledgment on an issue first, and I skipped that step. #14200 already has the fix and a regression test; I'm filing this so the discussion has somewhere to live and so you can say "no, do it differently" without anyone having reviewed code first. Happy to close the PR and wait if you'd prefer to reset the order properly.

Separate, deliberately out of scope: output_type="np" passes float [0, 1] arrays into the same function, where the int cast collapses them to 0. That needs a decision about that branch's contract, so I left it alone rather than guessing.

System Info

diffusers main, Python 3.12, numpy 2.x

Who can help?

@yiyixuxu @sayakpaul

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions