[LDM3D] Fix rgblike_to_depthmap truncating 16-bit depth back to 8 bits

[chuenchen309]

What does this PR do?

VaeImageProcessorLDM3D.rgblike_to_depthmap combines two 8-bit channels into a 16-bit depth value (green * 256 + blue, range 0–65535). It correctly casts up to a wider integer type for the multiplication, but then casts the result back to the input's dtype (uint8) via .to(original_dtype) / .astype(original_dtype), dropping the entire high byte — so it returned only depth % 256.

numpy_to_depth feeds this into Image.fromarray(..., mode="I;16"), which requires 16-bit-wide data, so with a uint8 buffer VaeImageProcessorLDM3D.postprocess(image, output_type="pil") — the normal way an LDM3D pipeline returns a depth image for a 6-channel output — crashes with ValueError: buffer is not large enough.

from diffusers.image_processor import VaeImageProcessorLDM3D
import torch, numpy as np
p = VaeImageProcessorLDM3D()

rgb = np.zeros((2, 2, 3), dtype=np.uint8); rgb[..., 1] = 5; rgb[..., 2] = 200
p.rgblike_to_depthmap(rgb).max()   # before: 200   after: 1480  (5*256+200)

img = torch.zeros(1, 6, 4, 4); img[:, 4] = 5/255.; img[:, 5] = 200/255.
p.postprocess(img, output_type="pil")   # before: ValueError: buffer is not large enough
                                         # after:  returns a valid mode="I;16" depth image

The fix keeps the 16-bit result instead of truncating: return the wider integer type directly (numpy → uint16 to match the mode="I;16" consumer, torch → the int32 combination). The function's whole purpose is to produce a 16-bit depth map from 8-bit RGB, so casting back to the 8-bit input dtype was never correct.

This regression was introduced by #12546 (which fixed a real NumPy 2.0 uint8 * 256 overflow but replaced it with this truncation). tests/others/test_image_processor.py had no VaeImageProcessorLDM3D coverage, which is why it shipped — this adds a small regression test class.

Note (kept out of this PR to stay focused): the output_type="np" branch of postprocess passes float [0, 1] arrays into rgblike_to_depthmap, where the int cast collapses them to 0. That's an independent caller-side issue in the same function's contract and is better handled separately.

Before submitting

• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes? (no user-facing doc change — internal correctness fix)
• Did you write any new necessary tests? (added Ldm3dImageProcessorTest: 16-bit combination for numpy + torch, and the pil postprocess path; red before the fix, green after. Full test_image_processor.py: 12 passed. ruff 0.9.10 check + format clean.)

Who can review?

@yiyixuxu @sayakpaul

---

Found via an AI-assisted (Claude) review pass over image_processor.py; independently reproduced, fixed, and verified before submitting.

[github-actions]

Hi @chuenchen309, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

[chuenchen309]

Belated, but owed: I used an AI agent on this PR and didn't run your self-review skill or tick the AI-agent boxes in the template before opening it. Doing it properly now rather than leaving the checkboxes silently unfilled.

Self-review report — rubric: .ai/review-rules.md + .ai/AGENTS.md, over git diff main...HEAD.

Blocking issues

None found.

Non-blocking issues

1. Comment density above what the diff earns — src/diffusers/image_processor.py:1050-1072. The change is 4 lines of code carrying 8 lines of comment. Two of them restate the code rather than explain it:

   # Calculate the 16-bit depth map: high-byte * 256 + low-byte.
   depth_map = image_safe[:, :, 1] * 256 + image_safe[:, :, 2]

   The expression says that. Per review-rules.md ("State the reason so the comment stands alone, or drop it"), the reason worth keeping is only the non-obvious one — why the result must not be cast back to the input dtype. I'd cut it to one comment per branch. Happy to push that if you agree; leaving it as-is for now so the diff you may already have looked at doesn't move under you.

2. Ephemeral context risk — the comments I kept are phrased against the old code ("must not be truncated back to the 8-bit input dtype"), which reads as an argument with the previous implementation rather than as standalone context. A future reader has no truncation to compare against. Same rule as above.

Dead code (advisory)

path:line	verdict	reason
image_processor.py:1048-1075 rgblike_to_depthmap	Used	Called from postprocess → numpy_to_depth (:1103) on the output_type="pil" LDM3D path; exercised end-to-end by the added test.

No new model/pipeline here, so the new-model trace doesn't apply.

Suggestions / additional info

• Docs: no user-facing surface changed (internal correctness fix), so no docs/ update is implied.
• Agent docs: nothing new — the gotcha here (a cast-back-to-input-dtype silently truncating a widened result) is generic Python/NumPy, not a diffusers-specific rule worth recording.

Left for the actual review, deliberately not fixed

• output_type="np" passes float [0,1] arrays into rgblike_to_depthmap, where the int cast collapses them to 0. Real, independent of this bug, and fixing it means deciding what that branch's contract should be — a call for a maintainer, not something to guess at inside a truncation fix. Flagged in the description; raising it here explicitly since your skill asks for exactly this category.

Verdict

READY, with the comment trim above as the one thing I'd change if you want it. No blocking issues; the behavioural claim (16-bit result reaches the mode="I;16" consumer intact) is covered by the added regression test, red before and green after.

[chuenchen309]

Following up on my own self-review note above: I went and read the Coding with AI agents guide properly, which the template links and I hadn't opened. It asks for three things in the PR description that I didn't provide:

1. A coordination link to an issue where a maintainer acknowledged the work — "wait for an explicit acknowledgment from a maintainer on that issue before opening a PR."
2. The test commands and their output — not just "tests pass".
3. Self-review notes — done above, belatedly.

On (1): I opened the PR without an issue at all. Filed #14206 now so the discussion has somewhere to live, but I'm aware that's backwards — the point of coordinating first is that you get to say "no" or "do it differently" before anyone reviews code. Happy to close this PR and wait on #14206 instead if you'd rather reset the order properly; no argument from me.

On (2), the commands and output:

$ pytest tests/others/test_image_processor.py -q
12 passed

$ pytest tests/others/test_image_processor.py -q -k Ldm3d     # on main, before the fix
FAILED ... test_ldm3d_rgblike_to_depthmap_numpy   - ValueError: buffer is not large enough
FAILED ... test_ldm3d_rgblike_to_depthmap_torch   - assert 16-bit value survived
FAILED ... test_ldm3d_postprocess_pil             - ValueError: buffer is not large enough
3 failed

$ ruff check . && ruff format --check .
All checks passed!

The three added cases are red on main and green on this branch; the full file is green either way apart from them, so nothing else moved.