Fix multi-view OOM on Turing GPUs: select bf16 autocast only where bf16 is native

[jannisringwald]

What

One-line behavioral change in DepthAnything3.forward (src/depth_anything_3/api.py): pass including_emulation=False to torch.cuda.is_bf16_supported() when choosing the autocast dtype.

# before
autocast_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
# after
autocast_dtype = (
    torch.bfloat16
    if torch.cuda.is_bf16_supported(including_emulation=False)
    else torch.float16
)

Why

torch.cuda.is_bf16_supported() returns True on GPUs that only emulate bf16, e.g. Turing / sm_75 (T4). On those GPUs the bf16 pick interacts badly with SDPA in the global (cross-view) attention blocks:

1. bf16 autocast is selected because is_bf16_supported() returns True;
2. SDPA's memory-efficient kernel rejects bf16 on sm_75;
3. attention silently falls back to the math backend;
4. the math backend materializes the full [H, N, N] attention-score matrix in fp32.

At multi-view token counts that is one enormous allocation. Measured on a 16 GB T4 (vitb, 12 heads, 378×504 inputs → ~973 tokens/view, 14 views → N = 13,622):

autocast dtype	attention path	peak allocation for scores	outcome
bf16 (current)	math-backend fallback	8.30 GiB (12 × 13,622² × 4 B)	OOM
fp16 (this PR)	memory-efficient kernel	~0.06 GiB	completes

With including_emulation=False, fp16 is selected on those GPUs, the memory-efficient kernel engages, and 14-view inference completes with plenty of headroom.

Precision

fp16 carries more mantissa bits than bf16 (10 vs 7) and matmul accumulation stays fp32 under autocast; we saw depth-output parity between the two dtypes on runs that fit in memory. On Ampere and newer, is_bf16_supported(including_emulation=False) still returns True, so behavior there is unchanged — the change only affects GPUs that would otherwise hit the math-backend fallback.

Compatibility

The including_emulation kwarg exists since torch 2.4.0 (July 2024). pyproject.toml declares torch>=2; if you'd rather keep torch 2.0–2.3 importable, I'm happy to guard the call with a try/except TypeError fallback instead — let me know which you prefer.

Related: #268 removes a latent per-head mask materialization of the same magnitude in the same attention blocks.

🤖 Generated with Claude Code