Skip to content

bf16 medium DiT: FMHA-fused TRT engine build recipe + CLI (bf16 = medium default, fp16-mixed kept)#73

Merged
Cortexelus merged 1 commit into
mainfrom
bf16-medium-dit-release
Jul 15, 2026
Merged

bf16 medium DiT: FMHA-fused TRT engine build recipe + CLI (bf16 = medium default, fp16-mixed kept)#73
Cortexelus merged 1 commit into
mainfrom
bf16-medium-dit-release

Conversation

@Cortexelus

Copy link
Copy Markdown
Collaborator

bf16 medium DiT — FMHA-fused TRT engine

Adds a bf16 DiT precision for medium as the new speed default, and wires --precision through the CLI and SA3Inference. fp16-mixed (canonical) and fp32 remain fully selectable.

What changed

  • Build recipe (build/build_from_onnx.py, build/build.py): new sa3-m-bf16 target. Reuses the same raw fp32 sa3-m/dit.onnxno new ONNX export — and builds with BuilderFlag.BF16 + EXPLICIT_BATCH (not STRONGLY_TYPED). bf16 carries fp32's dynamic range, so the FP32-softmax islands that fp16-mixed had to keep vanish and TRT 10.15's FMHA fuser fires (0 → 96 fused _gemm_mha_v2 nodes). Build-time precision recipe only — no graph change.
  • Precision registry + CLI (scripts/sa3_trt_core.py, scripts/sa3_trt.py): PRECISIONS gains bf16; per-model default resolution via default_precision(). --precision now defaults to None → resolves to bf16 for medium, fp16mixed for sm-music/sm-sfx. bf16 is medium-only — sm-music/sm-sfx use standard attention and already fuse in fp16-mixed, so requesting bf16 for them errors out.
  • README: --precision section with the per-model default table.

Speed (H100 SXM, medium)

~1.77× @ L=256 and ~4.70× @ L=4096 vs fp16-mixed, within the perceptual re-seed floor (FAD 0.59× floor, CLAP within spread, n=128).

Reproducibility caveat

bf16 is not seed-reproducible vs fp16-mixed. The medium DiT uses differential attention (cancellation-sensitive), so bf16 is a different-but-equal take per seed — quality within the re-seed floor, but exact samples differ. Use --precision fp16mixed when you need bit-reproducibility or max per-step fidelity.

Engine distribution

The dit_bf16.trt engine is gitignored and ships via HF (stabilityai/stable-audio-3-optimized), auto-downloaded on first use. No engine binary is included in this PR.

Notes

  • The TRT DiT engines are static batch=1 (the ONNX bakes batch=1), so CFG runs as a sequential cond+uncond dual-pass at batch=1 for every precision — unchanged by bf16.
  • Separate / orthogonal bug: a pre-existing medium + same-l + cfg>1 malloc crash exists and is not related to this change (it also reproduces on fp16-mixed). Tracked separately.

Branched off bbd30b5.

@Cortexelus
Cortexelus merged commit 217e853 into main Jul 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant