bf16 medium DiT: FMHA-fused TRT engine build recipe + CLI (bf16 = medium default, fp16-mixed kept)#73
Merged
Merged
Conversation
…ium default, fp16-mixed kept)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bf16 medium DiT — FMHA-fused TRT engine
Adds a bf16 DiT precision for medium as the new speed default, and wires
--precisionthrough the CLI andSA3Inference.fp16-mixed(canonical) andfp32remain fully selectable.What changed
build/build_from_onnx.py,build/build.py): newsa3-m-bf16target. Reuses the same raw fp32sa3-m/dit.onnx— no new ONNX export — and builds withBuilderFlag.BF16+EXPLICIT_BATCH(notSTRONGLY_TYPED). bf16 carries fp32's dynamic range, so the FP32-softmax islands that fp16-mixed had to keep vanish and TRT 10.15's FMHA fuser fires (0 → 96 fused_gemm_mha_v2nodes). Build-time precision recipe only — no graph change.scripts/sa3_trt_core.py,scripts/sa3_trt.py):PRECISIONSgainsbf16; per-model default resolution viadefault_precision().--precisionnow defaults toNone→ resolves to bf16 for medium,fp16mixedfor sm-music/sm-sfx. bf16 is medium-only — sm-music/sm-sfx use standard attention and already fuse in fp16-mixed, so requestingbf16for them errors out.--precisionsection with the per-model default table.Speed (H100 SXM, medium)
~1.77× @ L=256 and ~4.70× @ L=4096 vs fp16-mixed, within the perceptual re-seed floor (FAD 0.59× floor, CLAP within spread, n=128).
Reproducibility caveat
bf16 is not seed-reproducible vs fp16-mixed. The medium DiT uses differential attention (cancellation-sensitive), so bf16 is a different-but-equal take per seed — quality within the re-seed floor, but exact samples differ. Use
--precision fp16mixedwhen you need bit-reproducibility or max per-step fidelity.Engine distribution
The
dit_bf16.trtengine is gitignored and ships via HF (stabilityai/stable-audio-3-optimized), auto-downloaded on first use. No engine binary is included in this PR.Notes
medium + same-l + cfg>1malloc crash exists and is not related to this change (it also reproduces on fp16-mixed). Tracked separately.Branched off
bbd30b5.