Skip to content

fix(ds4): use monolithic backend for single-device launches, not only for HIP#526

Merged
davide221 merged 1 commit into
Luce-Org:mainfrom
Graffioh:codex/deepseek4-monolithic-single-device
Jul 21, 2026
Merged

fix(ds4): use monolithic backend for single-device launches, not only for HIP#526
davide221 merged 1 commit into
Luce-Org:mainfrom
Graffioh:codex/deepseek4-monolithic-single-device

Conversation

@Graffioh

@Graffioh Graffioh commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • route DeepSeek 4 single-device launches through DeepSeek4Backend on both CUDA and HIP
  • reserve LayerSplitBackend for explicit multi-device placement or a configured remote target shard
  • preserve automatic CUDA/Halo layer splitting whenever remote target-shard IPC is enabled

Why

CUDA launches previously always selected DeepSeek4LayerSplitAdapter. Without an explicit or remote split, its automatic sizing capped the local shard at 42 layers, so a single H200 executed a virtual [0, 42) + [42, 43) split on the same GPU.

That virtual boundary adds split orchestration overhead and prevents the full-model fused decode path, whose eligibility requires the same range to start at layer 0 and own the output layer. A single local device should instead use the monolithic backend; devices that cannot fit the full model can use its existing hybrid-expert fallback.

Expected behavior

Configuration Backend Behavior
H200/Halo single DeepSeek4Backend Full monolithic
3090 single DeepSeek4Backend Full attempt → hybrid fallback
3090 + Halo IPC LayerSplitBackend Auto-split CUDA/Halo
Multi-GPU explicit LayerSplitBackend Layer split

Validation

@Graffioh
Graffioh force-pushed the codex/deepseek4-monolithic-single-device branch from 5fc0d09 to 0171c69 Compare July 15, 2026 06:04
@Graffioh Graffioh changed the title Route single-device DeepSeek4 through monolithic backend fix(ds4): use monolithic backend for single-device launches Jul 15, 2026
@Graffioh Graffioh changed the title fix(ds4): use monolithic backend for single-device launches fix(ds4): use monolithic backend for single-device launches, not only for HIP Jul 15, 2026
@Graffioh
Graffioh marked this pull request as ready for review July 15, 2026 06:59

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Re-trigger cubic

@davide221
davide221 force-pushed the codex/deepseek4-monolithic-single-device branch from 0171c69 to 6e5b149 Compare July 21, 2026 00:01
@davide221

Copy link
Copy Markdown
Contributor

Rebased onto current main ae98774 and resolved the backend_factory.cpp conflict at head 6e5b149. The resolution preserves current DS4 prefill validation/config propagation and applies the intended routing rule: single local targets use DeepSeek4Backend; explicit or remote splits use LayerSplitBackend.

Lucebox4 validation (exact pushed head 6e5b149):

  • git diff --check: PASS
  • CUDA 13.3 / sm_86 Release: dflash_server and test_deepseek4_unit built successfully
  • CUDA deepseek4_unit: PASS
  • ROCm 7.2.2 / gfx1151 Release: dflash_server and test_deepseek4_unit built successfully
  • HIP deepseek4_unit: PASS
  • Model-backed RTX 3090 generation using DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf: PASS
    • DeepSeek4Backend selected (no virtual layer split)
    • expected 81.7 GiB full CUDA allocation failure cleanly fell back to hybrid placement
    • hybrid initialized with 2322 hot / 8686 cold experts
    • real chat request returned HTTP 200 and exactly OK
    • 5.75 s prefill; 4.4 tok/s decode; 6.20 s total

The isolated test server was stopped after validation. GitHub CI was approved and is running.

@davide221
davide221 merged commit 62880e2 into Luce-Org:main Jul 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants