Does this support using quantized models like AWQ for the main model?

[xbl916]

Below is my startup script.
START_CMD="vllm serve
/model/llm/Qwen3.5-35B-A3B-AWQ
--served-model-name Qwen3.5-VL
--gpu-memory-utilization 0.95
--max-num-seqs 8
--tensor-parallel-size 2
--generation-config auto
--reasoning-parser qwen3
--enable-auto-tool-choice --tool-call-parser qwen3_coder
--speculative-config {"method":"dflash","num_speculative_tokens":15,"model":"/model/llm/Qwen3.5-35B-A3B-DFlash"}
--distributed-executor-backend mp
--allowed-local-media-path /
--chat-template-content-format auto
--enable-prefix-caching
--kv-cache-dtype auto
--max-num-batched-tokens 8192
--max-model-len auto
--attention-backend flash_attn
--host 0.0.0.0
--port 9997"

I noticed that its speed has become even slower, suggesting that the adoption rate of speculation is very low.
(APIServer pid=11250) INFO 04-08 07:59:24 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=11250) INFO 04-08 07:59:24 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 1.50, Accepted throughput: 1.00 tokens/s, Drafted throughput: 30.00 tokens/s, Accepted: 10 tokens, Drafted: 300 tokens, Per-position acceptance rate: 0.300, 0.200, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 3.3%

[leonardcser]

Same question here with Qwen/Qwen3.5-27B-FP8. I don't get much of a speed up and the acceptance rate seems suspiciously low.

vllm serve Qwen/Qwen3.5-27B-FP8 \
        --served-model-name Qwen/Qwen3.5-27B \
        --enable-auto-tool-choice \
        --tool-call-parser qwen3_coder \
        --reasoning-parser qwen3 \
        --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 15}' \
        --attention-backend flash_attn \
        --max-num-batched-tokens 32768

(APIServer pid=790630) INFO 04-09 09:47:29 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.42, Accepted throughput: 32.70 tokens/s, Drafted throughput: 345.01 tokens/s, Accepted: 327 tokens, Drafted: 3450 tokens, Per-position acceptance rate: 0.678, 0.413, 0.222, 0.087, 0.017, 0.004, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 9.5%

UPDATE: running full precision Qwen/Qwen3.5-27B has similar results.

[jianc99]

Could you run our benchmark to check whether the acceptance lengths are aligned? If they are, then the lower acceptance length you're seeing is likely due to out-of-distribution data, since our training data mainly focuses on coding and math. If they're not aligned, there might be some issues with the inference backend.

[jianc99]

For quantized target model, I think DFlash should be able to work. This is my benchmark result of Qwen3.5-27B-FP8 on humaneval:

Successful requests:                     164       
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  395.08    
Total input tokens:                      24600     
Total generated tokens:                  165556    
Request throughput (req/s):              0.42      
Output token throughput (tok/s):         419.05    
Peak output token throughput (tok/s):    57.00     
Peak concurrent requests:                3.00      
Total token throughput (tok/s):          481.31    
---------------Time to First Token----------------
Mean TTFT (ms):                          91.81     
Median TTFT (ms):                        66.50     
P99 TTFT (ms):                           127.55    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.23      
Median TPOT (ms):                        2.10      
P99 TPOT (ms):                           3.75      
---------------Inter-token Latency----------------
Mean ITL (ms):                           18.16     
Median ITL (ms):                         17.93     
P99 ITL (ms):                            20.07     
---------------Speculative Decoding---------------
Acceptance rate (%):                     46.52     
Acceptance length:                       7.98      
Drafts:                                  20754     
Draft tokens:                            311310    
Accepted tokens:                         144822    
Per-position acceptance (%):
  Position 0:                            92.42     
  Position 1:                            82.01     
  Position 2:                            72.59     
  Position 3:                            63.95     
  Position 4:                            56.51     
  Position 5:                            50.32     
  Position 6:                            45.27     
  Position 7:                            40.82     
  Position 8:                            36.81     
  Position 9:                            33.44     
  Position 10:                           30.36     
  Position 11:                           27.66     
  Position 12:                           24.73     
  Position 13:                           21.82     
  Position 14:                           19.10     

[leonardcser]

Running the benchmark with vllm and Qwen3.5-27B-FP8 yields ~250 tok/s/request from the ~50 tok/s/request without SD.

==================================================
Backend:          vllm
Dataset:          gsm8k
Num prompts:      128
Concurrency:      1
Latency:          938.2s
Output tokens:    206025
Throughput:       219.59 tok/s
==================================================

Drafted throughput: 419.98 tokens/s, Accepted: 1868 tokens, Drafted: 4200 tokens, Per-position acceptance rate: 0.904, 0.800, 0.729, 0.639, 0.539, 0.475, 0.400, 0.361, 0.336, 0.304, 0.282, 0.275, 0.246, 0.211, 0.171, Avg Draft acceptance rate: 44.5%

I noticed that when using the model in agentic tasks it drops to ~60 tok/s/request with num_speculative_tokens = 15 and ~90 tok/s/request with num_speculative_tokens = 5. Sending the same prompts as the benchmark datasets along with temperature = 0 and top-p = 1 yields to ~150 tok/s/request.

So I can successfully reproduce the results. There might be a distribution discrepancy in the draft model for agentic tasks.

[jianc99]

Hi @leonardcser thanks for reporting this. I think the lower throughput on agentic tasks is mainly due to a distribution mismatch: we do not have agentic traces in our training data, and these workloads often involve ultra-long contexts beyond DFlash’s training context length. Both are difficult for us to address 😢, since collecting agentic traces at scale is costly, and long-context training requires massive compute support. Our current solution on SGLang is using sliding window attention for the draft model, which will improve the acceptance rate on ultra-long context. And using smaller number of draft tokens to reduce compute waste, which will be helpful for end-to-end throughput.

That said, I think these results still show the potential of diffusion drafting. We plan to release the training code and continue improving DFlash, including working with industry partners to scale training to longer contexts and add agentic data so it can perform better on more complex workloads in the future. Please stay tuned!

[leonardcser]

In any case, well done and thanks!