Skip to content

cvector-generator: GGML_ASSERT diff_filtered.size() == n_layers - 1 fails on all Qwen models #3

Open
Open
cvector-generator: GGML_ASSERT diff_filtered.size() == n_layers - 1 fails on all Qwen models#3

Description

@zhangchengdu007-oss
zhangchengdu007-oss
opened on Jul 12, 2026

Git commit

Bug Description

llama-cvector-generator.exe from turboquant-plus v0.2.0 (CUDA 12.4, build 91b4f8c) crashes with assertion failure on all Qwen architecture models. Same error with both GPU (-ngl 99) and CPU (-ngl 0) modes, with and
without -fit off.

Error

GGML_ASSERT((int) diff_filtered.size() == n_layers - 1) failed
At tools/cvector-generator/cvector-generator.cpp:221

Models Tested (both fail)

  • Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf
  • Qwen3.6-27B-16GB-VRAM-MTP-mini-IQ4_XS.gguf

Steps to Reproduce

llama-cvector-generator.exe \
  -m Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf \
  -ngl 0 \
  -fit off \
  --positive-file positive.txt \
  --negative-file negative.txt \
  --output test.triattention \
  --method mean

The tool loads the model, begins evaluating prompt 1/N, then crashes with the assertion.

Environment

- Windows 11, RTX 3080 Laptop 16GB
- CUDA 13.x runtime
- turboquant-plus-tqp-v0.2.0-windows-x64-cuda12.4
- Also tested: llama-turboquant-triattention-win-cu13-x64 (no cvector-generator included in that build)

Expected Behavior

cvector-generator should complete calibration and output a .triattention file for use with --triattention-stats.

Additional Context

The TriAttention CUDA 13 build (llama-turboquant-triattention-win-cu13-x64.zip) doesn't include cvector-generator.exe at all, so there's no way to generate calibration files for the triattention-enabled server when using Qwen
models.

### Operating systems

Windows

### GGML backends

CUDA

### Problem description & steps to reproduce


## Bug Description

`llama-cvector-generator.exe` from turboquant-plus v0.2.0 (CUDA 12.4, build 91b4f8c) crashes with assertion failure on all Qwen architecture models. Same error with both GPU (`-ngl 99`) and CPU (`-ngl 0`) modes, with and
without `-fit off`.

## Error

GGML_ASSERT((int) diff_filtered.size() == n_layers - 1) failed
At `tools/cvector-generator/cvector-generator.cpp:221`

## Models Tested (both fail)

- `Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf`
- `Qwen3.6-27B-16GB-VRAM-MTP-mini-IQ4_XS.gguf`

## Steps to Reproduce

```bash
llama-cvector-generator.exe \
  -m Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf \
  -ngl 0 \
  -fit off \
  --positive-file positive.txt \
  --negative-file negative.txt \
  --output test.triattention \
  --method mean

The tool loads the model, begins evaluating prompt 1/N, then crashes with the assertion.

Environment

- Windows 11, RTX 3080 Laptop 16GB
- CUDA 13.x runtime
- turboquant-plus-tqp-v0.2.0-windows-x64-cuda12.4
- Also tested: llama-turboquant-triattention-win-cu13-x64 (no cvector-generator included in that build)

Expected Behavior

cvector-generator should complete calibration and output a .triattention file for use with --triattention-stats.

Additional Context

The TriAttention CUDA 13 build (llama-turboquant-triattention-win-cu13-x64.zip) doesn't include cvector-generator.exe at all, so there's no way to generate calibration files for the triattention-enabled server when using Qwen
models.

### First Bad Commit


## Bug Description

`llama-cvector-generator.exe` from turboquant-plus v0.2.0 (CUDA 12.4, build 91b4f8c) crashes with assertion failure on all Qwen architecture models. Same error with both GPU (`-ngl 99`) and CPU (`-ngl 0`) modes, with and
without `-fit off`.

## Error

GGML_ASSERT((int) diff_filtered.size() == n_layers - 1) failed
At `tools/cvector-generator/cvector-generator.cpp:221`

## Models Tested (both fail)

- `Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf`
- `Qwen3.6-27B-16GB-VRAM-MTP-mini-IQ4_XS.gguf`

## Steps to Reproduce

```bash
llama-cvector-generator.exe \
  -m Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf \
  -ngl 0 \
  -fit off \
  --positive-file positive.txt \
  --negative-file negative.txt \
  --output test.triattention \
  --method mean

The tool loads the model, begins evaluating prompt 1/N, then crashes with the assertion.

Environment

- Windows 11, RTX 3080 Laptop 16GB
- CUDA 13.x runtime
- turboquant-plus-tqp-v0.2.0-windows-x64-cuda12.4
- Also tested: llama-turboquant-triattention-win-cu13-x64 (no cvector-generator included in that build)

Expected Behavior

cvector-generator should complete calibration and output a .triattention file for use with --triattention-stats.

Additional Context

The TriAttention CUDA 13 build (llama-turboquant-triattention-win-cu13-x64.zip) doesn't include cvector-generator.exe at all, so there's no way to generate calibration files for the triattention-enabled server when using Qwen
models.

### Compile command

```shell
## Bug Description

`llama-cvector-generator.exe` from turboquant-plus v0.2.0 (CUDA 12.4, build 91b4f8c) crashes with assertion failure on all Qwen architecture models. Same error with both GPU (`-ngl 99`) and CPU (`-ngl 0`) modes, with and
without `-fit off`.

## Error

GGML_ASSERT((int) diff_filtered.size() == n_layers - 1) failed
At `tools/cvector-generator/cvector-generator.cpp:221`

## Models Tested (both fail)

- `Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf`
- `Qwen3.6-27B-16GB-VRAM-MTP-mini-IQ4_XS.gguf`

## Steps to Reproduce


llama-cvector-generator.exe \
  -m Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf \
  -ngl 0 \
  -fit off \
  --positive-file positive.txt \
  --negative-file negative.txt \
  --output test.triattention \
  --method mean

The tool loads the model, begins evaluating prompt 1/N, then crashes with the assertion.

Environment

- Windows 11, RTX 3080 Laptop 16GB
- CUDA 13.x runtime
- turboquant-plus-tqp-v0.2.0-windows-x64-cuda12.4
- Also tested: llama-turboquant-triattention-win-cu13-x64 (no cvector-generator included in that build)

Expected Behavior

cvector-generator should complete calibration and output a .triattention file for use with --triattention-stats.

Additional Context

The TriAttention CUDA 13 build (llama-turboquant-triattention-win-cu13-x64.zip) doesn't include cvector-generator.exe at all, so there's no way to generate calibration files for the triattention-enabled server when using Qwen
models.

Relevant log output

## Bug Description

  `llama-cvector-generator.exe` from turboquant-plus v0.2.0 (CUDA 12.4, build 91b4f8c) crashes with assertion failure on all Qwen architecture models. Same error with both GPU (`-ngl 99`) and CPU (`-ngl 0`) modes, with and
  without `-fit off`.

  ## Error

  GGML_ASSERT((int) diff_filtered.size() == n_layers - 1) failed
  At `tools/cvector-generator/cvector-generator.cpp:221`

  ## Models Tested (both fail)

  - `Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf`
  - `Qwen3.6-27B-16GB-VRAM-MTP-mini-IQ4_XS.gguf`

  ## Steps to Reproduce

  
  llama-cvector-generator.exe \
    -m Qwen3.5-9B-DeepSeek-V4-Flash-MTP-Q8_0.gguf \
    -ngl 0 \
    -fit off \
    --positive-file positive.txt \
    --negative-file negative.txt \
    --output test.triattention \
    --method mean

  The tool loads the model, begins evaluating prompt 1/N, then crashes with the assertion.

  Environment

  - Windows 11, RTX 3080 Laptop 16GB
  - CUDA 13.x runtime
  - turboquant-plus-tqp-v0.2.0-windows-x64-cuda12.4
  - Also tested: llama-turboquant-triattention-win-cu13-x64 (no cvector-generator included in that build)

  Expected Behavior

  cvector-generator should complete calibration and output a .triattention file for use with --triattention-stats.

  Additional Context

  The TriAttention CUDA 13 build (llama-turboquant-triattention-win-cu13-x64.zip) doesn't include cvector-generator.exe at all, so there's no way to generate calibration files for the triattention-enabled server when using Qwen
  models.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions