Skip to content

feat: use pre-generated custom dataset for benchmarking MTP with chat template#63

Open
richardhuo-nv wants to merge 1 commit intosa-submission-q2-2026from
rihuo/add_custom_dataset_for_sa
Open

feat: use pre-generated custom dataset for benchmarking MTP with chat template#63
richardhuo-nv wants to merge 1 commit intosa-submission-q2-2026from
rihuo/add_custom_dataset_for_sa

Conversation

@richardhuo-nv
Copy link
Copy Markdown
Collaborator

@richardhuo-nv richardhuo-nv commented Apr 23, 2026

This is a workaround for adding the chat template for MTP based benchmarking for GLM-5 model.

GLM-5 is using a custom tokenizer, invoking it multiple times will cause pmix failure at runtime.

Workaround is pre-generate the dataset with chat template and use it for MTP based benchmarking.

The config is like this:

benchmark:
  type: "sa-bench"
  isl: 1024
  osl: 1024
  concurrencies: "8192"
  req_rate: "inf"
  dataset_name: "custom"
  dataset_path: "/glm5_datasets/glm5-1024-1024-100000-ratio-1_for_serve.json"
  custom_tokenizer: "glm_moe_dsa"

extra_mount:
  - "/lustre/fsw/core_dlfw_ci/rihuo/glm5_dataset:/glm5_datasets"

fix

fix

fix

fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant