Update VLM Reamde for v6.1 Specifications#2591
Open
Victor49152 wants to merge 5 commits into
Open
Conversation
Contributor
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
wangshangsam
suggested changes
May 22, 2026
|
|
||
| ### Configure the benchmark | ||
|
|
||
| Example configs live under [endpoints/examples/08_Qwen3-VL-235B-A22B_Example](https://github.com/mlcommons/endpoints/tree/381d13bbd27d6d52306813a51dc4e44295222d7e/examples/08_Qwen3-VL-235B-A22B_Example). Set the endpoint URL in the YAML file to match your server address and port: |
Contributor
There was a problem hiding this comment.
To be explicit, we should probably specify which fields in that yaml file might be adjusted by the submitter, and which fields are frozen (in which case if a submission changed it, the submission is invalid) -> which brings up a point that maybe we need a compliance test to validate the yaml file?
Author
There was a problem hiding this comment.
Updated this part. I believe the yaml config should be verified by submission checker?
wangshangsam
suggested changes
May 22, 2026
Co-authored-by: Shang Wang <shangw@nvidia.com>
wangshangsam
approved these changes
May 23, 2026
Contributor
wangshangsam
left a comment
wangshangsam
left a comment
There was a problem hiding this comment.
Some nits but LGTM overall!
|
|
||
| #### Fields that the submitter **should** update to match their server status: | ||
|
|
||
| - Served model name: |
Contributor
There was a problem hiding this comment.
Suggested change
| - Served model name: | |
| - Served model name (to match the actual, probably quantized, model checkpoint): |
| name: "Qwen/Qwen3-VL-235B-A22B-Instruct" | ||
| ``` | ||
|
|
||
| - Endpoints url and port: |
Contributor
There was a problem hiding this comment.
Suggested change
| - Endpoints url and port: | |
| - The URL and port number of the endpoint: |
Comment on lines
+72
to
+76
| ```yaml | ||
| load_pattern: | ||
| type: "poisson" | ||
| target_qps: 6.5 | ||
| ``` |
Contributor
There was a problem hiding this comment.
Suggested change
| ```yaml | |
| load_pattern: | |
| type: "poisson" | |
| target_qps: 6.5 | |
| ``` | |
| ```yaml | |
| load_pattern: | |
| target_qps: 6.5 | |
| ``` |
Removing type: "poisson" to avoid the misunderstanding that type could be changed.
|
|
||
| #### Fileds that the submitter **MUST NOT** change for valid results: | ||
|
|
||
| - Sampling parameters that specified in scection [Reference Implementation Specification](#reference-implementation-specification) |
Contributor
There was a problem hiding this comment.
Suggested change
| - Sampling parameters that specified in scection [Reference Implementation Specification](#reference-implementation-specification) | |
| - Sampling parameters specified in the section [Reference Implementation Specification](#reference-implementation-specification) |
|
|
||
| - Sampling parameters that specified in scection [Reference Implementation Specification](#reference-implementation-specification) | ||
|
|
||
| - Datasets (Neither performance or accuracy dataset) |
Contributor
There was a problem hiding this comment.
Suggested change
| - Datasets (Neither performance or accuracy dataset) | |
| - Datasets (neither for performance evaluation nor for accuracy evaluation) |
|
|
||
| #### Fields that the submitter **may** customize for performance tuning: | ||
|
|
||
| - Target_qps (for server and interactive mode): |
Contributor
There was a problem hiding this comment.
Suggested change
| - Target_qps (for server and interactive mode): | |
| - Target QPS (for the server and interactive scenarios): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.