Skip to content

[FIX] Do not charge baseline trials against the optimization n_trials budget (+ test)#21

Open
VincentG1234 wants to merge 1 commit into
mainfrom
fix/exclude-baseline-trials-budget
Open

[FIX] Do not charge baseline trials against the optimization n_trials budget (+ test)#21
VincentG1234 wants to merge 1 commit into
mainfrom
fix/exclude-baseline-trials-budget

Conversation

@VincentG1234
Copy link
Copy Markdown
Collaborator

Summary

Baseline runs should not consume the configured optimization trial budget (n_trials). The synchronous path in _run_baseline_trials still incremented completed_trials, which caused the study to stop early (e.g. an incomplete grid). This now matches _collect_completed_trials, which already counts only optimization trials.

Changes

  • auto_tune_vllm/core/study_controller.py

    • Removed completed_trials += 1 after each baseline finishes (success, failure, timeout, exception).
    • Docstring clarifies that n_trials applies to optimization trials only.
  • tests/core/test_study_controller.py (generic module for future StudyController tests)

    • Fake backend that completes a trial on the first poll.
    • Test: two entries in concurrency_levelscompleted_trials stays 0 after _run_baseline_trials.

Expected behavior after merge

  • You can set n_trials (and grid cardinality in grid mode) without manually adding +1 / +k for baselines.
  • Multiple concurrency_levels still mean multiple baseline Optuna trials, but they do not eat the optimization budget.

How to test

  • pytest tests/core/test_study_controller.py

Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant