Skip to content

bug(router): explicit stats.baselineModel is overridden by the default scenario #250

Description

@MicroMilo

Summary

router.stats.baselineModel is parsed, but createRouterRuntime overrides it with router.scenarios.default whenever a default scenario exists.

Why it matters

The purpose of router.stats.baselineModel is to define the no-router baseline used for saved-cost and baseline-cost calculations. Most router configs have a default scenario, so the explicit baseline is silently ignored in normal use. The dashboard or stats output can then report savings against the routed default model instead of the user-selected baseline model.

Evidence

  • src/router/config/schema.ts:61 documents baselineModel as the provider/model ref used for saved-cost baseline calculation.
  • src/router/config/parseRouterConfig.ts:491 enters parseStats.
  • src/router/config/parseRouterConfig.ts:532 parses raw.baselineModel.
  • src/router/config/parseRouterConfig.ts:538 returns the parsed baselineModel.
  • src/router/RouterRuntime.ts:90 constructs TokenStatsCollector.
  • src/router/RouterRuntime.ts:93 prefers config.scenarios.default as the collector baseline whenever a default scenario exists.
  • src/router/RouterRuntime.ts:95 only falls back to config.stats?.baselineModel when there is no default scenario.
  • src/router/stats/TokenStatsCollector.ts:68 stores the collector baseline from config?.baselineModel.
  • src/router/stats/TokenStatsCollector.ts:317 uses that baseline during baseline-cost calculation.
  • src/router/stats/TokenStatsCollector.ts:321 selects the baseline provider.
  • src/router/stats/TokenStatsCollector.ts:322 selects the baseline model.

Validation

Validation level: dynamic reproduction.

Minimal reproduction:

  1. Create router config with:
    • router.scenarios.default = cheap/c
    • router.stats.baselineModel = exp/e
  2. Call createRouterRuntime(...).
  3. Inspect the created stats collector baseline.

Key output:

  • Parsed config preserves the explicit baseline.
  • Runtime stats collector receives { provider: "cheap", model: "c" }.
  • Expected explicit baseline was { provider: "exp", model: "e" }.

Boundary: this repro does not need to perform a real model call. It validates the runtime construction path where stats collector config is derived. It does not change scenario routing behavior; it only observes which baseline is passed to stats.

Expected behavior

An explicitly configured router.stats.baselineModel should take precedence over router.scenarios.default. The default scenario should only be used as a fallback baseline when no explicit stats baseline is configured.

Existing coverage checked

No matching fix was found.

Checked adjacent work:

Search terms checked included router.stats.baselineModel, TokenStatsCollector baselineModel, and savedCost baseline.

Suggested fix

Change the stats collector construction priority in createRouterRuntime so it uses:

  1. config.stats?.baselineModel, when explicitly present.
  2. config.scenarios?.default, only as fallback.
  3. No baseline override otherwise.

This keeps routing behavior unchanged while making the stats contract match the parser/schema contract.

Suggested tests

  • Explicit stats.baselineModel wins when scenarios.default also exists.
  • scenarios.default is used as fallback when stats.baselineModel is absent.
  • Explicit stats.baselineModel works when no default scenario exists.
  • Baseline-cost calculation uses the explicit baseline provider/model after runtime construction.

Submitted with Codex.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions