Skip to content

feat(providers): pin Gemini and Grok models via config.json - #162

Merged
antonbabenko merged 3 commits into
masterfrom
feat/provider-model-config
Jul 27, 2026
Merged

feat(providers): pin Gemini and Grok models via config.json#162
antonbabenko merged 3 commits into
masterfrom
feat/provider-model-config

Conversation

@antonbabenko

Copy link
Copy Markdown
Owner

What

Adds providers.gemini.model and providers.grok.model so both providers can be pinned from deliberation's own config, instead of only through env vars and each CLI's private settings file.

Precedence is uniform for both: per-call model > providers.<name>.model > env var > built-in default. The key is read once at MCP start (unlike the hot-reloading models map), so a change needs a restart.

Codex is deliberately excluded. It resolves its model from ~/.codex/config.toml and deliberation never overrides it, so it keeps the enabled-only simpleProvider shape. Gemini needed its own geminiProvider def to carry a model.

Gemini

agy >= 1.0.9 has a --model flag. The bridge previously accepted a model param and dropped it on the floor (model is accepted but never reaches argv); it now passes it through, so a run no longer inherits whatever ~/.gemini/settings.json happens to hold.

The built-in default stays the portable auto-gemini-3 router alias. Concrete agy ids differ per install, and a pinned id that does not exist locally fails every call.

Pinning a concrete id is still preferable where you can: the alias routes server-side, and agy models shows the catalog also carries Claude Opus/Sonnet and GPT-OSS entries. A routed call can therefore seat a non-Gemini model in the Gemini slot, which quietly collapses cross-model independence in /ask-all and /consensus while Claude is already the arbiter. Nothing in the result envelope would show it.

Model ids fuse family and reasoning effort (gemini-3.6-flash-high); a bare family is rejected with requires --effort.

Grok

The documented default grok-4.3 no longer matches the served model, so the built-in moves to grok-4.5. The standalone bridge now reads the same config key as the unified server rather than staying env-only, so /ask-grok and the deliberation server cannot drift apart.

The bug this nearly shipped with

resolveProviders in server/openrouter/config.js projected providers down to { enabled }, silently dropping the new key before it reached the composition root. A valid config edit would have been accepted and done nothing:

gemini.model : undefined                  <- before
gemini.model : "gemini-3.6-flash-high"    <- after

It now carries model, drops blank or non-string values rather than forwarding a bogus id, and ignores the key for openrouter (whose models map owns selection). PM1-PM4 cover the projection so it cannot quietly narrow again.

Test plan

  • npm run check (typecheck + suite): 595 pass, 0 fail
  • M1 rewritten. It asserted the exact contract this PR reverses (model param accepted, never reaches argv) and failed as expected before being updated
  • M1b added for the default-pin path
  • PM1-PM4 added for the config projection
  • Verified end to end against a real config, not just the edit: ["--sandbox","--model","gemini-3.6-flash-high","-p"]
  • Live CLI verification of the id namespace: gemini-3.6-flash-high resolves, gemini-3.6-flash errors with requires --effort
  • Host artifacts regenerated via scripts/sync-hosts.js

Docs

TECHNICAL.md (bridge section + both env rows), SETUP.md (example block + new precedence section), rules/model-selection.md, commands/ask-gemini.md, commands/ask-grok.md, commands/ask-all.md.

Removed the hardcoded model: id from the ask-gemini example. Pinning a concrete id in a shared command is the same portability trap the default avoids.

https://claude.ai/code/session_01Q7Zv2sYjAsBhacqakr1ftJ

Adds `providers.gemini.model` and `providers.grok.model` so both providers
can be pinned from deliberation's own config instead of only through env
vars and each CLI's private settings.

Gemini: agy >= 1.0.9 has a `--model` flag, so the bridge now passes the
model through to argv instead of accepting it and dropping it. A run no
longer inherits whatever `~/.gemini/settings.json` happens to hold. The
built-in default stays the portable `auto-gemini-3` router alias, because
concrete agy ids differ per install and a pinned id that does not exist
locally fails every call. Pinning a concrete id is still preferable where
possible: the alias routes server side and agy's catalog also carries
Claude and GPT-OSS entries, so a routed call can seat a non-Gemini model
in the Gemini slot and silently collapse cross-model independence in
ask-all and consensus.

Grok: the documented default `grok-4.3` no longer matches the served
model, so the built-in moves to `grok-4.5`. The standalone bridge reads
the same config key as the unified server rather than staying env-only.

Precedence for both, uniform: per-call `model` > `providers.<name>.model`
> env var > built-in default. The key is read once at MCP start, so a
change needs a restart. Codex is deliberately excluded, it resolves its
model from `~/.codex/config.toml` and deliberation never overrides it.

`resolveProviders` previously projected `providers` down to `{ enabled }`,
which silently dropped the new key on the way to the composition root, so
a valid config edit would have been accepted and done nothing. It now
carries `model`, drops blank or non-string values rather than forwarding a
bogus id, and ignores the key for openrouter, whose models map owns
selection.

Tests: `M1` asserted the old "model never reaches argv" contract and is
rewritten for the new one, plus `M1b` for the default path and `PM1`-`PM4`
for the config projection. 595 pass.

Docs: TECHNICAL.md, SETUP.md, rules/model-selection.md, and the ask-*
commands. Removed the hardcoded model id from the ask-gemini example,
since pinning a concrete id in a shared command is the same portability
trap. Host artifacts regenerated.

Claude-Session: https://claude.ai/code/session_01Q7Zv2sYjAsBhacqakr1ftJ

@antonbabenko antonbabenko left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: REQUEST CHANGES

(Posted as a comment - GitHub blocks request-changes on your own PR.)

Config plumbing is sound and the resolveProviders projection fix is well covered by PM1-PM4. Two HIGH issues block merge.

Self-review caveat: authored and reviewed in the same session. Findings below were verified by execution, not by re-reading intent.

HIGH

H1 - M1b fails whenever GEMINI_DEFAULT_MODEL is set (environment-dependent test)

DEFAULT_MODEL = process.env.GEMINI_DEFAULT_MODEL || "auto-gemini-3" is computed at module load; the spawned bridge inherits the parent env and no test clears the var. Proven:

$ GEMINI_DEFAULT_MODEL="gemini-3.1-pro-low" node --test test/bridge.test.js
✖ M1b: absent model falls back to the pinned default, not settings.json
  actual:   'gemini-3.1-pro-low'
  expected: 'auto-gemini-3'

Not hypothetical: GROK_DEFAULT_MODEL=grok-4.5 is already exported in this dev environment, so the sibling var being set is the normal case. Any contributor or CI runner with GEMINI_DEFAULT_MODEL set gets a red suite from an unrelated change.

Fix: pass a scrubbed child env to startBridge for M1b so the test pins the constant, not the machine.

H2 - Unconditional --model breaks agy < 1.0.9, no version guard

buildAgyArgs now always pushes --model. agy rejects unknown flags outright (agy models --json -> Error: flags provided but not defined: -json). Before this PR the bridge never passed it, so older installs worked; now every Gemini call fails at argv parse. TECHNICAL.md states ">= 1.0.9" in prose but nothing enforces or degrades.

I could not install an older agy to reproduce directly, so this is inferred from the same binary's parser behavior plus the removed model ... never reaches argv comment. The change is unguarded either way.

Fix: server/gemini/index.js:811 already shells out to agy --help for the health check. Capture it once, detect --model, and skip the flag or fail with an actionable upgrade message.

MEDIUM

M1 - Seeding model into config.default.json kills the env var for new installs. commands/setup.md:88 copies that file to the user config, and precedence is config > env, so GROK_DEFAULT_MODEL / GEMINI_DEFAULT_MODEL become dead for anyone who runs setup. This undercuts the "configurable via both" goal. Values match the built-ins so behavior is unchanged today, but consider omitting the key from the seeded default and documenting it in SETUP.md only.

M2 - Composition-root expression is ambiguous and double-reads config. (getConfig().providers && getConfig().providers.gemini || {}).model parses correctly as (a && b) || {} but reads poorly and calls getConfig() twice. Prefer getConfig()?.providers?.gemini?.model.

LOW

L1 - providers.*.model is read once at startup while the models map hot-reloads. Documented in code, SETUP.md, and the PR body, so a note rather than a defect. Reading it inside ask() (as the Grok bridge's configuredModel() already does) would drop the restart requirement and make the two paths consistent.

Verified as non-issues

  • runWithFiles delegates to runGrok, so the config pin applies to file-bearing Grok calls.
  • configuredModel() is try/catch wrapped, returns undefined on failure, adds no throw path.
  • Blank / non-string model dropped rather than forwarded (PM3); openrouter correctly ignores the key (PM4).
  • No circular import between the Grok bridge and the config module.
  • No secrets, injection surface, or auth changes; --model is a distinct argv element (no shell) and the -p <prompt> tail invariant holds.

Validation

Check Result
Type check Pass
Tests Pass (595 pass, 0 fail)
Lint / Build Skipped (no such scripts)
Host-artifact drift Pass

The suite is green on this machine only because GEMINI_DEFAULT_MODEL is unset. See H1.

https://claude.ai/code/session_01Q7Zv2sYjAsBhacqakr1ftJ

…dent

Addresses the two HIGH findings from the review of #162, and adds a
configurable Grok reasoning effort.

H1: `DEFAULT_MODEL` reads `GEMINI_DEFAULT_MODEL` at module load, so M1b
asserted whatever the machine exported rather than the built-in constant.
Any contributor or CI runner with that variable set got a red suite from
an unrelated change. The test now blanks the variable in the child env.
Verified: the whole suite stays green with GEMINI_DEFAULT_MODEL,
GROK_DEFAULT_MODEL, and GROK_REASONING_EFFORT all set to foreign values.

H2: `buildAgyArgs` pushed `--model` unconditionally, but agy rejects
unknown flags outright, so every Gemini call would have failed on agy
older than 1.0.9. `agySupportsModelFlag()` now probes `agy --help` once
(the same call the startup health check already makes), caches the result
for the process, and omits the flag with a one-time stderr warning when
absent. Degrading to the pre-pin behavior beats failing every call. A
probe that throws is treated as supported, so a transient failure never
silently drops a pin the operator asked for. `fake-agy-nomodel.sh` errors
if it ever sees `--model`, so M1c fails loudly on a regression.

M2: the composition root hoists the provider config blocks once instead
of calling `getConfig()` twice inside a mixed `&&`/`||` expression.

Grok reasoning effort is now configurable via
`providers.grok.reasoningEffort`, on the same ladder as the model pin:
per-call > config > `GROK_REASONING_EFFORT` > `high`. Gemini gets no
equivalent key because agy fuses effort into the model id itself.

The first attempt put that lookup inside `resolveReasoningEffort`, which
made the pure resolver read the real user config and broke the existing
G20 test on any machine with a Grok effort pinned - the same
environment-dependence as H1. Config injection now happens at the call
sites (composition root for the core adapter, handler for the bridge) and
the resolver stays pure.

600 pass, and the suite is now verified hermetic against both a polluted
environment and a populated config.json.

Claude-Session: https://claude.ai/code/session_01Q7Zv2sYjAsBhacqakr1ftJ
@antonbabenko

Copy link
Copy Markdown
Owner Author

Review follow-up: H1, H2, M2 fixed (+ Grok effort config)

Pushed 12592f5. 600 pass, 0 fail.

H1 — fixed. M1b now blanks GEMINI_DEFAULT_MODEL in the child env, so it pins the built-in constant rather than the machine. Verified the whole suite stays green with all three vars set to foreign values:

$ GEMINI_DEFAULT_MODEL="gemini-3.1-pro-low" GROK_DEFAULT_MODEL="grok-9" \
  GROK_REASONING_EFFORT="low" npm run check
600 pass, 0 fail

H2 — fixed. agySupportsModelFlag() probes agy --help once (the call the startup health check already makes), caches per process, and omits --model with a one-time stderr warning on older binaries. Degrading to pre-pin behavior beats failing every call. A probe that throws is treated as supported, so a transient failure never silently drops a requested pin. New fixture fake-agy-nomodel.sh errors if it ever sees --model, so M1c fails loudly on a regression.

M2 — fixed. Composition root hoists geminiCfg / grokCfg once; no double getConfig(), no mixed &&/||.

M1 — still open by design. model (and now reasoningEffort) remain seeded in config.default.json as requested. Config still wins over env for anyone who runs setup. Values match the built-ins so nothing changes today; flagging that it stays a live tradeoff.

L1 — partly addressed. The new reasoningEffort key is read per call, so it needs no restart. The model pins are still start-time.

New: providers.grok.reasoningEffort

Same ladder as the model pin: per-call > config > GROK_REASONING_EFFORT > high. none omits the field so the model uses its own default. Gemini gets no equivalent key because agy fuses effort into the model id (gemini-3.6-flash-high vs -low).

A third issue of the same class, caught in this pass

The first implementation put the config lookup inside resolveReasoningEffort, which made a pure resolver read the real user config. That broke the pre-existing G20 test on any machine with a Grok effort pinned — the same environment-dependence as H1, reintroduced two commits later while fixing it.

Config injection moved to the call sites (composition root for the core adapter, handler for the bridge); the resolver is pure again and documented as such. Coverage split accordingly: G20 keeps testing the pure ladder, GK9 tests the injected pin, PM5-PM7 test the config projection.

The suite is now verified hermetic against both a polluted environment and a populated config.json.

https://claude.ai/code/session_01Q7Zv2sYjAsBhacqakr1ftJ

The PR review artifact belongs on the PR, not in version control. Its
content is preserved as a review comment on #162.

Claude-Session: https://claude.ai/code/session_01Q7Zv2sYjAsBhacqakr1ftJ
@antonbabenko
antonbabenko merged commit aff8026 into master Jul 27, 2026
1 check passed
@antonbabenko
antonbabenko deleted the feat/provider-model-config branch July 27, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant