[Bug]: Title: GAMOptSettings.random_state is unused — GAM exploration is non-deterministic

[LukaszCmielowski]

Bug Description

GAMOptSettings.random_state is unused — GAM exploration order is non-deterministic

Summary

GAMOptSettings exposes random_state (default 64), but GAMOptimizer never uses it. Initial hyperparameter exploration calls random.shuffle() on the global RNG, so repeated experiments with the same settings can evaluate different RAG config sequences (retrieval window, chunk count, etc.).

This blocks reproducible benchmarks and confounds prompt A/B tests (e.g. baseline vs PR #75) when GAM is enabled.

Root cause

In ai4rag/core/hpo/gam_opt.py:

@dataclass
class GAMOptSettings(OptimizerSettings):
    n_random_nodes: int = 4
    evals_per_trial: int = 1
    random_state: int = 64   # defined but unused

def evaluate_initial_random_nodes(self) -> None:
    ...
    random.shuffle(combinations_local)  # global RNG, ignores random_state

RandomOptimizer (random_opt.py) has the same pattern — no seed parameter, uses random.shuffle(combinations).

Docs list random_state: https://ibm.github.io/ai4rag/latest/api-reference/core/hpo/

Proposed fix

1. Wire random_state in GAM (and Random) optimizers

# GAMOptimizer.__init__
self._rng = random.Random(self.settings.random_state)

# evaluate_initial_random_nodes
self._rng.shuffle(combinations_local)

Add random_state: int = 64 to RandomOptSettings and use the same pattern in RandomOptimizer.

2. Optional: experiment-level seed

AI4RAGExperiment(..., experiment_seed: int | None = None) — if set, propagate to optimizer random_state when not explicitly provided.

3. Tests

• Mock objective with deterministic scores.
• Assert two GAMOptimizer.search() runs with identical search_space, GAMOptSettings(max_evals=N, n_random_nodes=M, random_state=42) evaluate the same ordered list of param dicts.

4. Docs

Update user-guide/optimizers.md with reproducibility guidance.

Related improvements (separate / follow-up)

Item	Why
Benchmark mode temperature=0	Reduce LLM score variance during eval
evaluate_fixed_patterns(patterns: list[dict]) or document known_observations for replay	Prompt-only A/B without GAM confounding
Persist exploration trace (evaluated params + scores per step)	Debug cross-run differences

References

• HPO API: https://ibm.github.io/ai4rag/latest/api-reference/core/hpo/
• Comparative analysis: RHOAI/AutoRAG William benchmark notebooks using GAMOptSettings(max_evals=10, n_random_nodes=4)

Environment

• ai4rag: move-autorag-components-code-to-ai4rag / fix-prompts branches
• Optimizer: GAMOptimizer via AI4RAGExperiment.search()
• Metric: faithfulness (Unitxt)

Steps to Reproduce

Run GAM based experiment multiple times.

Expected Behavior

Expected behavior

• GAMOptSettings(random_state=42) → same sequence of evaluated param dicts across runs (given deterministic objective scores).
• Document what random_state controls (config exploration order) vs what it does not control (LLM stochasticity at temperature=0.2).

Actual Behavior

Observed behavior

William benchmark comparative runs (max_evals=10, n_random_nodes=4) produced different “best” retrieval configs across runs on the same ai4rag branch — e.g. baseline best window=3, chunks=5 vs PR #75 best window=1, chunks=3. Pattern1…Pattern10 labels are evaluation ordinals, not fixed configs, so cross-run pattern names are not comparable.

Environment

- ai4rag version:
- Python version:
- OGX version:
- Vector Store (Milvus/ChromaDB):
- Operating System:

Error Logs

Additional Context

No response

[LukaszCmielowski]

https://github.com/IBM/ai4rag/pull/85/changes