Consistency improvements between adk eval and AgentEvaluator

[ftnext]

Please make sure you read the contribution guide and file the issues in the right place.
Contribution guide.

🔴 Required Information

Please ensure all items in this section are completed to allow for efficient
triaging. Requests without complete information may be rejected / deprioritized.
If an item is not applicable to you - please mark it as N/A

Is your feature request related to a specific problem?

Yes. In v1.24.1, evaluation behavior differs between adk eval and AgentEvaluator.evaluate,
which makes it harder to run and compare evals consistently across CLI and API/test workflows.

Concrete gaps I hit:

• adk eval has no --num_runs, so I need external loops to repeat runs.
• AgentEvaluator.evaluate does not provide built-in result persistence, so historical comparison is less convenient than CLI-based runs.
• In adk eval, --config_file_path is optional, but when omitted it falls back to in-code default criteria and does not auto-discover per-test test_config.json next to each eval/test file.
• adk eval resolves only agent_module.agent.root_agent and does not look for get_agent_async.

Describe the Solution You'd Like

I would like feature parity/consistency between both entry points:

1. Add --num_runs to adk eval (default 1) and aggregate results per eval case across runs.
2. Add optional result persistence support to AgentEvaluator.evaluate via an EvalSetResultsManager (local by default).
3. In adk eval, when --config_file_path is omitted, auto-discover test_config.json adjacent to each input eval/test file, then fall back to default criteria if no file is found.
4. Align agent resolution behavior between adk eval and AgentEvaluator: support both root_agent and get_agent_async consistently

Impact on your work

This impacts reproducibility and operational efficiency in day-to-day evals.

• Repeated runs are needed to reduce nondeterminism, but currently require custom scripting for CLI.
• Comparing historical outcomes is easier on one path than the other.
• The same dataset/config setup behaves differently depending on whether I use CLI or programmatic evaluation.
• Agent modules that are valid in one evaluation path can fail in the other, forcing workflow-specific agent definitions.

Willingness to contribute

Yes

---

🟡 Recommended Information

Describe Alternatives You've Considered

• Keeping current differences and documenting them: this still leaves manual repetition/persistence work and split workflows.
• Wrapper scripts outside ADK: works short-term, but duplicates logic and reduces maintainability.

Proposed API / Implementation

Pseudo-proposal:

# CLI
adk eval <agent_dir> <eval_set...> --num_runs=3 [--config_file_path=...]

# Internals (conceptual)
for i in range(num_runs):
  inference_results += run_inference(...)

aggregated = aggregate_eval_case_results(inference_results)
save_if_configured(aggregated)

# AgentEvaluator
await AgentEvaluator.evaluate(
    agent_module=...,
    eval_dataset_file_path_or_dir=...,
    num_runs=3,
    eval_set_results_manager=LocalEvalSetResultsManager(...),  # optional
)

# Config resolution for adk eval (conceptual)
if config_file_path:
  eval_config = load(config_file_path)
else:
  eval_config = discover_test_config_near_input_or_default(...)

# Conceptual shared loader used by both entry points.
agent_module = _get_agent_module(...)
if hasattr(agent_module.agent, "root_agent"):
  agent = agent_module.agent.root_agent
elif hasattr(agent_module.agent, "get_agent_async"):
  agent = await agent_module.agent.get_agent_async()
else:
  raise ValueError("Expected `root_agent` or `get_agent_async` in agent module.")

Additional Context

Goal: make eval behavior consistent regardless of entry point (adk eval vs AgentEvaluator.evaluate) so that the same eval assets produce comparable outcomes with less custom glue code.

[rohityan]

Hi @ftnext ,

1. The _get_eval_results_by_eval_id method explicitly uses inference_requests = [...] * num_runs to repeat inference requests, so the framework itself manages the repetition to address non-determinism.
2. The EvalSetResultsManager interface, specifies how evaluation results are stored and retrieved. Implementations like GcsEvalSetResultsManager and LocalEvalSetResultsManager handle the actual storage, either in Google Cloud Storage or locally on disk, respectively. This enables the persistence and reuse of evaluation results across different ADK components
3. The AgentEvaluator.evaluate static method is designed to automatically discover test_config.json files when the eval_dataset_file_path_or_dir argument points to a directory or a file.

[ftnext]

@ryanaiagent Thanks for the detailed references and for clarifying the current implementation.

I think we may be talking past each other slightly, so let me clarify the core request in this issue:
I’m not claiming these capabilities are missing in general; I’m pointing out behavior differences between adk eval and AgentEvaluator.evaluate that make workflows inconsistent.

Specifically:

1. num_runs
AgentEvaluator supports it (agreed).
   My request is that adk eval also supports num_runs so repeated runs do not require external scripting. (PR Support adk eval --num_runs N as AgentEvaluator #4411)
2. Result persistence
   I understand EvalSetResultsManager exists and is used by some components (agreed).
   My request is that AgentEvaluator.evaluate can optionally persist results through this manager, to match CLI-style history workflows. (PR Add optional eval set result persistence to AgentEvaluator #4414)
3. Config discovery
   I agree AgentEvaluator.evaluate auto-discovers test_config.json.
   My request is that adk eval should offer equivalent discovery behavior when --config_file_path is omitted, instead of only falling back to in-code defaults. (PR Auto-discover test_config.json only for single-file adk eval input #4412)
4. Agent resolution
   adk eval currently resolves agent_module.agent.root_agent directly, while AgentEvaluator can also use get_agent_async as a fallback.
   My request is to align this resolution behavior across both entry points. (PR fix(eval): support get_agent_async in adk eval #4413)

So the goal is entry-point consistency (adk eval vs AgentEvaluator), not to dispute the existence of the underlying primitives.

[rohityan]

Hi @sasha-gitg , can you PTAL .