Skip to content

Conversation

@slister1001
Copy link
Member

Summary

This PR enables scenarios to be initialized with an empty strategies list when include_baseline=True. Previously, empty strategies would raise a ValueError even when baseline was requested.

Problem

When using RedTeamAgent (or FoundryScenario) with include_baseline=True and scenario_strategies=[], the scenario would fail with:

ValueError: No valid FoundryStrategy strategies provided.

This prevented users from running baseline-only attacks, which is useful for:

  • Establishing baseline metrics before applying attack techniques
  • Quick sanity checks of target behavior
  • Comparing raw objective success rates against attack-modified prompts

Changes

  1. scenario_strategy.py: Added allow_empty parameter to prepare_scenario_strategies(). When True and an empty sequence is explicitly provided, returns an empty list instead of raising ValueError.

  2. scenario.py:

    • Updated initialize_async() to pass allow_empty=self._include_baseline so baseline-only execution is allowed when baseline is requested
    • Added _create_standalone_baseline() method that creates a baseline attack directly from dataset_config when no other atomic attacks exist to derive from
  3. Tests: Added 4 unit tests covering baseline-only execution scenarios

Usage Example

scenario = RedTeamAgent(
    include_baseline=True,
    attack_scoring_config=scoring_config,
)

await scenario.initialize_async(
    objective_target=target,
    scenario_strategies=[],  # Empty - baseline only
    dataset_config=dataset_config,
)

result = await scenario.run_async()
# Result contains only baseline attack results

Test Results

All 29 scenario tests pass, including 4 new tests for baseline-only execution.

This change allows scenarios to be initialized with an empty strategies list
when include_baseline=True. Previously, empty strategies would raise a
ValueError even when baseline was requested.

Changes:
- Add allow_empty parameter to prepare_scenario_strategies() in
  ScenarioStrategy. When True and an empty sequence is explicitly provided,
  returns an empty list instead of raising ValueError.

- Update Scenario.initialize_async() to pass allow_empty=include_baseline
  so baseline-only execution is allowed when baseline is requested.

- Add _create_standalone_baseline() method to Scenario that creates a
  baseline attack directly from dataset_config when no other atomic
  attacks exist to derive from.

- Add unit tests for baseline-only execution scenarios.

This enables use cases where users want to run only baseline attacks
without any additional attack strategies, such as for establishing
baseline metrics before applying attack techniques.
- Add return type annotation to _get_baseline_data() method
- Apply ruff formatting to scenario files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants