[Test Improver] tests: skip runtime factory tests when LLM not installed

[danielmeppiel]

🤖 Test Improver - automated AI assistant for test improvements.

Goal and Rationale

Four tests in test_runtime_factory.py hard-coded assumptions that the llm runtime is installed on the host system. In CI (and standard dev environments without the llm Python package), these tests fail unconditionally with:

ValueError: Runtime 'llm' is not available on this system
AssertionError: assert False is True

This creates noise in CI output and obscures real failures.

Approach

Added module-level skip markers that probe runtime availability at collection time using the factory's own runtime_exists() method - so the logic is consistent with production code:

_llm_available = RuntimeFactory.runtime_exists("llm")
skip_no_llm = pytest.mark.skipif(not _llm_available, reason="LLM runtime not installed on this system")

• LLM-specific tests (test_get_runtime_by_name_llm_real, test_create_runtime_with_name_real, test_runtime_exists_llm_true) now skip gracefully when LLM is absent
• test_get_available_runtimes_real_system refactored to be environment-agnostic (any list is valid)
• A new test_get_available_runtimes_includes_llm test captures the LLM-specific assertion, also skippable
• Tests for get_best_available_runtime and create_runtime (auto-detect) skip when no runtime at all is available

Coverage Impact

Before	After
4 failures (LLM not installed)	0 failures, 4 skipped (correctly)

The tests still run and assert correctness when the runtime is present.

Trade-offs

• Slightly more test code, but cleaner CI signal
• Skip reasons are explicit and actionable

Reproducibility

# Run the fixed tests
.venv/bin/pytest tests/unit/test_runtime_factory.py -v

# Full unit suite
.venv/bin/pytest tests/unit/ -q --ignore=tests/unit/test_audit_report.py \
  --ignore=tests/unit/test_deps_update_command.py \
  --ignore=tests/unit/test_plugin_exporter_schema.py

Test Status

• Before: 4 failures in test_runtime_factory.py
• After: 0 failures, 4 skipped (when LLM not installed), 6 passing
• Remaining 7 failures in test_policy_status.py are pre-existing (unrelated to this PR)

Generated by Daily Test Improver · ● 2.2M · ◷

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/daily-test-improver.md@b87234850bf9664d198f28a02df0f937d0447295

[Copilot]

Pull request overview

This PR makes tests/unit/test_runtime_factory.py resilient to environments where the llm runtime (and/or any runtime) is not installed, by skipping runtime-dependent tests instead of failing unconditionally during CI and local runs.

Changes:

• Added module-level pytest.mark.skipif markers to skip LLM-specific tests when llm is unavailable, and skip auto-detect tests when no runtime is available.
• Refactored test_get_available_runtimes_real_system to assert only environment-agnostic properties (returns a list).
• Added a dedicated LLM-inclusion assertion test that is also skippable when LLM is not present.

Show a summary per file

File	Description
tests/unit/test_runtime_factory.py	Adds collection-time runtime availability detection and skip markers to avoid CI failures when optional runtimes are not installed.

Copilot's findings

• Files reviewed: 1/1 changed files
• Comments generated: 1

[danielmeppiel]

🤖 Test Improver here.

This PR now has a merge conflict. More importantly, reviewing main's history shows that equivalent runtime-factory test fixes were incorporated into the codebase independently - tests/unit/test_runtime_factory.py in main now uses a _is_runtime_available helper pattern that covers the same ground.

All 9 runtime factory tests in main pass cleanly without this PR's changes. I'd recommend closing this PR as superseded. Happy to leave that decision to you.

Note

🔒 Integrity filter blocked 19 items

The following items were blocked because they don't meet the GitHub integrity level.

• #1281 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1277 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1262 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1248 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1246 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1245 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1234 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1227 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1224 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1146 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1117 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1113 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1028 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #1017 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #928 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #900 list_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 3 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Daily Test Improver · ● 5.9M · ◷

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/daily-test-improver.md@b87234850bf9664d198f28a02df0f937d0447295