Skip to content

fix: align BDD tests and mock server with real API behavior#198

Open
socksy wants to merge 5 commits intodevelopfrom
ben/fix-bdd-tests-match-real-server
Open

fix: align BDD tests and mock server with real API behavior#198
socksy wants to merge 5 commits intodevelopfrom
ben/fix-bdd-tests-match-real-server

Conversation

@socksy
Copy link
Contributor

@socksy socksy commented Feb 9, 2026

We want to run the BDD integration tests against the real Tower server (from the monorepo) to catch discrepancies between the mock and real API. This PR fixes the gaps found doing that:

  • Align mock server responses with real API (RFC 7807 error format, SSE warning events, realistic log streams, schedule update fields)
  • Update test assertions to match what the real server actually returns
  • Allow the test runner to target an external server via TOWER_URL env var, skipping mock server startup and session fixture reset

Summary by CodeRabbit

  • Tests
    • Updated CLI test expectations for error message formats and log output validation.
    • Enhanced test infrastructure to support external server mode alongside mock server testing.
    • Improved mock API server error responses with structured payloads and more realistic log stream generation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@coderabbitai
Copy link

coderabbitai bot commented Feb 9, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ben/fix-bdd-tests-match-real-server

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@tests/integration/features/steps/cli_steps.py`:
- Around line 51-56: The general exception handler in step_run_cli_command
doesn't set context.cli_stdout, causing attribute errors downstream; update the
broad except Exception handler to initialize context.cli_stdout = "" (matching
the TimeoutExpired handler) before re-raising so callers can always rely on
context.cli_stdout existing; locate the try/except in step_run_cli_command (and
the existing except subprocess.TimeoutExpired block) and add the initialization
there.
🧹 Nitpick comments (1)
tests/integration/run_tests.py (1)

27-48: Git-based external configuration detection may have edge cases.

The approach comparing file mtime to git commit time is clever, but consider these scenarios:

  1. If session.json was modified locally but not committed, git log returns the last commit time for that file, which could falsely indicate external configuration.
  2. If the file is newly created and not yet tracked by git, git log may output nothing, causing int(result.stdout.strip()) to fail on empty string.

The current exception handling catches most failures gracefully by returning False, which is safe default behavior. This is acceptable for the "Chill" review mode.

💡 Alternative approach using git status
def is_session_externally_configured(test_home):
    """Check if session.json has been modified from the committed version."""
    session_file = test_home / ".config" / "tower" / "session.json"
    if not session_file.exists():
        return False

    try:
        # Check if file differs from committed version
        result = subprocess.run(
            ["git", "diff", "--quiet", str(session_file)],
            capture_output=True,
        )
        # Non-zero exit means file has changes
        return result.returncode != 0
    except (subprocess.CalledProcessError, FileNotFoundError):
        return False

Comment on lines +51 to 56
context.cli_stdout = result.stdout
context.cli_return_code = result.returncode
except subprocess.TimeoutExpired:
context.cli_output = "Command timed out"
context.cli_stdout = ""
context.cli_return_code = 124
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Consider initializing cli_stdout in the general exception handler.

The timeout handler sets context.cli_stdout = "", but the general exception handler (lines 57-61) re-raises without setting it. If any downstream code assumes cli_stdout exists after step_run_cli_command, this could cause an AttributeError in edge cases.

🐛 Proposed fix
     except subprocess.TimeoutExpired:
         context.cli_output = "Command timed out"
         context.cli_stdout = ""
         context.cli_return_code = 124
     except Exception as e:
+        context.cli_stdout = ""
         print(f"DEBUG: Exception in CLI command: {type(e).__name__}: {e}")
         print(f"DEBUG: Command was: {full_command}")
         print(f"DEBUG: Working directory: {os.getcwd()}")
         raise
🤖 Prompt for AI Agents
In `@tests/integration/features/steps/cli_steps.py` around lines 51 - 56, The
general exception handler in step_run_cli_command doesn't set
context.cli_stdout, causing attribute errors downstream; update the broad except
Exception handler to initialize context.cli_stdout = "" (matching the
TimeoutExpired handler) before re-raising so callers can always rely on
context.cli_stdout existing; locate the try/except in step_run_cli_command (and
the existing except subprocess.TimeoutExpired block) and add the initialization
there.

socksy and others added 4 commits February 9, 2026 23:42
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@socksy socksy force-pushed the ben/fix-bdd-tests-match-real-server branch from c6f6074 to 88183c2 Compare February 9, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant