feat(e2e): Playwright E2E for all 6 Mothership question types + CI pipeline (#423)

[OgnjenGligoric]

Summary

Closes #423.

This PR adds end-to-end Playwright coverage for all six Mothership question delivery types and wires the test suite into CI so it runs automatically on every PR.

What was done

New question type coverage (Layer 5 E2E)

Previously only singleChoice, approval, and documentReview were covered. Added the remaining three:

• multiChoice — radio selection (same UI as singleChoice; type distinction handled server-side)
• freeText — textarea fill + submit
• priorityRanking — drag-and-drop list; JS serialises rankedItemsJson on submit; tests use default order (no drag simulation needed in headless)

Server: TestModeEndpoints.cs

Extended TestResponseRequest with an optional List<RankedItem>? RankedItems field so the /api/test/responses inject endpoint can persist priority ranking payloads directly, matching what the real respond form produces. Validation updated to accept rankedItems as a valid payload alongside selectedKey, freeText, and attachments.

Test seeder: Test-E2E-Mothership-QA.ps1

• Added fixture definitions for multiChoice, freeText, and priorityRanking
• Added -Headed switch for local debugging (opens real Chromium window)
• Corrected layer label from Layer 4 → Layer 5 throughout

Playwright spec: mothership-question-flow.spec.ts

• Extended Scenario interface with freeText? and rankedItems? submit fields
• Render test checks textarea[name="freeText"] and .rank-item for new types
• Submit test fills textarea / no-ops for ranking
• Inject test routes payload by type: freeText → freeText field, priorityRanking → rankedItems, others → selectedKey
• Verify assertions check type-appropriate response fields

CI: test.yml

Added test-mothership job (Layer 5, Ubuntu) that runs on every PR without manual trigger:

1. Builds the .NET server (Release)
2. Installs Playwright + Chromium
3. Starts Azurite, seeds answers and conversation-references containers
4. Starts DotbotServer with DOTBOT_TEST_MODE=true and BlobStorage__Backend=Local
5. Health-polls until server is up (fails the job if it never starts)
6. Runs Test-E2E-Mothership-QA.ps1 — seeds all 6 question types, mints magic-link JWTs, and runs 3 Playwright assertions per type (18 total)
7. Uploads Playwright HTML report as an artifact on failure

Docs: server/docs/MOTHERSHIP-E2E-SETUP.md

Updated to document all 6 question types and the rankedItems inject support. Running locally:

# Terminal 1
azurite --skipApiVersionCheck --location C:\azurite

# Terminal 2
az storage container create --name answers --connection-string "UseDevelopmentStorage=true"
az storage container create --name conversation-references --connection-string "UseDevelopmentStorage=true"
cd server/src/Dotbot.Server && dotnet run --launch-profile http-test

# Terminal 3 (from repo root)
$env:DOTBOT_SERVER_URL = "http://localhost:5048"
$env:DOTBOT_API_KEY    = "******"
pwsh tests/Test-E2E-Mothership-QA.ps1 -Headed   # omit -Headed for CI/headless

[OgnjenGligoric]

1. CI secrets in workflow

The test-mothership job needs two values to start the server in CI:

• BlobStorage__ConnectionString: 'UseDevelopmentStorage=true' (Azurite local emulator — not a real secret)
• Auth__JwtSigningKey: 'ci-test-signing-key-32-chars-min!!' (throwaway JWT key used only for test magic-link tokens)

These are committed in plain text in .github/workflows/test.yml. Since this is a test-only context with no access to real systems, it may be acceptable — but worth confirming:

Question: Should these be moved to GitHub Actions repository secrets (${{ secrets.DOTBOT_CI_JWT_KEY }} etc.), or is hardcoding throwaway CI-only values acceptable for this repo?

---

2. Why this was discovered — server startup error log

The server was silently crashing during CI. Added log capture to the health-check step to surface it. The actual error was:

[FTL] Application terminated unexpectedly
System.InvalidOperationException: Either BlobStorage:AccountUri or BlobStorage:ConnectionString must be configured

Root cause: --no-launch-profile wasn't set, so launchSettings.json was loaded but appsettings.Development.json was NOT read (env vars from the env: block weren't passed through correctly). Fixed by adding --no-launch-profile and supplying the required config explicitly via env vars.

Question: Is it acceptable to keep the server startup log capture (cat /tmp/dotbot-server.log) in the health-check step permanently, or should it be a separate debug step gated on failure only?

[IBondarenko-iwg]

@OgnjenGligoric Two issues worth fixing.

1. CI failure — missing Azurite connection string

Server crashes immediately with:

System.InvalidOperationException: Either BlobStorage:AccountUri or BlobStorage:ConnectionString must be configured

BlobStorage__Backend=Local is set but Program.cs:100 still validates that AccountUri or ConnectionString exists regardless of backend.

Fix: Add to the Start DotbotServer env block in test.yml:

BlobStorage__ConnectionString: 'UseDevelopmentStorage=true'
This is the standard Azurite connection string — validation passes naturally, no server code changes needed.

2. Layer inconsistency

Test-E2E-Mothership-QA.ps1 header says Layer 5. test.yml job name says Layer 5. But Run-Tests.ps1 puts it in the Layer 4 block alongside the real Claude E2E tests. This means Run-Tests.ps1 -Layer 5 does not run the Mothership tests, and -Layer 4 runs them only if ANTHROPIC_API_KEY is present (scheduled/manual CI only).

Since this test needs Playwright + Azurite but not Claude API credentials, it belongs in Layer 5. Move $mothershipExit from the if (4 -in $layersToRun) block to a if (5 -in $layersToRun) block in Run-Tests.ps1.

[Copilot]

Pull request overview

Adds a new Layer 5 end-to-end test suite for the Mothership “respond via magic link” web flow using Playwright, expands server test-mode support to inject priorityRanking responses, and wires the E2E suite into CI so it runs automatically on PRs.

Changes:

• Added a PowerShell seeder/runner that creates templates + instances for all 6 Mothership question types and runs Playwright assertions.
• Extended /api/test/responses (test mode) to accept and persist rankedItems payloads.
• Added a dedicated GitHub Actions job to spin up Azurite + DotbotServer and run the Playwright suite; added local setup docs.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/Test-E2E-Mothership-QA.ps1	New seeding + Playwright runner script for the Mothership respond flow across all question types.
tests/Run-Tests.ps1	Hooks the new Mothership E2E script into the layered test runner.
tests/e2e/specs/mothership-question-flow.spec.ts	New Playwright spec validating render/submit/storage for each seeded scenario.
tests/e2e/playwright.config.ts	Adds PLAYWRIGHT_HEADED-controlled headless toggle for Chromium runs.
server/src/Dotbot.Server/TestModeEndpoints.cs	Extends response injection validation/model to support rankedItems.
server/docs/MOTHERSHIP-E2E-SETUP.md	New/updated local setup documentation for running the Playwright + Azurite E2E suite.
.gitignore	Ignores Azurite local artifacts.
.github/workflows/test.yml	Adds test-mothership CI job to run the new Mothership Playwright E2E suite on PRs.

Comments suppressed due to low confidence (2)

tests/Test-E2E-Mothership-QA.ps1:244

• For question types without Options (e.g., freeText), $Qt.Options is $null, so $options becomes $null and options = @($options) serializes as [null]. The server validator rejects this (options[0] must not be null) because QuestionTemplate.Options is required. Build options as an empty array when there are no options (or conditionally omit the ForEach and set options = @()).

    $options = $Qt.Options | ForEach-Object {
        @{
            optionId    = [guid]::NewGuid().ToString()
            key         = $_.key
            title       = $_.label
        }
    }

    $body = @{
        questionId         = [guid]::NewGuid().ToString()
        version            = 1
        type               = $Qt.Type
        title              = $Qt.Title
        context            = "Playwright E2E test fixture"
        deliverableSummary = if ($Qt.ContainsKey('DeliverableSummary')) { $Qt.DeliverableSummary } else { $null }
        options            = @($options)
        project            = @{

tests/Test-E2E-Mothership-QA.ps1:272

• New-Instance hard-codes channel = "email", but DeliveryChannels:Email:Enabled is false by default (and the CI job doesn’t enable it), so /api/instances will return 400 “Delivery channel 'email' is not enabled…”. Use the default channel (omit channel) or switch to teams, or explicitly enable email in the CI server env if email is intended.

    # Use 'email' channel — delivery will fail (no SMTP) but the instance record
    # is persisted before delivery attempts, so /respond can still render it.
    $body = @{
        projectId       = $projectId
        questionId      = $QuestionId
        questionVersion = $Version
        channel         = "email"
        recipients      = @{ emails = @($testRecipient) }
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 5 comments.