From a82f6d84edeb24c203b054d8230554dabd47480b Mon Sep 17 00:00:00 2001 From: graslt Date: Sat, 28 Mar 2026 04:37:31 +0100 Subject: [PATCH 1/2] test: add comprehensive e2e scenario integration tests Add 24 end-to-end tests covering the full FirefoxClient API against a realistic multi-page web app fixture: todo CRUD, search, form submission, click/dblclick/hover, browser history, viewport resize, console & network monitoring, screenshots, tab management, stale UID detection, and error handling. All tests are self-contained with no ordering dependencies. Also update README with testing docs and known Firefox 148 startup crash on macOS ARM64 (Bug 2027228). --- README.md | 53 +- tests/fixtures/e2e-app.html | 339 ++++++++ .../e2e-scenario.integration.test.ts | 725 ++++++++++++++++++ 3 files changed, 1116 insertions(+), 1 deletion(-) create mode 100644 tests/fixtures/e2e-app.html create mode 100644 tests/integration/e2e-scenario.integration.test.ts diff --git a/README.md b/README.md index 0a577a2..5b85a0f 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,57 @@ npx @modelcontextprotocol/inspector node dist/index.js --headless --viewport 128 npm run inspector:dev ``` +## Testing + +```bash +# Run all tests once (unit + integration) +npm run test:run + +# Run only unit tests (fast, no Firefox needed) +npx vitest run tests/tools tests/firefox tests/utils tests/snapshot tests/cli tests/config tests/smoke.test.ts + +# Run only integration tests (launches real Firefox in headless mode) +npx vitest run tests/integration + +# Run the e2e scenario suite +npx vitest run tests/integration/e2e-scenario.integration.test.ts + +# Watch mode (re-runs on file changes) +npm test +``` + +### E2E scenario tests + +The file `tests/integration/e2e-scenario.integration.test.ts` contains end-to-end +tests that exercise the full `FirefoxClient` API against a realistic multi-page +web application (`tests/fixtures/e2e-app.html`). + +The fixture app has three pages (Todo List, Search, Registration Form) plus +always-visible hover/double-click targets. Each `describe` block launches its own +headless Firefox instance and tears it down after the tests. + +**Covered scenarios (24 tests):** + +| Scenario | What it tests | +| --------------------- | ---------------------------------------------------------------------- | +| Todo App Workflow | `takeSnapshot`, `fillByUid`, `clickByUid`, `evaluate` | +| Click Interactions | `clickByUid` (double-click), `hoverByUid` | +| Multi-Page Navigation | SPA page switching via UID clicks | +| Browser History | `navigateBack`, `navigateForward` | +| Viewport Resize | `setViewportSize` + dimension verification | +| Search Workflow | fill + click + result verification | +| Form Submission | `fillByUid`, `fillFormByUid` (batch), form submit | +| Console Monitoring | `getConsoleMessages`, `clearConsoleMessages` | +| Network Monitoring | `startNetworkMonitoring`, `getNetworkRequests`, `clearNetworkRequests` | +| Screenshot | `takeScreenshotPage` (base64 output) | +| Tab Management | `createNewPage`, `selectTab`, `closeTab`, `getTabs`, `refreshTabs` | +| Stale UID Detection | navigation invalidates old UIDs, `clearSnapshot` | +| Error Handling | invalid UID format, unknown UID, stale snapshot UID | + +### Known issues + +- **Firefox 148 startup crash on macOS ARM64** ([Bug 2027228](https://bugzilla.mozilla.org/show_bug.cgi?id=2027228)): Intermittent SIGSEGV in `RegisterFonts` thread (`RWLockImpl::writeLock()` null pointer) when launching Firefox in headless mode via Selenium. The crash is a race condition in Firefox font initialization and does not affect test results — Selenium recovers automatically. More likely to occur under fast sequential startup/shutdown cycles. + ## Troubleshooting - Firefox not found: pass `--firefox-path "/Applications/Firefox.app/Contents/MacOS/firefox"` (macOS) or the correct path on your OS. @@ -180,7 +231,7 @@ npm run inspector:dev ``` > **The Key Change:** On Windows, running a Node.js package via `npx` often requires the `cmd /c` prefix to be executed correctly from within another process like VSCode's extension host. Therefore, `"command": "npx"` was replaced with `"command": "cmd"`, and the actual `npx` command was moved into the `"args"` array, preceded by `"/c"`. This fix allows Windows to interpret the command correctly and launch the server. - + - **Solution 2** Instead of another layer of shell you can write the absolute path to `npx`: ```json diff --git a/tests/fixtures/e2e-app.html b/tests/fixtures/e2e-app.html new file mode 100644 index 0000000..83d7a5a --- /dev/null +++ b/tests/fixtures/e2e-app.html @@ -0,0 +1,339 @@ + + + + + + E2E Test Application + + + +

E2E Test Application

+
Ready
+ + + + +
+ + + + +
+ + +
+

Todo List

+
+ + +
+
+
+ 0 items + +
+
+ + +
+

Search

+
+ + +
+
+
+ + +
+

Registration

+
+
+ + +
Name is required
+
+
+ + +
Valid email is required
+
+
+ + +
+
+ + +
+ +
+
+
+ + + + diff --git a/tests/integration/e2e-scenario.integration.test.ts b/tests/integration/e2e-scenario.integration.test.ts new file mode 100644 index 0000000..92ecb8a --- /dev/null +++ b/tests/integration/e2e-scenario.integration.test.ts @@ -0,0 +1,725 @@ +/** + * E2E Scenario Integration Tests + * + * Comprehensive end-to-end tests exercising the full FirefoxClient API + * against a realistic multi-page web application (e2e-app.html fixture). + * + * Coverage: + * - Snapshot & UID workflow (take snapshot, find elements, resolve UIDs) + * - Click, double-click, hover by UID + * - Form filling (single field + batch fillFormByUid) + * - JavaScript evaluation via evaluate() + * - Multi-page SPA navigation via click + * - Browser history (navigateBack / navigateForward) + * - Viewport resize + * - Search workflow (fill + click + verify results) + * - Console message monitoring + * - Screenshot capture + * - Tab management (create, switch, close) + * - Stale UID detection after navigation + * - Error handling (invalid UID, unknown UID) + * - Network monitoring + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { + createTestFirefox, + closeFirefox, + waitForElementInSnapshot, + waitForPageLoad, + waitFor, +} from '../helpers/firefox.js'; +import type { FirefoxClient } from '@/firefox/index.js'; +import { resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const __dirname = fileURLToPath(new URL('.', import.meta.url)); +const fixturesPath = resolve(__dirname, '../fixtures'); +const appUrl = `file://${fixturesPath}/e2e-app.html`; + +// --------------------------------------------------------------------------- +// Todo App Workflow +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Todo App Workflow', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should load the app and take initial snapshot', async () => { + const snapshot = await firefox.takeSnapshot(); + + expect(snapshot.text).toContain('E2E Test Application'); + expect(snapshot.json.uidMap.length).toBeGreaterThan(0); + + const statusEl = snapshot.json.uidMap.find( + (e) => e.css.includes('#status') || e.css.includes('id="status"') + ); + expect(statusEl).toBeDefined(); + }, 10000); + + it('should add todo items via UID-based interaction', async () => { + const todoInput = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#todoInput') || e.css.includes('data-testid="todo-input"'), + 5000 + ); + + await firefox.fillByUid(todoInput.uid, 'Write BiDi tests'); + + const addBtn = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#addTodoBtn') || e.css.includes('data-testid="add-todo"'), + 5000 + ); + await firefox.clickByUid(addBtn.uid); + await waitForPageLoad(200); + + // Add second todo + const snapshot2 = await firefox.takeSnapshot(); + const todoInput2 = snapshot2.json.uidMap.find( + (e) => e.css.includes('#todoInput') || e.css.includes('data-testid="todo-input"') + ); + await firefox.fillByUid(todoInput2!.uid, 'Review PR'); + const addBtn2 = snapshot2.json.uidMap.find( + (e) => e.css.includes('#addTodoBtn') || e.css.includes('data-testid="add-todo"') + ); + await firefox.clickByUid(addBtn2!.uid); + await waitForPageLoad(200); + + const snapshot3 = await firefox.takeSnapshot(); + expect(snapshot3.text).toContain('Write BiDi tests'); + expect(snapshot3.text).toContain('Review PR'); + expect(snapshot3.text).toContain('2 items'); + }, 20000); + + it('should evaluate JavaScript to check app state', async () => { + const result = await firefox.evaluate('return document.getElementById("status").textContent'); + expect(result).toBeDefined(); + expect(typeof result).toBe('string'); + }, 10000); +}); + +// --------------------------------------------------------------------------- +// Click, Double-Click, Hover +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Click Interactions', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should double-click element by UID', async () => { + const dblBtn = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#dblClickTarget') || e.css.includes('data-testid="dblclick-target"'), + 5000 + ); + + await firefox.clickByUid(dblBtn.uid, true); + await waitForPageLoad(200); + + const result = await firefox.evaluate( + 'return document.getElementById("dblClickResult").textContent' + ); + expect(result).toBe('Double-clicked!'); + }, 15000); + + it('should hover over element by UID', async () => { + const hoverBtn = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#hoverTarget') || e.css.includes('data-testid="hover-target"'), + 5000 + ); + + await firefox.hoverByUid(hoverBtn.uid); + await waitForPageLoad(200); + + const result = await firefox.evaluate( + 'return document.getElementById("hoverResult").textContent' + ); + expect(result).toBe('Hovered!'); + }, 15000); +}); + +// --------------------------------------------------------------------------- +// Multi-Page Navigation +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Multi-Page Navigation', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should navigate between pages using nav buttons', async () => { + // Go to Search + const searchNav = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#navSearch'), + 5000 + ); + await firefox.clickByUid(searchNav.uid); + await waitForPageLoad(200); + + let snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Search'); + + // Go to Registration + const formNav = snapshot.json.uidMap.find((e) => e.css.includes('#navForm')); + await firefox.clickByUid(formNav!.uid); + await waitForPageLoad(200); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Registration'); + + // Back to Todo + const todoNav = snapshot.json.uidMap.find((e) => e.css.includes('#navTodo')); + await firefox.clickByUid(todoNav!.uid); + await waitForPageLoad(200); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Todo List'); + }, 20000); +}); + +// --------------------------------------------------------------------------- +// Browser History (navigateBack / navigateForward) +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Browser History', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should navigate back and forward between pages', async () => { + // Navigate to page A + await firefox.navigate(appUrl); + await waitForPageLoad(); + + let snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('E2E Test Application'); + + // Navigate to page B + const simpleUrl = `file://${fixturesPath}/simple.html`; + await firefox.navigate(simpleUrl); + await waitForPageLoad(); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Simple Test Page'); + + // Go back to page A + await firefox.navigateBack(); + await waitForPageLoad(); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('E2E Test Application'); + + // Go forward to page B + await firefox.navigateForward(); + await waitForPageLoad(); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Simple Test Page'); + }, 30000); +}); + +// --------------------------------------------------------------------------- +// Viewport Resize +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Viewport Resize', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should resize viewport and verify dimensions change', async () => { + // setViewportSize calls window().setRect() which sets the outer window size. + // innerWidth/innerHeight may differ due to browser chrome/toolbar offset, + // so we assert relative change rather than exact pixel values. + + await firefox.setViewportSize(800, 600); + await waitForPageLoad(200); + + const smallWidth = (await firefox.evaluate('return window.innerWidth')) as number; + const smallHeight = (await firefox.evaluate('return window.innerHeight')) as number; + + await firefox.setViewportSize(1200, 900); + await waitForPageLoad(200); + + const largeWidth = (await firefox.evaluate('return window.innerWidth')) as number; + const largeHeight = (await firefox.evaluate('return window.innerHeight')) as number; + + // Viewport should have grown in both dimensions + expect(largeWidth).toBeGreaterThan(smallWidth); + expect(largeHeight).toBeGreaterThan(smallHeight); + }, 15000); +}); + +// --------------------------------------------------------------------------- +// Search Workflow +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Search Workflow', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should search and display results', async () => { + const searchNav = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#navSearch'), + 5000 + ); + await firefox.clickByUid(searchNav.uid); + await waitForPageLoad(200); + + let snapshot = await firefox.takeSnapshot(); + const searchInput = snapshot.json.uidMap.find( + (e) => e.css.includes('#searchInput') || e.css.includes('data-testid="search-input"') + ); + await firefox.fillByUid(searchInput!.uid, 'bidi'); + + const searchBtn = snapshot.json.uidMap.find( + (e) => e.css.includes('#searchBtn') || e.css.includes('data-testid="search-btn"') + ); + await firefox.clickByUid(searchBtn!.uid); + await waitForPageLoad(200); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Bidirectional protocol'); + }, 15000); + + it('should show no results for unknown query', async () => { + // Self-contained: navigate to search page first + const searchNav = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#navSearch'), + 5000 + ); + await firefox.clickByUid(searchNav.uid); + await waitForPageLoad(200); + + let snapshot = await firefox.takeSnapshot(); + const searchInput = snapshot.json.uidMap.find( + (e) => e.css.includes('#searchInput') || e.css.includes('data-testid="search-input"') + ); + + await firefox.fillByUid(searchInput!.uid, 'nonexistent-xyz'); + + const searchBtn = snapshot.json.uidMap.find( + (e) => e.css.includes('#searchBtn') || e.css.includes('data-testid="search-btn"') + ); + await firefox.clickByUid(searchBtn!.uid); + await waitForPageLoad(200); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('No results found'); + }, 15000); +}); + +// --------------------------------------------------------------------------- +// Form Submission +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Form Submission', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should fill and submit registration form', async () => { + const formNav = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#navForm'), + 5000 + ); + await firefox.clickByUid(formNav.uid); + await waitForPageLoad(200); + + let snapshot = await firefox.takeSnapshot(); + + const nameInput = snapshot.json.uidMap.find((e) => e.css.includes('#regName')); + const emailInput = snapshot.json.uidMap.find((e) => e.css.includes('#regEmail')); + const bioInput = snapshot.json.uidMap.find((e) => e.css.includes('#regBio')); + const submitBtn = snapshot.json.uidMap.find((e) => e.css.includes('#regSubmitBtn')); + + await firefox.fillByUid(nameInput!.uid, 'Tomas Grasl'); + await firefox.fillByUid(emailInput!.uid, 'tomas@example.com'); + await firefox.fillByUid(bioInput!.uid, 'Firefox DevTools MCP contributor'); + + await firefox.clickByUid(submitBtn!.uid); + await waitForPageLoad(300); + + snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Registered'); + expect(snapshot.text).toContain('Tomas Grasl'); + }, 20000); + + it('should use fillFormByUid for batch form filling', async () => { + await firefox.navigate(appUrl); + await waitForPageLoad(); + + const formNav = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#navForm'), + 5000 + ); + await firefox.clickByUid(formNav.uid); + await waitForPageLoad(200); + + let snapshot = await firefox.takeSnapshot(); + + const nameInput = snapshot.json.uidMap.find((e) => e.css.includes('#regName')); + const emailInput = snapshot.json.uidMap.find((e) => e.css.includes('#regEmail')); + + await firefox.fillFormByUid([ + { uid: nameInput!.uid, value: 'Julian Descottes' }, + { uid: emailInput!.uid, value: 'julian@mozilla.com' }, + ]); + + const nameValue = await firefox.evaluate('return document.getElementById("regName").value'); + expect(nameValue).toBe('Julian Descottes'); + + const emailValue = await firefox.evaluate('return document.getElementById("regEmail").value'); + expect(emailValue).toBe('julian@mozilla.com'); + }, 20000); +}); + +// --------------------------------------------------------------------------- +// Console Monitoring +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Console Monitoring', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should capture console.log messages from the app', async () => { + // The app logs "[E2E App] Application loaded" on load — wait for it + await waitFor(async () => { + const messages = await firefox.getConsoleMessages(); + return messages.some((m) => m.text && m.text.includes('[E2E App]')); + }, 5000); + + const messages = await firefox.getConsoleMessages(); + const appLogMessage = messages.find((m) => m.text && m.text.includes('[E2E App]')); + expect(appLogMessage).toBeDefined(); + }, 10000); + + it('should capture dynamically generated console messages', async () => { + firefox.clearConsoleMessages(); + + await firefox.evaluate('console.log("BiDi test message", 42)'); + + // Wait for the BiDi event to arrive asynchronously + await waitFor(async () => { + const messages = await firefox.getConsoleMessages(); + return messages.some((m) => m.text && m.text.includes('BiDi test message')); + }, 5000); + + const messages = await firefox.getConsoleMessages(); + const testMessage = messages.find((m) => m.text && m.text.includes('BiDi test message')); + expect(testMessage).toBeDefined(); + }, 10000); +}); + +// --------------------------------------------------------------------------- +// Network Monitoring +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Network Monitoring', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.startNetworkMonitoring(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should capture network requests when navigating', async () => { + firefox.clearNetworkRequests(); + + // Navigate to network fixture which has fetch buttons + const networkUrl = `file://${fixturesPath}/network.html`; + await firefox.navigate(networkUrl); + await waitForPageLoad(); + + // Click the fetch GET button + const fetchBtn = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#fetchGet') || e.css.includes('fetchGet'), + 5000 + ); + await firefox.clickByUid(fetchBtn.uid); + + // Wait for request to be captured + await waitFor(async () => { + const requests = await firefox.getNetworkRequests(); + return requests.some((req) => req.url.includes('jsonplaceholder')); + }, 10000); + + const requests = await firefox.getNetworkRequests(); + const apiRequest = requests.find((req) => req.url.includes('jsonplaceholder')); + + expect(apiRequest).toBeDefined(); + expect(apiRequest?.method).toBe('GET'); + }, 20000); + + it('should clear network requests', async () => { + // Self-contained: generate a request first + firefox.clearNetworkRequests(); + + const networkUrl = `file://${fixturesPath}/network.html`; + await firefox.navigate(networkUrl); + await waitForPageLoad(); + + const fetchBtn = await waitForElementInSnapshot( + firefox, + (e) => e.css.includes('#fetchGet') || e.css.includes('fetchGet'), + 5000 + ); + await firefox.clickByUid(fetchBtn.uid); + + await waitFor(async () => { + const requests = await firefox.getNetworkRequests(); + return requests.some((req) => req.url.includes('jsonplaceholder')); + }, 10000); + + const requests = await firefox.getNetworkRequests(); + expect(requests.length).toBeGreaterThan(0); + + // Now clear and verify + firefox.clearNetworkRequests(); + + const cleared = await firefox.getNetworkRequests(); + expect(cleared.length).toBe(0); + }, 25000); +}); + +// --------------------------------------------------------------------------- +// Screenshot +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Screenshot', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should take a page screenshot as base64', async () => { + const screenshot = await firefox.takeScreenshotPage(); + + expect(screenshot).toBeDefined(); + expect(typeof screenshot).toBe('string'); + expect(screenshot.length).toBeGreaterThan(100); + }, 10000); +}); + +// --------------------------------------------------------------------------- +// Tab Management +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Tab Management', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should open a new tab and switch between tabs', async () => { + const simpleUrl = `file://${fixturesPath}/simple.html`; + const newTabIndex = await firefox.createNewPage(simpleUrl); + await waitForPageLoad(); + + expect(newTabIndex).toBeGreaterThan(0); + + // Verify new tab + const snapshot = await firefox.takeSnapshot(); + expect(snapshot.text).toContain('Simple Test Page'); + + // Switch back to first tab + await firefox.selectTab(0); + await waitForPageLoad(200); + + const snapshot2 = await firefox.takeSnapshot(); + expect(snapshot2.text).toContain('E2E Test Application'); + + // Close the second tab + await firefox.closeTab(newTabIndex); + }, 20000); + + it('should list tabs correctly', async () => { + await firefox.refreshTabs(); + const tabs = firefox.getTabs(); + + expect(tabs.length).toBeGreaterThanOrEqual(1); + + const currentIdx = firefox.getSelectedTabIdx(); + expect(currentIdx).toBeGreaterThanOrEqual(0); + expect(currentIdx).toBeLessThan(tabs.length); + }, 10000); +}); + +// --------------------------------------------------------------------------- +// Stale UID Detection +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Stale UID Detection', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should detect stale UIDs after navigation', async () => { + await firefox.navigate(appUrl); + await waitForPageLoad(); + + const snapshot = await firefox.takeSnapshot(); + const firstUid = snapshot.json.uidMap[0]?.uid; + expect(firstUid).toBeDefined(); + + // Navigate away — UIDs become stale + await firefox.navigate(`file://${fixturesPath}/simple.html`); + await waitForPageLoad(); + + // Old UID should throw + await expect(firefox.clickByUid(firstUid!)).rejects.toThrow(/(stale snapshot|UID not found)/); + }, 20000); + + it('should detect stale UIDs after clearSnapshot()', async () => { + await firefox.navigate(appUrl); + await waitForPageLoad(); + + const snapshot = await firefox.takeSnapshot(); + const uid = snapshot.json.uidMap[0]?.uid; + expect(uid).toBeDefined(); + + firefox.clearSnapshot(); + + await expect(firefox.clickByUid(uid!)).rejects.toThrow(); + }, 15000); +}); + +// --------------------------------------------------------------------------- +// Error Handling +// --------------------------------------------------------------------------- + +describe('E2E Scenario: Error Handling', () => { + let firefox: FirefoxClient; + + beforeAll(async () => { + firefox = await createTestFirefox(); + await firefox.navigate(appUrl); + await waitForPageLoad(); + }, 30000); + + afterAll(async () => { + await closeFirefox(firefox); + }); + + it('should throw on invalid UID format', async () => { + await expect(firefox.clickByUid('invalid-no-underscore')).rejects.toThrow(/Invalid UID format/); + }, 10000); + + it('should throw on unknown UID', async () => { + // Take a snapshot to set a valid snapshot ID + const snapshot = await firefox.takeSnapshot(); + const snapshotId = snapshot.json.snapshotId; + + // Use valid format but non-existent UID + await expect(firefox.clickByUid(`${snapshotId}_nonexistent`)).rejects.toThrow(/UID not found/); + }, 10000); + + it('should throw on stale snapshot UID', async () => { + const snapshot = await firefox.takeSnapshot(); + const snapshotId = snapshot.json.snapshotId; + + // Take another snapshot to bump ID + await firefox.takeSnapshot(); + + // Old snapshot ID is now stale + await expect(firefox.clickByUid(`${snapshotId}_button`)).rejects.toThrow(/stale snapshot/); + }, 10000); +}); From ad33fd72569076a290bfb7c75b50a71548cef363 Mon Sep 17 00:00:00 2001 From: graslt Date: Sat, 28 Mar 2026 04:40:16 +0100 Subject: [PATCH 2/2] docs: move testing details to docs/testing.md, keep README concise --- README.md | 48 +++----------------------------------- docs/testing.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 45 deletions(-) create mode 100644 docs/testing.md diff --git a/README.md b/README.md index 5b85a0f..582901b 100644 --- a/README.md +++ b/README.md @@ -165,53 +165,11 @@ npm run inspector:dev ## Testing ```bash -# Run all tests once (unit + integration) -npm run test:run - -# Run only unit tests (fast, no Firefox needed) -npx vitest run tests/tools tests/firefox tests/utils tests/snapshot tests/cli tests/config tests/smoke.test.ts - -# Run only integration tests (launches real Firefox in headless mode) -npx vitest run tests/integration - -# Run the e2e scenario suite -npx vitest run tests/integration/e2e-scenario.integration.test.ts - -# Watch mode (re-runs on file changes) -npm test +npm run test:run # all tests once (unit + integration) +npm test # watch mode ``` -### E2E scenario tests - -The file `tests/integration/e2e-scenario.integration.test.ts` contains end-to-end -tests that exercise the full `FirefoxClient` API against a realistic multi-page -web application (`tests/fixtures/e2e-app.html`). - -The fixture app has three pages (Todo List, Search, Registration Form) plus -always-visible hover/double-click targets. Each `describe` block launches its own -headless Firefox instance and tears it down after the tests. - -**Covered scenarios (24 tests):** - -| Scenario | What it tests | -| --------------------- | ---------------------------------------------------------------------- | -| Todo App Workflow | `takeSnapshot`, `fillByUid`, `clickByUid`, `evaluate` | -| Click Interactions | `clickByUid` (double-click), `hoverByUid` | -| Multi-Page Navigation | SPA page switching via UID clicks | -| Browser History | `navigateBack`, `navigateForward` | -| Viewport Resize | `setViewportSize` + dimension verification | -| Search Workflow | fill + click + result verification | -| Form Submission | `fillByUid`, `fillFormByUid` (batch), form submit | -| Console Monitoring | `getConsoleMessages`, `clearConsoleMessages` | -| Network Monitoring | `startNetworkMonitoring`, `getNetworkRequests`, `clearNetworkRequests` | -| Screenshot | `takeScreenshotPage` (base64 output) | -| Tab Management | `createNewPage`, `selectTab`, `closeTab`, `getTabs`, `refreshTabs` | -| Stale UID Detection | navigation invalidates old UIDs, `clearSnapshot` | -| Error Handling | invalid UID format, unknown UID, stale snapshot UID | - -### Known issues - -- **Firefox 148 startup crash on macOS ARM64** ([Bug 2027228](https://bugzilla.mozilla.org/show_bug.cgi?id=2027228)): Intermittent SIGSEGV in `RegisterFonts` thread (`RWLockImpl::writeLock()` null pointer) when launching Firefox in headless mode via Selenium. The crash is a race condition in Firefox font initialization and does not affect test results — Selenium recovers automatically. More likely to occur under fast sequential startup/shutdown cycles. +See [docs/testing.md](docs/testing.md) for full details on running specific test suites, the e2e scenario coverage, and known issues. ## Troubleshooting diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..8047ade --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,62 @@ +# Testing + +## Running tests + +```bash +# Run all tests once (unit + integration) +npm run test:run + +# Run only unit tests (fast, no Firefox needed) +npx vitest run tests/tools tests/firefox tests/utils tests/snapshot tests/cli tests/config tests/smoke.test.ts + +# Run only integration tests (launches real Firefox in headless mode) +npx vitest run tests/integration + +# Run the e2e scenario suite +npx vitest run tests/integration/e2e-scenario.integration.test.ts + +# Watch mode (re-runs on file changes) +npm test +``` + +## E2E scenario tests + +The file `tests/integration/e2e-scenario.integration.test.ts` contains end-to-end +tests that exercise the full `FirefoxClient` API against a realistic multi-page +web application (`tests/fixtures/e2e-app.html`). + +The fixture app has three pages (Todo List, Search, Registration Form) plus +always-visible hover/double-click targets. Each `describe` block launches its own +headless Firefox instance and tears it down after the tests. + +All tests are self-contained (no ordering dependencies) and use active polling +(`waitFor`) instead of fixed sleeps for async BiDi events. + +### Covered scenarios (24 tests) + +| Scenario | What it tests | +| --------------------- | ---------------------------------------------------------------------- | +| Todo App Workflow | `takeSnapshot`, `fillByUid`, `clickByUid`, `evaluate` | +| Click Interactions | `clickByUid` (double-click), `hoverByUid` | +| Multi-Page Navigation | SPA page switching via UID clicks | +| Browser History | `navigateBack`, `navigateForward` | +| Viewport Resize | `setViewportSize` + relative dimension verification | +| Search Workflow | fill + click + result verification | +| Form Submission | `fillByUid`, `fillFormByUid` (batch), form submit | +| Console Monitoring | `getConsoleMessages`, `clearConsoleMessages` | +| Network Monitoring | `startNetworkMonitoring`, `getNetworkRequests`, `clearNetworkRequests` | +| Screenshot | `takeScreenshotPage` (base64 output) | +| Tab Management | `createNewPage`, `selectTab`, `closeTab`, `getTabs`, `refreshTabs` | +| Stale UID Detection | navigation invalidates old UIDs, `clearSnapshot` | +| Error Handling | invalid UID format, unknown UID, stale snapshot UID | + +### Design principles + +- **Self-contained**: each test navigates to its own page, no inter-test dependencies +- **Active polling**: async events (console, network) use `waitFor` instead of fixed sleeps +- **Relative assertions**: viewport tests assert relative change, not exact pixel values (platform-dependent) +- **Isolated Firefox instances**: each `describe` block gets its own headless Firefox + +## Known issues + +- **Firefox 148 startup crash on macOS ARM64** ([Bug 2027228](https://bugzilla.mozilla.org/show_bug.cgi?id=2027228)): Intermittent SIGSEGV in `RegisterFonts` thread (`RWLockImpl::writeLock()` null pointer) when launching Firefox in headless mode via Selenium. The crash is a race condition in Firefox font initialization and does not affect test results — Selenium recovers automatically. More likely to occur under fast sequential startup/shutdown cycles.