Playwright AI Test Framework (TypeScript)

A production-ready Playwright test framework with AI-powered self-healing, visual regression testing, API mocking, and BrowserStack integration. TypeScript port of the Python version.

Targets The Internet test application.

Prerequisites

Node.js 22+ — Download
npm (included with Node.js)
Ollama (optional, for AI self-healing) — Install

Quick Start

# Clone the repository
git clone https://github.com/philmcneely/playwright-ai-framework-ts.git
cd playwright-ai-framework-ts

# Install dependencies
npm install

# Install Playwright browsers
npx playwright install --with-deps

# Copy environment template
cp .env.example .env.dev

# Run smoke tests
npm run test:smoke

Environment Configuration

The framework loads environment variables from .env files using dotenv. It loads .env first, then .env.<ENV> (defaulting to .env.dev).

Copy .env.example and customize per environment:

cp .env.example .env.dev    # Development
cp .env.example .env.test   # Test
cp .env.example .env.prod   # Production

Switch environments at runtime:

ENV=test npx playwright test

Key Environment Variables

Variable	Default	Description
`BASE_URL`	`https://the-internet.herokuapp.com`	Application under test
`ENV`	`dev`	Environment name (loads `.env.<ENV>`)
`BROWSER`	`chromium`	Browser engine: `chromium`, `firefox`, `webkit`
`HEADLESS`	`true`	Run browser headless
`SLOW_MO`	`100`	Slow down actions by N ms
`TIMEOUT`	`30000`	Global test timeout in ms
`RETRY_COUNT`	`3`	Number of retries on failure
`SCREENSHOT_ON_FAILURE`	`true`	Capture screenshot on failure
`VIDEO_ON_FAILURE`	`true`	Record video, retained on failure
`AI_HEALING_ENABLED`	`false`	Enable AI self-healing analysis
`OLLAMA_MODEL`	`llama3.1:8b`	Ollama model for healing
`OLLAMA_HOST`	`http://localhost:11434`	Ollama server URL
`OLLAMA_TEMPERATURE`	`0.1`	LLM temperature
`AI_HEALING_CONFIDENCE`	`0.7`	Confidence threshold for healing suggestions
`AI_HEALING_CONTEXT_WINDOW`	`5000`	Max DOM characters sent to the model
`BROWSERSTACK_ENABLED`	`false`	Run on BrowserStack
`BROWSERSTACK_USERNAME`	—	BrowserStack username
`BROWSERSTACK_ACCESS_KEY`	—	BrowserStack access key
`JIRA_ENABLED`	`false`	Enable Jira test reporting
`JIRA_BASE`	—	Jira instance URL
`JIRA_USER`	—	Jira username/email
`JIRA_TOKEN`	—	Jira API token
`JIRA_JQL`	`project = ABC AND status in ("To Do", "In Progress")`	JQL filter for ticket lookup
`JIRA_DRY_RUN`	`false`	Log instead of posting
`JIRA_TRANSITION_ON_PASS`	—	Transition name on test pass
`JIRA_TRANSITION_ON_FAIL`	—	Transition name on test fail
`OBSERVABILITY_ENABLED`	`false`	Enable test metrics collection
`DEBUG_MSG`	`false`	Verbose debug logging

Running Tests

All commands use @playwright/test via the scripts defined in package.json.

# Full test suite (default project: chromium)
npm test

# Smoke tests only
npm run test:smoke

# Visual regression tests
npm run test:visual

# Security / login attack tests
npm run test:security

# API mocking tests
npm run test:api

# Headed mode (browser visible)
npm run test:headed

# Debug mode (Playwright Inspector)
npm run test:debug

Passing Extra Options

Append flags after --:

# Run a specific test file
npx playwright test tests/login/test-login.spec.ts

# Run against Firefox
npx playwright test --project=firefox

# Filter by tag
npx playwright test --grep @smoke --project=chromium

# Type check and lint
npm run typecheck
npm run lint

Multi-Browser Support

Three browser projects are configured in playwright.config.ts:

Project	Engine	Device Profile
`chromium`	Chromium	Desktop Chrome
`firefox`	Firefox	Desktop Firefox
`webkit`	WebKit	Desktop Safari

Run against a specific browser:

npx playwright test --project=firefox

Run against all browsers:

npx playwright test --project=chromium --project=firefox --project=webkit

Visual Regression Testing

Uses pixelmatch and pngjs for pixel-level screenshot comparison. No external services required.

How It Works

First run — saves a baseline screenshot.
Subsequent runs — compares current screenshot against baseline.
If diff exceeds tolerance (or dimensions changed) — compare() returns passed: false and a diff image is saved. Assert on result.passed to fail the test.

File Locations

Directory	Purpose
`test_artifacts/visual/visual_baselines/`	Stored baseline images
`test_artifacts/visual/visual_current/`	Current run screenshots
`test_artifacts/visual/visual_diffs/`	Diff images (only on failure)

Usage in Tests

import { test, expect } from "../../fixtures/index.js";

test("homepage visual check @visual", async ({ visualRegression, page }) => {
  await page.goto("/");
  const result = await visualRegression.compare("homepage", {
    tolerance: 0.02,   // 2% pixel difference allowed
    fullPage: true,
  });
  expect(result.passed).toBeTruthy();
});

Options

selector — CSS selector for element-level screenshots
fullPage — capture the full scrollable page (default: false)
tolerance — max fraction of differing pixels (default: 0.01 = 1%)

Managing Baselines

Delete a baseline to regenerate it on the next run. The utility also exports helper functions:

import { resetBaseline, resetAllBaselines, listVisualFiles } from "../utils/visual-regression.js";

resetBaseline("homepage");    // Delete one baseline
resetAllBaselines();          // Delete all baselines
listVisualFiles();            // List all visual artifacts

API Mocking

The NetworkMocker utility wraps Playwright's route interception with a clean API for mocking HTTP requests.

Usage in Tests

The apiMocker fixture is available automatically through test.extend():

import { test, expect } from "../../fixtures/index.js";

test("mock API response @api", async ({ apiMocker, page }) => {
  await apiMocker.mockGet("**/api/users", { users: [{ id: 1, name: "Test" }] });
  await page.goto("/some-page");
  // The mocked response will be returned for any matching GET request
});

Available Methods

Method	Description
`mockGet(pattern, body, options?)`	Mock GET requests
`mockPost(pattern, body, options?)`	Mock POST requests
`mockPut(pattern, body, options?)`	Mock PUT requests
`mockDelete(pattern, body, options?)`	Mock DELETE requests
`mockFromFile(pattern, filePath, method?, options?)`	Load mock data from a JSON file
`mockWithFunction(pattern, handler, method?, options?)`	Dynamic response via callback
`simulateNetworkFailure(pattern)`	Abort matching requests
`simulateSlowNetwork(pattern, delayMs)`	Add latency before continuing
`simulateOffline()`	Abort all requests
`clearMocks()`	Remove all registered mocks
`getRequestLog()` / `getResponseLog()`	Inspect intercepted traffic

Mock Templates

Pre-built response templates for common HTTP scenarios:

import { getMockTemplate } from "../utils/network-mocking.js";

const template = getMockTemplate("unauthorized"); // 401 response
// Available: success, created, noContent, badRequest, unauthorized, forbidden, notFound, serverError

AI Self-Healing

When a test fails after all retries, the AI healing system queries a local Ollama instance to analyze the failure and suggest fixes. Reports are saved as Markdown files.

Setup

Install Ollama: https://ollama.ai
Pull a model:

# Text-only (faster, smaller)
ollama pull llama3.1:8b

# Vision + text (can analyze failure screenshots)
ollama pull llava:7b

Enable in your .env file:

AI_HEALING_ENABLED=true
OLLAMA_MODEL=llama3.1:8b

How It Works

AIHealingReporter is registered in playwright.config.ts as a custom reporter.
On each test failure, the reporter tracks retry counts.
After the final failure (all retries exhausted), it triggers healing:
- Reads the test source file
- Collects error details and any failure screenshots
- Sends a structured prompt to Ollama
- Parses the JSON response (with 6 fallback parsing strategies)
- Writes a Markdown report to test_artifacts/ai/ai_healing_reports/
- Optionally saves a healed .ts file if the model provides updated code

Running the Demo

A dedicated test exists to trigger AI healing intentionally:

AI_HEALING_ENABLED=true npx playwright test --grep @trigger_ai_healing --project=chromium

Reports appear in test_artifacts/ai/ai_healing_reports/.

Architecture Note

The Python version uses pytest_runtest_makereport hooks. The TypeScript version uses Playwright's custom Reporter API (AIHealingReporter), which receives onTestEnd callbacks with full access to test metadata, errors, and attachments.

BrowserStack Integration

Run tests on real browsers in the cloud via BrowserStack Automate.

Setup

Set credentials in your .env file:

BROWSERSTACK_ENABLED=true
BROWSERSTACK_USERNAME=your_username
BROWSERSTACK_ACCESS_KEY=your_access_key

Run tests:

npx playwright test --project=browserstack

How It Works

When BROWSERSTACK_ENABLED=true, an additional browserstack project is added to playwright.config.ts that connects via WebSocket to BrowserStack's CDP endpoint. The default capabilities target Chrome on macOS Sonoma. Customize in utils/browserstack.ts.

When BROWSERSTACK_ENABLED=false, the browserstack project is not registered and all tests run locally.

Jira Integration

Automatically post test results to Jira tickets and trigger workflow transitions on pass/fail. Conditionally enabled via environment variables — zero overhead when disabled.

Setup

Add the following to your .env.dev (or relevant .env file):

JIRA_ENABLED=true
JIRA_BASE=https://your-instance.atlassian.net
JIRA_USER=you@example.com
JIRA_TOKEN=your_api_token

Note: Set JIRA_ENABLED=false to disable entirely. Set JIRA_DRY_RUN=true to log what would be posted without making API calls.

How It Works

JiraReporter is registered in playwright.config.ts as a custom reporter.
Test names and title paths are scanned for Jira ticket patterns (e.g., PROJ-123).
On test completion (pass or final retry failure), a formatted comment is posted to the ticket.
If transition env vars are set, the ticket's workflow state is updated automatically.
Tests can also use test.info().annotations with type jira and the ticket ID as description.

Linking Tests to Tickets

Include the ticket ID anywhere in the test title path:

test("PROJ-123 login flow works correctly @smoke", async ({ page }) => {
  // ...
});

Or annotate the test:

test("login flow works correctly", async ({ page }) => {
  test.info().annotations.push({ type: "jira", description: "PROJ-123" });
  // ...
});

Architecture Note

The Python version uses pytest hooks (pytest_runtest_logreport). The TypeScript version uses Playwright's custom Reporter API (JiraReporter), which provides onTestEnd callbacks with full access to test metadata and retry state.

Test Observability

Structured per-test metrics collection with JSONL output, summary reports, and error categorization. Tracks flake rate, pass rate, slowest tests, heal events, and failures by category.

Setup

OBSERVABILITY_ENABLED=true

Note: Set OBSERVABILITY_ENABLED=false to disable metric collection entirely. Zero overhead when disabled.

What Gets Tracked

Each test execution records:

Field	Description
`testId`	Unique test identifier
`testName`	Human-readable test name
`suite`	Test suite/describe block
`status`	passed, failed, skipped, timedOut
`durationMs`	Execution time in milliseconds
`retryCount`	Number of retries for this test
`browser`	Browser engine used
`commitSha`	Git commit (auto-detected)
`branch`	Git branch (auto-detected)
`healEvent`	Whether AI healing was triggered
`errorCategory`	Auto-categorized: timeout, element-not-found, navigation, assertion, browser-crash, other

Output Files

File	Location
JSONL metrics	`test_artifacts/observability/metrics.jsonl`
Summary JSON	`test_artifacts/observability/summary.json`
Markdown report	`test_artifacts/observability/report.md`

Running with Observability

OBSERVABILITY_ENABLED=true npx playwright test --project=chromium

The summary is printed to the console at session end and a Markdown report is written to test_artifacts/observability/report.md.

CI/CD Setup

A GitHub Actions workflow is provided at .github/workflows/smoke-full.yml.

Pipeline Structure

push/PR to main ──> Smoke Job ──> (pass) ──> Full Job
                                  (fail) ──> Stop

Smoke job:

Installs Node 22, runs npm ci, installs Chromium only
Runs tests matching @smoke against the chromium project
Uploads playwright-report/ as an artifact

Full job:

Depends on smoke passing
Installs all browsers
Runs the complete test suite
Uploads the full report

Secrets

No secrets are required for the default configuration (tests target a public site). If you add authenticated tests, configure repository secrets and reference them in the workflow env block.

Directory Structure

playwright-ai-framework-ts/
├── .env.example                    # Environment variable template
├── .github/
│   └── workflows/
│       └── smoke-full.yml          # CI/CD pipeline
├── config/
│   ├── artifact-paths.ts           # Centralized artifact directory paths
│   ├── index.ts                    # Config barrel export
│   └── settings.ts                 # Environment-driven settings
├── data/
│   ├── index.ts                    # Data barrel export
│   └── test-data.ts                # Test personas, credentials, expected messages
├── fixtures/
│   └── index.ts                    # test.extend() with app, apiMocker, visualRegression
├── pages/
│   ├── app.ts                      # App facade (aggregates all page objects)
│   ├── base-page.ts                # Base page with shared helpers
│   ├── index.ts                    # Pages barrel export
│   ├── login-page.ts               # Login page object
│   └── secure-page.ts              # Secure area page object
├── tests/
│   ├── ai-healing/
│   │   └── test-ai-healing-trigger.spec.ts
│   ├── api/
│   │   └── test-api-mocking.spec.ts
│   ├── login/
│   │   ├── test-login.spec.ts
│   │   └── test-login-attacks.spec.ts
│   └── visual/
│       └── test-visual-regression.spec.ts
├── utils/
│   ├── ai-healing.ts               # Ollama healing service
│   ├── ai-healing-reporter.ts      # Custom Playwright Reporter
│   ├── browserstack.ts             # BrowserStack capability builder
│   ├── decorators.ts               # Utility decorators
│   ├── jira-client.ts              # Jira REST API client
│   ├── jira-reporter.ts            # Jira Playwright Reporter
│   ├── network-mocking.ts          # NetworkMocker + templates
│   ├── observability-reporter.ts   # Observability Playwright Reporter
│   ├── test-observability.ts       # Test metrics collector
│   └── visual-regression.ts        # pixelmatch-based visual comparison
├── test_artifacts/                  # Generated at runtime (gitignored)
│   ├── ai/ai_healing_reports/      # AI analysis Markdown reports
│   ├── observability/              # JSONL metrics, summary, report
│   └── visual/                     # Baselines, current screenshots, diffs
├── package.json
├── playwright.config.ts
└── tsconfig.json

Python to TypeScript Migration

This project is a full TypeScript port of the Python Playwright AI Framework. The following table maps each Python feature to its TypeScript equivalent.

Python Feature	TypeScript Equivalent
`pytest` + `pytest-asyncio`	`@playwright/test` (built-in async)
`conftest.py` (page fixture)	`fixtures/index.ts` via `test.extend()`
`config/settings.py`	`config/settings.ts`
`config/artifact_paths.py`	`config/artifact-paths.ts`
`pages/` (Page Object Model)	`pages/` (Page Object Model)
`data/test_data.py`	`data/test-data.ts`
`utils/ai_healing.py`	`utils/ai-healing.ts`
`pytest_runtest_makereport` hook	`AIHealingReporter` (custom Reporter)
`utils/visual_regression.py` (OpenCV)	`utils/visual-regression.ts` (pixelmatch + pngjs)
`utils/network_mocking.py`	`utils/network-mocking.ts`
`@screenshot_on_failure` decorator	Built-in Playwright config (`screenshot: "only-on-failure"`)
`@retry_decorator`	Built-in Playwright config (`retries`)
BrowserStack integration	`utils/browserstack.ts` + config project
`utils/jira_client.py`	`utils/jira-client.ts`
`utils/test_observability.py`	`utils/test-observability.ts`
Jira via `pytest_runtest_logreport` hook	`JiraReporter` (custom Reporter)
Observability via pytest hooks	`ObservabilityReporter` (custom Reporter)
`.github/workflows/`	`.github/workflows/smoke-full.yml`
`requirements_with_versions.txt`	`package.json` (npm)
Python virtual environment	Node.js (no venv needed)
`python-dotenv`	`dotenv` npm package

Key Architectural Differences

No conftest.py — Fixtures are defined via test.extend<Fixtures>() in fixtures/index.ts and imported by test files.
No pytest markers — Tags are embedded in test titles (e.g., @smoke, @visual) and filtered with --grep.
Visual regression — Python used OpenCV; TypeScript uses pixelmatch (pure JS, no native dependencies).
Reporter vs hooks — Python used pytest hooks; TypeScript uses Playwright's Reporter API, which provides structured callbacks for test lifecycle events.

Troubleshooting

Browser installation errors

Error: Executable doesn't exist at ...

Run npx playwright install --with-deps to download all browser binaries and system dependencies.

Missing environment variables

Error: BASE_URL is not defined

Ensure you have copied .env.example to .env.dev (or the file matching your ENV value) and that the variables are set.

Ollama not responding

[AI Healing] Ollama service not running, attempting to start...

Verify Ollama is installed: ollama --version
Start it manually: ollama serve
Check the model is pulled: ollama list
Test connectivity: curl http://localhost:11434/api/tags

Visual regression baseline mismatch

If baselines were generated on a different OS or screen resolution, delete the baseline and re-run to regenerate:

rm test_artifacts/visual/visual_baselines/<name>.png
npx playwright test --grep @visual

TypeScript compilation errors

The project uses NodeNext module resolution. Ensure imports include the .js extension:

// Correct
import { settings } from "./config/settings.js";

// Wrong — will fail at runtime
import { settings } from "./config/settings";

BrowserStack connection failures

Verify credentials: echo $BROWSERSTACK_USERNAME
Check that BROWSERSTACK_ENABLED=true is set
Ensure your BrowserStack account is active and has available parallel slots

Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
.github/workflows		.github/workflows
config		config
data		data
fixtures		fixtures
pages		pages
tests		tests
utils		utils
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
enhancements.md		enhancements.md
eslint.config.mjs		eslint.config.mjs
package-lock.json		package-lock.json
package.json		package.json
playwright.config.ts		playwright.config.ts
tsconfig.json		tsconfig.json

Folders and files

Latest commit

History

Repository files navigation

Playwright AI Test Framework (TypeScript)

Table of Contents

Prerequisites

Quick Start

Environment Configuration

Key Environment Variables

Running Tests

Passing Extra Options

Multi-Browser Support

Visual Regression Testing

How It Works

File Locations

Usage in Tests

Options

Managing Baselines

API Mocking

Usage in Tests

Available Methods

Mock Templates

AI Self-Healing

Setup

How It Works

Running the Demo

Architecture Note

BrowserStack Integration

Setup

How It Works

Jira Integration

Setup

How It Works

Linking Tests to Tickets

Architecture Note

Test Observability

Setup

What Gets Tracked

Output Files

Running with Observability

CI/CD Setup

Pipeline Structure

Secrets

Directory Structure

Python to TypeScript Migration

Key Architectural Differences

Troubleshooting

Browser installation errors

Missing environment variables

Ollama not responding

Visual regression baseline mismatch

TypeScript compilation errors

BrowserStack connection failures

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages