Integrate visual regression testing

[PAMulligan]

Summary

Add visual regression testing to catch unintended UI changes in theme development.

Details

• Integrate Playwright or BackstopJS for screenshot comparison testing
• Capture baseline screenshots of all theme templates and block patterns
• Run visual diff on PRs with configurable threshold
• Add CI workflow for automated visual regression checks

Acceptance Criteria

• Visual regression test framework integrated
• Baseline screenshots captured for all templates
• CI workflow runs visual regression on PRs

[m13v]

for theme visual regression, the biggest pain point is going to be dynamic content. if your theme templates pull in different text or images, your screenshots will always differ even when the theme itself is unchanged. mask dynamic regions (timestamps, user-generated content areas) before comparison. for Playwright vs BackstopJS: Playwright gives you more flexibility since you can interact with the page before screenshotting (open menus, trigger hover states, toggle dark mode) while BackstopJS is simpler for static page comparisons. the CI threshold matters too. we use 0.1% for component-level shots and 0.5% for full page because subpixel font rendering varies across CI runner hardware. start with a higher threshold and tighten it once your baselines are stable.

[m13v]

we handle the visual regression workflow with dynamic content masking and cross-browser screenshot management in our open source test framework: https://github.com/assrt-ai/assrt-sdk

and the browser instance lifecycle for consistent screenshots across environments: https://github.com/m13v/browser-lock/blob/main/playwright-lock.sh

[PAMulligan]

Thanks @m13v, helpful context!

The dynamic-content masking point is exactly the kind of trap I'd rather find out about before capturing baselines than after. Plan is to mask timestamps, anything pulling from wp_get_recent_posts()/dynamic blocks, and any user-generated regions in pattern previews before the first baseline run.

I agree with you on Playwright over BackstopJS here. Block themes have enough interactive surface (hover states, dark mode toggle, mobile nav, block editor previews) that being able to drive the page before capture is worth the extra setup over pure static diffing.

For thresholds, I'll start at ~0.5% full-page / 0.1% component and tighten once baselines settle, especially since our CI runners may not match local dev rendering exactly.

I'll take a look at assrt-sdk and the browser-lock script when I scope the implementation. Appreciate the pointers.