DO NOT MERGE: E2E gate validation test by fkb032 · Pull Request #45 · darkmatter/nixmac

Farhan (fkb032) · 2026-04-23T07:44:21Z

DO NOT MERGE

This is a draft validation PR for the new nixmac E2E gate. It exists only to trigger and inspect the GitHub Actions E2E status, PR comment, report links, and visual proof behavior.

What this is testing:

nixmac E2E Gate aggregate status/check
default Tauri WDIO scenario matrix
sticky PR comment output
report/proof artifact publishing
full-Mac lane gating behavior for release-sensitive paths if applicable

The final commit intentionally adds apps/native/e2e-tauri/DO_NOT_MERGE_E2E_GATE_TEST.md as a visible marker and native-path trigger.

Do not review for product merge. Do not merge this PR.

Co-authored-by: Copilot <copilot@github.com>

…p functions for clarity Co-authored-by: Copilot <copilot@github.com>

github-actions · 2026-04-23T07:44:54Z

	Fails
🚫	PR description is missing a `## Test Plan` (or `## Testing Instructions`) section. Add one describing how a reviewer can verify your change, or tag the PR `#trivial` if no testing is needed.
🚫	New Rust modules were added without tests. Add a `#[cfg(test)] mod tests { … }` block or a file under `apps/native/src-tauri/tests/`: `apps/native/src-tauri/src/e2e_support.rs`
🚫	This PR touches behavior-sensitive code that is documented in darkmatter/nixmac-web. Please either: Open a companion docs PR and check Docs updated in the PR description, or Check No docs update needed if the change doesn't affect user-facing behavior.

	Warnings
⚠️	PR is marked WIP / draft — do not merge until ready for review.
⚠️	Please assign this PR to someone (usually yourself).
⚠️	❗ Big PR (6454 lines changed). Consider splitting it into smaller, focused changes.
⚠️	New TypeScript source files were added without any new tests: `apps/native/src/utils/e2e-proof-mode.ts`

	Messages
📖	🤖 This PR touches CI / infra — reviewers please pay extra attention: `.github/workflows/e2e.yml`

📋 PR Overview


Lines changed	6454 (+6266 / -188)
Files	31 added, 39 modified, 0 deleted
Draft / WIP	yes
Has Test Plan	no
New UI components	no
New Storybook stories	no
New Rust modules	yes (1)
New TS source files	yes (1)
New tests	no
package.json touched	yes
Cargo.toml touched	no
Infra / CI touched	yes

Generated by 🚫 dangerJS against 24cd4b9

github-actions · 2026-04-23T07:56:19Z

nixmac E2E

Status: passed

Why this ran: required: e2e:all label present

Scope: Ran all configured scenarios: 13/13 configured scenarios (11/11 hosted, 2/2 full-Mac). PR-matched files are called out below as adversarial focus.

Coverage note: Scripted product-surface E2E pack runs first; PR-matched risk areas are called out as adversarial focus. Coverage is broad, not exhaustive.

Tested commit: b13f39985b8718bb8c881755235f51bbb8d63a66

PR-specific Focus

All configured scenarios ran; PR path matches are called out as extra focus.

Changed files considered: 95. Scenario-specific risk matches:

Auto-evolve basic package: Submits a mocked AI prompt and verifies nixmac reaches review with a generated Nix diff.
Discard and restore state: Exercises discard confirm/cancel behavior from the evolve review screen.
Manual evolve existing changes: Runs sequential prompts and verifies the second change preserves the first generated diff.
Settings and provider tabs: Opens Settings and verifies the General, AI Models, API Keys, and Preferences tabs plus key controls render.
Settings controls persistence: Mutates representative settings controls and verifies the persisted settings file reflects the UI changes.
Provider validation blocks prompt: Starts with vLLM selected and no base URL, then verifies the prompt stays blocked with settings guidance instead of submitting into an avoidable provider failure.
Provider failure recovery: Submits against a mock provider billing failure and verifies the provider failure is visible in the app instead of being swallowed.
Prompt keyboard and suggestions: Exercises visible prompt suggestions, send-button state, keyboard action proof, prompt history, and evolve review.
Feedback and issue reporting: Exercises header feedback mode and footer issue-report mode without sending external feedback.
History and settings navigation: Opens the history and settings surfaces and verifies the visible navigation controls return to the main prompt.
Onboarding existing repo: Connects an existing nix-darwin repo and verifies the app reaches the prompt screen.
Release DMG launch smoke: Launches the installed /Applications app on a real Mac and verifies a usable first screen.
Install Nix on clean machine: Runs the real Install Nix flow on a clean Mac and verifies Nix works afterward.

Matched files and known gaps

Auto-evolve basic package: Matched: apps/native/e2e-tauri/tests/data/add-fira-code.jsonl, apps/native/e2e-tauri/tests/data/add-font.jsonl, apps/native/e2e-tauri/tests/wdio/basic-prompts.spec.mjs, apps/native/e2e-tauri/tests/wdio/discard.spec.mjs, .... Known gap: Uses mocked model responses; it does not call live AI providers or apply/rebuild the generated config.
Discard and restore state: Matched: apps/native/e2e-tauri/tests/wdio/discard.spec.mjs, apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/src-tauri/src/store.rs, apps/native/src/components/widget/prompt-input.tsx, .... Known gap: Does not verify every rollback/history path or rebuild behavior after discard.
Manual evolve existing changes: Matched: apps/native/e2e-tauri/tests/data/add-fira-code.jsonl, apps/native/e2e-tauri/tests/data/add-font.jsonl, apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/e2e-tauri/tests/wdio/helpers/e2e-report.mjs, .... Known gap: Uses mocked model responses and does not run the final apply/rebuild step.
Settings and provider tabs: Matched: apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/e2e-tauri/tests/wdio/smoke.spec.mjs, apps/native/src/components/widget/settings-dialog.tsx. Known gap: Checks visibility of key settings controls; it does not mutate every field, verify real API keys, or call live providers.
Settings controls persistence: Matched: apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/e2e-tauri/tests/wdio/helpers/test-env.mjs, apps/native/e2e-tauri/tests/wdio/settings-controls.spec.mjs, apps/native/src-tauri/src/store.rs, .... Known gap: Uses local persisted settings; it does not validate live provider credentials or call external AI services.
Provider validation blocks prompt: Matched: apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/e2e-tauri/tests/wdio/helpers/test-env.mjs, apps/native/e2e-tauri/tests/wdio/provider-validation.spec.mjs, apps/native/src-tauri/src/store.rs, .... Known gap: Uses local vLLM validation and does not call an external provider.
Provider failure recovery: Matched: apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/e2e-tauri/tests/wdio/helpers/mock-vllm-server.mjs, apps/native/e2e-tauri/tests/wdio/helpers/test-env.mjs, apps/native/e2e-tauri/tests/wdio/provider-failure.spec.mjs, .... Known gap: Uses a deterministic mock provider failure; it does not exercise every provider's live failure shape.
Prompt keyboard and suggestions: Matched: apps/native/e2e-tauri/tests/data/add-fira-code.jsonl, apps/native/e2e-tauri/tests/data/add-font.jsonl, apps/native/e2e-tauri/tests/wdio/basic-prompts.spec.mjs, apps/native/e2e-tauri/tests/wdio/discard.spec.mjs, .... Known gap: Keeps model responses mocked and uses the existing reliable submit path after proving keyboard navigation in the prompt surface.
Feedback and issue reporting: Matched: apps/native/e2e-tauri/tests/wdio/feedback-report.spec.mjs, apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/src-tauri/src/feedback.rs, apps/native/src/components/widget/feedback-dialog.tsx, .... Known gap: Does not submit feedback to the backend or external services; it validates the user-facing collection flows. Uses DOM-click fallback for close/cancel controls because native Tauri WebDriver clicks are flaky on these dialog buttons.
History and settings navigation: Matched: apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/e2e-tauri/tests/wdio/history-navigation.spec.mjs, apps/native/src/components/widget/header.tsx, apps/native/src/components/widget/history/history-header.tsx, .... Known gap: Navigation coverage only; it does not restore historical commits or exercise every history item state. Uses DOM-click fallback for close/toggle controls because native Tauri WebDriver clicks are flaky on these small header buttons.
Onboarding existing repo: Matched: apps/native/e2e-tauri/tests/wdio/helpers/app-ui.mjs, apps/native/e2e-tauri/tests/wdio/helpers/e2e-report.mjs, apps/native/e2e-tauri/tests/wdio/helpers/mock-vllm-presets.mjs, apps/native/e2e-tauri/tests/wdio/helpers/mock-vllm-server.mjs, .... Known gap: Uses a prepared fixture repo; it does not cover every arbitrary user flake shape.
Release DMG launch smoke: Matched: tests/e2e/adapters/nixmac.sh, tests/e2e/fixtures/clean-machine.sh, tests/e2e/fixtures/nix-installed.sh, tests/e2e/lib/API.md, .... Known gap: Launch smoke only; it does not exercise Nix installation, settings, or evolve/apply flows.
Install Nix on clean machine: Matched: apps/native/src-tauri/src/nix.rs, tests/e2e/adapters/nixmac.sh, tests/e2e/fixtures/clean-machine.sh, tests/e2e/fixtures/nix-installed.sh, .... Known gap: Runs on one configured Mac runner and macOS version; it does not cover every hardware or OS variant.

Changed files sampled (12 of 95)

.github/actionlint.yaml
.github/workflows/e2e.yml
.oxlintrc.json
README.md
apps/native/e2e-tauri/.gitignore
apps/native/e2e-tauri/ADVERSARIAL_BEHAVIOR_QA.md
apps/native/e2e-tauri/ADVERSARIAL_VISUAL_QA.md
apps/native/e2e-tauri/DO_NOT_MERGE_E2E_GATE_TEST.md
apps/native/e2e-tauri/README.md
apps/native/e2e-tauri/process-compose.yaml
apps/native/e2e-tauri/report.schema.json
apps/native/e2e-tauri/scenarios/manifest.json

Scenario	Lane	Result	What it checks	Report	Proof
`auto_evolve_basic_package`	tauri-wdio	passed	Submits a mocked AI prompt and verifies nixmac reaches review with a generated Nix diff.	Report	Action-proof webview video
`discard_and_restore_state`	tauri-wdio	passed	Exercises discard confirm/cancel behavior from the evolve review screen.	Report	Action-proof webview video
`feedback_report_issue`	tauri-wdio	passed	Exercises header feedback mode and footer issue-report mode without sending external feedback.	Report	Action-proof webview video
`history_navigation`	tauri-wdio	passed	Opens the history and settings surfaces and verifies the visible navigation controls return to the main prompt.	Report	Screenshot
`install_nix_clean_machine`	full-mac	passed	Runs the real Install Nix flow on a clean Mac and verifies Nix works afterward.	Report	Full-screen recording
`manual_evolve_existing_changes`	tauri-wdio	passed	Runs sequential prompts and verifies the second change preserves the first generated diff.	Report	Action-proof webview video
`onboarding_existing_repo`	tauri-wdio	passed	Connects an existing nix-darwin repo and verifies the app reaches the prompt screen.	Report	Screenshot
`prompt_keyboard_and_suggestions`	tauri-wdio	passed	Exercises visible prompt suggestions, send-button state, keyboard action proof, prompt history, and evolve review.	Report	Action-proof webview video
`provider_failure_recovery`	tauri-wdio	passed	Submits against a mock provider billing failure and verifies the provider failure is visible in the app instead of being swallowed.	Report	Screenshot
`provider_validation_blocks_prompt`	tauri-wdio	passed	Starts with vLLM selected and no base URL, then verifies the prompt stays blocked with settings guidance instead of submitting into an avoidable provider failure.	Report	Screenshot
`release_dmg_app_translocation_smoke`	full-mac	passed	Launches the installed /Applications app on a real Mac and verifies a usable first screen.	Report	Full-screen recording
`settings_controls_persistence`	tauri-wdio	passed	Mutates representative settings controls and verifies the persisted settings file reflects the UI changes.	Report	Action-proof webview video
`settings_provider_change`	tauri-wdio	passed	Opens Settings and verifies the General, AI Models, API Keys, and Preferences tabs plus key controls render.	Report	Screenshot

Workflow run

Full E2E details

# Conflicts: # README.md # apps/native/src-tauri/src/completion_log.rs

# Conflicts: # apps/native/e2e-tauri/.gitignore # apps/native/e2e-tauri/README.md # apps/native/e2e-tauri/tests/data/add-font.jsonl # apps/native/e2e-tauri/tests/wdio/basic-prompts.spec.mjs # apps/native/e2e-tauri/tests/wdio/helpers/mock-vllm-presets.mjs # apps/native/e2e-tauri/tests/wdio/helpers/test-env.mjs

Scott McMaster (scottmcmaster) and others added 7 commits April 23, 2026 10:25

feat(tests): update mock VLLM preset to use new add-font.jsonl data file

54d581e

feat(tests): save/restore db in e2e tests

572e5c7

Co-authored-by: Copilot <copilot@github.com>

feat(docs): add section on mocking AI completion responses in E2E tests

bd72fd3

Co-authored-by: Copilot <copilot@github.com>

feat(tests): enhance error handling for JSON parsing and rename backu…

a9ada13

…p functions for clarity Co-authored-by: Copilot <copilot@github.com>

Merge remote-tracking branch 'origin/pr/43' into fkb/e2e-nixmac-testing

c94eb79

feat(e2e): add nixmac PR gate

7b12e3a

test(e2e): trigger gate validation PR

c1ddcb8

Farhan (fkb032) added the e2e:all Run the full nixmac E2E suite, including default WDIO and full-Mac flows label Apr 23, 2026

Farhan (fkb032) added 14 commits April 23, 2026 01:28

fix(e2e): run tauri gate on hosted macOS

c2fdf2d

fix(e2e): stabilize hosted failure reporting fixtures

353c0b7

fix(e2e): cover hosted summary mock calls

d1210cb

fix(e2e): add video proof to PR gate

1b9e300

fix(e2e): wait for hosted evolve diff

2224258

fix(e2e): stabilize hosted onboarding setup

0eab7b4

fix(e2e): clarify trigger reason and failure proof

17ba3f8

fix(e2e): capture readable proof artifacts

4b9316f

fix(e2e): harden proof capture path

769cc56

fix(e2e): publish working scenario reports

a43717f

Merge remote-tracking branch 'origin/main' into fkb/e2e-gate-test-pr

96ec65b

# Conflicts: # README.md # apps/native/src-tauri/src/completion_log.rs

Harden full-Mac E2E infra reporting

224f01d

Fix E2E report commit metadata

c24a73c

Serialize full-Mac E2E scenarios

25b2ffb

Farhan (fkb032) added e2e:all Run the full nixmac E2E suite, including default WDIO and full-Mac flows and removed e2e:all Run the full nixmac E2E suite, including default WDIO and full-Mac flows labels Apr 24, 2026

Farhan (fkb032) added 4 commits April 23, 2026 19:29

Make full-Mac video primary proof

e224867

Improve E2E failure report formatting

31a3d76

Make E2E gate scope PR-aware

ef7adce

Farhan (fkb032) added 3 commits April 23, 2026 21:56

Harden settings E2E text assertions

e85e72a

Stabilize onboarding E2E host seeding

24cd4b9

Frame E2E gate as adversarial PR focus

f8a12a0

Farhan (fkb032) added e2e:all Run the full nixmac E2E suite, including default WDIO and full-Mac flows and removed e2e:all Run the full nixmac E2E suite, including default WDIO and full-Mac flows labels Apr 24, 2026

Clarify all-label adversarial focus wording

b626172

Farhan (fkb032) mentioned this pull request Apr 24, 2026

DO NOT MERGE: Intentional E2E failure validation #47

Closed

Farhan (fkb032) added 21 commits April 24, 2026 09:44

test(e2e): retrigger gate validation

e55a93f

test(e2e): touch validation marker

8f67255

test(e2e): allow dispatch validation comments

e33410f

test(e2e): publish PR check report

91f2459

test(e2e): publish PR status context

e75b2f2

test(e2e): clarify proof recording labels

c708883

test(e2e): harden gate dispatch previews

b6c3727

test(e2e): expand PR gate coverage and proof recordings

d63b3df

ci(e2e): restore hosted WDIO app builds

d76faf9

test(e2e): stabilize feedback report scenario

07e039f

fix(e2e): repair hosted proof videos

c449ab2

fix(e2e): clean full-mac proof leftovers

794e7d8

fix(e2e): kill stale full-mac recorder windows

ad6cd9c

fix(e2e): launch full-mac recorder fresh

caeba50

fix(e2e): dismiss stale terminal automation prompt

e16292c

fix(e2e): dismiss recorder automation prompt

f8f909e

fix(e2e): click away terminal automation prompt

8961e20

fix(e2e): calibrate terminal prompt dismissal

a275330

test(e2e): harden visual gate against adversarial regressions

a5e7713

fix(e2e): publish rich PR gate summary

72f2c23

test(e2e): expand behavior gate coverage

b13f399

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DO NOT MERGE: E2E gate validation test#45

DO NOT MERGE: E2E gate validation test#45
Farhan (fkb032) wants to merge 50 commits intomainfrom
fkb/e2e-gate-test-pr

Farhan (fkb032) commented Apr 23, 2026

Uh oh!

github-actions Bot commented Apr 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Farhan (fkb032) commented Apr 23, 2026

DO NOT MERGE

Uh oh!

github-actions Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 PR Overview

Uh oh!

github-actions Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nixmac E2E

PR-specific Focus

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Apr 23, 2026 •

edited

Loading

github-actions Bot commented Apr 23, 2026 •

edited

Loading