Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 0 additions & 84 deletions .cirrus.yml

This file was deleted.

18 changes: 18 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,24 @@ jobs:
if: always()
run: kill $(cat /tmp/mock-pid) 2>/dev/null || true

macos-e2e:
name: macos e2e (L5)
runs-on: macos-latest
# Only on release tags or manual dispatch — these are slow and destructive.
if: startsWith(github.ref, 'refs/tags/v') || github.event_name == 'workflow_dispatch'
timeout-minutes: 45
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: "go.mod"

- name: Run macOS E2E (release tier)
run: make test-vm-release

cli-compat:
name: old-cli compat
runs-on: macos-latest
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ make build-release VERSION=0.25.0 # optimized + UPX
make test-unit # L1 (~15s) — pre-push hook
make test-integration # L2 (~75s) — real brew/git/npm in temp dirs
make test-e2e # L4 compiled binary
make test-vm-release # L5 VM (~20m) — before tagging
make test-vm-release # L5 destructive macOS (~20m) — before tagging
make test-destructive # L6 — actually installs
make test-coverage # coverage.out + coverage.html

Expand Down Expand Up @@ -60,7 +60,7 @@ internal/
ui/ # bubbletea Model pattern, lipgloss styling
updater/ # Auto-update: check GitHub → download → replace
test/{integration,e2e}/ # //go:build integration | e2e (+ vm, destructive, smoke)
testutil/ # shared helpers + Tart VM helpers
testutil/ # shared helpers + MacHost (destructive E2E on real macOS)
scripts/
install.sh # curl|bash installer
hooks/ # pre-commit, pre-push (install via `make install-hooks`)
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,14 @@ Tests are split across six tiers. Which one runs where:
| **L2 Integration** | Real `brew` / `git` / `npm` against temp dirs; real `httptest` servers | `make test-integration` (~75s) | CI on push/PR |
| **L3 Contract schema** | JSON schema validation against [openboot-contract](https://github.com/openbootdotdev/openboot-contract) | (runs in CI only) | CI on push/PR |
| **L4 E2E binary** | Compiled binary driven by scripts; `-tags=e2e` | `make test-e2e` | CI on release |
| **L5 E2E VM** | [Tart](https://github.com/cirruslabs/tart) macOS VMs (install Homebrew, run real flows) | `make test-vm-quick` (2 min) / `test-vm-release` (20 min) / `test-vm-full` (60 min) | Manual, before tagging a release |
| **L5 Destructive macOS** | Runs against a real macOS host (installs packages, modifies `~/.zshrc`, writes `defaults`) | `make test-vm-quick` / `test-vm-release` / `test-vm-full` — requires `CI=true` or `OPENBOOT_E2E_DESTRUCTIVE=1` | GH Actions `macos-latest` on release tags + manual dispatch |
| **L6 Destructive** | Actually installs real packages into a real system | `make test-destructive` / `test-smoke` | CI on release, plus manual `workflow_dispatch` |

Rules of thumb:

- **Local dev:** run nothing manually if hooks are installed. `make test-unit` on demand when you want a sanity check. Skip L2+ unless you're touching code that interacts with real brew/git/npm.
- **Before pushing:** `make test-unit` (the pre-push hook does this automatically).
- **Before tagging a release:** `make test-vm-release` locally (needs Tart).
- **Before tagging a release:** trigger the `macos-e2e` job via GitHub Actions (manual dispatch or tag push). To run locally on a throwaway macOS machine: `OPENBOOT_E2E_DESTRUCTIVE=1 make test-vm-release`.

## Git Hooks

Expand Down
24 changes: 14 additions & 10 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -44,31 +44,35 @@ test-all:
$(MAKE) test-coverage

# =============================================================================
# VM-based E2E tests (Tart VMs) — three levels
# Destructive macOS E2E tests — three levels
# =============================================================================
#
# These tests install real packages and modify ~/.zshrc / macOS defaults on
# the host they run on. They are intended for ephemeral macOS CI runners
# (GitHub Actions macos-latest) or a throwaway VM.
#
# On a developer machine `go test -tags="e2e,vm"` will skip unless you set
# OPENBOOT_E2E_DESTRUCTIVE=1 (see testutil/machost.go). Don't set that
# unless you mean it.

# L1: Quick validation (~2min) — run after code changes
# Runs TestVM_Infra only: boots a VM and checks SSH/arch/tools, no package installs
# L1: Quick sanity (~1min) — host/arch checks only, no package installs
test-vm-quick: build
go test -v -timeout 5m -tags="e2e,vm" -run "TestVM_Infra" ./test/e2e/...

# L2: Release validation (~20min) — run before tagging a release
# Core user journeys: dry-run safety, install + verify, diff/clean cycle,
# manual uninstall recovery, full setup, error messages
# L2: Release validation (~20min) — core user journeys
test-vm-release: build
go test -v -timeout 30m -tags="e2e,vm" \
-run "TestVM_Infra|TestVM_Journey_DryRun|TestVM_Journey_FirstTimeUser|TestVM_Journey_ManualUninstall|TestVM_Journey_DiffConsistency|TestVM_Journey_FullSetup|TestVM_Journey_ErrorMessages" \
-run "TestVM_Infra|TestVM_Journey_DryRunIsCompletelySafe|TestVM_Journey_FirstTimeUser|TestVM_Journey_FullSetupConfiguresEverything|TestE2E_DryRunMinimal|TestE2E_SnapshotCapture" \
./test/e2e/...

# L3: Full validation (~60min) — run for major releases or CI
# All 48 tests: journeys + edge cases + commands + interactive
# L3: Full validation (~60min) — everything under -tags="e2e,vm"
test-vm-full: build
go test -v -timeout 90m -tags="e2e,vm" ./test/e2e/...

# Aliases
test-vm: test-vm-release

# Single VM test by name (e.g. make test-vm-run TEST=TestVM_Journey_DryRun)
# Single test by name (e.g. make test-vm-run TEST=TestVM_Journey_DryRunIsCompletelySafe)
test-vm-run: build
go test -v -timeout 45m -tags="e2e,vm" -run $(TEST) ./test/e2e/...

Expand Down
2 changes: 1 addition & 1 deletion test/e2e/misc_e2e_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ func TestE2E_FullPreset_DryRun(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand Down
14 changes: 7 additions & 7 deletions test/e2e/openboot_e2e_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ func TestE2E_DryRunMinimal(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand All @@ -35,7 +35,7 @@ func TestE2E_DryRunDeveloper(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand All @@ -53,7 +53,7 @@ func TestE2E_SnapshotCapture(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand All @@ -71,7 +71,7 @@ func TestE2E_InvalidPreset(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand All @@ -91,7 +91,7 @@ func TestE2E_MissingGitConfig(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand All @@ -108,7 +108,7 @@ func TestE2E_SnapshotWithOutput(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand All @@ -127,7 +127,7 @@ func TestE2E_Diff_ThenClean_DryRun_SameSnapshot(t *testing.T) {
}

// Verify diff and clean produce consistent results from the same snapshot
vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand Down
4 changes: 2 additions & 2 deletions test/e2e/sync_shell_e2e_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ func TestE2E_Sync_Shell_CaptureShell(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
installOhMyZsh(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand Down Expand Up @@ -49,7 +49,7 @@ func TestE2E_Sync_Shell_NoPanic(t *testing.T) {
t.Skip("skipping VM test in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
installOhMyZsh(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand Down
2 changes: 1 addition & 1 deletion test/e2e/vm_edge_cases_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ func TestVM_Edge_ShellActuallyWorks(t *testing.T) {
t.Skip("skipping VM edge case in short mode")
}

vm := testutil.NewTartVM(t)
vm := testutil.NewMacHost(t)
vmInstallHomebrew(t, vm)
bin := vmCopyDevBinary(t, vm)

Expand Down
Loading
Loading