diff --git a/Docs/Architecture/01-overview.md b/Docs/Architecture/01-overview.md index abaa3cb..9999020 100644 --- a/Docs/Architecture/01-overview.md +++ b/Docs/Architecture/01-overview.md @@ -6,7 +6,7 @@ ## Purpose -`swift-mutation-testing` is a mutation testing CLI for Xcode + XCTest projects. It introduces controlled faults (mutants) into source code, runs the test suite for each one, and reports whether the tests detected the fault. The mutation score — the ratio of killed mutants to all testable mutants — measures the effectiveness of the test suite. +`swift-mutation-testing` is a mutation testing CLI for Swift projects (Xcode and SPM). It introduces controlled faults (mutants) into source code, runs the test suite for each one, and reports whether the tests detected the fault. The mutation score — the ratio of killed mutants to all testable mutants — measures the effectiveness of the test suite. The tool never modifies the original project. All mutations happen inside isolated sandbox copies in `$TMPDIR`. @@ -19,9 +19,9 @@ graph TD CLI["CLI\n(SwiftMutationTesting · CommandLineParser)"] CONFIG["Configuration\n(ConfigurationResolver · ProjectDetector)"] DISCOVERY["Discovery\n(DiscoveryPipeline · Operators · Schematization)"] - EXECUTION["Execution\n(MutantExecutor · BuildStage · TestExecutionStage)"] + EXECUTION["Execution\n(MutantExecutor · FallbackExecutor · IncompatibleMutantExecutor\nBuildStage · TestExecutionStage · TestResultResolver)"] REPORTING["Reporting\n(TextReporter · JsonReporter · HtmlReporter · SonarReporter)"] - INFRA["Infrastructure\n(ProcessLauncher · XCTestRunPlist · TestFilesHasher)"] + INFRA["Infrastructure\n(ProcessRunner · ProcessRequest · SPMProcessLauncher\nXCTestRunPlist · TestFilesHasher)"] CLI --> CONFIG CLI --> DISCOVERY @@ -37,9 +37,9 @@ graph TD | **CLI** | Argument parsing, subcommand routing, exit codes | | **Configuration** | Config file parsing, CLI merge, auto-detection of scheme and destination | | **Discovery** | Source file collection, AST parsing, mutant identification, schematization | -| **Execution** | Sandbox creation, build, simulator management, parallel test execution, result parsing, caching | +| **Execution** | Sandbox creation, build, simulator management, parallel test execution, result parsing (Xcode and SPM), fallback per-file builds, caching | | **Reporting** | Progress output, mutation report generation (text, JSON, HTML, Sonar) | -| **Infrastructure** | Process lifecycle management, xctestrun plist manipulation, test file hashing | +| **Infrastructure** | Process lifecycle management (`ProcessRunner`, `ProcessRequest`, `SPMProcessLauncher`), xctestrun plist manipulation, test file hashing | ## Entry Point @@ -68,17 +68,19 @@ flowchart LR subgraph Discovery FD[FileDiscoveryStage] --> PS[ParsingStage] PS --> MD[MutantDiscoveryStage] - MD --> SS[SchematizationStage] + MD --> MI[MutantIndexingStage] + MI --> SS[SchematizationStage] + MI --> IRS[IncompatibleRewritingStage] end subgraph Execution SF[SandboxFactory] --> BS[BuildStage] BS --> TES[TestExecutionStage] - BS -- build failed --> FBP[Per-file fallback] - TES --> RP[ResultParser] + BS -- build failed --> FBP[FallbackExecutor\nper-file rebuild] + TES --> TR[TestResultResolver] IME[IncompatibleMutantExecutor] end SS -- RunnerInput --> SF - SS -- incompatible mutants --> IME + IRS -- incompatible mutants --> IME ``` | Stage | Input | Output | @@ -86,18 +88,22 @@ flowchart LR | `FileDiscoveryStage` | `DiscoveryInput` | `[SourceFile]` | | `ParsingStage` | `[SourceFile]` | `[ParsedSource]` | | `MutantDiscoveryStage` | `[ParsedSource]` | `[MutationPoint]` | -| `SchematizationStage` | `[MutationPoint]`, `[ParsedSource]` | `RunnerInput` | +| `MutantIndexingStage` | `[MutationPoint]`, `[ParsedSource]` | `[IndexedMutationPoint]` | +| `SchematizationStage` | `[IndexedMutationPoint]`, `[ParsedSource]` | `[SchematizedFile]`, `[MutantDescriptor]` | +| `IncompatibleRewritingStage` | `[IndexedMutationPoint]`, `[ParsedSource]` | `[MutantDescriptor]` | | `SandboxFactory` | project path + schematized files | `Sandbox` | | `BuildStage` | `Sandbox` | `BuildArtifact` | | `TestExecutionStage` | `BuildArtifact` + mutants | `[ExecutionResult]` | +| `FallbackExecutor` | `RunnerInput` + `SimulatorPool` | `[ExecutionResult]` | | `IncompatibleMutantExecutor` | incompatible mutants | `[ExecutionResult]` | +| `TestResultResolver` | `TestLaunchResult` + `ProjectType` | `TestRunOutcome` | ## Invariants | Invariant | Enforcement | |---|---| | Original project is never modified | All mutations happen inside `$TMPDIR/xmr-/` sandbox | -| `xcodebuild build-for-testing` runs exactly once | `BuildStage` builds once; `TestExecutionStage` uses `test-without-building` | +| Build runs exactly once for the normal path | `BuildStage` builds once (Xcode: `build-for-testing`, SPM: `swift build --build-tests`); `TestExecutionStage` uses `test-without-building` (Xcode) or `swift test --skip-build` (SPM) | | No mutant results are lost or duplicated | `MutationCounter` tracks total; `withThrowingTaskGroup` accounts for every task | | Mutant positions are accurate | UTF-8 offsets are preserved from AST through to final report | | A cancelled task never permanently holds a simulator slot | `withTaskCancellationHandler` in `SimulatorPool.acquire` releases the slot on cancel | diff --git a/Docs/Architecture/02-discovery.md b/Docs/Architecture/02-discovery.md index 9879ea8..55bbf34 100644 --- a/Docs/Architecture/02-discovery.md +++ b/Docs/Architecture/02-discovery.md @@ -6,15 +6,18 @@ ## Design -The discovery pipeline is a **linear chain of pure stages**. Each stage receives an immutable input, produces an immutable output, and has no side effects. `DiscoveryPipeline` is the entry point and orchestrates the four stages sequentially. +The discovery pipeline is a **linear chain of pure stages**. Each stage receives an immutable input, produces an immutable output, and has no side effects. `DiscoveryPipeline` is the entry point and orchestrates the six stages sequentially. ```mermaid flowchart TD IN[DiscoveryInput] --> FD[FileDiscoveryStage] FD --> PA[ParsingStage] PA --> MD[MutantDiscoveryStage] - MD --> SC[SchematizationStage] + MD --> MI[MutantIndexingStage] + MI --> SC[SchematizationStage] + MI --> IR[IncompatibleRewritingStage] SC --> OUT[RunnerInput] + IR --> OUT ``` ## Stages @@ -52,16 +55,38 @@ Applies mutation operators to each parsed source and collects mutation points. R Each operator walks the AST with its own visitor and emits a `MutationPoint` for every applicable node. Points are collected from all operators and all files, then returned as a flat list. -### SchematizationStage +### MutantIndexingStage -Transforms mutation points into the final `RunnerInput` consumed by the execution pipeline. +Assigns unique sequential IDs to each mutation point and classifies them as schematizable or incompatible. | | | |---|---| | Input | `[MutationPoint]`, `[ParsedSource]` | -| Output | `RunnerInput` — schematized files, incompatible mutants, support file content | +| Output | `[IndexedMutationPoint]` — mutation point + unique ID + schematizable flag | + +Each mutation point receives an ID in the format `swift-mutation-testing_`, where `` is a zero-based global counter. `TypeScopeVisitor` determines whether a mutation falls inside a function body (schematizable) or outside (incompatible). The indexed points are consumed by the next two stages. + +### SchematizationStage -For each file, the stage separates schematizable mutations (inside function bodies) from incompatible ones (outside function bodies). Schematizable mutations are embedded into the source via `SchemataGenerator`. Incompatible mutations are stored as full file rewrites via `MutationRewriter`. See [Schematization](05-schematization.md) for a detailed breakdown. +Embeds all schematizable mutations into the source files via `SchemataGenerator`, producing `SchematizedFile` values and `MutantDescriptor` values for the execution pipeline. + +| | | +|---|---| +| Input | `[IndexedMutationPoint]`, `[ParsedSource]` | +| Output | `[SchematizedFile]`, `[MutantDescriptor]` — schematized files and schematizable mutant descriptors | + +For each file, the stage processes only the schematizable indexed points. Mutations are embedded into the source via `SchemataGenerator`, which rewrites function bodies to contain `switch __swiftMutationTestingID` blocks. See [Schematization](05-schematization.md) for a detailed breakdown. + +### IncompatibleRewritingStage + +Produces full-file rewrites for mutants that cannot be schematized (mutations outside function bodies, such as stored property initializers or global-scope expressions). + +| | | +|---|---| +| Input | `[IndexedMutationPoint]`, `[ParsedSource]` | +| Output | `[MutantDescriptor]` — incompatible mutant descriptors with pre-computed `mutatedSourceContent` | + +Each incompatible mutation point is applied to the source via `MutationRewriter`, producing a complete replacement source file stored in `MutantDescriptor.mutatedSourceContent`. These mutants are executed later by `IncompatibleMutantExecutor`, each requiring a separate build + test cycle. ## Mutation Operators @@ -87,11 +112,12 @@ Mutations can be suppressed on a per-scope basis using the inline annotation `// ``` DiscoveryInput -├── projectPath — Xcode project root +├── projectPath — project root (Xcode or SPM) +├── projectType — ProjectType (.xcode or .spm) ├── sourcesPath — root for Swift file discovery ├── excludePatterns — glob patterns to skip ├── operators — list of active operator identifiers -└── scheme, destination, timeout, concurrency, noCache +└── timeout, concurrency, noCache SourceFile ├── path — absolute path to the .swift file @@ -108,11 +134,30 @@ MutationPoint ├── originalText — token(s) before mutation ├── mutatedText — token(s) after mutation ├── operatorIdentifier -└── replacement — ReplacementKind enum +└── replacementKind — ReplacementKind enum + +IndexedMutationPoint +├── point — MutationPoint +├── id — unique ID (swift-mutation-testing_) +└── isSchematizable — whether the mutation is inside a function body + +MutantDescriptor +├── id — unique ID +├── filePath — absolute source file path +├── line, column — 1-based position +├── utf8Offset — byte offset +├── originalText — token(s) before mutation +├── mutatedText — token(s) after mutation +├── operatorIdentifier +├── replacementKind — ReplacementKind enum +├── description — human-readable mutation description +├── isSchematizable — schematizable or incompatible +└── mutatedSourceContent — pre-computed full source (incompatible only) RunnerInput ├── projectPath -├── scheme, destination, timeout, concurrency, noCache +├── projectType — ProjectType (.xcode or .spm) +├── timeout, concurrency, noCache ├── schematizedFiles — [SchematizedFile] (one per modified source file) ├── supportFileContent — __swiftMutationTestingID global declaration └── mutants — [MutantDescriptor] (all mutants, schematizable and incompatible) diff --git a/Docs/Architecture/03-execution.md b/Docs/Architecture/03-execution.md index 5d4094f..f2daed2 100644 --- a/Docs/Architecture/03-execution.md +++ b/Docs/Architecture/03-execution.md @@ -6,15 +6,16 @@ ## Design -`MutantExecutor` is the entry point for the execution pipeline. It separates mutants into two populations — schematizable and incompatible — and routes each through the appropriate path. +`MutantExecutor` is the entry point for the execution pipeline. It separates mutants into two populations — schematizable and incompatible — and routes each through the appropriate path. The executor supports both Xcode (`xcodebuild`) and SPM (`swift test`) project types via `ProjectType`. ```mermaid flowchart TD IN[RunnerInput] --> SF[SandboxFactory\ncreate sandbox] SF --> BS[BuildStage\nbuild-for-testing] BS -- success --> TES[TestExecutionStage\nparallel test-without-building] - BS -- compilationFailed --> FBP[Per-file fallback\none build per schematized file] - TES --> CACHE[CacheStore] + BS -- compilationFailed --> FBP[FallbackExecutor\none build per schematized file] + TES --> TR[TestResultResolver] + TR --> CACHE[CacheStore] FBP --> CACHE IN -- incompatible mutants --> IME[IncompatibleMutantExecutor\none full build+test per mutant] IME --> CACHE @@ -24,7 +25,12 @@ flowchart TD ## SandboxFactory -Creates an isolated copy of the Xcode project in `$TMPDIR/xmr-/` before every build. +Creates an isolated copy of the project in `$TMPDIR/xmr-/` before every build. Supports both Xcode and SPM projects. + +**Factory methods:** +- `create(projectPath:schematizedFiles:supportFileContent:)` — full sandbox with schematized files and support file injection (normal path) +- `createClean(projectPath:)` — clean sandbox without mutations (used by `IncompatibleMutantExecutor` for SPM shared sandbox) +- `create(projectPath:mutatedFilePath:mutatedContent:)` — sandbox with a single mutated file (incompatible mutants, Xcode path) **Copy strategy:** - Skips `.build`, `DerivedData`, and directories prefixed with `.xmr-` @@ -39,23 +45,29 @@ The original project is never touched. Cleanup removes the entire `xmr-*` direct ## BuildStage -Runs `xcodebuild build-for-testing` once for all schematizable mutants. +Runs a single build for all schematizable mutants. + +**Xcode path:** `xcodebuild build-for-testing` → find `.xctestrun` → parse plist → `BuildArtifact` + +**SPM path:** `swift build --build-tests` → `BuildArtifact` (no `.xctestrun` needed) ```mermaid flowchart TD - A[build-for-testing\n-scheme -destination\n-derivedDataPath] --> B{Exit code?} - B -- 0 --> C[Find .xctestrun in\nBuild/Products] - C --> D[Parse XCTestRunPlist] - D --> E[BuildArtifact] - B -- non-zero --> F[throw BuildError.compilationFailed] + A{ProjectType?} + A -- .xcode --> B[xcodebuild build-for-testing\n-scheme -destination\n-derivedDataPath] + A -- .spm --> C[swift build --build-tests] + B --> D{Exit code?} + C --> D + D -- 0 --> E[BuildArtifact] + D -- non-zero --> F[throw BuildError.compilationFailed] ``` | | | |---|---| -| Input | `Sandbox`, scheme, destination, timeout | -| Output | `BuildArtifact` — derived data path + `.xctestrun` URL + parsed plist | +| Input | `Sandbox`, project type, timeout | +| Output | `BuildArtifact` — derived data path + `.xctestrun` URL (Xcode) or sandbox path (SPM) | -`MutantExecutor` catches `BuildError.compilationFailed` and falls back to the per-file build path rather than aborting. Any other thrown error propagates up. +`BuildError` conforms to `LocalizedError`, providing structured error descriptions. `MutantExecutor` catches `BuildError.compilationFailed` and delegates to `FallbackExecutor` for per-file rebuilds rather than aborting. Any other thrown error propagates up. ## SimulatorPool @@ -68,6 +80,8 @@ flowchart TD `acquire()` returns an available `SimulatorSlot` or suspends the caller until one is released. A `withTaskCancellationHandler` wraps the suspension — if the owning task is cancelled, the slot is released immediately to avoid a permanent deadlock. +`SimulatorError` conforms to `LocalizedError` and covers three failure modes: `deviceNotFound(destination:)`, `bootTimeout(udid:)`, and `cloneFailed(udid:)`. Each provides a structured `errorDescription` for diagnostics. + ## TestExecutionStage Runs `xcodebuild test-without-building` for each mutant in parallel via `withThrowingTaskGroup`. @@ -93,40 +107,72 @@ flowchart TD **Dynamic concurrency:** the task group seeds N tasks initially, then adds one new task for each completed task, maintaining exactly N active tasks at all times. +## FallbackExecutor + +When the baseline build for all schematized files fails (`BuildError.compilationFailed`), `MutantExecutor` delegates to `FallbackExecutor`. This executor rebuilds one schematized file at a time — if one file causes a compilation error, the others can still be tested. + +```mermaid +flowchart TD + FILES["[SchematizedFile]"] --> LOOP["For each file"] + LOOP --> SF[SandboxFactory\nsingle-file sandbox] + SF --> BS[BuildStage] + BS -- success --> TES[TestExecutionStage\ntest mutants in this file] + BS -- failed --> UNVIABLE[Mark all mutants in file as .unviable] +``` + +For each schematized file, `FallbackExecutor` creates a sandbox containing only that file's schematization, builds it, and runs the test suite against its mutants. Files whose builds fail have all their mutants marked as `.unviable`. Results are cached via `CacheStore`. + ## IncompatibleMutantExecutor Handles mutants that cannot be schematized — mutations outside function bodies (e.g. in stored property initializers or global scope). Each incompatible mutant requires a full build + test cycle. ```mermaid flowchart TD - MUTANT[MutantDescriptor\nisSchematizable = false] --> SF2[SandboxFactory\ncreate mutant-only sandbox] + MUTANT[MutantDescriptor\nisSchematizable = false] --> PT{ProjectType?} + PT -- .xcode --> SF2[SandboxFactory\ncreate mutant-only sandbox] SF2 --> BS2[BuildStage\nbuild-for-testing] BS2 -- success --> TE2[xcodebuild test-without-building] BS2 -- compilationFailed --> UNVIABLE[.unviable] - TE2 --> RP2[ResultParser] + TE2 --> RP2[TestResultResolver] + PT -- .spm --> SHARED[Shared sandbox\nwrite mutated file → swift test] + SHARED --> SPM[SPMResultParser] ``` -Incompatible mutants run sequentially. Each creates its own sandbox via `SandboxFactory.create(projectPath:mutatedFilePath:mutatedContent:)`, which applies the single mutation directly without schematization. +**Xcode path:** Each incompatible mutant creates its own sandbox via `SandboxFactory.create(projectPath:mutatedFilePath:mutatedContent:)`, which applies the single mutation directly without schematization. Runs sequentially, each with a full build + test cycle. + +**SPM path:** Uses a shared sandbox created via `SandboxFactory.createClean(projectPath:)`. For each mutant, writes the mutated source content directly to the sandbox, runs `swift test`, and restores the original file. This avoids creating a new sandbox per mutant. -## ResultParser +## TestResultResolver -Determines the `ExecutionStatus` of a completed test run. +`TestResultResolver` determines the `TestRunOutcome` of a completed test run. It delegates to the appropriate parser based on project type. -| Condition | Status | +```mermaid +flowchart TD + TLR[TestLaunchResult] --> TR[TestResultResolver] + TR -- .xcode --> RP[ResultParser\nxcresulttool + output parsing] + TR -- .spm --> SP[SPMResultParser\noutput-only parsing] + RP --> OUT[TestRunOutcome] + SP --> OUT +``` + +**Xcode path (`ResultParser`):** Inspects stdout/stderr for XCTest and Swift Testing failure patterns, then parses the `.xcresult` bundle via `xcresulttool` for detailed failure information. The `.xcresult` bundle is deleted after parsing. + +**SPM path (`SPMResultParser`):** Parses exit code and stdout/stderr output only (no `.xcresult` bundles). Uses `TestOutputParser` to detect failure patterns. + +| Condition | Outcome | |---|---| | Exit code `-1` (killed by timeout) | `.timedOut` | -| Exit code `0` + no test failures detected | `.survived` | -| Exit code non-zero + test failure patterns in output | `.killed(let reason)` | -| Exit code non-zero + no parseable failure | `.killed(.other)` | - -`ResultParser` first inspects stdout/stderr for XCTest and Swift Testing failure patterns, then parses the `.xcresult` bundle via `xcresulttool` for detailed failure information. The `.xcresult` bundle is deleted after parsing. +| Exit code `0` | `.testsSucceeded` (survived) | +| Exit code non-zero + test failure pattern | `.testsFailed(failingTest:)` (killed) | +| Exit code non-zero + empty output | `.crashed` | +| Exit code non-zero + no parseable failure | `.unviable` | **Failure patterns detected:** | Framework | Pattern | |---|---| | XCTest | `Test Case '-[…]' failed` | -| Swift Testing | `Test "…" failed` | +| Swift Testing | `Test "…" failed`, `Issue recorded` | ## CacheStore @@ -176,7 +222,8 @@ score = killed / (killed + survived + timedOut + noCoverage) × 100 | `MutationCounter` | `actor` — tracks the current progress index | | `ConsoleProgressReporter` | `actor` — serialises output to stdout | | `TestExecutionStage` | `withThrowingTaskGroup` — N tasks, dynamically refilled | -| `ProcessLauncher` | `withTaskCancellationHandler` + `withCheckedThrowingContinuation` — kills process on cancel | +| `ProcessRunner` | `withTaskCancellationHandler` + `withCheckedThrowingContinuation` — kills process on cancel | +| `SPMProcessLauncher` | `ProcessLaunching` conformance backed by `ProcessRunner`; `killEscapedChildren` cleans up orphaned child processes via `sysctl` `KERN_PROCARGS2` inspection | | All data types | `Sendable` value types — safe to cross actor boundaries | --- diff --git a/Docs/Architecture/04-configuration.md b/Docs/Architecture/04-configuration.md index 23c7eac..3395f2c 100644 --- a/Docs/Architecture/04-configuration.md +++ b/Docs/Architecture/04-configuration.md @@ -8,7 +8,7 @@ `swift-mutation-testing` reads `.swift-mutation-testing.yml` from the project root. `ConfigurationFileParser` looks for the file at `/.swift-mutation-testing.yml`. If the file is absent, all values default to their built-in defaults and required options must be supplied via CLI. -`swift-mutation-testing init` generates a starter configuration file by running `ProjectDetector` to auto-detect the scheme, test target, and destination. +`swift-mutation-testing init` generates a starter configuration file by running `ProjectDetector` to auto-detect the project type, scheme, test targets, destination, and testing framework. ```yaml scheme: MyApp @@ -52,24 +52,51 @@ operators: ## Configuration Model +`RunnerConfiguration` is organized into three nested option groups: + ``` RunnerConfiguration -├── projectPath — absolute path to the Xcode project root -├── scheme — Xcode scheme (required) -├── destination — xcodebuild destination specifier (required) -├── testTarget — optional -only-testing filter -├── timeout — per-mutant test timeout in seconds (default: 60) -├── concurrency — parallel workers (default: ProcessInfo.activeProcessorCount - 1) -├── noCache — disable result caching (default: false) -├── output — path for JSON report (optional) -├── htmlOutput — path for HTML report (optional) -├── sonarOutput — path for Sonar report (optional) -├── sourcesPath — root directory for source file discovery (default: projectPath) -├── excludePatterns — glob patterns for files to exclude -├── operators — active mutation operator identifiers -└── quiet — suppress progress output (default: false) +├── projectPath — absolute path to the project root +├── build: BuildOptions +│ ├── projectType — ProjectType (.xcode(scheme:destination:) or .spm) +│ ├── testTarget — optional test target filter +│ ├── timeout — per-mutant test timeout (Xcode default: 120s, SPM default: 30s) +│ ├── concurrency — parallel workers (default: ProcessInfo.processorCount - 1) +│ ├── noCache — disable result caching (default: false) +│ └── testingFramework — TestingFramework (.xctest or .swiftTesting, default: .swiftTesting) +├── reporting: ReportingOptions +│ ├── output — path for JSON report (optional) +│ ├── htmlOutput — path for HTML report (optional) +│ ├── sonarOutput — path for Sonar report (optional) +│ └── quiet — suppress progress output (default: false) +└── filter: FilterOptions + ├── sourcesPath — root directory for source file discovery (default: projectPath) + ├── excludePatterns — glob patterns for files to exclude + └── operators — active mutation operator identifiers +``` + +### ProjectType + +```swift +enum ProjectType: Sendable, Equatable { + case xcode(scheme: String, destination: String) + case spm +} +``` + +Xcode projects require a scheme and destination. SPM projects are detected automatically when a `Package.swift` exists and no `.xcodeproj` or `.xcworkspace` is found. + +### TestingFramework + +```swift +enum TestingFramework: String, Sendable { + case xctest + case swiftTesting = "swift-testing" +} ``` +`ProjectDetector` automatically detects the testing framework by scanning test target source files for `import Testing` (Swift Testing) or `import XCTest` patterns. The detected framework influences test output parsing. + ## CLI Arguments All options correspond directly to `RunnerConfiguration` fields. CLI values override file values. @@ -108,23 +135,36 @@ flowchart TD R --> CFG[RunnerConfiguration] ``` -Required fields (`scheme`, `destination`) throw `UsageError` if absent in both sources. Optional fields fall back to their built-in defaults when absent in both. +For Xcode projects, `scheme` and `destination` are required and throw `UsageError` if absent in both sources. SPM projects require neither — `ProjectType.spm` is used automatically. Optional fields fall back to their built-in defaults when absent in both. ## Project Detection -`ProjectDetector` runs `xcodebuild -list` against the project to discover available schemes and test targets. `swift-mutation-testing init` uses it to pre-populate the generated configuration file. +`ProjectDetector` auto-detects the project type, scheme, test targets, destination, and testing framework. ```mermaid flowchart TD - A[ProjectDetector.detect] --> B[xcodebuild -list -json] - B --> C{Parsed?} - C -- yes --> D[DetectedProject\nscheme · testTarget · destination] - C -- no --> E[DetectedProject with nil fields] - D --> F[ConfigurationFileWriter\nwrites .swift-mutation-testing.yml] - E --> F + A[ProjectDetector.detect] --> B{.xcworkspace or\n.xcodeproj found?} + B -- yes --> C[xcodebuild -list -json] + C --> D[DetectedProject.xcode\nscheme · testTarget · destination] + B -- no --> E{Package.swift found?} + E -- yes --> F[swift package dump-package] + F --> G[DetectedProject.spm\ntestTargets] + E -- no --> H[DetectedProject with nil fields] + D --> I[detectDestination\niOS/tvOS/watchOS/visionOS/macOS] + I --> J[detectTestingFramework\nXCTest or Swift Testing] + G --> J + J --> K[ConfigurationFileWriter\nwrites .swift-mutation-testing.yml] + H --> K ``` -`DetectedProject` carries the best-guess scheme (first scheme found), first test target, and the default destination inferred from the project type. Fields are `nil` when detection fails, producing a template file with placeholder comments. +**Detection steps:** +1. `findContainer` — looks for `.xcworkspace` or `.xcodeproj` in the project directory +2. If found: `listProject` queries `xcodebuild -list` for schemes and test targets → `DetectedProject.xcode` +3. If not found: `listSPMTestTargets` queries `swift package dump-package` for test targets → `DetectedProject.spm` +4. `detectDestination` — scans project settings for platform SDKs (iOS, tvOS, watchOS, visionOS, macOS) +5. `detectTestingFramework` — scans test target source files for `import Testing` vs `import XCTest` to determine `TestingFramework` + +`DetectedProject` carries the best-guess scheme (first scheme found), test targets, destination, and testing framework. Fields are `nil` when detection fails, producing a template file with placeholder comments. ## Mutation Operators Reference diff --git a/Docs/Architecture/05-schematization.md b/Docs/Architecture/05-schematization.md index 624590a..094fdee 100644 --- a/Docs/Architecture/05-schematization.md +++ b/Docs/Architecture/05-schematization.md @@ -49,9 +49,11 @@ Multiple scopes within the same file are processed in reverse order by `bodyStar For **incompatible** mutants, `MutationRewriter` applies the single mutation directly to the source file's raw text using UTF-8 byte offsets, producing a complete replacement source file stored in `MutantDescriptor.mutatedSourceContent`. +Both `MutationRewriter` and `SchemataGenerator` use force-unwrapped UTF-8 conversions (`data(using: .utf8)!`, `String(data:encoding: .utf8)!`) because Swift source code is guaranteed to be valid UTF-8. This avoids unreachable error-handling paths. + ## TypeScopeVisitor -`TypeScopeVisitor` walks the SwiftSyntax AST and records every `FunctionBodyScope` — the UTF-8 byte range of each function body's `{`, its statements, and its closing `}`. +`TypeScopeVisitor` walks the SwiftSyntax AST and records every `FunctionBodyScope` — the UTF-8 byte range of each function body's `{`, its statements, and its closing `}`. It visits function declarations, initializers, deinitializers, and property/subscript accessors. ``` FunctionBodyScope diff --git a/Docs/Architecture/README.md b/Docs/Architecture/README.md index b80dec5..2bf446f 100644 --- a/Docs/Architecture/README.md +++ b/Docs/Architecture/README.md @@ -1,6 +1,6 @@ # Architecture Documentation -`swift-mutation-testing` is a CLI for mutation testing of Xcode + XCTest projects. It covers the full cycle: discovery (source file collection, AST parsing, mutant identification, schematization) followed by execution (sandbox creation, build, parallel test execution, result reporting). +`swift-mutation-testing` is a CLI for mutation testing of Swift projects (Xcode and SPM). It covers the full cycle: discovery (source file collection, AST parsing, mutant identification, indexing, schematization, incompatible rewriting) followed by execution (sandbox creation, build, parallel test execution, result reporting). ## Documents diff --git a/Docs/Assets/social-preview.png b/Docs/Assets/social-preview.png index 88c220d..7363aa4 100644 Binary files a/Docs/Assets/social-preview.png and b/Docs/Assets/social-preview.png differ diff --git a/Docs/Assets/social-preview.svg b/Docs/Assets/social-preview.svg index 678252a..b13404e 100644 --- a/Docs/Assets/social-preview.svg +++ b/Docs/Assets/social-preview.svg @@ -17,7 +17,8 @@ Swift Mutation Testing - Find untested behaviour in Swift codebases + Measure and improve test effectiveness in + Swift codebases using mutation testing discover mutants using operator visitors diff --git a/Docs/CodeBase/01-entry-point.md b/Docs/CodeBase/01-entry-point.md index 77c2715..9c5a400 100644 --- a/Docs/CodeBase/01-entry-point.md +++ b/Docs/CodeBase/01-entry-point.md @@ -10,19 +10,21 @@ @main struct SwiftMutationTesting { static func main() async - static func run(args: [String], launcher: any ProcessLaunching = ProcessLauncher()) async -> ExitCode - private static func execute(args: [String], launcher: any ProcessLaunching) async throws -> ExitCode - private static func discover(configuration: RunnerConfiguration) async throws -> RunnerInput + static func run(args: [String], launcher: (any ProcessLaunching)? = nil) async -> ExitCode + private static func execute(args: [String], launcher: (any ProcessLaunching)?) async throws -> ExitCode + private static func discover(configuration: RunnerConfiguration) async throws -> (RunnerInput, TimeInterval) static func writeReports(_ summary: RunnerSummary, configuration: RunnerConfiguration) + static func defaultLauncher(for projectType: ProjectType) -> any ProcessLaunching } ``` The program entry point. `main()` drops `CommandLine.arguments[0]` (the executable name) and delegates to `run(args:launcher:)`. -`run` catches three error categories before returning an exit code: +`run` catches two error categories before returning an exit code: - `UsageError` — prints `message` to stderr -- `SimulatorError.deviceNotFound` — prints destination string to stderr -- Any other `Error` — prints `localizedDescription` to stderr +- Any other `Error` — prints `localizedDescription` to stderr (errors conforming to `LocalizedError`, such as `SimulatorError` and `BuildError`, provide structured descriptions) + +`defaultLauncher(for:)` returns the appropriate process launcher based on project type: `XcodeProcessLauncher` for `.xcode`, `SPMProcessLauncher` for `.spm`. Used when no launcher is injected via the `launcher` parameter. `execute` is the primary execution path: @@ -81,7 +83,7 @@ struct UsageError: Error, Sendable { } ``` -Thrown by `CommandLineParser` for unknown flags and by `ConfigurationResolver` when required fields (`scheme`, `destination`) are absent in both CLI and file values. +Thrown by `CommandLineParser` for unknown flags and by `ConfigurationResolver` when required fields are absent in both CLI and file values (e.g. `scheme` and `destination` for Xcode projects). | Field | Type | Description | |---|---|---| diff --git a/Docs/CodeBase/02-configuration.md b/Docs/CodeBase/02-configuration.md index 30aed68..f8c03d5 100644 --- a/Docs/CodeBase/02-configuration.md +++ b/Docs/CodeBase/02-configuration.md @@ -71,31 +71,71 @@ struct ParsedArguments: Sendable { ```swift struct RunnerConfiguration: Sendable { let projectPath: String - let scheme: String - let destination: String - let testTarget: String? - let timeout: Double - let concurrency: Int - let noCache: Bool - let output: String? - let htmlOutput: String? - let sonarOutput: String? - let sourcesPath: String? - let excludePatterns: [String] - let operators: [String] - let quiet: Bool - - static let defaultTimeout: Double - static let defaultConcurrency: Int + let build: BuildOptions + let reporting: ReportingOptions + let filter: FilterOptions + + static let defaultXcodeTimeout: Double // 120.0 + static let defaultSPMTimeout: Double // 30.0 + static let defaultConcurrency: Int // max(1, processorCount - 1) + + struct BuildOptions: Sendable { + var projectType: ProjectType + var testTarget: String? + var timeout: Double + var concurrency: Int + var noCache: Bool + var testingFramework: TestingFramework // default: .swiftTesting + } + + struct ReportingOptions: Sendable { + var output: String? + var htmlOutput: String? + var sonarOutput: String? + var quiet: Bool + } + + struct FilterOptions: Sendable { + var sourcesPath: String? + var excludePatterns: [String] + var operators: [String] + } } ``` -Fully resolved configuration passed to both pipelines. All fields are immutable after construction. +Fully resolved configuration passed to both pipelines. Organized into three nested option groups: build, reporting, and filter. | Constant | Value | |---|---| -| `defaultTimeout` | `60.0` | -| `defaultConcurrency` | `max(1, ProcessInfo.activeProcessorCount - 1)` | +| `defaultXcodeTimeout` | `120.0` | +| `defaultSPMTimeout` | `30.0` | +| `defaultConcurrency` | `max(1, ProcessInfo.processorCount - 1)` | + +--- + +## Configuration/ProjectType.swift + +```swift +enum ProjectType: Sendable, Equatable { + case xcode(scheme: String, destination: String) + case spm +} +``` + +Xcode projects carry a scheme and destination. SPM projects require neither — `swift build` and `swift test` use the `Package.swift` manifest directly. + +--- + +## Configuration/TestingFramework.swift + +```swift +enum TestingFramework: String, Sendable { + case xctest + case swiftTesting = "swift-testing" +} +``` + +Detected automatically by `ProjectDetector` via source file scanning. Influences test output parsing patterns. --- @@ -109,7 +149,7 @@ struct ConfigurationResolver: Sendable { Merges `ParsedArguments` (CLI, higher priority) with `[String: String]` from the YAML parser (lower priority). CLI values always win. -Throws `UsageError` if `scheme` or `destination` is absent in both sources. +For Xcode projects, throws `UsageError` if `scheme` or `destination` is absent in both sources. SPM projects are auto-detected when a `Package.swift` exists and no `.xcodeproj`/`.xcworkspace` is found. **Operator resolution** (`resolveOperators`): @@ -156,23 +196,34 @@ Generates YAML content using `DetectedProject` values where available, falling b struct ProjectDetector: Sendable { init(launcher: any ProcessLaunching) func detect(at projectPath: String) async -> DetectedProject + private func findContainer(in: String) -> (flag: String, path: String)? + private func listProject(container:workingDirectory:) async -> (schemes: [String], projectName: String?, testTarget: String?) + private func listSPMTestTargets(in: String) async -> [String] + private func detectDestination(in: String) async -> String + private func detectTestingFramework(at:testTarget:) -> TestingFramework } ``` -Runs `xcodebuild -list -json` to discover schemes and test targets. +Auto-detects the project type, scheme, test targets, destination, and testing framework. ```mermaid flowchart TD - A[detect at projectPath] --> B{workspace or project?} - B -- workspace --> C[xcodebuild -list -json -workspace] - B -- project --> D[xcodebuild -list -json -project] - C & D --> E{parse JSON?} - E -- yes --> F[pick first scheme\npick first test target\nresolveDestination] - E -- no --> G[DetectedProject.empty] - F --> H[DetectedProject] + A[detect at projectPath] --> B{.xcworkspace or\n.xcodeproj found?} + B -- yes --> C[xcodebuild -list -json] + C --> D[DetectedProject.xcode\nscheme · testTarget · destination] + B -- no --> E{Package.swift found?} + E -- yes --> F[swift package dump-package] + F --> G[DetectedProject.spm\ntestTargets] + E -- no --> H[DetectedProject with nil fields] + D --> I[detectDestination\niOS/tvOS/watchOS/visionOS/macOS] + I --> J[detectTestingFramework\nXCTest or Swift Testing] + G --> J + J --> K[DetectedProject] ``` -`resolveDestination` queries `xcrun simctl list devices --json` and picks the first booted or available simulator for the detected platform. Falls back to hardcoded default destinations if detection fails. +`detectDestination` queries `xcrun simctl list devices --json` and picks the first booted or available simulator for the detected platform. Falls back to hardcoded default destinations if detection fails. + +`detectTestingFramework` scans test target source files for `import Testing` (Swift Testing) or `import XCTest` patterns to determine the testing framework in use. --- @@ -180,22 +231,24 @@ flowchart TD ```swift struct DetectedProject: Sendable { - let scheme: String? - let allSchemes: [String] + let kind: Kind let testTarget: String? - let destination: String? + let testingFramework: TestingFramework - static let empty: DetectedProject + enum Kind: Sendable { + case xcode(scheme: String?, allSchemes: [String], destination: String) + case spm(testTargets: [String]) + } } ``` | Field | Description | |---|---| -| `scheme` | First scheme found by `xcodebuild -list`, or `nil` | -| `allSchemes` | All schemes found | +| `kind` | `.xcode` with scheme, allSchemes, destination; or `.spm` with testTargets | | `testTarget` | First test target found, or `nil` | -| `destination` | Best-guess destination string, or `nil` | -| `empty` | Static instance with all fields `nil` / empty | +| `testingFramework` | Detected framework (`.xctest` or `.swiftTesting`) | + +Computed properties `scheme`, `allSchemes`, and `destination` extract values from `.xcode` kind for convenience. --- diff --git a/Docs/CodeBase/03-discovery-pipeline.md b/Docs/CodeBase/03-discovery-pipeline.md index d0a4822..8c5c0ef 100644 --- a/Docs/CodeBase/03-discovery-pipeline.md +++ b/Docs/CodeBase/03-discovery-pipeline.md @@ -13,15 +13,18 @@ struct DiscoveryPipeline: Sendable { } ``` -Entry point for the discovery phase. Runs four stages sequentially and assembles the `RunnerInput` for the execution pipeline. +Entry point for the discovery phase. Runs six stages sequentially and assembles the `RunnerInput` for the execution pipeline. ```mermaid flowchart TD IN[DiscoveryInput] --> FD[FileDiscoveryStage] FD --> PA[ParsingStage] PA --> MD[MutantDiscoveryStage\nwith resolved operators] - MD --> SC[SchematizationStage] + MD --> MI[MutantIndexingStage] + MI --> SC[SchematizationStage] + MI --> IR[IncompatibleRewritingStage] SC --> OUT[RunnerInput] + IR --> OUT ``` `allOperatorNames` is the ordered list of all registered operator identifiers. `ConfigurationFileWriter` uses it to populate the operators section of the generated YAML. @@ -47,8 +50,7 @@ When `input.operators` is empty, all seven operators are active. Otherwise only ```swift struct DiscoveryInput: Sendable { let projectPath: String - let scheme: String - let destination: String + let projectType: ProjectType let timeout: Double let concurrency: Int let noCache: Bool @@ -60,9 +62,8 @@ struct DiscoveryInput: Sendable { | Field | Description | |---|---| -| `projectPath` | Absolute path to the Xcode project root | -| `scheme` | Xcode scheme used for the build | -| `destination` | `xcodebuild` destination specifier | +| `projectPath` | Absolute path to the project root (Xcode or SPM) | +| `projectType` | `ProjectType` — `.xcode(scheme:destination:)` or `.spm` | | `timeout` | Per-mutant test timeout in seconds | | `concurrency` | Number of parallel test workers | | `noCache` | Disable result cache | @@ -137,46 +138,67 @@ Results are sorted by `filePath` then `utf8Offset`. --- +## Discovery/Pipeline/MutantIndexingStage.swift + +```swift +struct MutantIndexingStage: Sendable { + func run(mutationPoints: [MutationPoint], sources: [ParsedSource]) -> [IndexedMutationPoint] +} +``` + +Assigns a globally unique sequential index to each mutation point (sorted by file path, then UTF-8 offset) and classifies them as schematizable or incompatible using `TypeScopeVisitor`. The index becomes the mutant ID suffix in `"swift-mutation-testing_"`. + +--- + +## Discovery/Pipeline/IndexedMutationPoint.swift + +```swift +struct IndexedMutationPoint: Sendable { + let point: MutationPoint + let id: String + let isSchematizable: Bool +} +``` + +| Field | Description | +|---|---| +| `point` | The original mutation point | +| `id` | `"swift-mutation-testing_"` — unique per run | +| `isSchematizable` | `true` if the mutation falls inside a function body (determined by `TypeScopeVisitor`) | + +--- + ## Discovery/Pipeline/SchematizationStage.swift ```swift struct SchematizationStage: Sendable { - func run(mutationPoints: [MutationPoint], sources: [ParsedSource]) -> SchematizationResult + static let supportFileContent: String + func run(indexed: [IndexedMutationPoint], sources: [ParsedSource]) -> ([SchematizedFile], [MutantDescriptor]) } ``` -Transforms mutation points into the final representation consumed by the execution pipeline. +Embeds all schematizable mutations into the source files via `SchemataGenerator`. Returns a tuple of schematized files and schematizable mutant descriptors. ```mermaid flowchart TD - MP[MutationPoint sorted by file/offset] --> ASSIGN[assign global index\nclassify schematizable] - ASSIGN --> GROUP[group by file] + IP[IndexedMutationPoint\nisSchematizable = true] --> GROUP[group by file] GROUP --> SCHEMA[SchemataGenerator per file\n→ SchematizedFile] - GROUP --> REWRITE[MutationRewriter per incompatible\n→ mutatedSourceContent] - SCHEMA & REWRITE --> RESULT[SchematizationResult] + SCHEMA --> RESULT["([SchematizedFile], [MutantDescriptor])"] ``` -Assigns a globally unique sequential index to each mutation point (sorted by file path, then UTF-8 offset). The index becomes the mutant ID suffix in `"swift-mutation-testing_"`. - The static `supportFileContent` declares `__swiftMutationTestingID` as a computed property reading from `ProcessInfo.processInfo.environment["__SWIFT_MUTATION_TESTING_ACTIVE"]`. --- -## Discovery/Pipeline/SchematizationResult.swift +## Discovery/Pipeline/IncompatibleRewritingStage.swift ```swift -struct SchematizationResult: Sendable { - let schematizedFiles: [SchematizedFile] - let descriptors: [MutantDescriptor] - let supportFileContent: String +struct IncompatibleRewritingStage: Sendable { + func run(indexed: [IndexedMutationPoint], sources: [ParsedSource]) -> [MutantDescriptor] } ``` -| Field | Description | -|---|---| -| `schematizedFiles` | One entry per source file that contains schematizable mutations | -| `descriptors` | All mutants (schematizable and incompatible), sorted by index | -| `supportFileContent` | The `__swiftMutationTestingID` global declaration | +Produces full-file rewrites for mutants that cannot be schematized. Each incompatible mutation point is applied to the source via `MutationRewriter`, producing a complete replacement source file stored in `MutantDescriptor.mutatedSourceContent`. --- diff --git a/Docs/CodeBase/05-schematization.md b/Docs/CodeBase/05-schematization.md index 04247b8..f93f27d 100644 --- a/Docs/CodeBase/05-schematization.md +++ b/Docs/CodeBase/05-schematization.md @@ -8,11 +8,11 @@ ```swift struct SchemataGenerator: Sendable { - func generate(source: ParsedSource, mutations: [(index: Int, point: MutationPoint)]) -> String + func generate(source: ParsedSource, mutations: [(id: String, point: MutationPoint)]) -> String } ``` -Rewrites a source file to embed all its schematizable mutations into `switch __swiftMutationTestingID` blocks. Returns the complete rewritten source as a `String`. +Rewrites a source file to embed all its schematizable mutations into `switch __swiftMutationTestingID` blocks. Returns the complete rewritten source as a `String`. Uses force-unwrapped UTF-8 conversions because Swift source code is guaranteed to be valid UTF-8. ```mermaid flowchart TD @@ -41,7 +41,7 @@ default: } ``` -Mutant IDs follow `"swift-mutation-testing_"` where `index` is the global sequential index assigned by `SchematizationStage`. +Mutant IDs follow `"swift-mutation-testing_"` where `index` is the global sequential index assigned by `MutantIndexingStage`. --- @@ -55,7 +55,7 @@ struct MutationRewriter: Sendable { Applies a single mutation to a complete source file via raw UTF-8 byte replacement. Used exclusively for incompatible mutants. -Converts the source content to `Data`, replaces the subrange `utf8Offset ..< utf8Offset + originalText.utf8.count` with `mutatedText.data(using: .utf8)`, and converts back to `String`. +Converts the source content to `Data`, replaces the subrange `utf8Offset ..< utf8Offset + originalText.utf8.count` with `mutatedText.data(using: .utf8)!`, and converts back to `String`. Uses force-unwrapped UTF-8 conversions because Swift source code is guaranteed to be valid UTF-8. --- diff --git a/Docs/CodeBase/06-sandbox-build.md b/Docs/CodeBase/06-sandbox-build.md index 7855035..996f9b2 100644 --- a/Docs/CodeBase/06-sandbox-build.md +++ b/Docs/CodeBase/06-sandbox-build.md @@ -14,6 +14,10 @@ struct SandboxFactory: Sendable { supportFileContent: String ) async throws -> Sandbox + func createClean( + projectPath: String + ) async throws -> Sandbox + func create( projectPath: String, mutatedFilePath: String, @@ -22,14 +26,15 @@ struct SandboxFactory: Sendable { } ``` -Creates an isolated copy of the Xcode project in `$TMPDIR/xmr-/`. The original project is never modified. +Creates an isolated copy of the project in `$TMPDIR/xmr-/`. Supports both Xcode and SPM projects. The original project is never modified. -**Two overloads:** +**Three factory methods:** -| Overload | Used by | Description | +| Method | Used by | Description | |---|---|---| -| `(projectPath:schematizedFiles:supportFileContent:)` | `MutantExecutor` for schematizable path | Embeds all schematized files; injects support file; disables SwiftLint phases | -| `(projectPath:mutatedFilePath:mutatedContent:)` | `IncompatibleMutantExecutor` | Writes a single mutated file; no support file injection | +| `create(projectPath:schematizedFiles:supportFileContent:)` | `MutantExecutor` for schematizable path | Embeds all schematized files; injects support file; disables SwiftLint phases | +| `createClean(projectPath:)` | `IncompatibleMutantExecutor` for SPM shared sandbox | Clean sandbox without mutations; mutated files are written directly later | +| `create(projectPath:mutatedFilePath:mutatedContent:)` | `IncompatibleMutantExecutor` for Xcode path | Writes a single mutated file; no support file injection | **Copy strategy:** @@ -81,16 +86,24 @@ A lightweight wrapper around the sandbox root URL. ```swift struct BuildStage: Sendable { let launcher: any ProcessLaunching + func build( sandbox: Sandbox, scheme: String, destination: String, timeout: Double ) async throws -> BuildArtifact + + func buildSPM( + sandbox: Sandbox, + timeout: Double + ) async throws -> BuildArtifact } ``` -Runs `xcodebuild build-for-testing` once inside the sandbox. +Runs a single build inside the sandbox. + +**Xcode path (`build`):** ```mermaid flowchart TD @@ -104,7 +117,9 @@ flowchart TD E -- plist --> F[BuildArtifact] ``` -Auto-detects project format: prefers `-workspace` if a `.xcworkspace` exists, falls back to `-project` for `.xcodeproj`, or omits both for SPM packages. +Auto-detects project format: prefers `-workspace` if a `.xcworkspace` exists, falls back to `-project` for `.xcodeproj`. + +**SPM path (`buildSPM`):** Runs `swift build --build-tests` in the sandbox directory. Returns a `BuildArtifact` with the sandbox path (no `.xctestrun` needed). Derived data is placed at `/.xmr-derived-data` to keep it inside the sandbox directory. @@ -131,15 +146,19 @@ struct BuildArtifact: Sendable { ## Build/BuildError.swift ```swift -enum BuildError: Error, Sendable { - case compilationFailed +enum BuildError: Error, Equatable, LocalizedError { + case compilationFailed(output: String) case xctestrunNotFound + + var errorDescription: String? { get } } ``` +Conforms to `LocalizedError` to provide structured error descriptions that propagate through generic `catch` blocks. + | Case | Condition | Handling | |---|---|---| -| `compilationFailed` | `xcodebuild` exits with non-zero code | Caught by `MutantExecutor`; triggers fallback path | +| `compilationFailed(output:)` | Build exits with non-zero code | Caught by `MutantExecutor`; triggers `FallbackExecutor` | | `xctestrunNotFound` | No `.xctestrun` in `Build/Products`, or plist parse failure | Propagates; fatal | --- diff --git a/Docs/CodeBase/07-execution.md b/Docs/CodeBase/07-execution.md index 964fb7a..e44b0ac 100644 --- a/Docs/CodeBase/07-execution.md +++ b/Docs/CodeBase/07-execution.md @@ -8,20 +8,20 @@ ```swift struct MutantExecutor: Sendable { - init(configuration: RunnerConfiguration, launcher: any ProcessLaunching = ProcessLauncher()) + init(configuration: RunnerConfiguration, launcher: any ProcessLaunching) func execute(_ input: RunnerInput) async throws -> [ExecutionResult] } ``` -Entry point for the execution pipeline. Orchestrates sandbox creation, build, simulator pool setup, and test execution for both schematizable and incompatible mutants. +Entry point for the execution pipeline. Orchestrates sandbox creation, build, simulator pool setup, and test execution for both schematizable and incompatible mutants. Supports both Xcode and SPM project types. ```mermaid flowchart TD IN[RunnerInput] --> CACHE{all results cached?} CACHE -- yes --> RETURN[return cached results] CACHE -- no --> SANDBOX[SandboxFactory.create\nschematized sandbox] - SANDBOX --> BUILD[BuildStage.build] - BUILD -- compilationFailed --> FALLBACK[per-file fallback\none build per schematized file] + SANDBOX --> BUILD[BuildStage.build / buildSPM] + BUILD -- compilationFailed --> FALLBACK[FallbackExecutor\none build per schematized file] BUILD -- success --> POOL[SimulatorPool.setUp] FALLBACK --> POOL POOL --> NORMAL[TestExecutionStage\nschematizable mutants] @@ -32,7 +32,7 @@ flowchart TD **Normal path:** builds once, runs `TestExecutionStage` for all schematizable mutants in parallel. -**Fallback path:** triggered when `BuildStage` throws `compilationFailed`. For each schematized file, creates a separate sandbox and re-attempts the build for that file alone. Mutants in files that still fail to compile are marked `.unviable`. +**Fallback path:** triggered when `BuildStage` throws `compilationFailed`. Delegates to `FallbackExecutor`, which rebuilds one schematized file at a time. Mutants in files that still fail to compile are marked `.unviable`. **Incompatible path:** always runs after the schematizable path. Delegates to `IncompatibleMutantExecutor`. @@ -64,10 +64,7 @@ Bundle of shared collaborators passed between `MutantExecutor` and the stage typ ```swift struct TestExecutionStage: Sendable { - let launcher: any ProcessLaunching - let cacheStore: CacheStore - let reporter: any ProgressReporter - let counter: MutationCounter + let deps: ExecutionDeps func execute( mutants: [MutantDescriptor], @@ -144,17 +141,41 @@ Raw result from a single `xcodebuild test-without-building` invocation. --- +## Execution/FallbackExecutor.swift + +```swift +struct FallbackExecutor: Sendable { + let deps: ExecutionDeps + let configuration: RunnerConfiguration + + func execute( + input: RunnerInput, + pool: SimulatorPool, + testFilesHash: String + ) async throws -> [ExecutionResult] +} +``` + +When the baseline build for all schematized files fails (`BuildError.compilationFailed`), `MutantExecutor` delegates to `FallbackExecutor`. This executor rebuilds one schematized file at a time. + +```mermaid +flowchart TD + FILES["[SchematizedFile]"] --> LOOP["For each file"] + LOOP --> SF[SandboxFactory\nsingle-file sandbox] + SF --> BS[BuildStage] + BS -- success --> TES[TestExecutionStage\ntest mutants in this file] + BS -- failed --> UNVIABLE[Mark all mutants in file as .unviable] +``` + +For each schematized file, creates a sandbox containing only that file's schematization, builds it (Xcode or SPM), and runs the test suite against its mutants. Files whose builds fail have all their mutants marked as `.unviable`. Results are cached via `CacheStore`. + +--- + ## Execution/IncompatibleMutantExecutor.swift ```swift struct IncompatibleMutantExecutor: Sendable { - init( - launcher: any ProcessLaunching, - sandboxFactory: SandboxFactory, - cacheStore: CacheStore, - reporter: any ProgressReporter, - counter: MutationCounter - ) + let deps: ExecutionDeps func execute( _ mutants: [MutantDescriptor], @@ -165,40 +186,28 @@ struct IncompatibleMutantExecutor: Sendable { } ``` -Handles mutants that cannot be schematized. Runs sequentially — one full build + test cycle per mutant. +Handles mutants that cannot be schematized. Behaviour differs by project type. + +**Xcode path:** Each mutant creates its own sandbox via `SandboxFactory.create(projectPath:mutatedFilePath:mutatedContent:)`. Runs sequentially with a full build + test cycle per mutant. ```mermaid flowchart TD - MUTANT[MutantDescriptor\nisSchematizable = false] --> CACHE{cache hit?} + MUTANT[MutantDescriptor\nisSchematizable = false] --> PT{ProjectType?} + PT -- .xcode --> CACHE{cache hit?} CACHE -- yes --> CACHED[return cached result] - CACHE -- no --> CONTENT{mutatedSourceContent?} - CONTENT -- nil --> UNVIABLE[.unviable] - CONTENT -- present --> SF[SandboxFactory.create\nmutatedFilePath mutatedContent] + CACHE -- no --> SF[SandboxFactory.create\nmutatedFilePath mutatedContent] SF --> BS[BuildStage.build] - BS -- compilationFailed --> UNVIABLE2[.unviable] + BS -- compilationFailed --> UNVIABLE[.unviable] BS -- success --> SLOT[pool.acquire] - SLOT --> LAUNCH[xcodebuild test\n-xctestrun -destination] + SLOT --> LAUNCH[xcodebuild test-without-building] LAUNCH --> RELEASE[pool.release] - RELEASE --> PARSE[ResultParser.parse] + RELEASE --> PARSE[TestResultResolver] PARSE --> STORE[cacheStore.store] + PT -- .spm --> SPM[Shared sandbox\nwrite mutated file → swift test] + SPM --> SPMPARSE[SPMResultParser] ``` -Uses `xcodebuild test` (not `test-without-building`) because each incompatible mutant produces its own distinct binary. - ---- - -## Execution/IncompatibleTestLaunchResult.swift - -```swift -struct IncompatibleTestLaunchResult: Sendable { - let exitCode: Int32 - let output: String - let xcresultPath: String - let duration: Double -} -``` - -Identical structure to `TestLaunchResult`. Kept separate to allow divergence if the incompatible path's launch contract changes independently. +**SPM path:** Uses a shared sandbox created via `SandboxFactory.createClean(projectPath:)`. For each mutant, writes the mutated source content (`mutant.mutatedSourceContent!`) directly to the sandbox, runs `swift test`, and restores the original file. Pipeline invariants guarantee `mutatedSourceContent` is always non-nil for incompatible mutants. --- @@ -268,13 +277,17 @@ Provides simulator lifecycle utilities. ## Simulator/SimulatorError.swift ```swift -enum SimulatorError: Error, Sendable { +enum SimulatorError: Error, LocalizedError { case deviceNotFound(destination: String) case bootTimeout(udid: String) case cloneFailed(udid: String) + + var errorDescription: String? { get } } ``` +Conforms to `LocalizedError` to provide structured error descriptions that propagate through generic `catch` blocks. + | Case | Condition | |---|---| | `deviceNotFound` | No simulator matching the destination string | @@ -302,8 +315,7 @@ Tracks execution progress across concurrent tasks. `total` is set once at constr ```swift struct RunnerInput: Sendable { let projectPath: String - let scheme: String - let destination: String + let projectType: ProjectType let timeout: Double let concurrency: Int let noCache: Bool diff --git a/Docs/CodeBase/08-result-parsing-cache.md b/Docs/CodeBase/08-result-parsing-cache.md index fc5194c..08130ac 100644 --- a/Docs/CodeBase/08-result-parsing-cache.md +++ b/Docs/CodeBase/08-result-parsing-cache.md @@ -4,6 +4,26 @@ --- +## Execution/TestResultResolver.swift + +```swift +struct TestResultResolver: Sendable { + let launcher: any ProcessLaunching + + func resolve( + launch: TestLaunchResult, + projectType: ProjectType, + timeout: TimeInterval + ) async throws -> TestRunOutcome +} +``` + +Delegates to the appropriate parser based on project type: +- `.xcode` → `ResultParser` (xcresulttool + output parsing) +- `.spm` → `SPMResultParser` (output-only parsing) + +--- + ## Execution/Parsing/ResultParser.swift ```swift @@ -41,26 +61,24 @@ Exit code `-1` is the sentinel set by `ProcessLauncher` when it kills the proces ```swift enum TestRunOutcome: Sendable { case testsSucceeded - case testsKilled(reason: String) - case processCrashed + case testsFailed(failingTest: String) + case crashed case timedOut - case compilationFailed - case noCoverage + case unviable var asExecutionStatus: ExecutionStatus { get } } ``` -Intermediate result from `ResultParser`, converted to `ExecutionStatus` via `asExecutionStatus`. +Intermediate result from `TestResultResolver`/`ResultParser`/`SPMResultParser`, converted to `ExecutionStatus` via `asExecutionStatus`. | Case | Maps to | |---|---| | `testsSucceeded` | `.survived` | -| `testsKilled(reason:)` | `.killed(by: reason)` | -| `processCrashed` | `.killedByCrash` | +| `testsFailed(failingTest:)` | `.killed(by: failingTest)` | +| `crashed` | `.killedByCrash` | | `timedOut` | `.timeout` | -| `compilationFailed` | `.unviable` | -| `noCoverage` | `.noCoverage` | +| `unviable` | `.unviable` | --- @@ -89,6 +107,26 @@ Returns `.testsKilled(reason: )` for test failures, `.proce --- +## Execution/Parsing/SPMResultParser.swift + +```swift +struct SPMResultParser: Sendable { + func parse(exitCode: Int32, output: String) -> TestRunOutcome +} +``` + +Parses SPM test results from exit code and stdout/stderr output only (no `.xcresult` bundles). Uses `TestOutputParser` to detect failure patterns. + +| Condition | Outcome | +|---|---| +| Exit code `-1` | `.timedOut` | +| Exit code `0` | `.testsSucceeded` | +| Non-zero + test failure pattern | `.testsFailed(failingTest:)` | +| Non-zero + empty output | `.crashed` | +| Non-zero + no parseable failure | `.unviable` | + +--- + ## Execution/Parsing/XCResultParser.swift ```swift diff --git a/Docs/CodeBase/09-reporting-infrastructure.md b/Docs/CodeBase/09-reporting-infrastructure.md index 40d591b..a0c4a91 100644 --- a/Docs/CodeBase/09-reporting-infrastructure.md +++ b/Docs/CodeBase/09-reporting-infrastructure.md @@ -379,37 +379,79 @@ protocol ProcessLaunching: Sendable { ) async throws -> Int32 func launchCapturing( - executableURL: URL, - arguments: [String], - environment: [String: String]?, - workingDirectoryURL: URL, - timeout: Double + _ request: ProcessRequest ) async throws -> (exitCode: Int32, output: String) } ``` -Abstraction over process execution. `launch` discards output (stdout/stderr → `/dev/null`). `launchCapturing` captures combined stdout+stderr and returns it as a `String`. +Abstraction over process execution. `launch` discards output (stdout/stderr → `/dev/null`). `launchCapturing` accepts a `ProcessRequest` value, captures combined stdout+stderr, and returns it as a `String`. Return value `-1` from either method means the process was killed by the timeout handler. --- -## Infrastructure/ProcessLauncher.swift +## Infrastructure/ProcessRequest.swift + +```swift +struct ProcessRequest: Sendable { + let executableURL: URL + let arguments: [String] + let environment: [String: String]? + let additionalEnvironment: [String: String] + let workingDirectoryURL: URL + let timeout: Double +} +``` + +| Field | Description | +|---|---| +| `executableURL` | Path to the executable | +| `arguments` | Command-line arguments | +| `environment` | Full environment override (replaces inherited environment when non-nil) | +| `additionalEnvironment` | Key-value pairs merged into the existing environment | +| `workingDirectoryURL` | Working directory for the process | +| `timeout` | Maximum execution time in seconds | + +--- + +## Infrastructure/ProcessRunner.swift ```swift -struct ProcessLauncher: Sendable, ProcessLaunching { - func launch(...) async throws -> Int32 - func launchCapturing(...) async throws -> (exitCode: Int32, output: String) +struct ProcessRunner: Sendable { + var postTerminationCleanup: (@Sendable (Int32) -> Void)? + let onTimeout: @Sendable (Int32) -> Void + + func launch(executableURL:arguments:workingDirectoryURL:timeout:) async throws -> Int32 + func launchCapturing(_ request: ProcessRequest) async throws -> (exitCode: Int32, output: String) } ``` -Production implementation of `ProcessLaunching`. Uses `withTaskCancellationHandler` + `withCheckedThrowingContinuation` to bridge `Process.terminationHandler` into the Swift Concurrency runtime. +Low-level process execution engine. Uses `withTaskCancellationHandler` + `withCheckedThrowingContinuation` to bridge `Process.terminationHandler` into the Swift Concurrency runtime. + +**Timeout handling:** a `Task` sleeping for `timeout` seconds marks a `KilledByUsFlag` and calls `onTimeout(pid)`. The `terminationHandler` checks the flag and returns `-1` instead of the actual exit code. + +**Cancellation handling:** `onCancel` marks the flag and calls `onTimeout(pid)` immediately, ensuring the continuation is always resumed via the `terminationHandler`. + +**Post-termination cleanup:** `postTerminationCleanup` is called after every process termination (success or failure), used by `SPMProcessLauncher` to clean up escaped child processes. + +`launchCapturing` writes output to a temporary file (UUID-named) and reads it in the `terminationHandler` to avoid pipe buffer limits. Sets process group via `setpgid(pid, pid)` to enable group signaling. -**Timeout handling:** a `Task` sleeping for `timeout` seconds marks a `KilledByUsFlag` and calls `kill(-pid, SIGTERM)` + a delayed `kill(-pid, SIGKILL)`. The `terminationHandler` checks the flag and returns `-1` instead of the actual exit code. +--- + +## Infrastructure/SPMProcessLauncher.swift + +```swift +struct SPMProcessLauncher: Sendable, ProcessLaunching { + func launch(executableURL:arguments:workingDirectoryURL:timeout:) async throws -> Int32 + func launchCapturing(_ request: ProcessRequest) async throws -> (exitCode: Int32, output: String) +} +``` -**Cancellation handling:** `onCancel` marks the flag and terminates the process group immediately, ensuring the continuation is always resumed via the `terminationHandler`. +SPM-specific implementation of `ProcessLaunching`. Creates a `ProcessRunner` with: +- `onTimeout`: kills the process group via `kill(-pid, SIGKILL)` + `kill(pid, SIGKILL)` +- `postTerminationCleanup`: calls `killEscapedChildren(sandboxPath:)` to clean up orphaned child processes -`launchCapturing` writes output to a temporary file (UUID-named) and reads it in the `terminationHandler` to avoid pipe buffer limits. +**`killEscapedChildren(sandboxPath:)`** — inspects running processes via `sysctl` `KERN_PROCARGS2` to find any whose arguments contain the sandbox path prefix `xmr-`. Sends `SIGKILL` to matching processes to prevent resource leaks from spawned child processes that outlive the parent. --- diff --git a/Docs/CodeBase/README.md b/Docs/CodeBase/README.md index b908d34..e1bd2ad 100644 --- a/Docs/CodeBase/README.md +++ b/Docs/CodeBase/README.md @@ -9,14 +9,14 @@ Type-level reference for every public and internal type in `swift-mutation-testi | Document | Coverage | |---|---| | [01 — Entry Point](01-entry-point.md) | `SwiftMutationTesting`, `ExitCode`, `HelpText`, `UsageError` | -| [02 — Configuration](02-configuration.md) | `CommandLineParser`, `ParsedArguments`, `RunnerConfiguration`, `ConfigurationResolver`, `ConfigurationFileParser`, `ConfigurationFileWriter`, `ProjectDetector`, `DetectedProject` | -| [03 — Discovery Pipeline](03-discovery-pipeline.md) | `DiscoveryPipeline`, `DiscoveryInput`, `FileDiscoveryStage`, `FileDiscoveryError`, `ParsingStage`, `MutantDiscoveryStage`, `SchematizationStage`, `SchematizationResult`, `SourceFile`, `ParsedSource`, `MutationPoint`, `MutantDescriptor` | +| [02 — Configuration](02-configuration.md) | `CommandLineParser`, `ParsedArguments`, `RunnerConfiguration`, `BuildOptions`, `ReportingOptions`, `FilterOptions`, `ProjectType`, `TestingFramework`, `ConfigurationResolver`, `ConfigurationFileParser`, `ConfigurationFileWriter`, `ProjectDetector`, `DetectedProject` | +| [03 — Discovery Pipeline](03-discovery-pipeline.md) | `DiscoveryPipeline`, `DiscoveryInput`, `FileDiscoveryStage`, `FileDiscoveryError`, `ParsingStage`, `MutantDiscoveryStage`, `MutantIndexingStage`, `SchematizationStage`, `IncompatibleRewritingStage`, `SourceFile`, `ParsedSource`, `MutationPoint`, `IndexedMutationPoint`, `MutantDescriptor` | | [04 — Mutation Operators](04-mutation-operators.md) | `MutationOperator`, `MutationSyntaxVisitor`, `ReplacementKind`, all 7 operator structs and visitors, `SuppressionAnnotationExtractor`, `SuppressionFilter`, `SuppressionVisitor` | | [05 — Schematization](05-schematization.md) | `SchemataGenerator`, `MutationRewriter`, `TypeScopeVisitor`, `FunctionBodyScope`, `SchematizedFile` | | [06 — Sandbox & Build](06-sandbox-build.md) | `SandboxFactory`, `Sandbox`, `BuildStage`, `BuildArtifact`, `BuildError` | -| [07 — Execution](07-execution.md) | `MutantExecutor`, `ExecutionDeps`, `TestExecutionStage`, `TestExecutionContext`, `TestLaunchResult`, `IncompatibleMutantExecutor`, `IncompatibleTestLaunchResult`, `SimulatorPool`, `SimulatorSlot`, `SimulatorManager`, `SimulatorError`, `MutationCounter`, `RunnerInput`, `ExecutionResult`, `ExecutionStatus` | -| [08 — Result Parsing & Cache](08-result-parsing-cache.md) | `ResultParser`, `TestRunOutcome`, `TestOutputParser`, `XCResultParser`, `CacheStore`, `MutantCacheKey` | -| [09 — Reporting & Infrastructure](09-reporting-infrastructure.md) | `ProgressReporter`, `ConsoleProgressReporter`, `SilentProgressReporter`, `RunnerEvent`, `RunnerSummary`, `TextReporter`, `JsonReporter`, `HtmlReporter`, `SonarReporter`, all `MutationReport*` types, all `Sonar*` types, `ProcessLaunching`, `ProcessLauncher`, `XCTestRunPlist`, `TestFilesHasher` | +| [07 — Execution](07-execution.md) | `MutantExecutor`, `ExecutionDeps`, `TestExecutionStage`, `TestExecutionContext`, `TestLaunchResult`, `FallbackExecutor`, `IncompatibleMutantExecutor`, `SimulatorPool`, `SimulatorSlot`, `SimulatorManager`, `SimulatorError`, `MutationCounter`, `RunnerInput`, `ExecutionResult`, `ExecutionStatus` | +| [08 — Result Parsing & Cache](08-result-parsing-cache.md) | `TestResultResolver`, `ResultParser`, `SPMResultParser`, `TestRunOutcome`, `TestOutputParser`, `XCResultParser`, `CacheStore`, `MutantCacheKey` | +| [09 — Reporting & Infrastructure](09-reporting-infrastructure.md) | `ProgressReporter`, `ConsoleProgressReporter`, `SilentProgressReporter`, `RunnerEvent`, `RunnerSummary`, `TextReporter`, `JsonReporter`, `HtmlReporter`, `SonarReporter`, all `MutationReport*` types, all `Sonar*` types, `ProcessLaunching`, `ProcessRunner`, `ProcessRequest`, `SPMProcessLauncher`, `XCTestRunPlist`, `TestFilesHasher` | --- @@ -26,16 +26,20 @@ Type-level reference for every public and internal type in `swift-mutation-testi ``` DiscoveryInput - → FileDiscoveryStage → [SourceFile] - → ParsingStage → [ParsedSource] - → MutantDiscoveryStage → [MutationPoint] - → SchematizationStage → SchematizationResult + → FileDiscoveryStage → [SourceFile] + → ParsingStage → [ParsedSource] + → MutantDiscoveryStage → [MutationPoint] + → MutantIndexingStage → [IndexedMutationPoint] + → SchematizationStage → [SchematizedFile], [MutantDescriptor] + → IncompatibleRewritingStage → [MutantDescriptor] → RunnerInput RunnerInput → SandboxFactory → Sandbox → BuildStage → BuildArtifact - → TestExecutionStage → [ExecutionResult] + → TestExecutionStage → TestResultResolver → [ExecutionResult] + → FallbackExecutor (on build failure) → [ExecutionResult] + → IncompatibleMutantExecutor → [ExecutionResult] → RunnerSummary → Reporters ``` diff --git a/Docs/MUTATION-RESULTS.md b/Docs/MUTATION-RESULTS.md index c06d4d3..ea3976f 100644 --- a/Docs/MUTATION-RESULTS.md +++ b/Docs/MUTATION-RESULTS.md @@ -1,6 +1,6 @@ # Mutation Results -This document explains every possible outcome for a mutant, what causes it, and what it means for your test suite. It also explains the distinction between **schematizable** and **incompatible** mutants — a concept that affects how the tool runs and how to interpret the progress output. +This document explains every possible outcome for a mutant, what causes it, and what it means for your test suite. It also explains the distinction between **schematizable** and **incompatible** mutants — a concept that affects how the tool runs and how to interpret the progress output. All concepts apply equally to both Xcode and SPM projects. --- @@ -146,9 +146,9 @@ This is the most important internal distinction in how the tool operates. It dir ### Why the distinction exists -Running one full `xcodebuild build-for-testing` + `xcodebuild test-without-building` cycle per mutant would make mutation testing impractically slow for any real project. For a project with 200 mutants and a 20-second build, a naive approach would take over an hour just in build time. +Running one full build + test cycle per mutant would make mutation testing impractically slow for any real project. For a project with 200 mutants and a 20-second build, a naive approach would take over an hour just in build time. -The tool avoids this by **schematization**: rewriting source files to embed all mutations at once behind a runtime switch, building the project a single time, and then activating one mutant per test run by setting an environment variable. This reduces the total build cost to a single build regardless of the number of mutants. +The tool avoids this by **schematization**: rewriting source files to embed all mutations at once behind a runtime switch, building the project a single time, and then activating one mutant per test run by setting an environment variable. This reduces the total build cost to a single build regardless of the number of mutants. This works for both Xcode projects (`xcodebuild build-for-testing` / `test-without-building`) and SPM packages (`swift build --build-tests` / `swift test --skip-build`). ```swift // Original source @@ -169,7 +169,7 @@ func isAdult(age: Int) -> Bool { } ``` -The global `__swiftMutationTestingID` reads from `ProcessInfo.processInfo.environment["__SWIFT_MUTATION_TESTING_ACTIVE"]`. Each test run injects a different mutant ID into that environment variable via the `.xctestrun` plist. +The global `__swiftMutationTestingID` reads from `ProcessInfo.processInfo.environment["__SWIFT_MUTATION_TESTING_ACTIVE"]`. Each test run injects a different mutant ID into that environment variable — via the `.xctestrun` plist for Xcode projects, or via the process environment for SPM packages. ### What makes a mutant incompatible @@ -209,14 +209,14 @@ enum ExitCode: Int32 { } ``` -In all of these cases, the mutation site is not inside any executable scope that can host a `switch` statement. The tool falls back to applying the mutation directly to the source file, building the full project from scratch, and running `xcodebuild test` (not `test-without-building`). +In all of these cases, the mutation site is not inside any executable scope that can host a `switch` statement. The tool falls back to applying the mutation directly to the source file and running a full build + test cycle per mutant. For SPM projects, incompatible mutants use a shared sandbox — the mutated file is written, `swift test` runs, and the original is restored. ### Performance implications | | Schematizable | Incompatible | |---|---|---| -| Builds required | 1 (shared) | 1 per mutant | -| Test command | `test-without-building` | `test` | +| Builds required | 1 (shared) | 1 per mutant (Xcode) or shared sandbox (SPM) | +| Test command | `test-without-building` (Xcode) / `swift test --skip-build` (SPM) | full build + test per mutant | | Parallel execution | yes, N workers | sequential | | Typical cost | seconds per mutant | full build + test per mutant | diff --git a/Docs/USAGE.MD b/Docs/USAGE.MD index d7f7423..457d1e6 100644 --- a/Docs/USAGE.MD +++ b/Docs/USAGE.MD @@ -21,11 +21,11 @@ This guide covers every way to run and configure `swift-mutation-testing`, from ### Generate a configuration file ```bash -# Auto-detects scheme, destination, and test target +# Auto-detects project type, scheme, destination, test targets, and testing framework swift-mutation-testing init ``` -This writes `.swift-mutation-testing.yml` at the project root. Review and adjust the generated file before running. +This writes `.swift-mutation-testing.yml` at the project root. Review and adjust the generated file before running. For SPM packages, `scheme` and `destination` are not required and will be omitted. ### Run mutation testing @@ -33,6 +33,13 @@ This writes `.swift-mutation-testing.yml` at the project root. Review and adjust swift-mutation-testing ``` +### Run an SPM package + +```bash +# SPM packages are auto-detected — no scheme or destination needed +swift-mutation-testing /path/to/my-package +``` + ### Run on macOS without a simulator ```bash @@ -102,7 +109,7 @@ destination: platform=iOS Simulator,name=iPhone 16 # Limit test execution to a specific target (recommended when the project has UI tests) # testTarget: MyAppTests -# Per-mutant test timeout in seconds (default: 60) +# Per-mutant test timeout in seconds (default: 120 Xcode, 30 SPM) timeout: 60 # Number of parallel workers (default: max(1, CPU count - 1)) @@ -142,10 +149,10 @@ mutators: | Field | Type | Default | Description | |---|---|---|---| -| `scheme` | String | — | Xcode scheme to build and test (required) | -| `destination` | String | — | `xcodebuild` destination specifier (required) | +| `scheme` | String | — | Xcode scheme to build and test (required for Xcode projects, omit for SPM) | +| `destination` | String | — | `xcodebuild` destination specifier (required for Xcode projects, omit for SPM) | | `testTarget` | String | — | Limit test execution to this target (`-only-testing`) | -| `timeout` | Number | `60` | Per-mutant test timeout in seconds | +| `timeout` | Number | `120` (Xcode) / `30` (SPM) | Per-mutant test timeout in seconds | | `concurrency` | Number | `max(1, CPUs − 1)` | Number of parallel test workers | | `noCache` | Boolean | `false` | Disable result caching | | `output` | String | — | Path for Stryker-compatible JSON report | @@ -212,10 +219,10 @@ swift-mutation-testing [project-path] [options] | Option | Description | |---|---| -| `--scheme ` | Xcode scheme to build and test | -| `--destination ` | `xcodebuild` destination specifier | +| `--scheme ` | Xcode scheme to build and test (required for Xcode, omit for SPM) | +| `--destination ` | `xcodebuild` destination specifier (required for Xcode, omit for SPM) | | `--target ` | Limit test execution to this target | -| `--timeout ` | Per-mutant test timeout (default: 60) | +| `--timeout ` | Per-mutant test timeout (default: 120 Xcode, 30 SPM) | | `--concurrency ` | Parallel workers (default: CPUs − 1) | | `--no-cache` | Disable result caching | | `--output ` | Write Stryker-compatible JSON report | diff --git a/README.md b/README.md index 82f9f2b..b1e7f6c 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **Measure and improve test effectiveness in Swift codebases using mutation testing.** -`swift-mutation-testing` is a CLI for mutation testing of Xcode + XCTest projects. It modifies your source code in small, targeted ways — mutations — and runs your test suite against each one. A mutation that goes undetected reveals missing tests or weak assertions. The result is a mutation score that reflects how effectively your tests catch real bugs. +`swift-mutation-testing` is a CLI for mutation testing of Swift projects — both Xcode and Swift Package Manager. It modifies your source code in small, targeted ways — mutations — and runs your test suite against each one. A mutation that goes undetected reveals missing tests or weak assertions. The result is a mutation score that reflects how effectively your tests catch real bugs. ## Why @@ -18,12 +18,17 @@ Mutation testing introduces controlled changes to your code to verify that your ## Features -- Mutation testing for Xcode + XCTest projects -- Measures test effectiveness through mutation score -- Supports multiple mutation operators -- Provides detailed reports per file and mutation -- Configurable via YAML -- Can be integrated into CI pipelines +- Mutation testing for Xcode and SPM projects +- Supports both XCTest and Swift Testing frameworks +- 7 mutation operators (relational, boolean, logical, arithmetic, negate conditional, swap ternary, remove side effects) +- Schematization — builds once, tests all mutants via runtime switch +- Parallel test execution with configurable concurrency +- SHA256-based result caching across runs +- Multiple report formats: text, JSON (Stryker-compatible), HTML, SonarQube +- Simulator pool management for iOS/tvOS/watchOS targets +- Per-scope mutation suppression via `@SwiftMutationTestingDisabled` +- Configurable via YAML or CLI flags +- CI/CD ready with caching support ## Install @@ -37,11 +42,14 @@ Other installation methods — pre-built binary, build from source — are cover ## Quick start ```bash -# Generate a config file (auto-detects scheme and destination) +# Generate a config file (auto-detects project type, scheme, destination, and test targets) swift-mutation-testing init # Run mutation testing swift-mutation-testing + +# Run on an SPM package (no scheme or destination needed) +swift-mutation-testing /path/to/my-package ``` Example output: @@ -74,14 +82,27 @@ Total duration: 312.7s Drop a `.swift-mutation-testing.yml` in the project root: +**Xcode project:** + ```yaml scheme: MyApp destination: platform=iOS Simulator,name=iPhone 16 # testTarget: MyAppTests -timeout: 60 +# timeout: 120 +# concurrency: 4 +``` + +**SPM package** (scheme and destination are not needed): + +```yaml +# testTarget: MyPackageTests +# timeout: 30 # concurrency: 4 +``` -# Mutation operators — set active: false to disable +**Mutation operators** (both project types — all active by default): + +```yaml mutators: - name: RelationalOperatorReplacement active: true