Native parallel test execution

[sebastianbergmann]

Disclaimer

The purpose of this issue is to provide a space for exploring and discussing the architecture and design specifications of native parallel test execution implemented directly within PHPUnit. I would like to make it clear that I am not working on this because I think that existing solutions, such as Paratest and ParaUnit, are inadequate. I also want to make it clear that the existence of this issue does not mean that PHPUnit will support parallel test execution out of the box. For now, I am treating this as a thought experiment: what if parallel test execution were to be implemented natively in PHPUnit? Would that benefit end users? Would it be easier to maintain?

Status Quo

At present, PHPUnit can only execute a test suite sequentially: TextUI\TestRunner::run() flattens the suite and calls $test->run() on each test in turn (Framework\TestSuite::run()). On multi-core machines this leaves most of the hardware idle. Third-party tools (paratest, paraunit) parallelize by running PHPUnit itself as a black-box subprocess per test file and then parsing its log output to reconstruct results. That approach is inherently lossy and cannot reproduce PHPUnit's own reporting fidelity.

High-level implementation approach

PHPUnit already owns a high-fidelity mechanism for running a test in another process and reconstituting its outcome: the implementation of its process isolation feature. Native parallelism is therefore not a new subsystem bolted on top; it is the generalization of that implementation from "one synchronous child per test" to "N concurrent, reusable workers":

Concern	Class	Role
Serialize a test into a runnable child script	Framework\TestRunner\SeparateProcessTestRunner	Templates one TestCase + config + (optional) global state into PHP code
Spawn a child and collect its output	Util\PHP\JobRunner	proc_open, feed code, collect stdout/stderr
Reconstitute a child's outcome in the parent	Framework\TestRunner\ChildProcessResultProcessor	Forwards events, imports PassedTests, sets result/status/assertions, merges coverage
Transport	nonce-prefixed serialized stdClass written to a temp file	{events, passedTests, testResult, status, numAssertions, output, codeCoverage}
Replay child events in the parent	Event\Facade::forward(EventCollection)	Re-dispatches a collected event stream through the parent dispatcher
Collect events in a child	Event\Facade::initForIsolation() → CollectingDispatcher	Accumulates events; flush() returns + resets them

Every printer, logger, and report is a downstream subscriber of events. A child runs with a CollectingDispatcher, collects its events into an EventCollection, ships them home, and the parent replays them with forward(). Consequently, if a worker can ship its event stream to the parent, the parent's entire output/logging/result subsystem works unchanged:

TextUI\Application
        │  (selects runner when parallel workers > 1)
        ▼
TextUI\TestRunner ───────────────► TextUI\ParallelTestRunner
   (sequential, today)                      │
                                            ▼
                              Runner\Parallel\WorkerPool
                               │        │         │
                        Worker 1     Worker 2   Worker N   (PersistentWorker)
                               │        │         │
                          ┌────┴────────┴─────────┴─────┐
                          │  Scheduler / WorkQueue      │
                          └─────────────────────────────┘
                                       │  per-unit result envelopes
                                       ▼
                          Runner\Parallel\ResultAggregator
                          → Event\Facade::forward(...) in suite order
                          → PassedTests::import(), CodeCoverage::merge()

The parent process remains the single source of truth for output: it owns the OutputFacade, TestResultFacade, CodeCoverage, ResultCache, and the sealed event dispatcher. Workers never produce user-facing output; they only emit events that the parent replays.

JobRunner::runProcess() was synchronous: spawn one process, block in stream_get_contents(), proc_close(). A pool needs to keep N processes busy simultaneously.

Chosen mechanism: proc_open() and stream_select() over the workers' pipes.

Rationale:

• No new dependency or runtime requirement, it runs everywhere proc_open() already runs (the existing isolation requirement)
• Rejected alternatives: ext-parallel (requires ZTS; narrow availability) and ext-pcntl fork (no Windows; fragile fork-sharing of an initialized PHPUnit)

Granularity should be configurable, defaulting to per test class:

• per-suite: coarsest; good for several independent <testsuite>s
• per-class (default): preserves #[BeforeClass] / #[AfterClass] and intra-class ordering; matches paratest's default; bounds per-process overhead
• per-method: finest balancing, but breaks shared class fixtures (and likely has increased per-test coverage serialization cost, for example)

Distribution is a dynamic work-stealing queue, not static pre-chunking: each idle worker pulls the next unit, which self-balances against stragglers. The ResultCache (per-test durations, Runner\ResultCache\) can seed a longest-processing-time-first ordering so large units start early.

Step 1: Async-capable JobRunner

This has already been implemented in #6753.

Step 2: Persistent worker

This has already been implemented in #6753.

Step 3: Worker Pool and Scheduler

• Runner\Parallel\WorkerPool owns N PersistentWorkers and pumps their pipes with stream_select()
• Dynamic work-stealing queue; default per-class granularity; ordering seeded from ResultCache
• TextUI\ParallelTestRunner (or a branch in the existing runner) selects the parallel path when configured worker count > 1

Step 4: ResultAggregator with ordered replay

Workers finish out of order; events carry telemetry and the printers assume suite-ordered arrival. The aggregator buffers each unit's EventCollection and forwards it to the parent dispatcher in deterministic suite order, releasing a unit only once all prior units have flushed. Output then matches sequential mode byte-for-byte (good for CI diffs), at the cost of latency and memory.

Step 5: Correctness and robustness

• Dependencies: #[Depends] / ExecutionOrderDependency: dependent tests must land in the same worker in order (the TestSuiteSorter graph already computes this)
• Stop conditions: --stop-on-* / interruption need a broadcast halt; in-flight units are abandoned, so the stop point is no longer exactly
  deterministic (maybe "only" a documentation issue)
• Suite hooks: before-first-test / after-last-test hook methods run once in the parent (or a designated worker), not per worker
• Resource partitioning: expose a worker-identity environment variable so test fixtures can choose a per-worker database/port/temporary directory/...
• Crash/retry: detect a dead pipe / abnormal exit, attribute it to the in-flight unit, optionally retry once on a fresh worker, keep other workers running

[mondrake]

Very interesting. Just fyi, Drupal has it own paralleliser run-tests.sh, which actually allows both to allocate multiple parallel processes within one server, and to leverage on CIs 'parallel' concept to break up the tests to be executed across separate servers. This means, for instance, that currently Drupal functional tests run in 8 separate parallel CI bins each with 24 parallel concurrent processes - that means up to 192 tests being run at the same time (let alone other suites that also run in parallel). Again if anyone interested, this is the list of open issues for run-tests.sh

[sebastianbergmann]

Superseded by #6784.