Description
On Python 3.12+, running tasks with the multiprocessing runner (MRunner) produces a DeprecationWarning:
/usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=197896) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
This happens because the multiprocessing module defaults to fork on Linux, and if any threads are running, fork() can lead to deadlocks in the child process.
Background
Python 3.12 added this warning to flag unsafe usage of fork() in multi-threaded processes. Python 3.14 is expected to change the default start method from fork to forkserver on Linux.
Possible approaches
spawn: safest, but requires all task data to be picklable (related to cloudpickle optional dependency)
forkserver: middle ground, avoids the fork-in-threads issue but has its own quirks
- Make it configurable: let users choose the start method via
DOIT_CONFIG
Considerations
- Changing the default from
fork could break existing users' dodo.py files that rely on fork behavior (shared state, file handles, etc.)
- This is a backward-incompatible change that needs careful evaluation
- Related:
cloudpickle is already an optional dependency for pickling support with spawn
References
Description
On Python 3.12+, running tasks with the multiprocessing runner (
MRunner) produces aDeprecationWarning:This happens because the
multiprocessingmodule defaults toforkon Linux, and if any threads are running,fork()can lead to deadlocks in the child process.Background
Python 3.12 added this warning to flag unsafe usage of
fork()in multi-threaded processes. Python 3.14 is expected to change the default start method fromforktoforkserveron Linux.Possible approaches
spawn: safest, but requires all task data to be picklable (related tocloudpickleoptional dependency)forkserver: middle ground, avoids the fork-in-threads issue but has its own quirksDOIT_CONFIGConsiderations
forkcould break existing users'dodo.pyfiles that rely on fork behavior (shared state, file handles, etc.)cloudpickleis already an optional dependency for pickling support withspawnReferences
doit/runner.py(MRunnerclass)