Subinterpreters running on the main thread should handle signals

[ZeroIntensity]

Feature or enhancement

Proposal:

It's currently impossible to interrupt long-running operations inside subinterpreters:

from concurrent import interpreters

interp = interpreters.create()
interp.exec("import time; time.sleep(100)")  # Cannot CTRL^C!

This is because _Py_ThreadCanHandleSignals explicitly requires the main interpreter:

cpython/Include/internal/pycore_pystate.h

Lines 82 to 86 in 7040aa5

	static inline int
	_Py_ThreadCanHandleSignals(PyInterpreterState *interp)
	{
	return (_Py_IsMainThread() && _Py_IsMainInterpreter(interp));
	}

After thinking about this for a while, I can't see any real reason why stdlib interpreters shouldn't be able to handle signals (in the main thread). It's probably a slight compatibility break for legacy C API subinterpreters, so we can just disable it for them (and add the new option to handle signals behind an interpreter configuration setting).

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

Quickly discussed in-person a few months ago with @ericsnowcurrently and @colesbury. I don't think we came to a consensus, but there wasn't any pushback as far as I could tell.

Linked PRs

• gh-137173: Allow signal handling in isolated subinterpreters #137174

[ncoghlan]

What semantics do we want when there are multiple signal handling interpreters running in a process?

• an arbitrary (aka "the next one to run in the main thread") interpreter receives the signal, the others remain uninterrupted?
• all signal handling interpreters are interrupted?

My read of the draft PR is that it's implementing the former, but I'm not sure how that indeterminate behaviour is an improvement over "all signals will be handled in the main thread of the main interpreter".

Or is the rationale along the following lines?:

• if a subinterpreter is running outside the main thread, it will never receive a signal anyway, so this change doesn't matter.
• if a subinterpreter is running in the main thread, it's also blocking the main interpreter from running in that thread, so this change just means that the subinterpreter gets interrupted, allowing the event to be propagated out if it can't reasonably be handled locally in the subinterpreter

Does the fact that raising KeyboardInterrupt in the subinterpreter will turn into concurrent.interpreters.ExecutionFailed in the caller pose a problem for this approach?

Consider cases where the subinterpreter delay isn't as significant as the one in the OP - for 3.14, those will raise KeyboardInterrupt once control returns to the main interpreter. With the proposed change, they would raise ExecutionFailed instead.

[ZeroIntensity]

Part of the rationale is that we're slowly trying to avoid making the main interpreter special, and signal handling is a big step in the right direction.

Consider cases where the subinterpreter delay isn't as significant as the one in the OP - for 3.14, those will raise KeyboardInterrupt once control returns to the main interpreter. With the proposed change, they would raise ExecutionFailed instead.

I guess this could be a problem in theory, but I think it's much more likely that a subinterpreter isn't used for light operations. If you're concerned, we could specialize KeyboardInterrupt exceptions raised inside concurrent.interpreters to propagate as an actual KeyboardInterrupt.

[ncoghlan]

Yeah, I think having KeyboardInterrupt propagate as KeyboardInterrupt would alleviate my concern without adding much implementation complexity. If we save the underlying ExecutionFailed as the __context__ for the keyboard interrupt, it won't even lose the details of the interrupted execution inside the subinterpreter.

Folks that didn't want Ctrl-C to propagate would still be free to catch KeyboardInterrupt around the exec method call (which is what they would need to be doing in 3.14).

[encukou]

Does that generalize to all non-Exception BaseExceptions?
Is the ExceptionGroup/BaseExceptionGroup split a good precedent for “wrapping” arbitrary BaseExceptions?