Semantics of `PyThreadState_GetUnchecked` inside `tp_traverse` handlers

[davidhewitt]

Following the recent discussion about restrictions to tp_traverse handlers.

In #131264 (comment) @colesbury noted that

PyThreadState_GetUnchecked() != NULL seems like a sufficient way to decide if it's safe to call into the interpreter.

I have previously observed that PyThreadState_GetUnchecked will return non-null when inside a tp_traverse handler. Should it be changed to return NULL so that @colesbury 's statement remains valid inside tp_traverse handlers too?

If no, I have ways I can workaround in PyO3. But I worry about edge cases like a signal handler being hit inside of tp_traverse, maybe in that case code will need to check whether it is attached to the interpreter (with PyThreadStateGetUnchecked?) and then if not, safely fail to attach until the traverse has finished.

[colesbury]

I could be convinced otherwise, but at least for now I think that changing PyThreadState_GetUnchecked() to return NULL during tp_traverse introduces mores risks than it avoids.

I get that PyO3 wants to enforce correctness invariants, but I'm not sure you can do this thoroughly for tp_traverse. The restriction we stated was "no side effects". There are a number of macros or code patterns that don't have side effects, but would be unsafe to perform without a valid thread state:

• Py_VISIT(Py_TYPE(self))
• Py_VISIT(self->ob_item[i])

But I worry about edge cases like a signal handler being hit inside of tp_traverse

I don't think it's safe to call into Python APIs during a signal handler regardless of tp_traverse.