subprocess.Popen.communicate does not send input

[mrowqa]

Bug report

Bug description:

I will start with saying that documentation for subprocess.Popen is not clear. I was using subprocess.run(..., input=..., ...) and needed to monitor other things while the created process is running, so I changed it to subprocess.Popen. The only way to send input was Popen.communicate(input=...), however I wanted to have a timeout. My initial idea was .communicate(input=..., timeout=0), catch the exception, and then continue with .wait(timeout=STEP), however it was hanging. I worked it around by launching .communicate without timeout in a separate thread.

Then, I tested a few platforms and looked at the implementation. On Windows, my initial idea works. On Linux, it hangs. After looking into the implementation, I deduced I shouldn't use .wait(), but rather .communicate() and set the input only for the first invocation. If that's the intention, it would be great to have it documented.

Anyway, the following example hangs on a few Linux versions and few Python versions, including current main branch:

import subprocess
proc = subprocess.Popen(
    ["cat", "-"],
    stdin=subprocess.PIPE
)
try:
    proc.communicate(input="abcd\n".encode(), timeout=0)
except subprocess.TimeoutExpired:
    pass
print("it will hang:")
proc.communicate()  # initially: proc.wait()
print("finished")

It looks like a bug, and this line:

cpython/Lib/subprocess.py

Line 2108 in 9cd5427

if self.stdin and input:

which should be changed to:

                if self.stdin and not self.stdin.closed and self._input:

After this change, the example above no longer hangs. I'll open PR with the mentioned change.

CPython versions tested on:

3.12, 3.14, CPython main branch

Operating systems tested on:

Linux, Windows

Linked PRs

• gh-141473: Fix subprocess.Popen.communicate to send input to stdin upon a subsequent post-timeout call #141477
• [3.14] gh-141473: Fix subprocess.Popen.communicate to send input to stdin upon a subsequent post-timeout call (GH-141477) #142059
• [3.13] gh-141473: Fix subprocess.Popen.communicate to send input to stdin upon a subsequent post-timeout call (GH-141477) #142060
• gh-141473: Document not calling Popen.wait after Popen.communicate times out. #142101
• [3.14] gh-141473: Document not calling Popen.wait after Popen.communicate times out. (GH-142101) #142124
• [3.13] gh-141473: Document not calling Popen.wait after Popen.communicate times out. (GH-142101) #142125

[gpshead]

Good catch and reproducer. As always within the subprocess state machine I had to read over a lot to convince myself of its behavior and what was supported. Your proposed fix PR is simple and safe from what I can tell. I've also updated the docs in it to mention the caveat about input being a one-time only parameter.

I think what happened over time here is a mix: Popen.communicate was not originally intended to be called more than once as there had been no reason to do so - the process was supposed to have died by the time it returned. But the 3.3 addition of timeout support provided an actual use case including internally keeping track of the original input buffer and continuing to write what was left of that to the pipe on later post-timeout communicate() calls.

Popen does not expose any intentional public API to indicate of how much of the communicate() input= has been written to the stdin pipe after a timeout exception. Later calls to communicate if they also supply an input parameter per my read of the code today simply have that new input ignored. I'm documenting that as explicitly undefined behavior for now as part of this PR. We could choose to change that to raise or to try and accumulate more input into the internal buffer in the future but that'd need to be a deliberate feature. I doubt we should do so (complexity).

Either way, this is somewhat of a niche use case as once you want to let a process run on its own with a timeout with pipe(s) of data in and or out of it all at once, you either need to be using threads or async / selectors to avoid anything that blocks. Stepping on your own via repeated timeouts can artificially limit how rapidly the pipe data is written to the process more than would otherwise be natural.