Waiting in Python: wait(), communicate(), poll(), and the Deadlock Between Them

Q: How do I know whether thread.join() timed out?

Call is_alive() afterwards. join(timeout=...) returns None whether it succeeded or timed out, so there is no return value to inspect. There is also no way to kill a Python thread — use daemon=True or a shared flag the thread checks.

Use communicate() when you captured the output, wait() when you did not, and poll() when you need to check without blocking — and never use wait() with a pipe, because that is the deadlock everyone hits once.

Waiting is one of those operations that looks trivial until it hangs in production with no traceback and no CPU usage. The subprocess case is where most people meet it, so start there, then the thread, asyncio, and plain sleep versions of the same question.

The deadlock, first, because it is the important one
The three methods, and when each is right
Just use run() when you can
Timeouts, and cleaning up after one
Waiting on threads and futures
Waiting in async code
Why this shows up in production and not in testing
How this fits the rest of the stack
FAQ

The deadlock, first, because it is the important one

This code works on your test input and hangs forever on real data.

import subprocess

proc = subprocess.Popen(
    ["./generate-report"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)
proc.wait()                  # hangs
output = proc.stdout.read()  # never reached

The reason is buffer size. An OS pipe holds a limited amount of data — commonly 64KB. The child writes until the pipe is full, then blocks waiting for someone to read it. The parent is inside wait(), blocked until the child exits. Neither can move. The process tree sits there consuming nothing, which is why it does not look like a bug so much as a machine that stopped caring.

It works in testing because a small output fits in the buffer and the child exits before filling it. Cross that threshold and it hangs every time.

communicate() is the fix. It reads both pipes concurrently while waiting.

proc = subprocess.Popen(
    ["./generate-report"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
)
out, err = proc.communicate(timeout=300)
print(proc.returncode)

The rule is mechanical: if you passed PIPE, call communicate(). wait() is only safe when output goes to the terminal, a file, or DEVNULL.

The three methods, and when each is right

wait(timeout=None) — block until the process exits, return the exit code. Safe only without pipes.
communicate(input=None, timeout=None) — send stdin, read stdout and stderr to completion, wait for exit. Returns a tuple. The default choice when capturing output.
poll() — check whether the process has finished. Returns the exit code, or None if it is still running. Never blocks.

poll() is for when you want to do something else while waiting — update a progress display, service other work, check a shutdown flag.

import subprocess, time

proc = subprocess.Popen(["./long-job"], stdout=subprocess.DEVNULL)

while proc.poll() is None:
    time.sleep(0.5)          # the sleep is not optional
    report_progress()

print("finished with", proc.returncode)

That sleep matters. while proc.poll() is None: pass is a busy-wait that pins a CPU core to 100% doing nothing. It is a surprisingly common thing to find in a worker that someone described as “a bit heavy”.

Note the comparison is is None, not truthiness. A successful process returns exit code 0, which is falsy — while not proc.poll() exits immediately on failure and loops forever on success, which is exactly backwards.

Just use run() when you can

Most of the above is unnecessary for the common case. subprocess.run() handles the waiting, the pipe reading, and the error checking.

import subprocess

result = subprocess.run(
    ["git", "rev-parse", "HEAD"],
    capture_output=True,
    text=True,
    timeout=30,
    check=True,
)
print(result.stdout.strip())

check=True raises CalledProcessError on a non-zero exit, which is what you want almost every time. Without it, a failed command returns quietly and the script carries on with an empty string, producing an error somewhere unrelated.

timeout=30 is the one people leave off. Without it, run() waits forever, and a hung external command becomes a hung worker that no supervisor can distinguish from a busy one.

Reach for Popen only when you genuinely need the process running while you do something else. If you are going to wait for it anyway, run() is the same thing with fewer ways to get it wrong.

Timeouts, and cleaning up after one

A timeout raises, and the child is still running when it does. Killing it is your job.

import subprocess

proc = subprocess.Popen(["./flaky-tool"], stdout=subprocess.PIPE, text=True)
try:
    out, _ = proc.communicate(timeout=60)
except subprocess.TimeoutExpired:
    proc.kill()
    out, _ = proc.communicate()      # reap it, avoid a zombie
    raise RuntimeError("flaky-tool exceeded 60s")

The second communicate() after kill() is not redundant. It collects the exit status so the process is reaped rather than left as a zombie, and drains whatever was already in the pipes.

For a graceful stop, try terminate() first and escalate:

proc.terminate()                     # SIGTERM -- lets it clean up
try:
    proc.wait(timeout=10)
except subprocess.TimeoutExpired:
    proc.kill()                      # SIGKILL -- no choice given
    proc.wait()

One caveat worth knowing: kill() signals the process you started, not its children. If that process spawned its own, they survive as orphans. Killing the whole tree means starting it in a new process group with start_new_session=True and signalling the group with os.killpg.

Waiting on threads and futures

Threads use join(), which is the same idea with a different name.

import threading

t = threading.Thread(target=work, daemon=True)
t.start()
t.join(timeout=30)

if t.is_alive():
    print("still running -- join timed out")

The trap here: join(timeout=...) returns None whether it succeeded or timed out. There is no return value to check, so you must call is_alive() afterwards. Code that assumes a returned join means a finished thread is wrong, and it is wrong silently.

There is also no way to kill a Python thread. If it will not finish, your options are daemon=True so it dies with the process, or a shared flag the thread checks. Plan for that before you need it.

For pools, the futures API is cleaner:

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=4) as pool:
    futures = [pool.submit(fetch, url) for url in urls]
    for fut in as_completed(futures, timeout=120):
        result = fut.result()        # re-raises whatever the worker raised

as_completed yields results as they arrive rather than in submission order, so one slow item does not hold up the others. Exiting the with block waits for everything — which is convenient and worth knowing, because a return inside the block still blocks until the pool drains.

Waiting in async code

In asyncio, blocking calls are actively harmful — time.sleep() inside a coroutine stops the entire event loop, not just that task. Every other request your service is handling stops too.

import asyncio

async def main():
    await asyncio.sleep(1)                    # yields to the loop

    # wait for one thing with a limit
    async with asyncio.timeout(30):
        result = await fetch(url)

    # wait for several
    results = await asyncio.gather(*[fetch(u) for u in urls])

asyncio.timeout() (3.11 and later) cancels the operation when the limit passes. On older versions, asyncio.wait_for(coro, timeout=30) does the same job.

For a subprocess, use the asyncio version rather than blocking the loop on subprocess:

proc = await asyncio.create_subprocess_exec(
    "./report", stdout=asyncio.subprocess.PIPE,
)
out, _ = await proc.communicate()

gather fails fast by default — the first exception propagates while the rest keep running unobserved. Pass return_exceptions=True to collect them all instead, which is usually what you want when the tasks are independent.

Why this shows up in production and not in testing

Every failure above shares a shape: it depends on timing, output size, or an external system being slow, and none of those match on a laptop. The pipe deadlock needs more than 64KB of output. The missing timeout needs the external service to actually hang. The busy-wait needs a job long enough for someone to notice the CPU.

So the defensive version is not clever, it is just consistent. A timeout on every wait. check=True on every command whose failure matters. communicate() whenever a pipe is involved. is None rather than truthiness on poll().

The other half is being able to see it when it happens. A worker that hangs with no output is nearly undiagnosable; a worker that logs before starting a subprocess and after it returns tells you immediately which command it is stuck on. Services on RunxBuild stream stdout and stderr into runtime logs per deploy, so that log line is there when you go looking — and a hung deploy is visible as a build that stopped rather than one that failed.

How this fits the rest of the stack

A worker blocked on a subprocess still costs what a working one costs. Runtime, database, storage, and bandwidth each carry a number, and the RunxBuild hosting calculator puts them on one page so you can model the total before committing.

Useful related references:

FAQ

Why does subprocess.wait() hang?

Because you passed stdout=PIPE or stderr=PIPE. The OS pipe buffer fills, typically around 64KB, and the child blocks waiting for someone to read it while the parent blocks inside wait(). Use communicate(), which reads both pipes while waiting. It works in testing only because small output fits in the buffer.

What is the difference between wait, communicate, and poll?

wait() blocks until exit and is safe only without pipes. communicate() sends stdin, drains stdout and stderr, and waits — the right choice whenever you captured output. poll() checks whether the process finished and returns None if not, without ever blocking.

Why should I check poll() is None instead of just poll()?

A successful process exits with code 0, which is falsy. Writing while not proc.poll() loops forever on success and exits immediately on failure — the opposite of the intent. Always compare against None explicitly.

How do I kill a subprocess after a timeout?

Catch TimeoutExpired, call proc.kill(), then call communicate() again to reap the process and drain the pipes. For a graceful stop, try terminate() first and wait with a short timeout before escalating to kill(). Note that kill only signals the process you started, not any children it spawned.

How do I know whether thread.join() timed out?

Call is_alive() afterwards. join(timeout=…) returns None whether it succeeded or timed out, so there is no return value to inspect. There is also no way to kill a Python thread — use daemon=True or a shared flag the thread checks.