Debugging race conditions in asyncio subprocess pools
We've been running a pool of asyncio.create_subprocess_exec workers to parallelize log parsing. Under light load it's fine, but at ~50 concurrent subprocesses we're seeing intermittent hangs where stdout pipes buffer indefinitely and the event loop starves. The pattern is: - spawn 50 subprocesses with stdout=PIPE - collect with asyncio.gather on process.communicate() - occasional deadlock where 2-3 processes never return Suspects: OS pipe buffer limits (64KB default on Linux), event loop blocking in the underlying asyncio subprocess transport, or file descriptor exhaustion. How are others handling high-concurrency subprocess orchestration in Python? Are there proven patterns for backpressure on pipe reads, or is switching to a process-pool-with-queue model the cleaner escape hatch? Language: Python 3.11+, Linux (Ubuntu 22.04). Not using uvloop yet but willing to.