Multiprocessing Pipe send() blocks
Python Pipe
is an abstractions on top of OS nameless pipes.
An OS pipe is generally implemented as a memory buffer of a certain size within the kernel. By default, if the buffer fills up the next call to send/write will block.
If you want to be able to continue publishing data even if no consumer is consuming it, you should either use a multiprocessing.Queue
or asyncio
facilities.
The multiprocessing.Queue
employs a "limitless" buffer and a thread to push the data into the OS pipe. If the pipe gets full the caller will continue running as the published data will be piled up in the Queue
buffer.
IIRC, asyncio sets the pipe O_NONBLOCK
flag and waits for the pipe to be consumed. Additional messages are stored within a "limitless" buffer as for the multiprocessing.Queue
.