Can functions know if they are already multiprocessed in Python (joblib) Can functions know if they are already multiprocessed in Python (joblib) python python

Can functions know if they are already multiprocessed in Python (joblib)


Parallel in joblib should be able to sort these things out:
http://pydoc.net/Python/joblib/0.8.3-r1/joblib.parallel/

Two pieces from 0.8.3-r1:

# Set an environment variable to avoid infinite loopsos.environ[JOBLIB_SPAWNED_PROCESS] = '1'

Don't know why they go from a variable referring to the environmental, to the env. itself.. But as you can see. The feature is already implemented in joblib.

# We can now allow subprocesses againos.environ.pop('__JOBLIB_SPAWNED_PARALLEL__', 0)


Here you can select other versions, if that's more relevant:
http://pydoc.net/Python/joblib/0.8.3-r1/


The answer to the specific question is: I don't know of a ready-made utility.

A minimal(*) core refactoring would to be add a named parameter to your function currently creating child processes. The default parameter would be your current behavior, and an other value would switch to a behavior compatible with how you are running tests(**).

(*: there might be other, may be better, design alternatives to consider but we do not have enough information)(**: one may say that the introduction of a conditional behavior would require to test that as well, and we are back to square one...)


Check multiprocessing.current_process().daemon -- it will return True if the current process is a spawned one. (Answering own question)