Python multiprocessing - AssertionError: can only join a child process

I can change the old code to not use os.fork() but I'd also like to know why this caused this problem and if there's a workable solution.

The key to understanding the problem is knowing exactly what fork() does. CPython docs state "Fork a child process." but this presumes you understand the C library call fork().

Here's what glibc's manpage says about it:

fork() creates a new process by duplicating the calling process. The new process, referred to as the child, is an exact duplicate of the calling process, referred to as the parent, except for the following points: ...

It's basically as if you took your program and made a copy of its program state (heap, stack, instruction pointer, etc) with small differences and let it execute independent of the original. When this child process exits naturally, it will use exit() and that will trigger atexit() handlers registered by the multiprocessing module.

What can you do to avoid it?

• omit os.fork(): use multiprocessing instead, like you are exploring now
• probably effective: import multiprocessing after executing fork(), only in the child or parent as necessary.
• use _exit() in the child (CPython docs state, "Note The standard way to exit is sys.exit(n). _exit() should normally only be used in the child process after a fork().")

https://docs.python.org/2/library/os.html#os._exit

It seems to me that you are threading it one time too many. I would not thread it from run_in_parallel, but simply calling run_server_client with the proper arguments, because they will thread inside.