Mixing threads, fork, and mutexes, what should I watch out for?

See pthread_atfork, especially RATIONALE section, for a discussion of problems related to fork in multithreaded environment. It also gives a hint on what supposed to be valid before and after fork within a child and a parent.

UPDATE: the RATIONALE section is non-normative, and it turned out to be in conflict with other parts of the standard. See this defect report by Dave Butenhof for more details.

Immediate exec after fork is supposed to be safe for any state of multithreaded program (that is, any threads holding any mutexes). As of the things possible between fork and exec, the situation is complicated:

The most important thing is that only one thread (that which called fork) is duplicated in the child process. Consequently, any mutex held by another thread at the moment of fork becomes locked forever. That is (assuming non-process-shared mutexes) its copy in the child process is locked forever, because there is no thread to unlock it.

Releasing mutex after fork is safe when it's possible, that is, if the forking thread owns the mutex in the first place. That's how pthread_atfork handlers usually work: locking mutexes before fork, unlocking in child and unlocking in parent.

As of acquiring a mutex that the process owned before fork (remember, we discuss a copy in the child's address space): if it was owned by a forking thread, it's recursive locking (works for PTHREAD_MUTEX_RECURSIVE); if it was owned by another thread, it remains locked forever and can't be reacquired.

By registering appropriate pthread_atfork handlers, third-party libraries can provide a guarantee of being safe to use between fork and exec. (I would expect it mostly from programming language runtimes, not for general purpose libraries).

After some more research, I would recommend to avoid relying in any way on pthread_atfork, and doing nothing but async-signal-safe calls between fork and exec (abandoning fork/exec for posix_spawn would be even better).

The problem is, fork itself can be invoked in signal handler. It precludes any nontrivial use of pthread_atfork, even if its RATIONALE explicitly mentions unlocking mutexes and recreating threads (!) in a child process.

I think that a "grey area" of different possible interpretations remains:

1. For pthread_atfork handlers in a program which is known to never call fork in a signal handler.
2. For non-pthread-atfork actions happening around the fork call which is not in a signal handler.

But it's crystal clear which reading is to be used for portable applications.

It is safe to exec() after fork() provided the mutex is owned by the program that will get replaced by the exec(). If the mutex is part of a library and is protecting a resource that must be accessed serially then it should call pthread_atfork() to register callbacks:

1. A function that gets called prior to the fork itself and in the context of the thread that makes the fork() system call. Typically this function will grab the lock on mutexes that protect critical sections, thereby guaranteeing that during the fork no thread is inside a critical section
2. A function to be called in the context of the thread invoking fork(), after the child process has been created but before the fork() system call returns. This function can then unlock the mutexes in the parent process.
3. A function to be called in the context of the thread of the child process if/when the process forks - it gets called after the child process has been created but before the fork() system call returns. The child process can then unlock its copy of the mutex.

The POSIX standard limits the type of system calls that are allowed after a fork() and before an exec() to so called async-signal-safe system calls. Creating a thread isn't explicitly listed as an async-signal-safe system call, therefore POSIX doesn't allow a child process to create threads after a fork() and before exec(). Ironically, unlocking a mutex isn't explicitly listed as an async-signal-safe system call either, and thereofre isn't strictly allowed after a fork() in the child process if the intention of the fork() is to then exec() - probably an oversight in the standard.