Forking / Multi-Threaded Processes | Bash

In bash scripts (non-interactive) by default JOB CONTROL is disabled so you can't do the the commands: job, fg, and bg.

Here is what works well for me:

#!/bin/shset -m # Enable Job Controlfor i in `seq 30`; do # start 30 jobs in parallel  sleep 3 &done# Wait for all parallel jobs to finishwhile [ 1 ]; do fg 2> /dev/null; [ $? == 1 ] && break; done

The last line uses "fg" to bring a background job into the foreground. It does this in a loop until fg returns 1 ($? == 1), which it does when there are no longer any more background jobs.

I don't know of any explicit fork call in bash. What you probably want to do is append &to a command that you want to run in the background. You can also use & on functions that you define within a bash script:

do_something_with_line(){  line=$1  foo  foo2  foo3}for line in filedo  do_something_with_line $line &done

EDIT: to put a limit on the number of simultaneous background processes, you could try something like this:

for line in filedo  while [`jobs | wc -l` -ge 50 ]  do    sleep 5  done  do_something_with_line $line &done

I don't like using wait because it gets blocked until the process exits, which is not ideal when there are multiple process to wait on as I can't get a status update until the current process is done. I prefer to use a combination of kill -0 and sleep to this.

Given an array of pids to wait on, I use the below waitPids() function to get a continuous feedback on what pids are still pending to finish.

declare -a pidswaitPids() {    while [ ${#pids[@]} -ne 0 ]; do        echo "Waiting for pids: ${pids[@]}"        local range=$(eval echo {0..$((${#pids[@]}-1))})        local i        for i in $range; do            if ! kill -0 ${pids[$i]} 2> /dev/null; then                echo "Done -- ${pids[$i]}"                unset pids[$i]            fi        done        pids=("${pids[@]}") # Expunge nulls created by unset.        sleep 1    done    echo "Done!"}

When I start a process in the background, I add its pid immediately to the pids array by using this below utility function:

addPid() {    local desc=$1    local pid=$2    echo "$desc -- $pid"    pids=(${pids[@]} $pid)}

Here is a sample that shows how to use:

for i in {2..5}; do    sleep $i &    addPid "Sleep for $i" $!donewaitPids

And here is how the feedback looks:

Sleep for 2 -- 36271Sleep for 3 -- 36272Sleep for 4 -- 36273Sleep for 5 -- 36274Waiting for pids: 36271 36272 36273 36274Waiting for pids: 36271 36272 36273 36274Waiting for pids: 36271 36272 36273 36274Done -- 36271Waiting for pids: 36272 36273 36274Done -- 36272Waiting for pids: 36273 36274Done -- 36273Waiting for pids: 36274Done -- 36274Done!