Bash: Running the same program over multiple cores

You could use

for f in *.fa; do    myProgram (options) "./$f" "./$f.tmp" &donewait

which would start all of you jobs in parallel, then wait until they all complete before moving on. In the case where you have more jobs than cores, you would start all of them and let your OS scheduler worry about swapping processes in an out.

One modification is to start 10 jobs at a time

count=0for f in *.fa; do    myProgram (options) "./$f" "./$f.tmp" &    (( count ++ ))            if (( count = 10 )); then        wait        count=0    fidone

but this is inferior to using parallel because you can't start new jobs as old ones finish, and you also can't detect if an older job finished before you manage to start 10 jobs. wait allows you to wait on a single particular process or all background processes, but doesn't let you know when any one of an arbitrary set of background processes complete.

With GNU Parallel you can do:

parallel myProgram (options) {} {.}.tmp ::: *.fa

From: http://git.savannah.gnu.org/cgit/parallel.git/tree/README

= Full installation =

Full installation of GNU Parallel is as simple as:

./configure && make && make install

If you are not root you can add ~/bin to your path and install in~/bin and ~/share:

./configure --prefix=$HOME && make && make install

Or if your system lacks 'make' you can simply copy src/parallelsrc/sem src/niceload src/sql to a dir in your path.

= Minimal installation =

If you just need parallel and do not have 'make' installed (maybe thesystem is old or Microsoft Windows):

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallelchmod 755 parallelcp parallel semmv parallel sem dir-in-your-$PATH/bin/

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

# Wait while instance count less than $3, run additional instance and exitfunction runParallel () {    cmd=$1    args=$2    number=$3    currNumber="1024"    while true ; do        currNumber=`ps -e | grep -v "grep" | grep " $1$" | wc -l`        if [ $currNumber -lt $number ] ; then            break        fi        sleep 1    done    echo "run: $cmd $args"    $cmd $args &}loop=0# We will run 12 sleep commands for 10 seconds each # and only five of them will work simultaneouslywhile [ $loop -ne 12 ] ; do    runParallel "sleep" 10 5    loop=`expr $loop + 1`done