Parallel processing from a command queue on Linux (bash, python, ruby... whatever)
On the shell, xargs can be used to queue parallel command processing. For example, for having always 3 sleeps in parallel, sleeping for 1 second each, and executing 10 sleeps in total do
echo {1..10} | xargs -d ' ' -n1 -P3 sh -c 'sleep 1s' _And it would sleep for 4 seconds in total. If you have a list of names, and want to pass the names to commands executed, again executing 3 commands in parallel, do
cat names | xargs -n1 -P3 process_nameWould execute the command process_name alice, process_name bob and so on.
I would imagine you could do this using make and the make -j xx command.
Perhaps a makefile like this
all : usera userb userc....usera: imapsync userauserb: imapsync userb....make -j 10 -f makefile
Parallel is made exatcly for this purpose.
cat userlist | parallel imapsyncOne of the beauties of Parallel compared to other solutions is that it makes sure output is not mixed. Doing traceroute in Parallel works fine for example:
(echo foss.org.my; echo www.debian.org; echo www.freenetproject.org) | parallel traceroute