Parallel execution of shell processes Parallel execution of shell processes windows windows

Parallel execution of shell processes


Edit - I modified the script to optionally display the output of each process

Here is a native batch solution that reliably runs a list of commands in parallel, never launching more than n processes at a time.

It even has a mechanism built in to distribute the processes to specific CPUs or remote machines via PSEXEC, but I haven't tested that feature.

The trick to make this work is to START each command through a CMD process that redirects either stdout or an undefined handle to a lock file. The process will maintain an exclusive lock on the file until it terminates. It doesn't matter how the process terminates (normal exit, crash, killed process), the lock will be released as soon as it does.

The master script can test if the process is still active by attempting to redirect to the same lock file. The redirection will fail if the process is still active, succeed if it has terminated.

By default, the script ignores the output of each process. If started with the /O option as the 1st parameter, then it displays the output of each process, without interleaving.

My demo sets the process limit to 4, and simply runs a series of PING commands of varying length.

I've tested this on XP, Vista, and Windows 7.

@echo offsetlocal enableDelayedExpansion:: Display the output of each process if the /O option is used:: else ignore the output of each processif /i "%~1" equ "/O" (  set "lockHandle=1"  set "showOutput=1") else (  set "lockHandle=1^>nul 9"  set "showOutput="):: The list of commands could come from anywhere such as another file:: or the output of another command. For this demo I will list the:: commands within this script - Each command is prefixed with :::::: ping /n 05 ::1::: ping /n 20 ::1::: ping /n 10 ::1::: ping /n 15 ::1::: ping /n 07 ::1::: ping /n 05 ::1::: ping /n 20 ::1::: ping /n 10 ::1::: ping /n 15 ::1::: ping /n 07 ::1:: Define the maximum number of parallel processes to run.:: Each process number can optionally be assigned to a particular server:: and/or cpu via psexec specs (untested).set "maxProc=4":: Optional - Define CPU targets in terms of PSEXEC specs::           (everything but the command):::: If a CPU is not defined for a proc, then it will be run on the local machine.:: I haven't tested this feature, but it seems like it should work.:::: set cpu1=psexec \\server1 ...:: set cpu2=psexec \\server1 ...:: set cpu3=psexec \\server2 ...:: etc.:: For this demo force all CPU specs to undefined (local machine)for /l %%N in (1 1 %maxProc%) do set "cpu%%N=":: Get a unique base lock name for this particular instantiation.:: Incorporate a timestamp from WMIC if possible, but don't fail if:: WMIC not available. Also incorporate a random number.  set "lock="  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (    set "lock=%%T"    goto :break  )  :break  set "lock=%temp%\lock%lock%_%random%_":: Initialize the counters  set /a "startCount=0, endCount=0":: Clear any existing end flags  for /l %%N in (1 1 %maxProc%) do set "endProc%%N=":: Launch the commands in a loop:: Modify the IN () clause as needed to retrieve the list of commands  set launch=1  for /f "tokens=* delims=:" %%A in ('findstr /b ":::" "%~f0"') do (    if !startCount! lss %maxProc% (      set /a "startCount+=1, nextProc=startCount"    ) else (      call :wait    )    set cmd!nextProc!=%%A    if defined showOutput echo -------------------------------------------------------------------------------    echo !time! - proc!nextProc!: starting %%A    2>nul del %lock%!nextProc!    %= Redirect the lock handle to the lock file. The CMD process will     =%    %= maintain an exclusive lock on the lock file until the process ends. =%    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! %%A  )  set "launch=":wait:: Wait for procs to finish in a loop:: If still launching then return as soon as a proc ends:: else wait for all procs to finish  :: redirect stderr to null to suppress any error message if redirection  :: within the loop fails.  for /l %%N in (1 1 %startCount%) do 2>nul (    %= Redirect an unused file handle to the lock file. If the process is    =%    %= still running then redirection will fail and the IF body will not run =%    if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (      %= Made it inside the IF body so the process must have finished =%      if defined showOutput echo ===============================================================================      echo !time! - proc%%N: finished !cmd%%N!      if defined showOutput type "%lock%%%N"      if defined launch (        set nextProc=%%N        exit /b      )      set /a "endCount+=1, endProc%%N=1"    )  )  if %endCount% lss %startCount% (    1>nul 2>nul ping /n 2 ::1    goto :wait  )2>nul del %lock%*if defined showOutput echo ===============================================================================echo Thats all folks^^!

Here is output from a sample run that ignores process output

12:24:07.52 - proc1: starting  ping /n 05 ::112:24:07.52 - proc2: starting  ping /n 20 ::112:24:07.53 - proc3: starting  ping /n 10 ::112:24:07.54 - proc4: starting  ping /n 15 ::112:24:11.60 - proc1: finished  ping /n 05 ::112:24:11.60 - proc1: starting  ping /n 07 ::112:24:16.66 - proc3: finished  ping /n 10 ::112:24:16.66 - proc3: starting  ping /n 05 ::112:24:17.68 - proc1: finished  ping /n 07 ::112:24:17.68 - proc1: starting  ping /n 20 ::112:24:20.72 - proc3: finished  ping /n 05 ::112:24:20.72 - proc3: starting  ping /n 10 ::112:24:21.75 - proc4: finished  ping /n 15 ::112:24:21.75 - proc4: starting  ping /n 15 ::112:24:26.82 - proc2: finished  ping /n 20 ::112:24:26.82 - proc2: starting  ping /n 07 ::112:24:29.86 - proc3: finished  ping /n 10 ::112:24:32.89 - proc2: finished  ping /n 07 ::112:24:35.92 - proc4: finished  ping /n 15 ::112:24:36.93 - proc1: finished  ping /n 20 ::1Thats all folks!

Here is the output if run with the /O option showing process output

-------------------------------------------------------------------------------12:24:51.02 - proc1: starting  ping /n 05 ::1-------------------------------------------------------------------------------12:24:51.02 - proc2: starting  ping /n 20 ::1-------------------------------------------------------------------------------12:24:51.03 - proc3: starting  ping /n 10 ::1-------------------------------------------------------------------------------12:24:51.04 - proc4: starting  ping /n 15 ::1===============================================================================12:24:55.10 - proc1: finished  ping /n 05 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms-------------------------------------------------------------------------------12:24:55.10 - proc1: starting  ping /n 07 ::1===============================================================================12:25:00.17 - proc3: finished  ping /n 10 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms-------------------------------------------------------------------------------12:25:00.19 - proc3: starting  ping /n 05 ::1===============================================================================12:25:01.22 - proc1: finished  ping /n 07 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 7, Received = 7, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms-------------------------------------------------------------------------------12:25:01.23 - proc1: starting  ping /n 20 ::1===============================================================================12:25:04.27 - proc3: finished  ping /n 05 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms-------------------------------------------------------------------------------12:25:04.28 - proc3: starting  ping /n 10 ::1===============================================================================12:25:05.30 - proc4: finished  ping /n 15 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms-------------------------------------------------------------------------------12:25:05.32 - proc4: starting  ping /n 15 ::1===============================================================================12:25:10.38 - proc2: finished  ping /n 20 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms-------------------------------------------------------------------------------12:25:10.40 - proc2: starting  ping /n 07 ::1===============================================================================12:25:13.44 - proc3: finished  ping /n 10 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms===============================================================================12:25:16.48 - proc2: finished  ping /n 07 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 7, Received = 7, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms===============================================================================12:25:19.52 - proc4: finished  ping /n 15 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms===============================================================================12:25:20.54 - proc1: finished  ping /n 20 ::1Pinging ::1 with 32 bytes of data:Reply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msReply from ::1: time<1msPing statistics for ::1:    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),Approximate round trip times in milli-seconds:    Minimum = 0ms, Maximum = 0ms, Average = 0ms===============================================================================Thats all folks!


Try start:

start "title of the process" "P:\ath\to.exe"

It opens a new window with the given title and executes the BAT, CMD or EXE file. You can also set the priority, set the same environment etc.

Files being not executeable are opened with the associated program.

Further reading: Start -> Run

cmd /k start /?

Start is available at least since WinME.

Good luck!


Sounds more like you want to use Powershell 2. However, you can spawn new cmd windows (or other processes) by using start, see also this answer. Although you probably have to use some other tools and a little trickery to create something like a "process pool" (to have only a maximum of n instances running at a time). You could achieve the latter by using tasklist /im and counting how many are already there (for loop or wc, if applicable) and simply wait (ping -n 2 ::1 >nul 2>&1) and re-check again whether you can spawn a new process.

I have cobbled together a little test batch for this:

@echo offfor /l %%i in (1,1,20) do call :loop %%igoto :eof:loopcall :checkinstancesif %INSTANCES% LSS 5 (    rem just a dummy program that waits instead of doing useful stuff    rem but suffices for now    echo Starting processing instance for %1    start /min wait.exe 5 sec    goto :eof)rem wait a second, can be adjusted with -w (-n 2 because the first ping returns immediately;rem otherwise just use an address that's unused and -n 1)echo Waiting for instances to close ...ping -n 2 ::1 >nul 2>&1rem jump back to see whether we can spawn a new process nowgoto loopgoto :eof:checkinstancesrem this could probably be done better. But INSTANCES should contain the number of running instances afterwards.for /f "usebackq" %%t in (`tasklist /fo csv /fi "imagename eq wait.exe"^|find /c /v ""`) do set INSTANCES=%%tgoto :eof

It spawns a maximum of four new processes that execute in parallel and minimized. Wait time needs to be adjusted probably, depending on how much each process does and how long it is running. You probably also need to adjust the process name for which tasklist is looking if you're doing something else.

There is no way to properly count the processes that are spawned by this batch, though. One way would be to create a random number at the start of the batch (%RANDOM%) and create a helper batch that does the processing (or spawns the processing program) but which can set its window title to a parameter:

@echo offtitle %1"%2" "%3"

This would be a simple batch that sets its title to the first parameter and then runs the second parameter with the third as argument. You can then filter in tasklist by selecting only processes with the specified window title (tasklist /fi "windowtitle eq ..."). This should work fairly reliable and prevents too many false positives. Searching for cmd.exe would be a bad idea if you still have some instances running, as that limits your pool of worker processes.

You can use %NUMBER_OF_PROCESSORS% to create a sensible default of how many instances to spawn.

You can also easily adapt this to use psexec to spawn the processes remotely (but wouldn't be very viable as you have to have admin privileges on the other machine as well as provide the password in the batch). You would have to use process names for filtering then, though.