Powershell: Run multiple jobs in parralel and view streaming results from background jobs

You may take a look at the module SplitPipeline.It it specifically designed for such tasks. The working demo code is:

# import the module (not necessary in PS V3)Import-Module SplitPipeline# some servers (from 1 to 10 for the test)$servers = 1..10# process servers by parallel pipelines and output results immediately$servers | Split-Pipeline {process{"processing server $_"; sleep 1}} -Load 1, 1

For your task replace "processing server $_"; sleep 1 (simulates a slow job) with a call to your script and use the variable $_ as input, the current server.

If each job is not processor intensive then increase the parameter Count (the default is processor count) in order to improve performance.

Not a new question but I feel it is missing an answer including Powershell using workflows and its parallel possibilities, from powershell version 3. Which is less code and maybe more understandable than starting and waiting for jobs, which of course works good as well.

I have two files: TheScript.ps1 which coordinates the servers and BackgroundJob.ps1 which does some kind of check. They need to be in the same directory.

The Write-Output in the background job file writes to the same stream you see when starting TheScript.ps1.

TheScript.ps1:

workflow parallelCheckServer {    param ($Servers)    foreach -parallel($Server in $Servers)    {        Invoke-Expression -Command ".\BackgroundJob.ps1 -Server $Server"    }}parallelCheckServer -Servers @("host1.com", "host2.com", "host3.com")Write-Output "Done with all servers."

BackgroundJob.ps1 (for example):

param (    [Parameter(Mandatory=$true)] [string] $server)Write-Host "[$server]`t Processing server $server"Start-Sleep -Seconds 5

So when starting the TheScript.ps1 it will write "Processing server" 3 times but it will not wait for 15 seconds but instead 5 because they are run in parallel.

[host3.com]  Processing server host3.com[host2.com]  Processing server host2.com[host1.com]  Processing server host1.comDone with all servers.

In your ForEach loop you'll want to grab the output generated by the Jobs already running.

Example Not Tested

$sb = {     "Starting Job on $($args[0])"     #Do something     "$($args[0]) => Do something completed successfully"     "$($args[0]) => Now for something completely different"     "Ending Job on $($args[0])"}Foreach($computer in $computers){    Start-Job -ScriptBlock $sb -Args $computer | Out-Null    Get-Job | Receive-Job}

Now if you do this all your results will be mixed. You might want to put a stamp on your verbose output to tell which output came from.

Or

Foreach($computer in $computers){    Start-Job -ScriptBlock $sb -Args $computer | Out-Null    Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}}while((Get-Job -State Running).count){    Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}    start-sleep -seconds 1}

It will show all the output as soon as a job is finished. Without being mixed up.