Running commands in background from TCL script and formatting output

If the commands don't have state that depends on each other, you can parallelize them. There are many ways to do this, but one of the easier is to use the thread package's thread pooling (which requires a threaded Tcl, the norm on many platform nowadays):

package require Threadset pool [tpool::create -maxworkers 4]# The list of *scripts* to evaluateset tasks {    {command 1}    {command 2}    ...    {command n}}# Post the work items (scripts to run)foreach task $tasks {    lappend jobs [tpool::post $pool $task]}# Wait for all the jobs to finishfor {set running $jobs} {[llength $running]} {} {    tpool::wait $pool $running running}# Get the results; you might want a different way to print the results...foreach task $tasks job $jobs {    set jobResult [tpool::get $pool $job]    puts "TASK: $task"    puts "RESULT: $jobResult"}

The main tweakable is the size of the thread pool, which defaults to a limit of 4. (Set it via the -maxworkers option to tpool::create which I've listed explicitly above.) The best value to choose depends on how many CPU cores you've got and how much CPU load each task generates on average; you'll need to measure and tune…

You can also use the -initcmd option to pre-load each worker thread in the pool with a script of your choice. That's a good place to put your package require calls. The workers are all completely independent of each other and of the master thread; they do not share state. You'd get the same model if you ran each piece of code in a separate process (but then you'd end up writing more code to do the coordinating).

[EDIT]: Here's a version that will work with Tcl 8.4 and which uses subprocesses instead.

namespace eval background {}proc background::task {script callback} {    set f [open |[list [info nameofexecutable]] "r+"]    fconfigure $f -buffering line    puts $f [list set script $script]    puts $f {fconfigure stdout -buffering line}    puts $f {puts [list [catch $script msg] $msg]; exit}    fileevent $f readable [list background::handle $f $script $callback]}proc background::handle {f script callback} {    foreach {code msg} [read $f] break    catch {close $f}    uplevel "#0" $callback [list $script $code $msg]}proc accumulate {script code msg} {    puts "#### COMMANDS\n$script"    puts "#### CODE\n$code"    puts "#### RESULT\n$msg"    # Some simple code to collect the results    if {[llength [lappend ::accumulator $msg]] == 3} {        set ::done yes    }}foreach task {    {after 1000;subst hi1}    {after 2000;subst hi2}    {after 3000;subst hi3}} {    background::task $task accumulate}puts "WAITING FOR TASKS..."vwait done

Notes: the tasks are Tcl commands that produce a result, but they must not print the result out; the fabric code (in background::task) handles that. These are subprocesses; they share nothing with one another, so anything you want them to do or be configured with must be sent as part of the task. A more sophisticated version could keep a hot pool of subprocesses around and in general work very much like a thread pool (subject to the subtle differences due to being in a subprocess and not a thread) but that was more code than I wanted to write here.

Result codes (i.e., exception codes) are 0 for “ok”, 1 for “error”, and other values in less common cases. They're exactly the values documented on the Tcl 8.6 catch manual page; it's up to you to interpret them correctly. (I suppose I should also add code to make the ::errorInfo and ::errorCode variable contents be reported back in the case of an error, but that makes the code rather more complex…)