Is there a multiprocessing module for Perl?
forks provides the same awesome interface as threads, but uses processes instead of threads.
use forks; # Or: use threads;use Thread::Queue;my $q = Thread::Queue->new();my @workers;for (1..NUM_WORKERS) { push @workers, async { while (defined(my $job = $q->dequeue())) { ... } };}$q->enqueue(...);$q->enqueue(undef) for @workers;$_->join for @workers;Comparing forks with Forks::Super.
Keep in mind, these are suppose to the be the cases where Forks::Super excels!
use Forks::Super;sub do_something { my @args = @_; ... }$process = fork { sub => \&do_something, args => [@args] };$process->wait;can be written as
use forks;sub do_something { my @args = @_; ... }$process = async { do_something(@args) };$process->join;---
use Forks::Super;my $x = 42;my @y = ();my %z = ();sub do_something_else { $x = 19; @y = qw(foo bar); %z = (foo => 'bar');}$process = fork { sub => 'do_something_else', share => [\$x, \@y, \%z ] };$process->wait;can be written as
use forks;use forks::shared;my $x :shared = 42;my @y :shared = ();my %z :shared = ();sub do_something_else { $x = 19; @y = qw(foo bar); %z = (foo => 'bar');}$process = async { do_something_else() };$process->join;---
use Forks::Super;use IO::Handle;pipe my $child_read, my $parent_write;pipe my $parent_read, my $child_write;$parent_write->autoflush(1);$child_write->autoflush(1);sub square { while (my $x = <$child_read>) { chomp($x); print {$child_write} $x ** 2, "\n"; } close $child_write;}$process = fork { sub => 'square' };print { $parent_write } "9\n";chomp( my $result = <$parent_read> ); # 81close $parent_write;$process->wait;can be written as
use forks;use Threads::Queue;my $req = Threads::Queue->new();my $resp = Threads::Queue->new();sub square { $_[0] ** 2 }$process = async { while (defined(my $x = $req->dequeue())) { $resp->enqueue( square($x) ); }};$req->enqueue(9);my $result = $resp->dequeue(); # 81$resp->enqueue(undef);$process->join;---
use Forks::Super;sub square_root { sleep 1 && seek STDIN,0,1 while eof(STDIN); # ok, this is a workaround for an existing bug :-( while (my $x = <STDIN>) { chomp($x); print sqrt($x), "\n"; }}$process = fork { sub => 'square_root', child_fh => 'in,out,block' };$process->write_stdin("81\n");chomp( $result = $process->read_stdout() ); # 9$process->close_fh('stdin');$process->wait;can be written as
use forks;use Threads::Queue;my $req = Threads::Queue->new();my $resp = Threads::Queue->new();$process = async { while (defined(my $x = $req->dequeue())) { $resp->enqueue( sqrt($x) ); }};$req->enqueue(81);my $result = $resp->dequeue(); # 9$resp->enqueue(undef);$process->join;
I think Forks::Super comes pretty close. It has a few features for running an arbitrary subroutine (or external command) in a background process, monitoring and signalling the background process, and making interprocess communication a little less painful.
use Forks::Super;sub do_something { my @args = @_; ... }$process = fork { sub => \&do_something, args => [@args] };$process->wait;my $x = 42;my @y = ();my %z = ();sub do_something_else { $x = 19; @y = qw(foo bar); %z = (foo => 'bar');}$process = fork { sub => 'do_something_else', share => [\$x, \@y, \%z ] };$process->wait;# $x, @y, and %z are now updated with changes made in background process# create your own pipes to use for IPCuse IO::Handle;pipe my $child_read, my $parent_write;pipe my $parent_read, my $child_write;$parent_write->autoflush(1);$child_write->autoflush(1);sub square { while (my $x = <$child_read>) { print {$child_write} $x ** 2, "\n"; } close $child_write;}$process = fork { sub => 'square' };print {$parent_write} "9\n";my $result = <$parent_read>; # should be "81\n";close $parent_write;# or use the standard I/O handles for IPCsub square_root { sleep 1 && seek STDIN,0,1 while eof(STDIN); # ok, this is a workaround for an existing bug :-( while (my $x = <STDIN>) { print sqrt($x), "\n"; }}$process = fork { sub => 'square_root', child_fh => 'in,out,block' };$process->write_stdin("81\n");$result = $process->read_stdout(); # => "9\n"Both the multiprocessing module and Forks::Super have a lot of features. Which ones are you specifically interested in?
I am the author of Forks::Super and my goal is to include any features for parallel processing that people find useful, so if there's a feature in multiprocessing that you want in Perl, let me know.