Re: Can Perl write multiple files at the same time?

Replies are listed 'Best First'.

Re^2: Can Perl write multiple files at the same time?
by marioroy (Prior) on Dec 13, 2014 at 02:39 UTC

There is another example Re^4: Using MCE to write to multiple files..

The MCE->sendto method is available. A nice thing is that MCE caches the file handle, therefore only opening the file handle once during the run. The sendto method appends, thus the reason for unlink.

Below, using the Flow model as input_data is not required to run.

   use MCE::Flow max_workers => 8;

   my $file1 = "/path/to/file1.txt";
   my $file2 = "/path/to/file2.txt";

   unlink $file1 if -e $file1;
   unlink $file2 if -e $file2;

   mce_flow sub {
      my $wid = MCE->wid;

      if (MCE->wid % 2 == 0) {
         MCE->sendto("file:$file1", "sunny day from $wid\n");
      } else {
         MCE->sendto("file:$file2", "raining day from $wid\n");
      }
   };
[download]

Another way is via MCE->print, MCE->printf, MCE->say. MCE serializes data from workers to the manager process. The output is done by the manager process. These methods are beneficial when many workers write to same file simultaneously.

   use MCE::Loop chunk_size => 1, max_workers => 8;

   my @input = (100..199);

   open my $fh_1, '>', '/path/to/file1.txt';
   open my $fh_2, '>', '/path/to/file2.txt';

   mce_loop {
      my ($mce, $chunk_ref, $chunk_id) = @_;

      if ($chunk_id % 2 == 0) {
         MCE->say($fh_1, "id: $chunk_id: input $_");
      } else {
         MCE->say($fh_2, "id: $chunk_id: input $_");
      }

   } @input;

   close $fh_1;
   close $fh_2;
[download]

In case the reader missed it, the syntax for mce_flow requires the sub in front of the opening brace unlike mce_loop which takes a block. The reason is that mce_flow can take many anonymous blocks; e.g. mce_flow sub { ... }, sub { ... }.

[reply]
[d/l]
[select]


There's more than one way to do things
	PerlMonks