Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Can Perl write multiple files at the same time?

by westrock2000 (Beadle)
on Oct 20, 2014 at 13:33 UTC ( #1104438=perlquestion: print w/replies, xml ) Need Help??

westrock2000 has asked for the wisdom of the Perl Monks concerning the following question:

I have a bunch of USB drives attached to my computer that I use as backup. Currently files are written sequentially and it takes a while. Is there a way to tell Perl to write 6 files each to separate drives at the same time?
  • Comment on Can Perl write multiple files at the same time?

Replies are listed 'Best First'.
Re: Can Perl write multiple files at the same time?
by MidLifeXis (Monsignor) on Oct 20, 2014 at 15:40 UTC
Re: Can Perl write multiple files at the same time?
by johngg (Canon) on Oct 20, 2014 at 15:11 UTC

    Alternatively, have a look at fork and wait.

    Cheers,

    JohnGG

Re: Can Perl write multiple files at the same time?
by tune (Curate) on Oct 20, 2014 at 13:54 UTC
    You need to learn threads to achieve that.

    --
    tune

Re: Can Perl write multiple files at the same time?
by no_slogan (Deacon) on Oct 20, 2014 at 15:09 UTC
    But ithreads are now discouraged, which leaves you with fork. You might try opening all 6 files at once and writing one block at a time to each of them -- that might work ok, depending on how your os does buffering.

      But ithreads are now discouraged, which leaves you with fork. You might try opening all 6 files at once and writing one block at a time to each of them -- that might work ok, depending on how your os does buffering.

      No it doesn't, that just nonsense, don't spread it around Re^2: Splitting large array for threads.

        And who are you that we should take your word for it, even though you provide no reasoning behind your statement?
Re: Can Perl write multiple files at the same time? Yes of course, but...
by Discipulus (Abbot) on Oct 21, 2014 at 06:51 UTC
    as said you can but, if i understand well you are experiencing low performance writing to an USB drive: this is quite normal i dont think that multiple threads can resolve the slowness.

    A decent write speed can be around 4Mb/s (and up to 46Mb/s) but for large files. If you are writing many little files performance drop down drastically. Maybe multiple threads for small files can result in less speed then sequential write.

    You can find more numbers here and here too

    HtH
    L*


    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
      I assumed the OP was writing several different files to several different USB drives, in which case you might be able to get an improvement.
Re: Can Perl write multiple files at the same time?
by biohisham (Priest) on Oct 20, 2014 at 22:14 UTC

    You might as well give the module Many Core Engines (MCE) a go.



    A 4 year old monk
      The MCE 1.515 was moved to back pan some time back. Here is the URL.

      https://metacpan.org/pod/MCE

      There is another example Re^4: Using MCE to write to multiple files..

      The MCE->sendto method is available. A nice thing is that MCE caches the file handle, therefore only opening the file handle once during the run. The sendto method appends, thus the reason for unlink.

      Below, using the Flow model as input_data is not required to run.

      use MCE::Flow max_workers => 8; my $file1 = "/path/to/file1.txt"; my $file2 = "/path/to/file2.txt"; unlink $file1 if -e $file1; unlink $file2 if -e $file2; mce_flow sub { my $wid = MCE->wid; if (MCE->wid % 2 == 0) { MCE->sendto("file:$file1", "sunny day from $wid\n"); } else { MCE->sendto("file:$file2", "raining day from $wid\n"); } };

      Another way is via MCE->print, MCE->printf, MCE->say. MCE serializes data from workers to the manager process. The output is done by the manager process. These methods are beneficial when many workers write to same file simultaneously.

      use MCE::Loop chunk_size => 1, max_workers => 8; my @input = (100..199); open my $fh_1, '>', '/path/to/file1.txt'; open my $fh_2, '>', '/path/to/file2.txt'; mce_loop { my ($mce, $chunk_ref, $chunk_id) = @_; if ($chunk_id % 2 == 0) { MCE->say($fh_1, "id: $chunk_id: input $_"); } else { MCE->say($fh_2, "id: $chunk_id: input $_"); } } @input; close $fh_1; close $fh_2;

      In case the reader missed it, the syntax for mce_flow requires the sub in front of the opening brace unlike mce_loop which takes a block. The reason is that mce_flow can take many anonymous blocks; e.g. mce_flow sub { ... }, sub { ... }.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1104438]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (3)
As of 2019-06-19 03:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Is there a future for codeless software?



    Results (83 votes). Check out past polls.

    Notices?
    • (Sep 10, 2018 at 22:53 UTC) Welcome new users!