Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Crash with ForkManager on Windows

by amitsq (Beadle)
on Sep 20, 2017 at 15:34 UTC ( #1199760=perlquestion: print w/replies, xml ) Need Help??
amitsq has asked for the wisdom of the Perl Monks concerning the following question:

I wrote a Link Checker, to check which Links are dead on a webpage. To get Server informations like Status Code and last modified Date, I used the head function of LWP::simple. Now this takes a lot of time. To speed up the process I’d like to run the processes simultaneous. I used the modul ForkManager and I am very impressed. It’s easy to use and fasten up the program. But it is not reliable. I have run the script multiple times to see the crash happens unregulary. Sometimes after 9 times of executions, then again after the first time, while in between it works fine and correct. The question is why?

The Program crashes with a windows msg opening, saying “Perl Command Line Interpreter has been stop working”.

If I click on show further informations, following Informations get shown:
Problem signature: Problem Event Name: APPCRASH Application Name: perl.exe Application Version: 5.16.3.1604 Application Timestamp: 534c69c3 Fault Module Name: ntdll.dll Fault Module version: 6.1.7601.23864 Fault Module Timestamp: 595fa942 Exception Code: c0000022 Exception offset: 00000000000c8078 OS version: 6.1.7601.2.1.0.272.7 Locale ID: 1031 Additional Information 1: fdd8 Additional Information 2: fdd8457bc7039603b6dc9eb824e5dab9 Additional Information 3: 8044 Additional Information 4: 80447c7063d0a8ad48b87e45c4d37c4f
Though I assume this is an incompatibility problem with lwp::Simple head function, because if I wouldn’t include it in the parallising process, it doesn’t crash so far. But this is also the point in the script, which takes the most time in processing. So I wonder if there is an error in my script or any working alternative? ( I already tried HTTP::Async, which I couldn’t get work at all).


I use ActivePerl 64 bit (v5.16.3 built for MSWindows32-x64-multi-thread) on Windows Server 2008 R2 Standard.

#!d:\perl\bin\perl.exe use LWP::Simple; use strict; use warnings; use Parallel::ForkManager; my $pm= new Parallel::ForkManager(8); #didn’t matter if it’s 1 or 5 o +r whatever else .. crash gets caused anyways foreach my $url ( 'http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif', 'http://hooboy.no-such-host.int/', 'http://www.yahoo.com', 'http://www.ora.com/ask_tim/graphics/asktim_header_main.gif', 'http://www.guardian.co.uk/', 'http://www.pixunlimited.co.uk/siteheaders/Guardian.gif' ) { $pm->start and next; my ($type, $length, $mod) = head($url); unless (defined $type) { print "$url is done \n"; ## I obviously reduced the output msgs $pm->finish; next; } if ($mod) { print "$url is done \n"; $pm->finish; } else { print "$url is done \n"; $pm->finish; } $pm->finish; } $pm->wait_all_children;
Thanks for your time.

Replies are listed 'Best First'.
Re: Crash with ForkManager on Windows
by Discipulus (Monsignor) on Sep 20, 2017 at 21:07 UTC
    Hello amitsq,

    I'm not at all a parallel monk and here there are, very skilled.

    > my $pm= new Parallel::ForkManager(8); #didn’t matter if it’s 1 or 5 or whatever else

    your original program crash here too and in the same way, but with 1 does not.

    I have borrowed the below modifying it a little from example from P::FM docs and the resulting code runs with no crashes. At least something to start with:

    use Parallel::ForkManager; use LWP::Simple; my $pm = Parallel::ForkManager->new(10); LINKS: for my $link ('http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif', 'http://hooboy.no-such-host.int/', 'http://www.yahoo.com', 'http://www.ora.com/ask_tim/graphics/asktim_header_main.gif', 'http://www.guardian.co.uk/', 'http://www.pixunlimited.co.uk/siteheaders/Guardian.gif') { $pm->start and next LINKS; my ($content_type, $document_length, $modified_time, $expires, $ +server) = head($link); print "$link fetched: declared lenght $document_length, \n"; $pm->finish; }; # output http://hooboy.no-such-host.int/ fetched: declared lenght , http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif fetched: declared leng +ht 5611, http://www.guardian.co.uk/ fetched: declared lenght 628397, http://www.ora.com/ask_tim/graphics/asktim_header_main.gif fetched: de +clared lenght , http://www.pixunlimited.co.uk/siteheaders/Guardian.gif fetched: declar +ed lenght , http://www.yahoo.com fetched: declared lenght 12,

    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

      i tried your reduced code, 100% copy - no change and it crashes. I might run it 20 times with no crashes, but then the crashes happened like before.

      Even once i got a very strange error message on the terminal:
      http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif 1354074654, Number found where operator expected at D:/Perl64/lib/File/Glob.pm lin +e 13, near "over 4" (Do you need to predeclare over?) Semicolon seems to be missing at D:/Perl64/lib/File/Glob.pm line 16. Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 17, near "<GLOB_NOSORT> is" (Missing operator before is?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 18, near ") rather" (Missing operator before rather?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 24, near ") returns" (Missing operator before returns?) Can't locate object method "_init_implementor" via package "URI::http" + at D:/Perl64/lib/URI.pm line 137. Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 25, near "&File::Glob::GLOB_ERROR will" (Missing operator before will?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 26, near "&File::Glob::GLOB_ERROR is" (Missing operator before is?) Number found where operator expected at D:/Perl64/lib/File/Glob.pm lin +e 29, near "over 4" (Do you need to predeclare over?) Semicolon seems to be missing at D:/Perl64/lib/File/Glob.pm line 32. Semicolon seems to be missing at D:/Perl64/lib/File/Glob.pm line 36. Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 41, near ") has" (Missing operator before has?) Operator or semicolon missing before &File::Glob::ERROR at D:/Perl64/l +ib/File/Glob.pm line 43. Ambiguous use of & resolved as operator & at D:/Perl64/lib/File/Glob.p +m line 43. Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 45, near ") deviates" (Missing operator before deviates?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 45, near ") behaviour" (Missing operator before behaviour?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 46, near ") will" (Missing operator before will?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 47, near "continue processing" (Do you need to predeclare continue?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 47, near "<GLOB_ERR> flag" (Missing operator before flag?) Number found where operator expected at D:/Perl64/lib/File/Glob.pm lin +e 54, near "over 4" (Do you need to predeclare over?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 60, near ") isn't" (Missing operator before isn't?) Semicolon seems to be missing at D:/Perl64/lib/File/Glob.pm line 66. Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 67, near "use of backslash" (Do you need to predeclare use?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 68, near ") solution" (Missing operator before solution?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 68, near "use forward slashes" (Do you need to predeclare use?) String found where operator expected at D:/Perl64/lib/File/Glob.pm lin +e 70, near "match "normal practice"" (Do you need to predeclare match?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 70, near ""normal practice" on" (Missing operator before on?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 71, near ") only" (Missing operator before only?) String found where operator expected at D:/Perl64/lib/File/Glob.pm lin +e 72, near "metacharacters '['" (Do you need to predeclare metacharacters?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 77, near "use the real" (Do you need to predeclare use?) Array found where operator expected at D:/Perl64/lib/File/Glob.pm line + 89, at end of line Semicolon seems to be missing at D:/Perl64/lib/File/Glob.pm line 90. Array found where operator expected at D:/Perl64/lib/File/Glob.pm line + 91, at end of line Array found where operator expected at D:/Perl64/lib/File/Glob.pm line + 92, at end of line Array found where operator expected at D:/Perl64/lib/File/Glob.pm line + 93, at end of line Semicolon seems to be missing at D:/Perl64/lib/File/Glob.pm line 93. Number found where operator expected at D:/Perl64/lib/File/Glob.pm lin +e 96, near ") 1989" (Missing operator before 1989?) Bareword found where operator expected at D:/Perl64/lib/File/Glob.pm l +ine 96, near "1993 The"
      No idea where that pop up from. It only happened once in all the times trying.

        When seeing errors (Bareword found where operator expected), try pre-loading the module before running parallel. See, this post.

        use LWP::Simple; # On the Windows platform, pre-loading essential modules # (required at runtime) may be necessary for extra stability. if ( $INC{'LWP/UserAgent.pm'} && !$INC{'Net/HTTP.pm'} ) { require IO::Handle; require Net::HTTP; require Net::HTTPS; } # run parallel ...
Re: Crash with ForkManager on Windows
by kcott (Chancellor) on Sep 21, 2017 at 08:45 UTC

    G'day amitsq,

    I suspect your problems are related to the additional $pm->finish and next statements.

    Your instantiation of the Parallel::ForkManager object may also be an issue. What you have is different from every example shown in the documentation; also see "perlobj: Invoking Class Methods" (including its "Indirect Object Syntax" subsection).

    Using exactly the same message to identify your different cases is pointless: use meaningful messages.

    I rewrote your code like this (pm_1199760_parallel_forkmgr_link_checker.pl):

    #!/usr/bin/env perl -l use strict; use warnings; use LWP::Simple; use Parallel::ForkManager; my @urls = qw{ http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif http://hooboy.no-such-host.int/ http://www.yahoo.com http://www.ora.com/ask_tim/graphics/asktim_header_main.gif http://www.guardian.co.uk/ http://www.pixunlimited.co.uk/siteheaders/Guardian.gif }; my $max_processes = $ARGV[0] || 8; my $pm = Parallel::ForkManager::->new($max_processes); print 'Start link checking.'; for my $url (@urls) { $pm->start and next; my ($type, undef, $mod) = head($url); if (defined $type) { print "PASS: $url"; print ' MOD: ', $mod || 0; } else { print "FAIL: $url"; } $pm->finish; } $pm->wait_all_children; print 'Link checking completed.';

    Here's the output with no argument:

    $ pm_1199760_parallel_forkmgr_link_checker.pl Start link checking. FAIL: http://hooboy.no-such-host.int/ PASS: http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif MOD: 1354074654 PASS: http://www.guardian.co.uk/ MOD: 0 FAIL: http://www.ora.com/ask_tim/graphics/asktim_header_main.gif PASS: http://www.yahoo.com MOD: 0 FAIL: http://www.pixunlimited.co.uk/siteheaders/Guardian.gif Link checking completed.

    I then ran it with arguments of 8, 5, 3, 2 and 1. Lke the first run (with no argument), all ran to completion: no crashes! With the exception of 1, they produced the same results (although the order of output varied).

    Running with an argument of 1 rather defeats the purpose of running processes in parallel; however, as you said you'd used it, I tried it also. Here's what I got:

    $ pm_1199760_parallel_forkmgr_link_checker.pl 1 Start link checking. PASS: http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif MOD: 1354074654 FAIL: http://hooboy.no-such-host.int/ PASS: http://www.yahoo.com MOD: 0 FAIL: http://www.ora.com/ask_tim/graphics/asktim_header_main.gif FAIL: http://www.guardian.co.uk/ FAIL: http://www.pixunlimited.co.uk/siteheaders/Guardian.gif Link checking completed.

    I ran that a second time: the result was the same. I'll leave you to investigate that further.

    — Ken

      Hello kcott,

      where are you running your code?

      Unfortunately on Perl (v5.14.2) built for MSWin32-x86-multi-thread on win7 and Parallel::ForkManager 1.19 I get more than 50% of crashes, using your code.

      Testing more times mine, as posted above, I get ~25% of crashes.. (not saying mine is better: better is no crash at all ;=)

      Unfortunately windows is a shaggy beast and I'm not able to debug this behaviour further: what I can add is that during installation I got some warning about disappeared PIDs even if the overall testing process ended ok:

      t/00-report-prereqs.t ..... ok t/01-utf8-all.t ........... skipped: This is a bug in perl itself on W +indows t/02-callback.t ........... # This test can take 10-20 seconds, please + wait. Started at Thu Sep 21 11:23:08 2017 t/02-callback.t ........... ok t/03-callback-data.t ...... # This test can take 2-6 seconds, please w +ait. Started at Thu Sep 21 11:23:25 2017 t/03-callback-data.t ...... ok t/basic-methods.t ......... ok t/changing-pids.t ......... ok t/start_child.t ........... ok t/waitpid-conflict.t ...... # code outside of P::FM stole -1152 child process '-1152' disappeared. A call to `waitpid` outside of Para +llel::ForkManager might have reaped it. t/waitpid-conflict.t ...... ok t/waitpid-waitonechild.t .. # code outside of P::FM stole -5456 # found warning: child process '-5456' disappeared. A call to `waitpid +` outside of Parallel::ForkManager might have reap ed it. t/waitpid-waitonechild.t .. 1/? # expected to find warning: (?^:child +process '\d+' disappeared. A call to `waitpid` out side of Parallel::ForkManager might have reaped it\.) t/waitpid-waitonechild.t .. ok t/waitpid_blocking.t ...... ok All tests successful.

      L*

      PS modified the title a bit for better later search

      There are no rules, there are no thumbs..
      Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

        G'day Discipulus,

        Thanks for the follow-up. While it's clear the OP was using Windows, I wasn't aware that this was a Windows-specific problem: I don't see anything like "This worked on XYZ, but not on Windows".

        I'm using:

        Perl 5.26.0 (built for darwin-thread-multi-2level) Parallel::ForkManager 1.19 LWP::Simple 6.26

        I had to install Parallel::ForkManager and the cpan output is still on one of my screens. Here's my equivalent lines to those you posted:

        t/00-report-prereqs.t ..... ok t/01-utf8-all.t ........... skipped: Need utf8::all for this test cras +hing on Windows # This test can take 10-20 seconds, please wait. Started at Thu Sep 21 + 17:07:29 2017 t/02-callback.t ........... ok # This test can take 2-6 seconds, please wait. Started at Thu Sep 21 1 +7:07:45 2017 t/03-callback-data.t ...... ok t/basic-methods.t ......... ok t/changing-pids.t ......... ok t/start_child.t ........... ok # code outside of P::FM stole 2369 child process '2369' disappeared. A call to `waitpid` outside of Paral +lel::ForkManager might have reaped it. t/waitpid-conflict.t ...... ok # code outside of P::FM stole 2371 t/waitpid-waitonechild.t .. ok t/waitpid_blocking.t ...... ok All tests successful.

        You appear to have had problems with t/waitpid-waitonechild.t that I didn't have; everything else looks fairly similar.

        To be honest, I started the installation, went and made a coffee, came back and saw:

        YANICK/Parallel-ForkManager-1.19.tar.gz /usr/bin/make install -- OK cpan[2]>

        I rarely review the (often many screenfuls of) output unless the installation failed.

        — Ken

      Taking your code, i get always the same output regardless which param i take. But it crashes too.

        Feedback, so far, is showing crashes on Windows Perl versions 5.14 and 5.16; however, 5.24 on Windows did not crash.

        "CPAN Testers Matrix: Parallel-ForkManager 1.19" is showing failures (for "mswin32") up to 5.22.3, but no failures for 5.24.0 or higher. There's virtually no failures for any other platforms: two very old "linux" ones for 5.8.x; and one other for "cygwin" 5.22.2 (which I believe only runs on Windows anyway).

        You should probably dig a bit deeper into this than I have, but problems do not appear to be solely related to ActivePerl; while looking around I came across this GitHub bug report: "Parallel::Forkmanager 1.19 cpan install fails on Windows Server 2016 Strawberry Perl 5.16.3". (Don't take that one report as being in any way conclusive.)

        Obviously, installation failures are not the same as crashes, but a trend is suggested. Again, something for you to look into further.

        I'd suggest installing a newer Perl (at least 5.24.0; but 5.26.0 — the highest stable version — would be better) and trying the code again with that. Also ensure you've got the latest versions of the modules you're using.

        — Ken

Re: Crash with ForkManager on Windows
by holli (Monsignor) on Sep 20, 2017 at 19:57 UTC
    Try wrapping the relevant parts of your code in try/catch blocks and see if it is dying somewhere unexpectedly.


    holli

    You can lead your users to water, but alas, you cannot drown them.
      ok i did that. Can't see why value shouldn't be defined though :/
      Can't call method "start" on an undefined value at line 18. Can't call method "finish" on an undefined value at .\multithreadsTe\3 +.pl line 25. </code which basically are following lines in my code: <code> line 18: $pm->start and next LINKS; line 25: $pm->finish;
      my modified code again:
      #!d:\perl\bin\perl.exe use Parallel::ForkManager; use LWP::Simple; use Try::Tiny; try{ my $pm = Parallel::ForkManager->new(6);} catch {print "error 1 \n"; exit;}; LINKS: for my $link ('http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif', 'http://hooboy.no-such-host.int/', 'http://www.yahoo.com', 'http://www.ora.com/ask_tim/graphics/asktim_header_main.gif', 'http://www.guardian.co.uk/', 'http://www.pixunlimited.co.uk/siteheaders/Guardian.gif') { try { $pm->start and next LINKS; } catch { warn $_; print "### ", $link, " \n error 2 \n"; next;}; try{ my ($content_type, $document_length, $modified_time, $expires, $ +server) = head($link); print "$link fetched: declared lenght $document_length, \n"; } catch{print "error 3 \n";}; try{$pm->finish; } catch{ warn $_; print "error 4 \n";}; };
        Can't see why value shouldn't be defined though
        Because you defined $pm lexically in the try - block and you don't have strict on.

        Edit:
        I just ran your code with the issue fixed but experience no crashes (Win 8.1, ActivePerl 5.24). I think there is something broken with your installation of the module and or Perl itself. I suggest a clean reinstall of both.


        holli

        You can lead your users to water, but alas, you cannot drown them.
        By wrapping the
        my $pm = Parallel::ForkManager->new(6);
        into a try block, $pm now is local to that new block … I think strict would have told you about it.
Re: Crash with ForkManager on Windows
by karlgoethebier (Monsignor) on Sep 22, 2017 at 05:06 UTC

    You might also take a look at Yet another example to get URLs in parallel. Perhaps something like this works for you. You can simplify it for your needs (no role etc.). I vaguely remember that i tried something like this with LWP::Simple and it didn't work. I forgot why ;-)

    Hope this helps. Regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Re: Crash with ForkManager on Windows -- further readings
by Discipulus (Monsignor) on Sep 21, 2017 at 22:19 UTC
Re: Crash with ForkManager on Windows
by marioroy (Priest) on Sep 22, 2017 at 08:56 UTC

    Update: Modified script. Completes in 1 second with Cygwin Perl v5.22.4 and less than 1 second on Unix.

    Update: Modified list of modules to pre-load. On the Windows platform, run with Perl v5.26 or later for best results.

    Running parallel is problematic and not fun on the Windows platform. To increase the chance for failure, I quadrupled the input size.

    # Strawberry Perl on Windows 7 VM # LWP::Simple included with Perl # Parallel::ForkManager v1.19 # Testing involves running multiple times. # Failing indicates the script crashed one or more times. perl-5.10.1.2 - LWP::Simple v5.827 : pass, slow (> 14 seconds) perl-5.12.3.0 - LWP::Simple v5.835 : pass, slow (> 14 seconds) perl-5.14.4.1 - LWP::Simple v6.00 : fail perl-5.16.3.1 - LWP::Simple v6.00 : fail perl-5.18.4.1 - LWP::Simple v6.00 : fail perl-5.20.3.3 - LWP::Simple v6.15 : fail perl-5.22.3.1 - LWP::Simple v6.15 : fail perl-5.24.2.1 - LWP::Simple v6.26 : fail perl-5.26.0.2 - LWP::Simple v6.26 : pass, fast (~ 3 seconds)

    A solution is pre-loading essential modules (required at runtime) by LWP::Simple before running parallel.

    use strict; use warnings; use LWP::Simple; # Pre-load essential modules for extra stability. if ( $INC{'LWP/UserAgent.pm'} && !$INC{'Net/HTTP.pm'} ) { require IO::Handle; require Net::HTTP; require Net::HTTPS; } my @urls = ( 'http://hooboy.no-such-host.int/', 'http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif', 'http://www.guardian.co.uk/', 'http://www.ora.com/ask_tim/graphics/asktim_header_main.gif', 'http://www.pixunlimited.co.uk/siteheaders/Guardian.gif', 'http://www.yahoo.com', ) x 4; use Parallel::ForkManager; my $pm = new Parallel::ForkManager(8); if ( $^O ne 'MSWin32' ) { $pm->set_waitpid_blocking_sleep(0); } foreach my $url ( @urls ) { $pm->start and next; my ($type, $length, $mod) = head($url); # if (!defined $type) { # ... # } # elsif ($mod) { # ... # } # else { # ... # } print "$url is done\n"; $pm->finish; } $pm->wait_all_children;

    Results.

    # Strawberry Perl on Windows 7 VM # LWP::Simple included with Perl # Parallel::ForkManager v1.19 # Testing involves running multiple times. perl-5.10.1.2 - LWP::Simple v5.827 : pass, > 14 seconds perl-5.12.3.0 - LWP::Simple v5.835 : pass, > 14 seconds perl-5.14.4.1 - LWP::Simple v6.00 : pass, > 7 seconds perl-5.16.3.1 - LWP::Simple v6.00 : pass, > 7 seconds perl-5.18.4.1 - LWP::Simple v6.00 : pass, > 7 seconds perl-5.20.3.3 - LWP::Simple v6.15 : pass, > 6 seconds perl-5.22.3.1 - LWP::Simple v6.15 : pass, > 6 seconds perl-5.24.2.1 - LWP::Simple v6.26 : pass, > 6 seconds perl-5.26.0.2 - LWP::Simple v6.26 : pass, ~ 3 seconds perl v5.22.4 on Cygwin - LWP::Simple v6.27 : pass, 1 second ;-)

    Perl 5.26 provides the best performance, completing in 3 seconds.

    Regards, Mario

      Update: Modified script. Completes in 1 second with Cygwin Perl v5.22.4 and less than 1 second on Unix.

      Update: Modified list of modules to pre-load. On the Windows platform, run with Perl v5.26 or later for best results.

      The following does the same thing using MCE::Hobo. Pre-loading essential modules is necessary, even for Perl v5.26. The posix_exit option applies to UNIX-like OSes and is helpful in the event an underlying module or dependency isn't multi-process safe.

      use strict; use warnings; use LWP::Simple; # Pre-load essential modules for extra stability. if ( $INC{'LWP/UserAgent.pm'} && !$INC{'Net/HTTP.pm'} ) { require IO::Handle; require Net::HTTP; require Net::HTTPS; } my @urls = ( 'http://hooboy.no-such-host.int/', 'http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif', 'http://www.guardian.co.uk/', 'http://www.ora.com/ask_tim/graphics/asktim_header_main.gif', 'http://www.pixunlimited.co.uk/siteheaders/Guardian.gif', 'http://www.yahoo.com', ) x 4; use MCE::Hobo 1.831; MCE::Hobo->init( max_workers => 8, posix_exit => 1, ); foreach my $url ( @urls ) { mce_async { my ($type, $length, $mod) = head($url); print "$url is done\n"; }; } MCE::Hobo->waitall;

      Regards, Mario

        Update: Modified script. Completes in 1 second with Cygwin Perl v5.22.4 and less than 1 second on Unix.

        Update: Modified list of modules to pre-load. On the Windows platform, run with Perl v5.26 or later for best results.

        The following is a demonstration using MCE::Loop. The behaviour is similar to running a pool of workers. Workers request the manager process the next URL to process. The job_delay option gives each worker time to load any missing modules at runtime before another worker starts processing. The delay occurs one time by each worker.

        use strict; use warnings; use LWP::Simple; # Pre-load essential modules for extra stability. if ( $INC{'LWP/UserAgent.pm'} && !$INC{'Net/HTTP.pm'} ) { require IO::Handle; require Net::HTTP; require Net::HTTPS; } my @urls = ( 'http://hooboy.no-such-host.int/', 'http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif', 'http://www.guardian.co.uk/', 'http://www.ora.com/ask_tim/graphics/asktim_header_main.gif', 'http://www.pixunlimited.co.uk/siteheaders/Guardian.gif', 'http://www.yahoo.com', ) x 4; use MCE::Loop; MCE::Loop->init( max_workers => 8, chunk_size => 1, posix_exit => 1, ); mce_loop { my ($type, $length, $mod) = head( my $url = $_ ); print "$url is done\n"; } \@urls; MCE::Loop->finish;

        Regards, Mario

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1199760]
Approved by stevieb
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2018-05-21 19:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?