Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Surprised by limitation of Win32 "system(1" hack

by eyepopslikeamosquito (Chancellor)
on Jun 24, 2015 at 05:26 UTC ( #1131761=perlquestion: print w/replies, xml ) Need Help??
eyepopslikeamosquito has asked for the wisdom of the Perl Monks concerning the following question:

On Windows, running this program:

use strict; use warnings; $| = 1; my $cmd = qq{$^X -le "print 'pid=', \$\$"}; for my $i (1..70) { print "$i: run $cmd ---------\n"; my $rc = system(1, $cmd); sleep(1); print "$i: done (rc=$rc).\n"; $rc < 0 and print "$i: ($!)\n"; }
surprised me by producing:

1: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=4600 1: done (rc=384). 2: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=3036 2: done (rc=4432). 3: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=7156 3: done (rc=6364). 4: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=5604 4: done (rc=4000). 5: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=5784 5: done (rc=2820). 6: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=1820 6: done (rc=6592). 7: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=6672 7: done (rc=6484). 8: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=6452 8: done (rc=4800). 9: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" -------- +- pid=3340 9: done (rc=6652). 10: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4620 10: done (rc=6524). 11: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4240 11: done (rc=224). 12: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=3200 12: done (rc=208). 13: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4120 13: done (rc=4112). 14: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=5324 14: done (rc=1312). 15: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=7032 15: done (rc=6272). 16: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4164 16: done (rc=6884). 17: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4504 17: done (rc=6404). 18: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6292 18: done (rc=6396). 19: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=748 19: done (rc=416). 20: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6804 20: done (rc=1296). 21: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6696 21: done (rc=5712). 22: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4028 22: done (rc=7148). 23: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=3220 23: done (rc=4852). 24: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6788 24: done (rc=92). 25: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6932 25: done (rc=3176). 26: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=960 26: done (rc=5332). 27: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4832 27: done (rc=6928). 28: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=1088 28: done (rc=4500). 29: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=3648 29: done (rc=6456). 30: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6756 30: done (rc=3568). 31: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=332 31: done (rc=1484). 32: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=5624 32: done (rc=5944). 33: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=3612 33: done (rc=6576). 34: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=828 34: done (rc=4408). 35: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=3116 35: done (rc=3172). 36: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4944 36: done (rc=5956). 37: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4752 37: done (rc=3592). 38: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6668 38: done (rc=2780). 39: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6612 39: done (rc=4128). 40: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=2500 40: done (rc=6760). 41: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6808 41: done (rc=4616). 42: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=5252 42: done (rc=2640). 43: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4396 43: done (rc=6368). 44: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=5088 44: done (rc=2604). 45: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6828 45: done (rc=6664). 46: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6012 46: done (rc=6680). 47: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6136 47: done (rc=2980). 48: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6676 48: done (rc=6556). 49: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=5764 49: done (rc=1412). 50: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6720 50: done (rc=3468). 51: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6908 51: done (rc=6156). 52: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=228 52: done (rc=2332). 53: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=5204 53: done (rc=4728). 54: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6236 54: done (rc=5952). 55: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6128 55: done (rc=6924). 56: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=5988 56: done (rc=3900). 57: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6656 57: done (rc=6604). 58: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6432 58: done (rc=32). 59: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=432 59: done (rc=6324). 60: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=2376 60: done (rc=7056). 61: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6384 61: done (rc=3432). 62: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=7152 62: done (rc=5652). 63: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=6764 63: done (rc=6480). 64: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- pid=4336 64: done (rc=5028). 65: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- 65: done (rc=-1). 65: (Resource temporarily unavailable) 66: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- 66: done (rc=-1). 66: (Resource temporarily unavailable) 67: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- 67: done (rc=-1). 67: (Resource temporarily unavailable) 68: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- 68: done (rc=-1). 68: (Resource temporarily unavailable) 69: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- 69: done (rc=-1). 69: (Resource temporarily unavailable) 70: run C:\Strawberry\perl\bin\perl.exe -le "print 'pid=', $$" ------- +-- 70: done (rc=-1). 70: (Resource temporarily unavailable)

That is, there appears to be a limit of 64 processes with the Win32 system(1 hack. Anyone know why? This is just a guess, but I remember that Win32 WaitForMultipleObjects has a limit of 64 handles.

Super search found Getting a return code from waitpid in Windows. I was using the "system(1" hack only out of laziness. I should be able to work around this limitation ... though if I need to go to the bother of calling waitpid etc. I will probably just bite the bullet and use Win32::Process instead.

Replies are listed 'Best First'.
Re: Surprised by limitation of Win32 "system(1" hack
by BrowserUk (Pope) on Jun 24, 2015 at 12:24 UTC
    though if I need to go to the bother of calling waitpid etc. I will probably just bite the bullet and use Win32::Process instead.

    Why? Why is "the bother of calling waitpid" so onerous?

    This just ran to completion in 10 seconds or so(Note:1000):

    use strict; use warnings; $| = 1; my $cmd = qq{$^X -le "print 'pid=', \$\$"}; for my $i ( 1 .. 1000 ) { print "$i: run $cmd ---------\n"; my $rc = system( 1, $cmd ); waitpid( $rc, 0 ); }

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
Re: Surprised by limitation of Win32 "system(1" hack
by Anonymous Monk on Jun 24, 2015 at 07:11 UTC
Re: Surprised by limitation of Win32 "system(1" hack
by sundialsvc4 (Abbot) on Jun 24, 2015 at 11:52 UTC

    I’d program the system so that it first put those 70-odd commands into a list, then launched a “configurable-n” number of children (yes, use system_detached ...), then waited for any one of them to complete before popping-off and starting the next command.   So that the number of workers alive at any moment never exceeds this n, no matter how many commands are on the list to be executed.

    (In the design I am thinking of, the children don’t themselves take work from this list.   The parent process uses it to create and launch the next command-to-do.)

    I think you’ll find that there is a “sweet spot” number of children that should be allowed to be active at one time, and that this number (found by experimentation ...) is likely to be relatively small.   The total time required, to do the total amount of work that is to be done, will be optimal at or around this point.   If, instead, you “just throw children at it,” the total amount of time might be much longer because the children are competing with each other (especially for I/O).   In the general case, the performance curve of such things becomes “an elbow” at the so-called thrash point.   It’s nice and linear up to that point, then it all goes sour exponentially fast.   Regulating the number of workers, independently of the amount of work-to-do, is a reliable and controllable way to prevent that.   Yeah, it means waitpid().

    Incidentally, on Unix/Linux systems, the trusty xargs command ordinarily has a -P number_of_children option which allows you to specify the size of a worker-pool.   On such systems, the need for this entire Perl script(!) might have been avoided.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1131761]
Approved by Athanasius
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (6)
As of 2017-11-21 01:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In order to be able to say "I know Perl", you must have:













    Results (294 votes). Check out past polls.

    Notices?