Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Perl system command memory usage in threads

by rmahin (Scribe)
on Jul 31, 2015 at 18:24 UTC ( [id://1137039]=perlquestion: print w/replies, xml ) Need Help??

rmahin has asked for the wisdom of the Perl Monks concerning the following question:

SOLVED Updated to perl 5.20.2 and all problems vanished.

Hello! Been noticing some strange memory issues with a script I'm working on, and have come up with this to highlight the issues

use strict; use warnings; use threads; use threads::shared; $|++; my $DONE :shared = 0; my $lock :shared; my $execMethod = $ARGV[0] || 0; if($execMethod !~ /[12345]/){ print "Must pass an exec method:\n"; print "1 = backticks\n"; print "2 = backticks synchronized\n"; print "3 = open\n"; print "4 = open synchronized\n"; print "5 = system\n"; exit 1; } sub execute{ my $cmd = shift; if($execMethod == 1){ `$cmd` }elsif($execMethod == 2){ lock $lock; `$cmd` }elsif($execMethod == 3){ open(my $fs, "-|", $cmd); foreach(<$fs>){}; close $fs; }elsif($execMethod == 4){ lock $lock; open(my $fs, "-|", $cmd); foreach(<$fs>){}; close $fs; }elsif($execMethod == 5){ system($cmd . ">nul"); } } sub worker{ while(!$DONE){ execute('echo hello world'); } } my @workers = map threads->create( \&worker), (1..30); print "Press <enter> to terminate\n"; <STDIN>; $DONE = 1; $_->join() for @workers;

Script is invoked with <script>.pl executionMethod

Question 1 relates to memory usage. Methods 2, 4, and 5 all exhibit a permanent memory creep. Is there a way to fix this and let this script run forever? If the solution is periodically join threads, this is not feasible. I have tried this a couple years ago and came across some bug but outlined in this node and changing to an approach like this would require far too much testing to ensure no other bugs occur. If there really is no solution, would be good to know :)

Question 2: For methods 1 and 3, why do i need to synchronize them? If I do not, the script just hangs up usually after only a few seconds. Synchronizing this access kind of seems to defeat the purpose of running system commands in threads, is there something I can do differently?

*I have been running this on Windows 2008 R2, using perl 5.18.2

Thanks in advance for all your help

UPDATE 1: Here is the log file from perfmon running test 5 for a little over a minute and eating roughly 150MB.

UPDATE 2:

perl -v output

This is perl 5, version 18, subversion 1 (v5.18.1) built for MSWin32-x +64-multi-thread (with 1 registered patch, see perl -V for more detail) Copyright 1987-2013, Larry Wall Binary build 1800 [297570] provided by ActiveState http://www.ActiveSt +ate.com Built Sep 20 2013 15:07:17 Perl may be copied only under the terms of either the Artistic License + or the GNU General Public License, which may be found in the Perl 5 source ki +t. Complete documentation for Perl, including FAQ lists, should be found +on this system using "man perl" or "perldoc perl". If you have access to + the Internet, point your browser at http://www.perl.org/, the Perl Home Pa +ge.

systeminfo

Host Name: VCLOUD291 OS Name: Microsoft Windows Server 2008 R2 Enterprise OS Version: 6.1.7601 Service Pack 1 Build 7601 OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Server OS Build Type: Multiprocessor Free Registered Owner: Windows User Registered Organization: Product ID: 55041-507-5915375-84291 Original Install Date: 4/23/2014, 12:10:03 PM System Boot Time: 7/31/2015, 1:49:06 PM System Manufacturer: Microsoft Corporation System Model: Virtual Machine System Type: x64-based PC Processor(s): 1 Processor(s) Installed. [01]: Intel64 Family 6 Model 46 Stepping 6 +GenuineIntel ~1862 M hz BIOS Version: American Megatrends Inc. 090004 , 3/19/2009 Windows Directory: C:\Windows System Directory: C:\Windows\system32 Boot Device: \Device\HarddiskVolume1 System Locale: en-us;English (United States) Input Locale: en-us;English (United States) Time Zone: (UTC-07:00) Arizona Total Physical Memory: 4,096 MB Available Physical Memory: 1,635 MB Virtual Memory: Max Size: 6,143 MB Virtual Memory: Available: 3,364 MB Virtual Memory: In Use: 2,779 MB Page File Location(s): C:\pagefile.sys Domain: WORKGROUP Logon Server: \\VCLOUD291 Hotfix(s): 46 Hotfix(s) Installed.

Replies are listed 'Best First'.
Re: Perl system command memory usage in threads
by BrowserUk (Patriarch) on Jul 31, 2015 at 22:49 UTC
    Question 2: For methods 1 and 3, why do i need to synchronize them? If I do not, the script just hangs up usually after only a few seconds.

    I can confirm this problem. I believe it to be a bug in Perl. I think it is to do with the implementation of the wait/waitpid emulation on windows which limits the number of concurrent child processes to 64. I'll try to tie that down a bit further tomorrow and raise a perlbug for it if I succeed.

    Meanwhile, on the basis of what I've seen of your code, you should probably try MCE for this task.


    Anyone got any experience of this phone's predecessor?

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!

      Awesome, glad this one wasn't just me! Again, this problem also seemed to clear up int 5.20, so maybe its been fixed? Or at least masked a bit better :P

      That MCE sounds really intriguing. In the full scale version of my code I make use of Thread::Queue and a pool threads (as you have taught me over the years), in your experience how large are the performance/reliability benefits of using MCE over such a model? Thanks!

    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Perl system command memory usage in threads
by BrowserUk (Patriarch) on Jul 31, 2015 at 19:14 UTC

      See Update 1, if I'm doing something wrong in my data collection, or you need additional info please let me know!

Re: Perl system command memory usage in threads
by marioroy (Prior) on Aug 01, 2015 at 18:47 UTC

    Hi all.

    The following is a modified version using MCE from trunk. Thus, will work when MCE 1.7 is released. All 5 exec methods ran without the memory creeping issues with or without threads.

    The Perl version is 5.16.2 on Mac OS X 10.9.5. Also, tested on a CentOS 7 VM with Perl v5.16.3 without any problems. Finally, tested on a Windows 7 VM with Strawberry Perl v5.16.3.1 and v5.18.4.1. No problems with either.

    use strict; use warnings; use threads; # using threads is optional use MCE::Mutex; use MCE::Flow; use MCE::Shared; my $DONE : Shared = 0; my $lock = new MCE::Mutex; my $execMethod = $ARGV[0] || 0; if($execMethod !~ /[12345]/){ print "Must pass an exec method:\n"; print "1 = backticks\n"; print "2 = backticks synchronized\n"; print "3 = open\n"; print "4 = open synchronized\n"; print "5 = system\n"; exit 1; } sub execute{ my $cmd = shift; if($execMethod == 1){ `$cmd`; }elsif($execMethod == 2){ $lock->lock; `$cmd`; $lock->unlock; }elsif($execMethod == 3){ open(my $fs, "-|", $cmd); foreach(<$fs>){}; close $fs; }elsif($execMethod == 4){ $lock->lock; open(my $fs, "-|", $cmd); foreach(<$fs>){}; close $fs; $lock->unlock; }elsif($execMethod == 5){ system($cmd . ">nul"); } } sub worker{ while(!$DONE){ execute('echo hello world'); } } sub console{ print "Press <enter> to terminate\n"; <STDIN>; $DONE = 1; } # two subtasks; 30 workers, and 1 for console mce_flow { max_workers => [ 30, 1 ] }, \&worker, \&console;

    The OP's demonstration is possible with MCE. It requires 1.7 which will be released soon.

    Kind regards, Mario.

Re: Perl system command memory usage in threads
by Anonymous Monk on Jul 31, 2015 at 20:15 UTC
    Give us some numbers here ... how much memory, how much creep, over how much time, and what makes you call it permanent?

      See Update 1. The rate of growth shown there stays the same throughout the life cycle of the script (which is what I mean by permanent).

      If you need anything else let me know!

        Is that the exact code from the OP?

        Because when I run that code (Vista;(v5.18.4) built for MSWin32-x64-multi-thread) with:

        • Option 2: 22.4MB constant regardless of how long I run it.
        • Option 4: 22.3MB.
        • Option 5: 23.3MB.

        Which brand (AS/Strawberry/self-built) of Perl are you using?


        Anyone got any experience of this phone's predecessor?

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
        In the absence of evidence, opinion is indistinguishable from prejudice.
        I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1137039]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (2)
As of 2024-03-19 06:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found