Re: Perl system command memory usage in threads

[marioroy]

Hi all.

The following is a modified version using MCE from trunk. Thus, will work when MCE 1.7 is released. All 5 exec methods ran without the memory creeping issues with or without threads.

The Perl version is 5.16.2 on Mac OS X 10.9.5. Also, tested on a CentOS 7 VM with Perl v5.16.3 without any problems. Finally, tested on a Windows 7 VM with Strawberry Perl v5.16.3.1 and v5.18.4.1. No problems with either.

use strict;
use warnings;
use threads;       # using threads is optional

use MCE::Mutex;
use MCE::Flow;
use MCE::Shared;

my $DONE : Shared = 0;
my $lock = new MCE::Mutex;

my $execMethod = $ARGV[0] || 0;
if($execMethod !~ /[12345]/){
    print "Must pass an exec method:\n";
    print "1 = backticks\n";
    print "2 = backticks synchronized\n";
    print "3 = open\n";
    print "4 = open synchronized\n";
    print "5 = system\n";
    exit 1;
}
sub execute{
    my $cmd = shift;
    if($execMethod == 1){
        `$cmd`;
    }elsif($execMethod == 2){
        $lock->lock;
        `$cmd`;
        $lock->unlock;
    }elsif($execMethod == 3){
        open(my $fs, "-|", $cmd);
        foreach(<$fs>){};
        close $fs;
    }elsif($execMethod == 4){
        $lock->lock;
        open(my $fs, "-|", $cmd);
        foreach(<$fs>){};
        close $fs;
        $lock->unlock;
    }elsif($execMethod == 5){
        system($cmd . ">nul");
    }
}

sub worker{
    while(!$DONE){
        execute('echo hello world');
    }
}

sub console{
    print "Press <enter> to terminate\n";
    <STDIN>;
    $DONE = 1;
}

# two subtasks; 30 workers, and 1 for console
mce_flow { max_workers => [ 30, 1 ] }, \&worker, \&console;

The OP's demonstration is possible with MCE. It requires 1.7 which will be released soon.

Kind regards, Mario.