Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re^2: Consumes memory then crashs

by allhellno (Novice)
on Mar 24, 2012 at 13:20 UTC ( #961399=note: print w/ replies, xml ) Need Help??


in reply to Re: Consumes memory then crashs
in thread Consumes memory then crashs

I was trying to get it to only allow 20 workings threads at one time, but here is the original

use LWP::Simple; $names = 'zezima'; $re69='((?:[a-z0-9_]+))'; open (lookup, '>>rstlookup.txt'); while ($names =~ m/$re69/isg) { $name = $1; $lookup = get("http://rscript.org/lookup.php?type=track&time=62899 +200&user=".$name."&skill=all"); print "Looking up $name...\n"; $reg1='(ERROR)'; $re1='(gain)'; $re2='(:)'; $re3='(Overall)'; $re4='(:)'; $re5='(\\d+)'; $re6='(:)'; $re7='(\\d+)'; $re=$re1.$re2.$re3.$re4.$re5.$re6.$re7; if ($lookup =~ m/$re/isg) { print lookup "$name $7\n"; } elsif ($lookup =~ m/$reg1/isg) { print lookup "$name doesn't exist + \n" } else{ print lookup "$name 0\n"; } } close (lookup);


Comment on Re^2: Consumes memory then crashs
Download Code
Re^3: Consumes memory then crashs
by BrowserUk (Pope) on Mar 24, 2012 at 14:54 UTC

    There doesn't seem any good reason to build up your regex a bit at a time in separate variables; then concatenate those bits into another variable; and then interpolate that into a regex:

    $re1='(gain)'; $re2='(:)'; $re3='(Overall)'; $re4='(:)'; $re5='(\\d+)'; $re6='(:)'; $re7='(\\d+)'; $re=$re1.$re2.$re3.$re4.$re5.$re6.$re7; if ($lookup =~ m/$re/isg) {

    And even less reason to repeat the same 3-stage exercise over and over again each time around a loop. Especially as it means that the regex engine needs to recompile the combined regex every time around that loop even though nothing changes.

    And why capture 7 different bits of the string you are matching against:

    $re = '(gain)(:)(Overall)(:)(\\d+)(:)(\\d+)'; ... "$name $7\n"

    when 5 of them are constants; and you are only using one of them?

    Equally, the is nothing to be gained from assigning a constant string to a variable and then interpolating it into a regex:

    $reg1='(ERROR)'; ... elsif ($lookup =~ m/$reg1/isg)

    Cleaning up those; and a few other things up; adding strict and -w; and moving the regexing into a subroutine (to make later threading easier), I get:

    #! perl -slw use strict; use LWP::Simple; sub lookup { my( $hf, $name ) = @_; my $lookup = get( "http://rscript.org/lookup.php?type=track&time=62899200&user=$ +name&skill=all" ); print "Looking up $name...\n"; if( $lookup =~ m/gain:Overall:\d+:(\d+)/isg ) { print { $fh } "$name $7\n"; } elsif( $lookup =~ m/(ERROR)/isg ) { print { $fh } "$name doesn't exist \n" } else{ print { $fh } "$name 0\n"; } } my $names = 'zezima'; open( LOOKUP, '>>rstlookup.txt' ) or die $!; while( my( $name ) = $names =~ m/([a-z0-9_]+)/isg ) { lookup( \*LOOKUP, $name ); } close( LOOKUP );

    Which will probably not run much more quickly as you are IO-bound, but (I hope you'll agree) is easier to read and will at least consume less cpu.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      small change
      my( $fh, $name ) = @_;
      Now the problem is it only looks up the first name infinitely
      while( my( $name ) = $names =~ m/([a-z0-9_]+)/isg ) { lookup( \*LOOKUP, $name ); }
      Nothing seems wrong to me so I changed it slightly
      while($names =~ m/([a-z0-9_]+)/isg ) { my $name = $1; lookup( \*LOOKUP, $name ); }
      Now I am back to where I started, 1 response ever 1 or 2 seconds, too slow. Is there a simple solution to thread this properly?

        For quickly looking up a bunch of addresses, AnyEvent::DNS may be of help.

        Now I am back to where I started, 1 response ever 1 or 2 seconds, too slow. Is there a simple solution to thread this properly?

        Yes. Try this:

        #! perl -slw use strict; use threads; use threads::shared; use Thread::Queue; use LWP::Simple; my $sem :shared; sub lookup { my( $fh, $name ) = @_; my $lookup = get( "http://rscript.org/lookup.php?type=track&time=62899200&user=$ +name&skill=all" ); print "Looking up $name...\n"; if( $lookup =~ m/gain:Overall:\d+:(\d+)/isg ) { lock $sem; print { $fh } "$name $1\n"; } elsif( $lookup =~ m/(ERROR)/isg ) { lock $sem; print { $fh } "$name doesn't exist \n" } else{ lock $sem; print { $fh } "$name 0\n"; } } our $THREADS //= 4; my $names = 'zezima fred bill john jack'; my $Q = new Thread::Queue; open( LOOKUP, '>>rstlookup.txt' ) or die $!; my @threads = map async( sub { while( my $name = $Q->dequeue ) { lookup( \*LOOKUP, $name ); } } ), 1 .. $THREADS; while( $names =~ m/([a-z0-9_]+)/isg ) { $Q->enqueue( $1 ); sleep 1 while $Q->pending > $THREADS * 2; } $Q->enqueue( (undef) x $THREADS ); $_->join for @threads; close( LOOKUP ); __END__ [15:38:57.93] C:\test>junk39 Looking up john... Looking up bill... Looking up fred... Looking up zezima... Looking up jack... [15:39:03.07] C:\test>type rstlookup.txt bill 0 fred 135601422 zezima 417155645 john 0 jack 8133157

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://961399]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2014-08-02 06:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Who would be the most fun to work for?















    Results (55 votes), past polls