Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: count sort & output II

by blokhead (Monsignor)
on Dec 19, 2002 at 17:18 UTC ( [id://221175]=note: print w/replies, xml ) Need Help??


in reply to count sort & output II

You're running this as a CGI script, are you not? If so, then this line:
unless (++$counter % 10) { print "Press Enter"; <STDIN> }
..won't work. Whoever suggested using this must have assumed you were running this script from the command line. If you want to get back only 10 results at a time from your CGI script, you need to either:
  • Reparse the entire log file each time you get a request for 10 results, or
  • Store the processed results somewhere, maybe a database
Your CGI script can't store your hash in between HTTP requests, since it is unloaded from memory after every request1. I'll assume you don't want to wait 2 minutes for each 10 records, so I'd recommend writing to a cache file somewhere, storing all of the referrer => num records in your preferred sorted order. On each request to your script, just get the appropriate 10 lines from the file, if it exists, or else regenerate it if it doesn't exist. There should also be a mechanism to force a regeneration if the contents of the cache get stale.

You may want to do some debugging to see where the major slowdown is.. If it's the while loop, there's probably not a lot you can do, but if it's in the sorting and copying of the hash keys (BTW, how big does this hash end up?), you may want to consider a non-hash-based solution. Your for-loop has to make a copy of all the hash keys in memory, which may take a long time, considering your HTTP-referer strings are all probably fairly long. Other monks might have some good ideas about improving this portion of the code, but I'm at a loss at the moment.

Good luck,

blokhead

1: Of course, this is not true if your script is running under mod_perl, but it doesn't look like it

Replies are listed 'Best First'.
Re: Re: count sort & output II
by mkent (Acolyte) on Dec 19, 2002 at 22:16 UTC
    Thanks. You're right, my Apache server isn't using mod_perl and actually I didn't know that might make a difference, so thanks for the explanation.

    For a cache, this is what I came up with. I'm sure there's a better way, so any suggestions welcome. It doesn't speed up the processing at all.

    # Sort and print for (sort {$referers{$b} <=> $referers{$a}} keys %referers) { if ($counter <= 10) { open (FILE, ">storage1.txt") || die('Could not open $storage1.t +xt: $1'); print "$_ - $referers{$_}<p>"; print FILE "$_ - $referers{$_}<p>"; ++$counter; } elsif ($counter > 10 && $counter <= 20) { if ($counter == 11) { print "<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;< +a href=\"storage2.txt\"><font color=\"FF0000\">Next</font></a><br>"; print FILE "<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n +bsp;<a href=\"storage2.txt\"><font color=\"FF0000\">Next</font></a><b +r>"; open (FILE2, ">storage2.txt") || die('Could not open $stora +ge2.txt: $1'); print FILE2 "$_ - $referers{$_}<p>"; ++$counter; } } elsif ($counter > 20 && $counter <= 30) { if ($counter == 21) { print "<p>a href=\"storage2.txt\"><font color=\"FF0000\">Pr +evious</font></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a h +ref=\"storage3.txt\"><font color=\"FF0000\">Next</font></a><br>"; open (FILE3, ">storage3.txt") || die('Could not open $stora +ge3.txt: $1'); print FILE3 "$_ - $referers{$_}<p>"; ++$counter; } } elsif ($counter > 30 && $counter <= 40) { if ($counter == 31) { print "<p>a href=\"storage3.txt\"><font color=\"FF0000\">Pr +evious</font></a><br>"; open (FILE4, ">storage4.txt") || die('Could not open $stora +ge4.txt: $1'); print FILE4 "$_ - $referers{$_}<p>"; } } } close FILE; close FILE2; close FILE3; close FILE4;

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://221175]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (2)
As of 2024-06-16 10:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.