Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re: Untangling Log Files

by davorg (Chancellor)
on Feb 08, 2007 at 12:14 UTC ( [id://598984]=note: print w/replies, xml ) Need Help??

in reply to Untangling Log Files

You shouldn't slurp files unless you actually need all of the file in memory at the same time. Your normal action should be to process the file a record at a line.

I'd do something like this:

my %fh; # store handles of new log files open OLDLOG, $path_to_old_log or die $!; while (<OLDLOG>) { my $proc = extract_process_id_from_old_log_record($_); unless (exists $fh{$proc}) { open $fh{$proc}, '>', "$proc.log" or die $!; } print $fh{$proc} $_; }

Update: Reread the first line. It made no sense, so I fixed it.

Replies are listed 'Best First'.
Re^2: Untangling Log Files
by sauoq (Abbot) on Feb 08, 2007 at 16:03 UTC
    open $fh{$proc}, '>', "$proc.log" or die $!;

    Uhm... I don't care for that. It's very likely that there will be more pids in each file than open descriptors permitted by resource limits... and if you fail to open you die, probably half-done and with no way to pick up where you left off.

    I don't have an immediate fix though someone else suggested closing one handle randomly which would, I guess, work. (So long as you changed your open to '>>' and remembered to delete it from your hash.)

    Personally, I'd probably take a less elegant—call it more braindead—approach as this seems to be a one-off thing anyway, and just close the filehandle and open a new one whenever the pid changed from the previous record.

    Update: Well, I just re-read the OP and now I think I may have misinterpreted the bit about "30 processes" the first time around. If there are only 30 pids in the log files, then I like your approach just fine and my criticisms are all moot.

    "My two cents aren't worth a dime.";

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://598984]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (4)
As of 2024-06-15 11:15 GMT
Find Nodes?
    Voting Booth?

    No recent polls found

    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.