Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Performing a tail(1) in Perl (reading the last N lines of a file)

by Chmrr (Vicar)
on Apr 25, 2002 at 17:23 UTC ( #162048=note: print w/ replies, xml ) Need Help??


in reply to Performing a tail(1) in Perl (reading the last N lines of a file)

I'd seen Tie::File touted recently, so I decided to see how it stood up. I added the following subroutine:

sub tie_file { my @lines; tie @lines, 'Tie::File', $file or die "$!"; map {"$_\n"} @lines[-$limit..-1]; }

Pretty much as simple as it gets. Unfortunatly, it's not quite so hot on the performance:

[chmrr@supox chmrr]$ wc -l /var/log/messages 4442 /var/log/messages [chmrr@supox chmrr]$ perl -w tailbm.pl -10 100 /var/log/messages Benchmark: running f_rb_obj, f_rb_obj_u, f_rb_tie, f_rb_tie_u, file_ta +il, grinder, lastn, lastn_getc, tie_file, each for at least 10 CPU se +conds... f_rb_obj: 11 wallclock secs (10.35 usr + 0.22 sys = 10.57 CPU) @ 66 +2.54/s (n=7003) f_rb_obj_u: 11 wallclock secs (10.54 usr + 0.12 sys = 10.66 CPU) @ 65 +0.19/s (n=6931) f_rb_tie: 11 wallclock secs (10.48 usr + 0.20 sys = 10.68 CPU) @ 54 +5.13/s (n=5822) f_rb_tie_u: 12 wallclock secs (10.39 usr + 0.16 sys = 10.55 CPU) @ 53 +7.25/s (n=5668) file_tail: 12 wallclock secs (10.60 usr + 0.14 sys = 10.74 CPU) @ 33 +3.71/s (n=3584) grinder: 10 wallclock secs (10.49 usr + 0.07 sys = 10.56 CPU) @ 26 +.23/s (n=277) lastn: 12 wallclock secs ( 7.79 usr + 2.63 sys = 10.42 CPU) @ 26 +.97/s (n=281) lastn_getc: 11 wallclock secs ( 8.83 usr + 1.46 sys = 10.29 CPU) @ 28 +.38/s (n=292) tie_file: 11 wallclock secs (10.15 usr + 0.12 sys = 10.27 CPU) @ 3 +.89/s (n=40)

Well, it's good to know, at least.

Update: Heh -- I started doing this test a'fore broquaint posted; great minds think alike, eh?

Update 2: On a whim, I decided to test:

sub backticks { split /$/m, `tail -$limit $file`; }

..as well. Even given the overhead of the shell, it's still darn fast:

[chmrr@supox chmrr]$ perl -w tailbm.pl 500 100 /var/log/messages Benchmark: timing 500 iterations of backticks, f_rb_obj, f_rb_obj_u, f +_rb_tie, f_rb_tie_u, file_tail, grinder, lastn, lastn_getc, tie_file. +.. backticks: 1 wallclock secs ( 0.43 usr 0.16 sys + 0.61 cusr 0.30 +csys = 1.50 CPU) @ 847.46/s (n=500) f_rb_obj: 1 wallclock secs ( 0.71 usr + 0.04 sys = 0.75 CPU) @ 66 +6.67/s (n=500) f_rb_obj_u: 1 wallclock secs ( 0.77 usr + 0.00 sys = 0.77 CPU) @ 64 +9.35/s (n=500) f_rb_tie: 1 wallclock secs ( 0.89 usr + 0.02 sys = 0.91 CPU) @ 54 +9.45/s (n=500) f_rb_tie_u: 1 wallclock secs ( 0.91 usr + 0.02 sys = 0.93 CPU) @ 53 +7.63/s (n=500) file_tail: 1 wallclock secs ( 1.47 usr + 0.01 sys = 1.48 CPU) @ 33 +7.84/s (n=500) grinder: 19 wallclock secs (18.93 usr + 0.22 sys = 19.15 CPU) @ 26 +.11/s (n=500) lastn: 19 wallclock secs (14.20 usr + 4.34 sys = 18.54 CPU) @ 26 +.97/s (n=500) lastn_getc: 20 wallclock secs (15.09 usr + 2.64 sys = 17.73 CPU) @ 28 +.20/s (n=500) tie_file: 143 wallclock secs (126.62 usr + 1.81 sys = 128.43 CPU) @ + 3.89/s (n=500)

perl -pe '"I lo*`+$^X$\"$]!$/"=~m%(.*)%s;$_=$1;y^`+*^e v^#$&V"+@( NO CARRIER'


Comment on Re: Performing a tail(1) in Perl (reading the last N lines of a file)
Select or Download Code
Re: Performing a tail(1) in Perl (reading the last N lines of a file)
by Dominus (Parson) on Apr 26, 2002 at 03:42 UTC
    Says Chmrr:
    Tie::File ... Pretty much as simple as it gets. Unfortunatly, it's not quite so hot on the performance:
    Not too surprising, because unlike most of the methods you benchmarked, Tie::File does not read the file backwards. It reads it forwards, starting at the beginning, and constructs some data structures along the way. For a long file like /var/adm/messages, this will take some time, because it grovels over the entire file.

    The payoff would come if you then asked it to tell you what was on line 12,345, which it would do instantly---or if you asked it to modify line 12,345, which the other modules won't do at all.

    --
    Mark Dominus
    Perl Paraphernalia

      Have you considered modifying File::Tail to defer reading the file such that if the only subscripts ever given to the tied array are negative, it reads the file backwards? Not asking for the feature; just throwing out ideas.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://162048]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (9)
As of 2014-11-27 10:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (183 votes), past polls