Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Is there a way to read a file backwards?

by gaggio (Friar)
on Jul 31, 2000 at 07:24 UTC ( #25200=perlquestion: print w/ replies, xml ) Need Help??
gaggio has asked for the wisdom of the Perl Monks concerning the following question:

This is a question that has been bothering me for a long time: is it possible to read a file backwards - let's say, if I want to read only the 10 last lines of a file without having to parse the whole file - and, if yes, how would that be speedwise?

Thanks for any help.

Comment on Is there a way to read a file backwards?
Re: Is there a way to read a file backwards?
by mikfire (Deacon) on Jul 31, 2000 at 07:38 UTC
    In short, no. Files really aren't meant to be read that way. There is, to the best of my knowledge, no way of doing this. The only way I have seen is basically "suck it into an array and pop the last N lines off".

    mikfire

Re: Is there a way to read a file backwards?
by btrott (Parson) on Jul 31, 2000 at 07:45 UTC
    Take a look at File::ReadBackwards.

    From the docs:

    $bw = File::ReadBackwards->new( 'log_file' ) or die "can't read 'log_file' $!" ; while( defined( $log_line = $bw->readline ) ) { print $log_line ; }
      You could also do it manually(without the module).
      <code>
        $line_count=-1;
      
        while(<>)
        {  $line_count++;  }
      
        print "There are $line_count lines in the file.";
      
        # To change the line number for the current filehandle
        # set the '$.' var to desired # not to exceed $line_count
      <code>
      
        I don't think that setting $. will help you much. This variable is incremented on each read, but it isn't used by perl to track its place in the file. In fact, $. isn't even file specific. I quote from the Camel Book,
        The current input line number of the last filehandle that was read. An explicit close on the filehandle resets the line number. Since <> never does an explicit close, line numbers increase across ARGV files
Re: Is there a way to read a file backwards?
by jlp (Friar) on Jul 31, 2000 at 07:55 UTC
    If you're not worried about memory consumption, you could just do:

     my @reverse_lines = reverse <FH>;

    This will read the whole file into memory, and is thus not very memory efficient, but it is probably the simplest way to do what you want.

    UPDATE: Whoops, that'll teach me to not read closely enough; I just noticed you said you didn't want to read the whole file; sorry. :\

Re: Is there a way to read a file backwards?
by chromatic (Archbishop) on Jul 31, 2000 at 08:41 UTC
    For the sake of completeness, you might also use backticks to capture the output of the Unix tail command (provided you're on a platform where it's available). Speedwise it's not too bad, depending on the cost of invoking a new process compared to file access times and such. You might also use a combination of seek and sysread, but you'll have to search for line endings (newlines) yourself. The search will be somewhat expensive.

    Go with File::Backwards.

Re: Is there a way to read a file backwards?
by vnpandey (Scribe) on Jul 31, 2000 at 11:21 UTC
    Yes There is surely a way to it...like this...First find the size of files in bytes which is very easy as you know.. After that you just read the last 100bytes of the file(this number will be best depending on how long are your lines in the file) This can be easily done as follows..Suppose your file has 10000 bytes now reading the last 100 bytes is just pointing your file pointer on the 9900 th byte...and grab the 100 bytes then

    #!usr/local/bin/perl open(FILE ,"name_of_the_file_to_be_read"); seek(FILE, 9900,0); #point the pointer to 9900th byte from the s +tarting( $ read(FILE,$ab,100); #note the last 100 bytes get stored in $ab close(FILE);
    Then as you know search for the newline charcter in variable ab(or as the case may be fixing the input record seperator as reqd) keep in mind that there will be a newline character in the end too.. after finding the new line character you can easily know at which position it is in these 100 bytes.. (say it is on 15th byte in this 100 byte u read) then you can easily retrieve the desired 16th to 100 th byte by using substr() function.. thus you have read the last line.. if you want to read the last but second line from the end. you may please read in similar fashion from 9815bytes to 9915 bytes i.e. just fixing ur pointer at 9815 form the start) provided you have the results as I have told. and proceed further.. for any line number from the end... In case you r not able to get the new line character in the last 100bytes(i.e. the last line is more than 100 bytes long read the last 200 bytes or so and get the last line similarly... The advantage is that it saves you a lot of memory space as you are reading the minimum bytes and also speed if files are too long and one is intrested in contents in the last portion of the file(say last 5%)

    Also There is a module to read the file back-wards.. as I have been told by autark...I am quoting his reply...

    Indeed, there is a module called File::ReadBackwards available through CPAN. From the description in the man file: This module reads a file backwards line by line. It is simple to use, memory efficient and fast. It supports both an object and a tied handle interface. It is intended for processing log and other similar text files which typically have their newest entries appended to them. By default files are assumed to be plain text and have a line ending appropriate to the OS. But you can set the input record separator string on a per file basis. Hope it helps....Please comment if this helps you...

    And note the metod above can be really optimised depending upon your need I have just stated the algorithm..The no of bytes etc.. all depends upon your local requirements..also suppose you want to read the 6th line from the last you don't have to nessearily read the last line & then second last & so on but you just read the last 500 bytes and find how many new line characters are there (you require 5 or 6 depending that there is new line character in the end of last line) when you do have this then read from the first new-line character in variable ab to the nest new line character i.e. you can really optimise it...That's all.. :-vnpandey

    Edit: g0n - formatting

      Please do not vote any further on this reply above as I just made a mistake in putting correctly the codes (I had put / after code whereas I should have done it before code).. It won't happen again....Thanks:-vnpandey
        Dear Mr. Pandey... It is a very good thing that you accept your mistakes Infact that is the spirit of monks...but please be a little careful too as you can imagine it is very trivial just to put your codes...Well I am sure monks will forgive you but take care next time........o.k.....
        Dear Mr. Pandey... One more thing is that u urself can see instead of getting positive votes on your write-up in nutshell you got negative votes.. (-17 and +6) inspite of a very nice reply just because of one trivial carelessness..Infact I can see that you tried to put your things in codes from your post but perhabs did not bother to look on site how to for rechecking in case of doubt...So u yourself loose points apart form causing inconvienieance to others...As far as mentioning you sources is concerned..(as there is a comment for that) yes it is good to mention if u want it but anyway everybody learns things from some-where and it is always not possible to mention all the sources (as the similar answer to your question was given by me earlier) So I don't mind u not mentioning my names etc... That is perfectly o.k....Any way hapy perling...:-mrt

      This is exactly what I needed. A pointer in the direction to go for last-segment only file reading. Now I need to look into it and figure out what to extract from it. I was wasking about Reading the last line of a file in a Pure Way.

      --hackmare.

      Edit: g0n - fixed link

Re: Is there a way to read a file backwards?
by vnpandey (Scribe) on Jul 31, 2000 at 11:23 UTC
    Yes There is surely a way to it...like this...First find the size of files in bytes which is very easy as you know.. After that you just read the last 100bytes of the file(this number will be best depending on how long are your lines in the file) This can be easily done as follows..Suppose your file has 10000 bytes now reading the last 100 bytes is just pointing your file pointer on the 9900 th byte...and grab the 100 bytes then
    #!usr/local/bin/perl open(FILE ,"name_of_the_file_to_be_read"); seek(FILE, 9900,0); #point the pointer to 9900th byte from the s +tarting( $ read(FILE,$ab,100); #note the last 100 bytes get stored in $ab close(FILE);
    Then as you know search for the newline charcter in variable ab(or as the case may be fixing the input record seperator as reqd) keep in mind that there will be a newline character in the end too.. after finding the new line character you can easily know at which position it is in these 100 bytes.. (say it is on 15th byte in this 100 byte u read) then you can easily retrieve the desired 16th to 100 th byte by using substr() function.. thus you have read the last line.. if you want to read the last but second line from the end. you may please read in similar fashion from 9815bytes to 9915 bytes i.e. just fixing ur pointer at 9815 form the start) provided you have the results as I have told. and proceed further.. for any line number from the end... In case you r not able to get the new line character in the last 100bytes(i.e. the last line is more than 100 bytes long read the last 200 bytes or so and get the last line similarly... The advantage is that it saves you a lot of memory space as you are reading the minimum bytes and also speed if files are too long and one is intrested in contents in the last portion of the file(say last 5%) Also There is a module to read the file back-wards.. as I have been told by autark...I am quoting his reply... Indeed, there is a module called File::ReadBackwards available through CPAN. From the description in the man file: This module reads a file backwards line by line. It is simple to use, memory efficient and fast. It supports both an object and a tied handle interface. It is intended for processing log and other similar text files which typically have their newest entries appended to them. By default files are assumed to be plain text and have a line ending appropriate to the OS. But you can set the input record separator string on a per file basis. Hope it helps....Please comment if this helps you... And note the metod above can be really optimised depending upon your need I have just stated the algorithm..The no of bytes etc.. all depends upon your local requirements..also suppose you want to read the 6th line from the last you don't have to nessearily read the last line & then second last & so on but you just read the last 500 bytes and find how many new line characters are there (you require 5 or 6 depending that there is new line character in the end of last line) when you do have this then read from the first new-line character in variable ab to the nest new line character i.e. you can really optimise it...That's all.. :-vnpandey
      Two things: one, you can modify your original post here. The only time you can't is if you start a new thread. Just click on the title.

      The second thing is that it is polite to mention your sources.

      Seek's 3rd argument indicates what kind of seek to do. 0 means from the front of the file, 1 means relative to the current position and 2 means from the end of the file. By using the "seek from EOF" option you can use the above algorithm without having to compute the filesize first.
Re: Is there a way to read a file backwards?
by gaggio (Friar) on Jul 31, 2000 at 17:43 UTC
    I think that btrott definitely has the right answer. It does not make sense to me to reinvent the wheel (an expression that merlyn likes also) when somebody coded a module that "cooks everything for you".

    I will go with File::ReadBackwards, as it does exactly what I want.

    Thanks all for your help, though!
Re: Is there a way to read a file backwards?
by Cirollo (Friar) on Jul 31, 2000 at 18:30 UTC
    This isn't really a Perl thing, just a general unix tip (although you could run this from a Perl script...dirty!) There is a userland util called 'tac' that does an upside-down 'cat'. It's come in handy on numerous occasions, and I often find myself using it when I need to see more info than I get from tail -f.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://25200]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (12)
As of 2014-07-11 16:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (232 votes), past polls