Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Reading a file live!

by negativ_m (Novice)
on Mar 07, 2014 at 07:18 UTC ( #1077337=perlquestion: print w/replies, xml ) Need Help??
negativ_m has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I have a small problem and i don;t have a solution for it, any help will be greatly appreciated.

I have a master script which i run. From my master script i call another script and i redirect the outputs into a file. So that file is constantly updating with new lines. I want to read the file while it is being updated and parse the text...basically i need to store the lines that are constantly added, and use them in my master script.

How can i do this without blocking my master script?

Thank you in advance.

Replies are listed 'Best First'.
Re: Reading a file live!
by Corion (Pope) on Mar 07, 2014 at 08:39 UTC

    How about just using File::Tail (which works under Unixish-enough OSes), or spawning a pipe from tail -f?

      Hi Corion,

      Thank you for the suggestion.

      I know that with File::Tail it should work.

      But i really wanna make my life harder and try something else :).

Re: Reading a file live! (poll)
by Anonymous Monk on Mar 07, 2014 at 08:00 UTC

    How can i do this without blocking my master script?

    see Proc::Background and Poll :) non-blocking is called polling

    for example

    use Proc::Background; my $proc1 = Proc::Background->new($command, $arg1, $arg2); while( 1 ){ sleep 1; my $newsize = -s $fhorfilename; # if( ( $newsize - $oldsize ) > $minimum ){ if( ( $newsize - $oldsize ) > $buffersize ){ } my $readed = read $fh, $buffer, $buffersize; doStuff( $readed , $fh, $buffer, $buffersize ); ... last if not $proc1->alive; }


      Thank you for the suggestions.

      But do you know another way to do it, without using this modules (Proc and Schedule) ?

      Thank you

        Of course, you don't have to use Proc::Background, it's just a nicety and saves you dealing with all the low-level wrangling involved with using fork in a robust fashion. But by all means feel free to use fork natively if you would rather. TIMTOWTDI, after all.

        But do you know another way to do it, without using this modules (Proc and Schedule) ?

        Why? Maybe I'll have some ideas :)

        But, this is like the lowest common demoninator of portability and ease :)

Re: Reading a file live!
by Laurent_R (Canon) on Mar 07, 2014 at 07:48 UTC

    In principle, it should not be necessarily blocking your main script, but this depends on many things, like what is your OS, how your secundary script is writing into the file, whether it closes the file when it is done, whether your secundary script is permanent or temporary, whether you are waiting for any specific output in the file, too many things that we don'y know.

    I also wonder whether this is the best way of doing things. If you want to read the output of the secundary script in your main script, why do you need a secundary script at all? I am not saying it is a bad idea, I don't know enough, I am just questionning whether it is a good idea. Please tell us more about what you are trying to achieve.

      I will try to explain better.

      #main script


      ...  #i am doing something

      system ("perl $var > file.txt &"); #here i call my other script and i redirect the output into the file.txt. My script will run some time but in my master script i need to use the newly added lines (always) from the txt file

      I need to read that file always ...i can do that but the problem is that is blocking my main script and i can;t use the new lines in my master script. In a way i am trying to read that file "live" get the lines and use them inside my master script.

      Hope you understand what i am trying to say.

Re: Reading a file live!
by golux (Hermit) on Mar 07, 2014 at 19:38 UTC
    Hi negativ_m,

    My suggestion would be to assign a variable to the length of the file prior to the first append. Then, just check the filesize; when it grows you know you've got more text to read:

    my $len = (-s "file.txt") || 0; system ("perl $var > file.txt &"); # ... later .... my $newlen = (-s "file.txt") || 0; if ($newlen > $len) { my @newlines = read_new_lines("file.txt", $len); $len = $newlen; # Adjust length to new end of file }

    The subroutine read_new_lines() would look something like this:

    sub read_new_lines { my ($fn, $offset) = @_; use IO::File; my $fh = new IO::File($fn); $fh or die "Failed to read '$fn' ($!)\n"; seek($fh, $offset, 1) or die "Failed seek ($!)\n"; chomp(my @lines = <$fh>); return (@lines); }

    I use this trick a lot for webpages which need to monitor continuously updating logfiles.

    say  substr+lc crypt(qw $i3 SI$),4,5
Re: Reading a file live!
by RedElk (Hermit) on Mar 07, 2014 at 15:46 UTC

    Seems like you could define a subroutine that reads the file in question (and does whatever processing is required). Then strategically place calls to that subroutine within your master script.

    But then again, I may need more coffee.

Re: Reading a file live!
by Laurent_R (Canon) on Mar 08, 2014 at 16:29 UTC

    At least on Unix and Unix-like systems, it seems to me that your master script should not be blocked. This is a quick test. First, the subprogram, which is turning off IO buffering and writing a number every 3 seconds:

    # sub_program use strict; use warnings; { my $fh = select STDOUT; # making STDOUT "hot" (no IO buffering) $| = 1; select $fh; } for my $i (0..100) { print "$i \n"; sleep 3; }
    Now the master program, which is reading every 10 seconds the full file produced by the subprogram and prints out the last line:
    # main program use strict; use warnings; system ("perl > file.txt &"); my $last_line; for my $i (0..100) { sleep 10; local @ARGV = ("file.txt"); $last_line = $_ while <>; print "iteration $i : $last_line"; }
    Now running the master program:
    $ perl iteration 0 : 3 iteration 1 : 6 iteration 2 : 9 iteration 3 : 13 iteration 4 : 16 ...
    As you can see, the master program ("") is perfectly able to read the file generated by the subprogram, even though the subprogram is still running. Maybe your problem is that you did not think about turning off IO buffering.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1077337]
Approved by Corion
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (2)
As of 2018-07-21 06:28 GMT
Find Nodes?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?

    Results (444 votes). Check out past polls.