Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Active Munging via a pipe (code)

by deprecated (Priest)
on Jan 16, 2002 at 23:23 UTC ( #139294=perlquestion: print w/ replies, xml ) Need Help??
deprecated has asked for the wisdom of the Perl Monks concerning the following question:

I run two services which produce log files on the scale of multigigabytes: Postgres and http://opennap.sourceforge.net/opennap. Taking huge zipped up tarballs of these logs and parsing them during off hours or when I come home from work gets tedious. I'd rather just have a perl daemon set up that actually read them as they were generated and stuffed em in a database in the format I want.

So the approach I came up with was using mkfifo(1) and this eensy-weensy loop:

#!/usr/local/bin/perl open FIFO, "<foo" or die "$!\n"; while (<FIFO>) { chomp; print "$_\n"; }
The problem is, this doesnt actually stay open and alive. It dies after the first "echo 'hello' > foo".

Is there a way I can make perl "listen" to a file and actually have it be parsing data and doing inserts on a db?

thanks,
brother dep.

--
Laziness, Impatience, Hubris, and Generosity.

Comment on Active Munging via a pipe (code)
Select or Download Code
Re: Active Munging via a pipe (code)
by broquaint (Abbot) on Jan 16, 2002 at 23:45 UTC
    The problem with looping on the input from named pipes is that after the first read you're left with an EOF for the second read1, so the loop exits. Subsequently you can't loop on a named pipe without working some serious voodoo (over-riding CORE::GLOBAL::eof()?). So this provides with many excuses to do dubious things such as using goto(), labels and (dum dum dum) infinite loops. Personally I'd go with an infinite loop opening the FIFO on each iteration, or take have a good long think about it. Good luck!
    HTH

    broquaint

    1unless you have something writing faster than perl can read

(Ovid) Re: Active Munging via a pipe (code)
by Ovid (Cardinal) on Jan 17, 2002 at 00:03 UTC

    This type of stuff isn't exactly my strong suit, but it sounds like the the while loop is exiting on EOF. If that's the case, you need to clear the EOF flag. The classic example, modified from the Cookbook:

    for (;;) { print while (<FIFO>) sleep FOR_A_WHILE; # some constant seek <FIFO>, 0, 1; # resets EOF }

    Is that what you are needing? It should effectively be a tail.

    Cheers,
    Ovid

    Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

Re: Active Munging via a pipe (code)
by PrakashK (Pilgrim) on Jan 17, 2002 at 00:15 UTC
    How about using File::Tail?
    use File::Tail; $file=File::Tail->new("/some/log/file"); while (defined($line=$file->read)) { print "$line"; }
    HTH,

    /prakash

Re: Active Munging via a pipe (code)
by gt8073a (Hermit) on Jan 17, 2002 at 00:18 UTC

    Is there a way I can make perl "listen" to a file and actually have it be parsing data and doing inserts on a db?

    This is seriously off the top of my head, and I have never done anything similiar, but instead of using open( BLAH, $file ) and while ( <FIFO> ) couldn't you use sysopen, sysread and then sleep on eof? You can "know" where you are in the file, and just sleep when you get an EOF. I am sure there are buffering and pre-munging munging issues involved, but maybe this will help.

    Sorry, I do not have any code to help you with, but someone else prolly can . . .

    Will perl for money
    JJ Knitis
    (901) 756-7693
    gt8073a@industrialmusic.com

Re: Active Munging via a pipe (code)
by count0 (Friar) on Jan 17, 2002 at 01:54 UTC
    Since nobody has mentioned it yet.. it's definitely worth noting that perlipc has a section on named pipes (FIFOs).
Re: Active Munging via a pipe (code)
by Zaxo (Archbishop) on Jan 17, 2002 at 06:51 UTC

    Here's yet another way, demonstrated from the shell:

    Make the pipe, and...

    $ mkfifo logpipe $ perl -e 'open PIPE, "< logpipe"; { select 1<<fileno(PIPE),undef,unde +f,undef; print <PIPE>;redo}' &
    ... start a simple server in the background. select makes it sleep till PIPE is readable. redo in a bare block loops forever. Kick the tires:
    $ echo foo >logpipe foo $ echo far >logpipe $ far echo fie >logpipe fie $
    The asynchronous operation of the server is shown in the staggered output. Clean up with
    $ fg perl -e 'open PIPE, "< logpipe"; {select 1<<fileno PIPE,undef,undef,un +def;print <PIPE>;redo}' # do a Ctrl-C here $
    I think this is the Right Way® to handle your log file problem. Sleeping till there is something to read will save a lot of load on the host.

    Update: Changed to use mkfifo instead of mknod, and repaired a pasto in the timeout arg of select.

    After Compline,
    Zaxo

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://139294]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (7)
As of 2014-12-29 12:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (187 votes), past polls