Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Database vs. tail -f

by Coruscate (Sexton)
on Apr 19, 2003 at 22:58 UTC ( [id://251733]=perlquestion: print w/replies, xml ) Need Help??

Coruscate has asked for the wisdom of the Perl Monks concerning the following question:

I am currently in a situation where multiple clients need to make data available to one server application. Both the server and client are running on the same machine, so this makes it very easy to handle. My question is: which way is better? I've thought of having the clients append to a flat text file, while the server uses tail -f to get at the data. My second thought was using a database. Here is example structure of my thoughts. First, the database method:

################## # Server my $id = 0; while (1) { # do mysql query: "SELECT FROM msgs WHERE id>?", $id for my $msg ($results_of_mysql_query) { # do something with this line of data } sleep 1; } ################## # Client: while (<STDIN>) { chomp; # do mysql insert: "INSERT INTO msgs VALUES(?,?)", '', $_; }

Then let's take a look at the flat text file way of getting this done:

################## # Server: open my $msgs, "| tail -f data.txt" or die "ouch: $!"; while (<$msgs>) { # do something with this line of data } ################## # Client: use Fcntl ':flock'; while (<STDIN>) { chomp; open my $msgs, '>>', 'data.txt' or die "ouch: $!"; flock $msgs, LOCK_EX; print( $msgs, ($_ . "\n") ); close $msgs; }

So out of these two methods, which one strikes you as being more efficient? Or can you think of a different method altogether that beats the pants of both of these? As well, is there any flaw in my logic of the file locking for the flat file method? Will `tail` still be capable of reading the appended data, even if I take an exclusive lock on the file while appending is done? Thanks :)


If the above content is missing any vital points or you feel that any of the information is misleading, incorrect or irrelevant, please feel free to downvote the post. At the same time, please reply to this node or /msg me to inform me as to what is wrong with the post, so that I may update the node to the best of my ability.

Replies are listed 'Best First'.
Re: Database vs. tail -f
by Abigail-II (Bishop) on Apr 19, 2003 at 23:53 UTC
    I'd say neither is very efficient. Your database solution is very inefficient. If one client has added 10000 rows, and then all clients are silent for an hour, your server is plowing through 36 million rows - neither of them "new".

    If client and server are running on the same machine, why not use a named pipe to communicate? Or a socket?

    Abigail

Re: Database vs. tail -f
by Coruscate (Sexton) on Apr 19, 2003 at 23:03 UTC

    My brain was a little screwy at the moment, so replace line 7 of the flat text file client code (print( $msgs, ($_ . "\n") );) with print $msgs $_ . "\n";. Besides that, I tested out the flat text file. The file locking works good, I'm just wondering if it will be fine with multiple clients at the same moment.


    If the above content is missing any vital points or you feel that any of the information is misleading, incorrect or irrelevant, please feel free to downvote the post. At the same time, please reply to this node or /msg me to inform me as to what is wrong with the post, so that I may update the node to the best of my ability.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://251733]
Approved by shotgunefx
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others browsing the Monastery: (4)
As of 2024-03-19 07:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found