Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

[threads] Open a file in one thread and allow others to write to it

by Gangabass (Priest)
on Nov 16, 2009 at 10:15 UTC ( #807405=perlquestion: print w/ replies, xml ) Need Help??
Gangabass has asked for the wisdom of the Perl Monks concerning the following question:

Hi, Monks.

First of all i have read Threads: How to share a FileHandle to Write and Threading and File Handles. But i'm still don't know how to make this task:

  • I need get some info from web site and save it to log file
  • But i must count lines in log file and if it is too big (say 1000 lines) i need to close it and continue with another file (changing suffix: log1.log, log2.log etc.)

Sharing file handle is impossible so i try to write to shared variable. But this doesn't work too (it just create ugly file name \$shared_variable).

How i can do that? Now i'm thinking about one writing thread which will communicate with others and write data to log file and create new one. Can you give me hint how i can do this?

Thanks.
Roman

Update

Thanks to all. Your suggestions really help. I'm using this code right now:

use strict; use warnings; use threads; use Thread::Queue; use IO::File; my $q = Thread::Queue->new(); sub worker { my $counter = 1; my $number_of_lines = 0; open my $fh, ">", "log" . $counter++ . ".txt" or die $!; $fh->autoflush(1); while (1) { my $thread_id = $q->dequeue(); if ( $number_of_lines >= 20 ) { close $fh; open $fh, ">", "log" . $counter++ . ".txt" or die $!; $fh->autoflush(1); $number_of_lines = 0; } print $fh "This is a very long string containing $thread_id\n" +; $number_of_lines++; } } threads->new( \&worker )->detach(); my @threads; foreach my $thread_id ( 0 .. 10 - 1 ) { my $thread = threads->create( 'start_thread', $thread_id ); push @threads, $thread; } foreach my $thread (@threads) { $thread->join(); } sub start_thread { my ( $thread_id ) = @_; while (1) { $q->enqueue($thread_id); sleep 1; } }

Comment on [threads] Open a file in one thread and allow others to write to it
Download Code
Re: [threads] Open a file in one thread and allow others to write to it
by roboticus (Canon) on Nov 16, 2009 at 11:24 UTC
    Gangabass:

    Not an answer to your question, but I tend to use time rather than log file size. I typically name my log files like "foolog.YYYYMMDD_HHMM" and then in the logging function if the log file is old enough, I close it & open a new one.

    ...roboticus

      I call it log but this is just text file containing data parsed from Web page. And it's name format is defined by the user in the config file.

        Why not have a shared variable which you write the 'log' to, say an array which holds the information for each line entry, and a thread which keep an eye on the size of the shared array and dumps it out when it hits whatever size you want and resets it.

        This way you don't need to worry about shared filehandles, only a shared in-memory variable ( IPC::Sharable seems to be the usual way of doing this ), and you have separated the 'logging' function from the 'collection' function.

        Just a something something...
Re: [threads] Open a file in one thread and allow others to write to it
by BrowserUk (Pope) on Nov 16, 2009 at 12:41 UTC

    This opens a file, starts 4 threads; writes to that file from all 4 threads (with locking) at random for a short while then stops the threads. It then reopens the file and prints it to stdout:

    #! perl -slw use strict; use Time::HiRes qw[ time sleep ]; use threads; use threads::shared; our $THREADS ||= 4; my $sem :shared; open LOG, '>', 'log.txt' or die $!; sub logit { lock $sem; return printf LOG @_; } sub thread { my $tid = threads->self->tid; my $stop = shift; warn $tid, ": starting:", time(), " Stoptime: $stop\n"; while( sleep( rand( 1 ) ) && time() < $stop ) { logit "%2d: The time is %f\n", $tid, time; } warn $tid, ": stopping at ", time(), "\n"; } my @threads = map threads->create( \&thread, time() + int( rand 10 ) ), 1 .. $THREADS; warn "threads started; waiting\n"; $_->join for @threads; warn "threads done\n"; open LOG, '<', 'log.txt' or die $!; print while <LOG>; close LOG;

    What more do you need?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

        S'funny, cos the code I posted is very well tested (going back to 5.8.0 and right upto 5.8.9 & 5.10.1 (64-bit)!

        The mistake you're making is reading "Filehandles cannot be made 'shared'." to mean 'filehandles cannot be shared'. Those phrases do not be not mean the same thing.

        Filehandles are process global entities and therefore can be cloned. Which means that each thread created after a filehandle comes into existance gets it's own copy of the global.

        Try the code I posted. Tell me how you get on :)


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

      Sorry, you didn't understand me (my error). Of course i know that file handler is cloned between threads. But i need more: i need to open file handler in the thread and allow other threads to write to it.

      Or shortly i need a solution to write to log file (from each thread). This is easy. But i'm also need to check for log size. So if it arrive some value i need to close old log file and open new one. And this event (maximum log file) occur in one of the thread (i have shared variable which increment after each write to log).

Re: [threads] Open a file in one thread and allow others to write to it
by gulden (Monk) on Nov 16, 2009 at 12:42 UTC
    A simplified approach without log file rotation:
    #!/opt/coolstack/bin/perl -w use strict; use warnings; use threads; use Thread::Queue; my $file = "logfile.log"; my $queue = Thread::Queue->new() or die "$!"; my $thr = threads->create( sub{ my $q = shift; my $line; open(my $fh,">>", $file) or die "Cant open file $! [$file]"; print $fh "$line\n" while $line = $queue->dequeue(); }, $queue ); my @thrs; push @thrs , threads->create(sub{ $queue->enqueue(threads->tid ."> Log Message: $_") for (1..100); }) for (1..10); $_->join for @thrs; $queue->enqueue(undef); # End Logger Thread $thr->join; exit;
    A contentious debate is always associated with a lack of valid arguments.
Re: [threads] Open a file in one thread and allow others to write to it
by zentara (Archbishop) on Nov 16, 2009 at 13:18 UTC
    ....funny no one mentioned that you can share a filehandle between threads thru the fileno, see FileHandles and threads

    ...here is an old node..... just put the logfile's fileno into a shared variable....then any thread can open it...... watch out for concurrency problems though

    #!/usr/bin/perl use warnings; use strict; use threads; use threads::shared; # original idea from BrowserUK at # http://perlmonks.org?node_id=493754 for my $file ( map{ glob $_ } @ARGV ) { open my $fh, '<', $file or warn "$file : $!" and next; printf "From main: %s", scalar <$fh> for 1 .. 10; printf "Fileno:%d\n", fileno $fh; threads->create( \&thread, fileno( $fh ) )->detach; printf 'paused:';<STDIN>; } sub thread{ my( $fileno ) = @_; open my $fh, "<&=$fileno" or warn $! and die; printf "%d:%s", threads->self->tid, $_ while defined( $_ = <$fh> ); close $fh; }

    ...this bit of code might interest you

    ======== zentara wrote: > I wouldn't give up too quickly on letting the threads close and > rename the file. > > First, all threads share the same filehandles thru the filenos. [snipped outline for shared writing to logfile] Heureka! My problem was not getting the logging sorted out. I didn't think that + you could close the filehandle from each thread. But one is actually required to, before the file is free to rename. I should have tried it + before. Thanks for your input! Thomas Here a small example that works as intended: use strict; use warnings; use threads; use Thread::Queue; use IO::File; my $q = Thread::Queue->new(); sub worker { my $fh; while (1) { my $fno = $q->dequeue(); if ( $fno ) { print "worker: opening fileno:$fno\n"; $fh = IO::File->new_from_fd($fno, '>') or die $!; $fh->autoflush(1); print $fh "worker was here\n"; } else { $fh->close(); } } } threads->new(\&worker)->detach(); my $fname = 'test.log'; my $fh; $fh = IO::File->new($fname, '>') or die $!; $fh->autoflush(1); print $fh "main was here\n"; $q->enqueue($fh->fileno()); sleep 1; $q->enqueue(0); sleep 1; $fh->close(); rename($fname, "$fname.1") or die $!; $fh = IO::File->new($fname, '>') or die $!; $fh->autoflush(1); print $fh "main was here\n"; $q->enqueue($fh->fileno()); sleep 1;

    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://807405]
Approved by BioLion
Front-paged by almut
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (9)
As of 2014-07-22 19:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (126 votes), past polls