Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re^10: [threads] Open a file in one thread and allow others to write to it

by gulden (Monk)
on Nov 16, 2009 at 16:07 UTC ( #807500=note: print w/ replies, xml ) Need Help??


in reply to Re^9: [threads] Open a file in one thread and allow others to write to it
in thread [threads] Open a file in one thread and allow others to write to it

The below output file, generated by your code is not enough to clarify you?

1: The time is 1258377733.481062 1: The time is 1258377733.481164 1: The time is 1258377733.481337 1: The time is 1258377733.4 3: The time is 1258377733.473267 3: The time is 1258377733.473595 3: The time is 1258377733.473749 3: The time is 1258377733.473885 3: The time is 1258377733.474008 3: The time is 1258377733.474110 3: The time is 1258377733.474212 3: The time is 1258377733.474313

Your code without the sleep()'s:

#! perl -slw use strict; use Time::HiRes qw[ time sleep ]; use threads; use threads::shared; our $THREADS ||= 4; my $sem :shared; open LOG, '>', 'log.txt' or die $!; sub logit { lock $sem; return printf LOG @_; } sub thread { my $tid = threads->self->tid; my $stop = shift; warn $tid, ": starting:", time(), " Stoptime: $stop\n"; while( time() < $stop ) { logit "%2d: The time is %f\n", $tid, time; } warn $tid, ": stopping at ", time(), "\n"; } my @threads = map threads->create( \&thread, time() + int( rand 10 ) ), 1 .. $THREADS; warn "threads started; waiting\n"; $_->join for @threads; warn "threads done\n"; open LOG, '<', 'log.txt' or die $!; print while <LOG>; close LOG;

My position in this forum is to share ideas,opinions,knowledge,... and of course, "to learn", with the help of all the monks. And only joining strengths it will be possible to achieve real knowledge. Sometimes this forum is used to humiliate members in order to praise to himself, which is a pity.

A contentious debate is always associated with a lack of valid arguments.


Comment on Re^10: [threads] Open a file in one thread and allow others to write to it
Select or Download Code
Re^11: [threads] Open a file in one thread and allow others to write to it
by BrowserUk (Pope) on Nov 16, 2009 at 16:43 UTC
    The below output file, generated by your code is not enough to clarify you?

    It is, now you've posted it. Which is what I asked you to do 3 posts ago!

    And the system you are running this on?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      We have had a similar thread in the past 764973.
Re^11: [threads] Open a file in one thread and allow others to write to it
by ikegami (Pope) on Nov 16, 2009 at 17:45 UTC

    There's no race condition there. Notice the timestamps aren't in order? You never ask Perl to write to disk, so it writes in 4k chunks, whether the chunk ends in the middle of a line or not.

    Simplest solution: Change

    open LOG, '>', 'log.txt' or die $!;
    to
    use IO::Handle qw( ); open LOG, '>', 'log.txt' or die $!; LOG->autoflush(1);

    If you're still having jumbled output after fixing the above problem, there might be a problem keeping file pointers in sync. Use the following before the print:

    use Fcntl qw( :seek ); seek(LOG, 0, SEEK_END);

    I don't think it'll come to that, though.

      The AUTOFLUSH does not solve the problem

      The SEEK works !!! and it works without the lock in $sem and the AUTOFLUSH in LOG. However, IMHO the lock in $sem must be used

      In overall i've understand your tip, but it's not clear for me how all the file pointers are syncked in a thread concurrency level, without conflicts. With the lock in $sem it's obvious , because only one thread can SEEK the Filehandle and write to it. However, I'm getting the log file syncked without using the lock in $sem. But these are other discussions

      Thanks for the tip it will be very useful for me...

        and it works without the AUTOFLUSH in LOG.

        seek apparently forces a flush. Assuming it's safe to rely on that, it's not good enough. The catch is that it's forcing the flush at the wrong time. You want to flush immediately after the print, not at the start of the thread's next print.

        and it works without the lock in $sem

        You're relying on your OS to do writes atomically. That might be reliable for small writes, but why chance it. If you do find a performance bottleneck, you might be better off using a separate log file for each thread and merge them later.

Re^11: [threads] Open a file in one thread and allow others to write to it
by BrowserUk (Pope) on Nov 16, 2009 at 19:21 UTC
    Hum, "buffered IO" and "buffer-fulls"

    Remember what I was saying about buffering?

    Your modification of my code, mildy modified:

    #! perl -slw use strict; use Time::HiRes qw[ time sleep ]; use threads; use threads::shared; our $THREADS ||= 4; my $sem :shared; open LOG, '>', 'log.txt' or die $!; my $old = select LOG; $|++; select $old; sub logit { lock $sem; return printf LOG @_; } sub thread { my $tid = threads->self->tid; my $stop = shift; warn $tid, ": starting:", time(), " Stoptime: $stop\n"; while( time() < $stop ) { logit "%2d: The time is %f\n", $tid, time; } warn $tid, ": stopping at ", time(), "\n"; } my @threads = map threads->create( \&thread, time() + int( rand 10 ) ), 1 .. $THREADS; warn "threads started; waiting\n"; $_->join for @threads; warn "threads done\n";

    When I run that under *nix:

    mehere@mehere-desktop:~/perl$ perl -v This is perl, v5.10.0 built for x86_64-linux-gnu-thread-multi ... mehere@mehere-desktop:~/perl$ wc -l log.txt 173746 log.txt

    It created a 173000 line file that is flawless!


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      This is the solution 807525, until proven otherwise.
      A contentious debate is always associated with a lack of valid arguments.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://807500]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (12)
As of 2014-10-23 10:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (125 votes), past polls