Vexing Race Condition

[Maelstrom]

I have a long running script that logs it's process to a file and an FCGI that reads said file every 200ms when prompted to by ajax calls and it seems to mostly work but occasionally fails. I believe I've recreated the problem with the following 2 SSCCE's

one.pl

use Fcntl qw (:DEFAULT :flock :seek);
my $file = '/dev/shm/tmpcount.txt';

for (1 .. 180) {
  print "$_\n";
  &filewrite($file,$_);
  sleep 1;
}

sub filewrite {                            
  my ($file,$enc) = @_;
  open(my $fh,">", $file) || die "Can't open $file for reading: $!";
  my  $mode = LOCK_EX; 
  flock($fh,$mode) or die "Couldn't lock $file for write";
  print $fh $enc;
  close $fh;
}

two.pl

use Fcntl qw (:DEFAULT :flock :seek);

my $file = '/dev/shm/tmpcount.txt';
my $c;

do {
  if (-e $file) {
    $c = &fileread($file);
    die "C is empty" unless ($c);
    print "count is $c \n";
  } else { print "File not found\n"; }
} until ($c == 180);

sub fileread {                                                        
+                         # Read using perl IO 
my $file = shift;
open(my $fh, "<", $file) || die "Can't open $file for reading: $!";
my $mode = LOCK_SH;
flock($fh,$mode) or die "couldn't lock";
my $string = do { local $/; <$fh> };
close $fh;
return $string;
}

When I use warn instead of die the script appears to work, as it is it will invariably complain that $c is empty at some point. What I think is happening is one.pl opens the file and truncates it but before it gets to calling flock two.pl swoops in and reads an empty file. I suppose the obvious solution is semaphores but as I'm trying to develop a "safe" file handling library I'm wondering if there's another way to go about it. My first thought was an atomic open and lock with sysopen but apparently linux can't do that. I'm wondering if opening the file readwrite and truncating after flock would solve this or would possible race conditions remain? I'm aware if multiple processes are doing read-mutate-write then a semaphore lock is the only way to go but I'm curious if 1 writing and 1 reading process can work without them.

[tybalt89]

Try this. Path::Tiny::spew writes to a temporary file and then atomic renames to the final name. This should never allow an empty file.

#!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11164885
use warnings;
use Path::Tiny;

#one.pl

my $file = '/dev/shm/tmpcount.txt';

for (1 .. 180) {
  print "$_\n";
#  &filewrite($file,$_);
  path($file)->spew($_); # NOTE writes to temp file, then atomic renam
+es over
  sleep 1;
}

#two.pl

my $file = '/dev/shm/tmpcount.txt';
my $c;

do {
  if (-e $file) {
#    $c = &fileread($file);
    $c = path($file)->slurp;
    die "C is empty" unless ($c);
    print "count is $c \n";
  } else { print "File not found\n"; }
} until ($c == 180);

[Maelstrom]

Of all the wheels I regret re-inventing Path::Tiny may be the most regrettable. The atomic renaming seems to have eradicated this race condition entirely. Cheers.

[NERDVANA]

Completely aside from your question but very related to your problem, I solve a similar task with my module Log::Progress. I run my long-running task as a pair of processes - the parent sets up the environment and runs the main worker process, while monitoring the log file which is the child process's STDOUT and STDERR. The long-running script just keeps writing to STDOUT without ever locking or touching file names. The parent uses Log::Progress to incrementally process the output file, building a Progress data structure that describes the overall progress, and progress of sub-tasks. When the child exits for any reason, the parent gets to capture the exit code and even see if the child aborted with a low-level error like libc running out of memory. Meanwhile, the parent is writing updates to a database record, and the web workers report the progress to clients reading from that database record. If you are using Postgres, you can even have a trigger on that table which pushes notifications through a websocket to the clients :-)

Another technique to read log files is to write javascript that makes HTTP Range requests to just keep looking for additional data on the end of the file. Then you don't even need a perl handler, apache can serve it for you. Of course, this assumes that it's OK for users to see the entire stdout of a background process, which in many cases could contain secret data.

That might be way more complication than you want to get into for this project, but I figured it was worth mentioning some of your other options.

[ikegami]

open(my $fh,">", $file) truncates the file before you lock it.

[Marshall]

As ikegami points out, the file open truncates the file before you have a chance to lock it.

A common solution is to use a different file for the cooperative locking. Make a file, "lockfile_logx" and get and release locks on that file's file handle to protect all of the operations on $file.

[tybalt89]

Or use Path::Tiny's trick: open the file for append, lock it, truncate to 0, then write new data :)

( Untested, let me know if it works :)

[Maelstrom]

Actually Path:Tiny's trick is renaming a temporary file. My idea of opening read-write, locking then truncating turns out to be a non starter as mode +> clobbers the file and mode +< fails if the file doesn't already exist, I was able to run the script with no errors after touching the file although in the wild that would be another race condition. Opening the file in append mode as you suggest does work, so it's an option although using rename is probably more robust as it will also work for processes that don't use flock.

[Maelstrom]

It was my understanding that using a different file was referred to as a semaphore lock. I'm less sure after googling the term but I'm sure I've seen people refer to a 'semaphore file'.

[cavac]

Other have already provided you with a solution to your code problems. If you want an alternative to using temporary files for sending the progress of your script to some other program, Net::Clacks might be of help to you Net::Clacks for IPC (2024 tutorial).

Even Santa uses it to monitor his factory: https://perladvent.org/2024/2024-12-13.html