Beefy Boxes and Bandwidth Generously Provided by pair Networks vroom
No such thing as a small change

Re: Auto flock

by davido (Archbishop)
on Nov 20, 2012 at 06:56 UTC ( #1004666=note: print w/ replies, xml ) Need Help??

in reply to Auto flock

I can sympathize with your desire to avoid physically touching many lines of code, possibly spread throughout a larger application. Each line you touch is one more opportunity to introduce a bug. Each occurrence of a naked 'open' in your code is one more opportunity to miss applying your flock logic. And it takes time to pour over a lot of code trying to spot those issues. I tend to agree, however, with those who suggest a search and replace approach, mostly because the alternative seems like a fragile hack that could have unintended consequences. Nevertheless, I found the question interesting, and did a little digging.

None of the ideas I'm presenting here should be considered "production ready". They're just brainstorming to see where it leads. Perhaps others might find one of the methods I mention motivating enough to pursue it further.

It seems like if we can have an 'autodie' pragma, then an autoflock shouldn't be such a far fetched idea. But after looking over the source for autodie, which uses Fatal to do its heavy lifting, I have to admit that my first read-through wasn't enough to understand how it works. I certainly didn't learn enough from it to sit down and write my own 'autoflock'. YMMV. ;)

The next thought was layers. It's possible to write custom layers in pure Perl using PerlIO::via. You might be able to use the open pragma to ensure that your custom layer (PerlIO::via::autoflock?) gets applied. But I think that the open pragma is only applied to two-arg versions of open, so if you're (correctly) using three-arg opens, this approach may not pay off.

Then there's the idea of overriding CORE::open and CORE::close do perform the flocking and unflocking. I found Re^2: Override the open builtin globally which shows a fairly robust approach to overriding CORE::GLOBAL::open and CORE::GLOBAL::close. It's possible to adapt the code shown in that node to do our flocking for us. Here's an example (minimally tested, and with some serious problems that I'll mention below):

use strict; use warnings; BEGIN { use Fcntl qw( :flock SEEK_END ); sub lock { my ($fh) = @_; flock($fh, LOCK_EX) or die "Unable to obtain lock: $!\n"; # and, in case someone appended while we were waiting... seek($fh, 0, SEEK_END) or die "Unable to seek to EOF: $!\n"; } sub unlock { my ($fh) = @_; flock($fh, LOCK_UN) or die "Unable to unlock: $!\n"; } *CORE::GLOBAL::open = sub (*;$@) { use Symbol (); my $handle = Symbol::qualify_to_ref($_[0], scalar caller); $_[0] = $handle unless defined $_[0]; # pass up to caller my $result; if (@_ == 1) { $result = CORE::open $handle; } elsif (@_ == 2) { $result = CORE::open $handle, $_[1]; } elsif (@_ == 3) { if (defined $_[2]) { $result = CORE::open $handle, $_[1], $_[2]; } else { $result = CORE::open $handle, $_[1], undef; # special case } } else { $result = CORE::open $handle, $_[1], $_[2], @_[3..$#_]; } lock $handle if defined $result; # CORE::open returns undef on fai +lure. # It could legitimately return '0 +' on # success. See 'Fatal' for detai +ls. return $result; }; *CORE::GLOBAL::close = sub (;*) { unlock( $_[0] ); return close ( $_[0] ); }; }

This probably has unanticipated consequences that I haven't considered (or am only now trying to flag), such as the fact that the behavior will ripple into any use of 'open' within your program (even in modules that you didn't write). Another problem is that the 'lock' and 'unlock' I provide here (mostly copied from flock) can throw exceptions, which means you'll now have to wrap your opens in try{}catch{} blocks, or rework this rough-draft solution to return undef when flock fails. But then it becomes really difficult to distinguish between undef being returned because 'open' failed, or because 'open' succeeded and 'flock' failed. This could be more serious than it sounds. If you return undef as 'flock' fails, you'll be tempted to believe that CORE::open failed, and consequently, that you don't need to close the filehandle. Now you've got a resource leak. So while this method would probably work, it has some challenges that would need to be worked out.

Another problem with this approach is that it makes assumptions about what type of lock you want, and also implicitly seeks to the end of the file. Unfortunately, that's a big issue. Seeking to the EOF after obtaining a lock is necessary if you're appending a file, but makes no sense if you're reading from it. So now you've got an asymmetry; your 'open' "does the right thing" for output files, but very much the wrong thing for input files. Be aware that this issue, as well as the means of differentiating between flock failure and open failure are both big huge deal-breakers, and they both apply to just about any "autoflock" solution I can think of. So, while the "open" logic could be modified to detect whether our open call is for input or for output, or whether our open call is even a "flockable" open, I'm thinking the best answer is "move on and forget about the autoflock idea."

I do like this question. There are probably other approaches aside from those I've considered in this node. It could turn out that (as I mentioned at the top of this node) any solution will end up being a fragile hack that introduced more problems than it's worth. Even though that may be the case, it's been sort of fun thinking through the idea. But in the end, you're probably better off using the manual search/replace function of your editor to iterate over each call to 'open' so that you can visually inspect it and decide whether your 'flock' is appropriate in that specific open. ...and then do the same for all of your closes.


Comment on Re: Auto flock
Select or Download Code
Re^2: Auto flock
by ColonelPanic (Friar) on Nov 20, 2012 at 10:50 UTC
    Thanks for this very useful post. I would just like to add that you don't have to override CORE::GLOBAL::open, etc. Instead, you can override open and close in only the current package:
    use subs 'open'; sub open (*;$@) { print "Open sesame!"; #Need logic as Dave outlined for calling CORE::open return CORE::open [...]; } open my $fh, '>file.txt' or die "Ouch!";

    This will solve one of the problems you mentioned, as now you are not overriding modules' use of open and close.

    The problem of lock failing wouldn't really be a problem as long as you handled it properly in your open function--just catch any errors there and close the filehandle if lock fails.

    The problem of seeking to the end of the file would be pretty easy to handle as well: detect that the file was opened for appending, and only seek to the end if that is true.

    Still, I do tend to think that this solution is more hacky than it's worth. Overriding system functions is not really recommended. It makes for confusing code, and there still may be unintended side effects.

    I think space_monk's suggestion of making your own, differently named, open and close functions that lock and using them where needed would be easier overall.

    When's the last time you used duct tape on a duct? --Larry Wall

      Your assessment of how to fix some of the concerns I raised is good. One could certainly build logic into the tool to deal with read versus write. And the failed flock issue could also be handled internally with better logic. There are other issues as well though. One is that code isn't usually written with the notion that 'open' could block, which is exactly what would happen with 'autoflock' if a file is already locked. That flock normally blocks to wait for a lock is expected behavior of that function, but not expected of open. That could break existing code.

      So those are the issues we know about. My fear is the edge cases we haven't considered and aren't aware of. If we were talking about overriding 'chop', I'd say no big deal. open is a little more complicated, and the stakes are higher.

      Yes, space_monk's suggestion is probably the one.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1004666]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (7)
As of 2014-04-19 16:12 GMT
Find Nodes?
    Voting Booth?

    April first is:

    Results (483 votes), past polls