Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re^3: Synchronisation between multiple scripts

by eye (Chaplain)
on Jan 16, 2009 at 09:23 UTC ( #736784=note: print w/replies, xml ) Need Help??

in reply to Re^2: Synchronisation between multiple scripts
in thread Synchronisation between multiple scripts

Does your advisary locking of a pid file scheme, handle recovery after a system crash?

If you review the mentioned thread, the first node by shmem details how the script should verify that the listed PID is actually running the proper script.

Polling directories is a silly way to do things.
Silliness is relative. If the files only need to be pushed to the server every hour, polling doesn't seem so bad.
How long do you wait after the filename appears, before you decide that the application writing that file has finished doing so?
How do you detect if the process writing the file has hung or crashed part way through writing it?
In the first approach, I specifically said " move files to the watched directory...." This is an important point; perhaps I should have explicitly stated that files should be moved to the directory rather than written to the directory.

Update: Corrected "rather than copied" to "rather than written" in the last sentence.

Update 2: Corrected HTML formatting of whitespace before update 1.

  • Comment on Re^3: Synchronisation between multiple scripts

Replies are listed 'Best First'.
Re^4: Synchronisation between multiple scripts
by BrowserUk (Pope) on Jan 16, 2009 at 11:23 UTC

    It is a relatively non-controversial fact that most people's attempts to implement robust & recoverable resource serialisation through advisory file locking, will fail under a myriad of exceptional failure scenarios.

    The most difficult of these to handle is complete systems failure at critical points in the handshaking. That is, when both the resource controller and the resource requester terminate simultaneously with no possibility for cleanup, back out or status logging.

    And even when starting with an expertly prepared and tested scheme of operations, the process of adaption of that scheme to the natural variations in application requirements, creates ample opportunities to introduce subtle but fatal errors.

    Upon restart, the advisory locks are simply forgotten; pid files may exist but the processes no longer do; or worse, the pids have been reallocated to completely unrelated processes.

    And the shared resource transaction can be in any state.

    Using the OPs example of a file transfer, the file in question may not yet exist in the application space; may exist in a partial form in the application space; may exist in a completed form in the application space;

    It may exist in the controller space in any of those three forms. That is, the name of the file may have been recorded within the transfer processes namespace, but the handover of the file may not yet have started; may have started but not completed; may have completed but not been cleaned up.

    And if the handover has completed and been cleaned up (in the application space) the transfer may not yet have started; may have started, but not yet been completed; may have completed, but not been cleaned up;

    In the absence of persistent and atomic, bilateral status recording, a complete systems failure can leave either end of the transaction with no information; or partial and differing information.

    I've yet to see a resource serialisation scheme, based around advisory file locking, that gets it right in all scenarios. And I once wasted nearly 3 months on a critical project waiting for the experts to do so.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://736784]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (4)
As of 2018-01-21 07:18 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (227 votes). Check out past polls.