Cool symlink hack

[grantm]

There's nothing Perl-specific about this hack, but our implementation is in Perl so here goes ...

We have an 'n-tier' architecture where some number of Apache/mod_perl/Mason frontend servers handle the user interaction and use XMLRPC (via Frontier::RPC) to call business functions on an Apache/mod_perl backend application server. This presents us with a couple of related configuration problems:

• Each frontend server needs to know the address of the backend server
• If the backend is down for maintenance, the frontend needs to know so that it can tell the users ("come back in 10 minutes" etc)

The quick solution to the first problem was to hard code the address using PerlSetEnv in the Apache config. Of course this meant that changing the address required a restart of the frontend server.

A better solution that happened to solve both problems was to use 'hanging' symlinks.

A hanging symlink is a symbolic link to a non-existant file. For example, you could create one like this:

ln -s 192.168.1.159:80 /var/run/backend.lnk

There is no file called '192.168.1.159:80'. The significance of that string is that it contains the IP address and port number of the backend server. To get that information out of the symlink requires one line of Perl:

my $host_port = readlink('/var/run/backend.lnk');

This translates into one system call that either returns a string on success or undef on failure. This is a much lower overhead than opening a file, reading a line of text and closing the file. There's also no need to worry about locking or concurrency issues (although see below for more on this).

In our case, the absence of a symlink implies the system is down for maintenance. However we did take things one step further and allowed a second type of value in the symlink, eg:

ln -s 2005-11-15-23:55 /var/run/backend.lnk

If readlink() returns a value in the form YYYY-MM-DD-HH:MM then the frontend can report two things to the user:

• the system is currently down for scheduled maintenance
• it is expected to be available again at 11:55pm

Our code checks that the time value is in the future. If not, we simply omit the second part of the message.

Race Conditions

There is one potential race condition in this setup. If you use the command 'ln -sf' command to replace a symlink with a new value, two system calls are issued - an unlink() followed by a symlink(). If you were changing the link from one backend server address to another, then there would be a very brief period in between where the symlink did not exist and the frontends would signal a 'system unavailable' message back to the user.

In our case, that's perfectly all right. However if you do want a guaranteed seamless switchover, Perl comes to the rescue:

symlink("192.168.1.160:80", "new_backend.lnk");
rename("new_backend.lnk", "backend.lnk");

The rename is an atomic operation, so readlink() calls will either see the value before the rename or the value after it.

So there you have it, only three lines of Perl but pretty cool nonetheless.