On Win32 trying to open a file for write that is locked is an 'Access is denied' error.
Perhaps that's the case if another process has opened the file and used mandatory locking. However, this snippet certainly behaves as expected on Win32 (ie: a second instance successfully opens the locked file and waits for the lock to be released):
use Fcntl qw(:flock);
open(FILE, ">sem.lck") or die $!;
print "Opened file\n";
flock(FILE, LOCK_EX) or die $!;
print "Got lock\n";
sleep(10);
close(FILE);
I do exactly this on Win32 all the time.
You're right, there is a potential race condition if you want to lock the file you're editing and you need to create it if it doesn't exist. The way around that is to use sysopen like this:
sysopen(FILE, $path, O_RDWR|O_CREAT|O_EXCL) or die $!;
The O_CREAT flag specifies that the file should be created if it does not exist and the O_EXCL specifies that the create should fail if the file does already exist. But this quickly gets complicated because you need to use one form if the file doesn't exist and another if it does. That's why it is so much simpler to use a separate file for locking than the file you're editing.