in reply to what are the common causes of "system" command failure on Vista
(Assuming the answer to my question above is yes.)
The problem is that you are creating asynchronous processes, but never gathering their return codes via wait or waitpid.
In order to provide a way to retrieve the exit codes from asynchronous processes, the windows implementation has an internal limit of 64 un-waited asynchronous processes. Call one of the above two calls once the process has finished, and you will "fix" the problem.
That said, if your code is starting one process at a time, and then hanging around polling a 'lock' file before continuing; why on earth are you using asynchronous processes and lock files?
If you just called the synchronous variant of system, you wouldn't need the lock files, or to call wait/pid.