system() is a convenience wrapper around fork/exec. Don't expect to use signals with it and have things work well. The classic example of this kind of failure is when someone uses alarm with system: The ALRM occurs, the parent then continues, but the child doesn't receive the ALARM and keeps running (on some platforms at least). I've seen similar cases where system returns an invalid result if I've already forked off an unrelated process and it happens to die when system() is executing.
If you want to trap signals, it's better to do this yourself For example, fork/exec the child and have the parent call waitpid. If the waitpid is interrupted by the alarm, have the parent explicitly kill the child. An alternative is not to use alarm at all, but just to call waitpid about once a second. When the time is up just kill the child process.
Update: Clarified a little