Perl's behavior isn't defined when you read the value of a variable and then auto-increment/decrement it later in the same statement. Try doing this instead and you'll get the behavior you expect:
$| = 0;
$| = 1;
In your first case, Perl evaluates the second $|, then decrements it, then evaluates the first one, leading to the unexpected behavior. The reason you don't end up with "-1" is that there is magic associated with $|. It doesn't actually hold a value, it just tracks whether it is set to a non-zero value or not. Try for instance $| = -3; print $| to see this behavior.
Read perlvar and perlop for more information on what is going on here.