The difference is that with 
is part of a character class, which means that anything (in this case a line) within the 
will constitute a match at this position. So in your code, [htm|asp]
anything that contains any of the following characters will match: h t m | a s p
On the other hand with parenthesis you are asking the regex engine to match anything with htm OR asp, and hence output.php doesn't match.
You might want to have a look at perlretut and perlre for more info.