Correct me if I'm wrong, but I was under the impression that the pointer itself doesn't know whether it's a pointer to a struct, or anything else. The pointer just points to a spot of memory. It's the syntax that you use that determines how the memory gets used.
Perl's references, on the other hand, know what they're referring to. If you try to use a hash reference as an array reference, perl can look at the reference type and say, "wait a minute, that's not an array reference!"
I don't know how much difference this makes in terms of usage, but it's certainly a difference between the two constructs.