You should have done the Perlish API stuff in wrappers written in Perl that called the SWIG / XS / Inline::C wrappers.
As Joost said, it depends. In the project I was referring to, SWIG simply didn't provide enough flexibilty to get the (sometimes weird and dynamic) data structures converted into anything which would have remotely made sense in Perl. In other words, doing this part in Perl essentially would've meant returning big chunks of memory and then fiddling with pack()/unpack() on the Perl side to create the objects as needed. This certainly wouldn't have made it less fragile. Changing the C side wasn't an option, as it was a closed source third party library.
Well, I don't want to go into further details. I'd just like to point out that in the end, the flexibility of XS did allow me to write less and faster, i.e. more-to-the-point code than SWIG did.
Ugh. The "cool" factor of writing ugly, fragile code.
Did you notice the smiley?