You should have done the Perlish API stuff in wrappers written in Perl that called the SWIG / XS / Inline::C wrappers. Minimize the amount of code you write in SWIG / XS / Inline::C and you'll be happier in the long run (as will your users). That's why I advocate using Inline::C (it encourages simple, C-friendly interfaces which encourages writing the complex interface in Perl).
Lastly, and somewhat less seriously: using XS would likely boost your ego with that warm and fuzzy feeling of having done it the way most 'real hackers' would do ;)
Ugh. The "cool" factor of writing ugly, fragile code.