I have used all three.
Inline::C will immediately work for simple things, and you'll be very gratified and encouraged -- enough so that you'll quickly expand to use it for all kinds of things and wrap your whole UI. Then you'll run into some small problem. Probably something to do with reference counts. You've made it so far already, it can't be that hard to just fix up the one little problem... soon you'll be learning large portions of XS, in a completely ass-backwards order, and just before you realize that you've completely lost track of what led you down that hellish path, you'll go insane.
XS will require a big investment of time and documentation-reading up front, which will do you no good whatsoever, since everything you try will break for bizarre and subtle reasons. You'll think you understand it, and get more and more confident as you read the documentation and feel everything falling neatly into place. Until you actually try to use it, when nothing will seem to work at all like what you read, even though you can't find anything that directly contradicts what you read. Looking at the generated C code will make you even more confused.
At some point, you'll realize that the only way to make any headway is to copy an existing XS interface and mutate it into what you want. This will mostly work, and you'll gradually make progress. Gradually but not steadily -- you will frequently encounter major roadblocks that will leave smatters of blood and brain pulp on the nearest brick wall. It's a bit like crossing a freeway with a chihuahua chewing on your ankle, getting flattened by several cars and an SUV on the way across. Still, everything you learn will be useful; you'll just have to gradually build up your knowledge in stages.
SWIG is bliss. It effortlessly wraps everything you want (and far more, since it traces through a bunch of headers you didn't really intend). It presents the API to you in nice clean Perl.
But it's slow. And 20% of the time, it doesn't work at all. Another 20% of the time, it almost works but does something blindingly idiotic. And when it doesn't work, it'll be like dissecting a frog in high school science class to figure out what's going wrong. You know -- those special high school science class frogs, that have groxbitner where the textbook says they should have lungs, sporkinits instead of intestines, and a blob of yerfikation in place of a heart. In other words, there are layers and layers of nonsensical internals specifically designed to make your brain dribble out your ears.
All clear now?