Actually, xsubpp produces set of code that does conversion from structures specific to perl to those understandable for your code. In fact, you can write C code that works with perl structures directly, in which case no conversion has to be done. Just make your function work with AV* or for example SV*.
Inline::C and Co. provide automatic conversion to some set of common types, just like XS does. If using these types is enough, then you don't need anything else but those modules. If you have to work with more complex data structures, then you have to understand perl's guts. Read perldoc perlguts. Then read perldoc perlxs to learn how to work with XS. Then read perldoc perlapi to see which functions are available to you for working with perl's structures. XS shall only simplify certain things for you.
After all, you can think of Perl as of C library providing certain API.