I share your double point of view. Using external code is sensitive but may be very rewarding.
Now, the other side of the coin is that Perl modules (as all Perl code) is distributed as source code. You can inspect all code to verify that it suits your requirements and make necessary changes.
In theory, you could accept the project of rewriting all the XML related modules you need in your shop, copy them from CPAN and tell the folks you wrote it yourself. (That wouldn't be a very nice thing to do, but as a theoretical example it works.)
Then imagine that you re-write the whole lot. Besides from taking a lot of time, your code will most likely not be as well tested as the code you find in the most popular CPAN modules. I would guess that you would introduce more bugs by re-writing the stuff than by relying on already tested code. (With all due respect. ;-)
In our shop we do extensive testing and we are happy with that.
My (and our) opinion is that the art is not to reinvent the wheel, but to chose which existing wheel you should get to your vehichle. I.e. chosing the right, mature and extensivly tested CPAN modules to use is where you should put your efforts.
Everything went worng, just as foreseen.