XML::Simple is designed for simple XML documents. It is not designed to be a simple to use XML parser. Trying to do "interesting" stuff with XML::Simple is a recipe for misery. In the long run (say in an hour) converting to something like XML::Twig will be less tedious and make your life much happier than persisting in trying to hammer XML::Simple into shape for a job it wasn't designed for.
Seriously, have a serious crack at using XML::Twig. If you haven't made solid progress in an hour come back for some tips.
Premature optimization is the root of all job security