This is why you do not see any change in the size of the process' memory allocation, when you free te XML::Simple object. But reast assured, that memory is available to the rest of the program should it require it.
The OP used (145908-35476)/4=27608 pages. Some of those pages HAVE to be returned to the OS and removed from the process, anything else is fragmentation leading to a DOS or a leak. All his memory could be speculative reserves for growth space for existing allocations by the memory allocator and therefore is not accessible.