A quick look with Data::Dumper shows me that Spreadsheet::ParseExcel has circular links in its structures. Namely, the S::P::Cell objects have a Format key which refers the parent object. Thus, if those links are not weakened, the structure will not be released properly when the objects go out of scope.
To fix this would probably require a patch involving the use of weaken in functions that make S::P::Cell objects, or DESTROY methods which do that when the object is about to be garbage collected.
If a patch is not an option for you, you could carefully fork a new process for each file or break up the task in some other way.