Yes, it doesn't make much sense to have the whole handler in a separate thread. What could be done is for the handler in the main thread to extract the data it needs, and then to do the processing in a separate thread. that is assuming that the data can be extracted just from the current element and that processing it doesn't change the original XML.
An other option might be to split the initial XML and then to process those in parallel. xml_split, a tool that comes with XML::Twig could do this.
That said, it is indeed stange that it takes so long to process the data. I somehow doubt that the XML parsing is responsible for this.