The xml feed can be quite huge, (~150Gb max.)

Often as not with XML files that big, the feed consists of one top level tag that contains a raft of much smaller, identical (except an ID) substructures:

<top>
    <sub name=1>
        ...
    </sub>
    <sub name=2>
        ...
    </sub>
    ...
</top>
[download]

It therefore becomes quite simple to do a preliminary parse of the datastream and break the huge dataset down into manageable chunks for processing:

Produces:

C:\test>\perl64-10\bin\perl 995446.pl
{
  "sub" => { name => 1, subsub => "\n           some stuff\n       " }
+,
}

{
  "sub" => { name => 2, subsub => "\n           some stuff\n       " }
+,
}

{
  "sub" => { name => 3, subsub => "\n           some stuff\n       " }
+,
}

{
  "sub" => { name => 4, subsub => "\n           some stuff\n       " }
+,
}

{
  "sub" => { name => 5, subsub => "\n           some stuff\n       " }
+,
}
[download]

Of course, this 'breaks the rules' of XML processing, and requires you to assume some knowledge of the details of the XML you will be processing. But then the kinds of details required are usually, a) easily discovered; b) rarely change; c) easily catered for when and if they do change.

So if you favour the pragmatism of getting the job done over more esoteric -- and revenue sink -- criteria such as 'being correct', bending the rules a little can save you a lot of time, effort and expense.