Parsing extremely large XML files

[Spitters]

Hey. I need to parse an extremely large XML database. First I want to generate simple ASCII lists of all values of certain elements on various levels in the XML structure. So that's a on-time action and I don't really mind if it takes a while to finish. But, later I also want to be able to receive elements from the database really fast, or find element values real-time and stuff. I don't have any experience with XML modules for Perl, so my questions are: 1) what's the most appropriate XML parser for processing large XML files? 2) can anyone provide some code for basic operations? Thanks a lot, Martijn

[samtregar]

You'll need to use a SAX parser (XML::SAX) or a SAX-like stream parser (XML::Parser) to keep from loading the whole file into memory when you parse. Also, you'll need to think carefully about what you do with the data as you parse it; it's no good using a stream parser if you just build a giant hash using it! I had success parsing 500MB of XML into a MySQL database with XML::Parser, but these days I would probably go to the shiny new SAX route.

-sam

[simeon2000]

Although I haven't done much XML parsing in my day, I am positive that SAX parsers are better fitted to looping through extremely large XML files than DOM parsers. The reason being is that SAX parses the content line-by-line and is parsed using event-handler code, whereas DOM loads it all into memory at one go.

Although there seem to be a plethora of SAX related modules on CPAN, I would reccomend starting with one like XML-SAX.

"Falling in love with map, one block at a time." - simeon2000

[cebrown]

Others have correctly commented that a SAX parser is the way to go. I'll focus in on something else -- you mention that you want to get elements from the XML really fast.

If that's the case, you ought to be thinking about parsing the document once into a database, then using a database for future access. They're really not all that scary, just grab and install Postgres or something else and you can be on your way.

Databases will be good for you today because you can define indexes for searching, and they will be good for you tomorrow because odds are your "really fast" searching will become "really fast, and pretty complex, and I want to make updates too..." as time goes on.

If you don't use a database, you will need to SAX parse your whole XML document every time, and although SAX is really fast, the I/O alone will kill you.

[mp]

Checkout XML::Twig, "A perl module for processing huge XML documents in tree mode". (See also XML::Twig website).