http://www.perlmonks.org?node_id=951822


in reply to Re: Incremental XML parsing
in thread Incremental XML parsing

In the XML::Twig docs, it says:
WARNING: this option is NOT used when parsing with the non-blocking parser (parse_start, parse_more, parse_done methods) which you probably should not use with XML::Twig anyway as they are totally untested!

Replies are listed 'Best First'.
Re^3: Incremental XML parsing
by mirod (Canon) on Feb 05, 2012 at 07:59 UTC

    XML::Twig can (and is actually designed to) parse big document. It doesn't do it by using an incremental parser, but by building a (non-DOM) tree for each element it parses. It lets you call handlers on elements, using the twig_handlers or twig_roots options. These handlers are called as soon as the element is completely parsed. In the handler you can process the element, then release the memory it used, by calling purge or flush (which also outputs the tree so far). This doesn't work when you need to have the entire tree available for processing, but in practice it does work in most cases, where you can treat the overall XML as a collection of independant elements. See the section Processing an XML document chunk by chunk in the docs.

    Using XML::Parser incremental parsing methods probably works in XML::Twig, I just never tested it, because that's not what XML::Twig needs. It does just fine with the regular interface, calling handlers during parsing.

    XML::LibXML offers a different interface to parse incrementally the XML, look into XML::LibXML::Reader.

      XML::LibXML offers a different interface to parse incrementally the XML, look into XML::LibXML::Reader.
      It's like nobody's reading the entire thread. I specifically asked about using XML::LibXML::Reader in my OP.

        It's like nobody's reading the entire thread. I specifically asked about using XML::LibXML::Reader in my OP.

        You also asked Or is there a better alternative?

Re^3: Incremental XML parsing
by Corion (Pope) on Feb 04, 2012 at 16:53 UTC
    Thatg warning is from the documentation of "keep_encoding". I don't understand how it applies to the problem of parsing a file withohut creating the complete DOM in memory. Can you please explain how it applies?
      Because it explicitly says that the non-blocking parser should not be used with XML::Twig and is untested. Yes, the author chose to include the only mention of the non-blocking parser in the documentation for an unrelated option, but the docs in general are quite confusing. As it stands, XML::LibXML::Reader has similar functionality to XML::Twig, but is much, much faster. But neither appear to be able to parse XML incrementally. Note this means not only to not produce the entire DOM at once, but also to be able to consume the input document in pieces, such as feeding data as it arrives over the wire.
        In the OP, I see only a requirement to parse large documents, in which case Twig (or XML::Rules) would likely be fine.