Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

Re: Best way to Download and Process a XML file

by BrowserUk (Pope)
on Sep 25, 2012 at 15:02 UTC ( #995564=note: print w/replies, xml ) Need Help??

in reply to Best way to Download and Process a XML file

The xml feed can be quite huge, (~150Gb max.)

Often as not with XML files that big, the feed consists of one top level tag that contains a raft of much smaller, identical (except an ID) substructures:

<top> <sub name=1> ... </sub> <sub name=2> ... </sub> ... </top>

It therefore becomes quite simple to do a preliminary parse of the datastream and break the huge dataset down into manageable chunks for processing:


C:\test>\perl64-10\bin\perl { "sub" => { name => 1, subsub => "\n some stuff\n " } +, } { "sub" => { name => 2, subsub => "\n some stuff\n " } +, } { "sub" => { name => 3, subsub => "\n some stuff\n " } +, } { "sub" => { name => 4, subsub => "\n some stuff\n " } +, } { "sub" => { name => 5, subsub => "\n some stuff\n " } +, }

Of course, this 'breaks the rules' of XML processing, and requires you to assume some knowledge of the details of the XML you will be processing. But then the kinds of details required are usually, a) easily discovered; b) rarely change; c) easily catered for when and if they do change.

So if you favour the pragmatism of getting the job done over more esoteric -- and revenue sink -- criteria such as 'being correct', bending the rules a little can save you a lot of time, effort and expense.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://995564]
[marto]: Discipulus the issue that should be adressed is that the page needs to be updated to reflect modern perl on Windows
[Discipulus]: but is really necessary to support these ancient versions? from 5.14 onward is not enough?
[Discipulus]: yes marto I understood
[Corion]: Discipulus: I'm slowly migrating my code to require 5.8.x ;) Most of my code works on 5.6, but Filter::signatures requires 5.10 I think
[Corion]: 5.14 is also a good target, widely available and fairly stable. Also, over five years old, which doesn't push people into a needless upgrade cycle
[Discipulus]: yes I saw your efforts Corion but the pain is worth?
[Corion]: Discipulus: IMO it's not that much pain, but most of the systems I use have a Perl 5.14 or newer on them nowadays
[marto]: the question is more, is Padre still the way new users should be directed?
[Discipulus]: ah ah I have all 5.26 atm but I play alone..
[Corion]: Except this RHEL box, which comes with Perl 5.10.1 , but there I installed my own 5.20.3 for the real Perl programs we run ;)

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (10)
As of 2018-06-25 08:49 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (126 votes). Check out past polls.