|laziness, impatience, and hubris|
Best way to Download and Process a XML fileby perl_gog (Initiate)
|on Sep 24, 2012 at 22:04 UTC||Need Help??|
perl_gog has asked for the
wisdom of the Perl Monks concerning the following question:
I am trying to:
a) download a xml file
b) do some basic processing on this xml file and
c) finally save it.
I was wondering what is the best option to do this:
Is there a way to process the file, as its being downloaded? (instead of having to save it temporarily, and then reading it to process it) ?
I dont know how to implement option#2. Do you have any suggestions?
The xml feed can be quite huge, (~150Gb max.) As, the feed is huge, I am hoping there's a better option compared to option #1, as writing this file first to some tmp file and then saving it again would mean more disk-activity.