<?xml version="1.0" encoding="windows-1252"?>
<node id="995446" title="Best way to Download and Process a XML file" created="2012-09-24 18:04:36" updated="2012-09-24 18:04:36">
<type id="115">
perlquestion</type>
<author id="995444">
perl_gog</author>
<data>
<field name="doctext">
Hello perl-monks, &lt;br&gt;
I am trying to: &lt;br&gt;
  a) download a xml file &lt;br&gt;
  b) do some basic processing on this xml file and &lt;br&gt;
  c) finally save it. &lt;br&gt;

&lt;br&gt;
I was wondering what is the best option to do this: &lt;br&gt;
&lt;br&gt;
option 1. &lt;br&gt;
&lt;code&gt;
  system("wget -O - 'http://host/getFeed.xml' &gt; /tmp/myfeed.xml"); //download + save as tmp file..
  open FH, "&lt;/tmp/myfeed.xml";
  while(&lt;FH&gt;) {
     //manipulation steps here..
  }
  close FH;
&lt;/code&gt;
&lt;br&gt;
Option 2: &lt;br&gt;
   Is there a way to process the file, as its being downloaded? (instead of having to save it temporarily, and then reading it to process it) ?&lt;br&gt;
&lt;br&gt;
I dont know how to implement option#2. Do you have any suggestions? &lt;br&gt;
The xml feed can be quite huge, (~150Gb max.) As, the feed is huge, I am hoping there's a better option compared to option #1, as writing this file first to some  tmp file and then saving it again would mean more disk-activity. &lt;br&gt;
&lt;br&gt;
thanks! &lt;br&gt;</field>
</data>
</node>
