<?xml version="1.0" encoding="windows-1252"?>
<node id="1003811" title="Re: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)" created="2012-11-14 09:20:08" updated="2012-11-14 09:20:08">
<type id="11">
note</type>
<author id="961">
Anonymous Monk</author>
<data>
<field name="doctext">
If you need to modify a flat-file, and the size is such that it can't be slurped into memory (and a 1GB file probably &lt;i&gt;can&lt;/i&gt; be ...), it's best to process the file one record at a time writing the modified records to another file.  Then, swap the file-names around.  If you slurp into memory, run a performance monitor to see if the virtual-memory subsystem starts thrashing, because VM swapping activity is a serious source of "invisible" disk I/O.  If you process a record at a time, memory won't be an issue but data movement (a gigabyte is read and written) will be.  But a one-gigabyte file should be processed sequentially in seconds, not 30 minutes.</field>
<field name="root_node">
1003734</field>
<field name="parent_node">
1003734</field>
<field name="reputation">
5</field>
</data>
</node>
