comment on

I have some proof-of-concept code in Matching in Huge files that you might be able to adapt if your substitutions span lines and you want to do them large-chunk-at-a-time.

Reading an 81Gb file in 8K chunks requires about 10.7 million reads. You can reduce that number by reading the file in larger chunks, via sysread().

Another thing you might look at is whether part of the performance hit you're seeing has to do with disk. If you're writing to the same physical drive that you're reading from, the OS has to move the disk head a lot. This takes time that can add up. Doing writes in larger chunks (via syswrite()) should help, though writing to a separate disk is preferrable. Writing to a heavily fragmented drive will also add time.

In reply to Re: Performance Question by dws
in thread Performance Question by Jarinn

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


P is for Practical
	PerlMonks