Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re^2: Perl always reads in 4K chunks and writes in 1K chunks... Loads of IO!

by NeilF (Sexton)
on Jan 01, 2006 at 11:01 UTC ( #520225=note: print w/replies, xml ) Need Help??


in reply to Re: Perl always reads in 4K chunks and writes in 1K chunks... Loads of IO!
in thread Perl always reads in 4K chunks and writes in 1K chunks... Loads of IO!

Definate progress! With my example I'd get around 1611 io processes reported by File Mon. With yours I only get 600!

Now, keep in mind I'm trying to read in records into an array which I would process and then write back. Your example does not do this so I changed it back to more like mine:-
open DF,"test.txt"; binmode DF; my @test = <DF>; close DF; my $rec; foreach(@test){$rec.=$_;} open (DF,'>test1.txt') || die ("Failed to open test1.txt: $!\n"); binmode DF; print DF $rec; close DF;
Now that worked fine, and I still only get 600 processes reported!

Note: I had to remove the "local $/;". I've never seen that before and it seemed to mean the lines were not read in as an array?

With further testing ALL the gains your version shows are in the write. If I replace your binary read with my original read I still get 600 io processes. Here's the current example code:-
open DF,"test.txt"; my @test=<DF>; close DF; # Print first 3 elements to prove read as an array print "Line 1=$test[0]<BR>Line 2=$test[1]<BR>Line 3=$test[2]<BR>"; # Can we get around this? Waste of processing & memory! my $rec; foreach(@test){$rec.=$_;} open (DF,'>test.txt') || die ("Failed to open test2.txt: $!\n"); binmode DF; print DF $rec; close DF;
So this is a major improvement in the write area, but the read is still (at least on my XP system) a big cause of most of the io processes as it's reading in 4K chunks!

Also, having to build the array into another variable ($rec) to write eats up memory and processing. Is there a way to better print out that array as a single combined variable? (If I "PRINT DF @test" instead of $rec this increases the io figure from 600 to 841!)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://520225]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (5)
As of 2022-05-19 14:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (71 votes). Check out past polls.

    Notices?