Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re: Performance Question

by Elian (Parson)
on May 08, 2002 at 13:54 UTC ( #165033=note: print w/replies, xml ) Need Help??

in reply to Performance Question

What's going on may depend on the OS and filesystem, but creating an 81G file by printing out line by line is going to be a killer. You're going to be constantly extending it piece by piece, probably ending up with a badly fragmented file. And even with the disk likely being a RAID volume of some sort (As it's got to be at least a 170G volume there) you're going to find the reads and writes competing for the disk heads a lot.

This is as much, or more, a system administration issue as a perl one. Find your sysadmin, get him/her to fill you in on the characteristics of the system, and work with that. You may find that pre-extending the file is a good thing. There may be system parameters that can be set that govern the size of the new chunk your OS allocates when it extends a file. There may be cache settings that can be tweaked on the disk controller or individual files. This is obviously an adequately beefy machine, so you just need to do some performance tuning, and probably at the OS level.

Hopefully you're not accessing the file via NFS or some other networked file system. Those will tend to kill your performance dead.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://165033]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (5)
As of 2020-03-29 21:15 GMT
Find Nodes?
    Voting Booth?
    To "Disagree to disagree" means to:

    Results (171 votes). Check out past polls.