Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re: sort large file

by Abigail-II (Bishop)
on Apr 01, 2004 at 13:02 UTC ( #341606=note: print w/replies, xml ) Need Help??

in reply to sort large file

Use the sort command, it's optimized to sort large files. It can figure out when to sort in memory, and when to use temporary files. It's a standard Unix tool, and has been ported to different OSses, including Windows.


Replies are listed 'Best First'.
Re: Re: sort large file
by z3d (Scribe) on Apr 01, 2004 at 15:33 UTC
    ...unless the file is truly huge. Sort has its limitations - for instance, on solaris it tends to use /var/tmp to store that temporary file, and if you're dealing with a few gigs worth of data, you can easily run out of even disk space for the tmp files (painful, true story). Not that I have a better solution :)

    "I have never written bad code. There are merely unanticipated features."
      Solaris sort offers -T to specify a directory for its temprary files.
      Sure, there will always be limitations. But it's usually cheaper/easier to add a few gigs of disk space than it is to add a few gigs of RAM. (And if you do it in Perl, multiply that by a factor as well). Typically, 'sort' will need free disk space about the same amount as the file to sort is large.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://341606]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (2)
As of 2018-08-16 04:39 GMT
Find Nodes?
    Voting Booth?
    Asked to put a square peg in a round hole, I would:

    Results (166 votes). Check out past polls.