Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: "Out of memory" problem

by mbethke (Hermit)
on Nov 30, 2012 at 21:49 UTC ( [id://1006523]=note: print w/replies, xml ) Need Help??


in reply to "Out of memory" problem

I would try and use the system's sort(1) first as it's already optimized for this kind of stuff. 500M 32-bit integers would barely fit your memory if you put them in a straight C-style array with none of Perl's overhead, so it's probably a good idea to do a disk-based mergesort. Split the file into half a dozen or so parts, run sort on them individually and then use sort once more with -m to merge the results.

If you really need to do it in Perl, you could have a look at File::Sort but I have no idea whether it works well.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1006523]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (3)
As of 2025-06-25 02:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuliAnonymous Monks are no longer allowed to use Super Search, due to an excessive use of this resource by robots.