Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: Optimizing Iterating over a giant hash

by jethro (Monsignor)
on Dec 25, 2009 at 23:47 UTC ( #814387=note: print w/replies, xml ) Need Help??

in reply to Optimizing Iterating over a giant hash

You might use a divide and conquer method.

Store the data sequentially into files, one file for each different $entry value. If that number is too big (bigger than the number of files you can have open at any one time), you might group the entry values with a suitable scheme.

The difference to DMBM:Deep is that you write to these files sequentially, it should be much faster because of buffering.

After that you can reread and work through the files one by one. Hopefully now a single file should fit into memory without having to resort to swapping (which probably makes your application slow at the moment)

  • Comment on Re: Optimizing Iterating over a giant hash

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://814387]
[Discipulus]: with eskimo greeting is a little bit impressing, anyway
[Eily]: I actually used the eskimo greeting :P, I just wrote the "canonical" form of the oneliner here
[Discipulus]: next time you can add `SARGV-1` and if still not impressed pass a 'rm -f..' as last arg
[Eily]: :)
[Eily]: or I might be tempted to send him an Acme::EyeDropsed file next time
[Eily]: Acme::Bleach is tempting, but it will look like I'm just using the one module that does precisely exactly what he wants

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (8)
As of 2017-04-26 15:10 GMT
Find Nodes?
    Voting Booth?
    I'm a fool:

    Results (482 votes). Check out past polls.