Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re: memory:: grep vs looping through hash

by sundialsvc4 (Abbot)
on Aug 18, 2014 at 12:54 UTC ( #1097848=note: print w/replies, xml ) Need Help??

in reply to memory:: grep vs looping through hash

The admonition to “try it” really is the best advice that can be given.   If your system has its memory-back up against a memory-wall, and if in your case grep is going to return long lists such that your system simply has nowhere to put them ... then you probably can’t use it.   You have to use a loop.   However, in that case, “you have bigger fish to fry,” because you are seriously over-loading your system.

There’s a reason why early systems used the term “SOS (as in ‘... --- ...’ ) = Short On Storage.”   If you seriously over-commit the RAM resource, then the system is going to start thrashing, and it will be the futile disk-I/O that sinks the boat, not any clever algorithm or the absence thereof.   If you literally have stuffed your machine full of all the RAM that a 64-bit motherboard can possibly take, then you simply have to expand the job to multiple machines, each similarly equipped.

And you are probably equally screwed whether you use a loop or not, because it is memory references that cause page-faults.   When memory-references that ought to take nanoseconds consistently start taking milliseconds ... “ding dong, you’re dead.”

  • Comment on Re: memory:: grep vs looping through hash

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1097848]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (4)
As of 2018-05-20 12:27 GMT
Find Nodes?
    Voting Booth?