Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Re: Re: Re: Randomize lines with limited memory

by BrowserUk (Pope)
on Nov 03, 2003 at 08:18 UTC ( #304046=note: print w/replies, xml ) Need Help??


in reply to Re: Re: Re: Randomize lines with limited memory
in thread Randomize lines with limited memory

Unfortunately, Tie::File is not a solution to this problem.

For a file of 100,000 lines, it completes, but takes around 800 times longer than List::Util::shuffle on an in memory array. So what, trade time for space. The problem is that the increase in time is exponential, meaning that shuffling the OP's 27,000,000 record file would take close to a week!

However, more fundemental than that is that Tie::File simply isn't able to cope with a file this large. It runs out of memory and segfaults trying this with a 1,000,000 record file on my machine, where as I can comfortably load a 2,000,000 record file completelly into memory and sort even using the copy semantics of List:Util::shuffle. This is true regardles of what settings (default, bigger or smaller) I tried using for the memory parameter.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
Hooray!
Wanted!

  • Comment on Re: Re: Re: Re: Randomize lines with limited memory

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://304046]
help
Chatterbox?
[Discipulus]: if i had to go to psycho i need to ask why i always end with over complex datastructure: now i have an HoHoAoH in my object
[Discipulus]: i will need to handle by reference later on
[Eily]: kinda sounds like father Christmas is late
[hippo]: (asking because I'm sure you said before about learning English) Did you know there are 2 different words for smaller quantities in English? They are "less" and "fewer". Fewer applies to a number of discrete things like days.
Eily notes also
[hippo]: So if you don't drink as much milk and not as often you are said to drink less milk on fewer days. Confusing, eh?

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (7)
As of 2018-01-23 09:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How did you see in the new year?










    Results (242 votes). Check out past polls.

    Notices?