Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Randomize lines with limited memory

by scmason (Monk)
on Nov 02, 2003 at 10:44 UTC ( #303902=note: print w/ replies, xml ) Need Help??


in reply to Randomize lines with limited memory

This is simple. Two methods:

1)Create an array with values 1 to 3.5 million. Randomize th.is array. Then start at the first element of the array, if it is (say) 1376, read the 1367th line from the file and write it to the new file. Then read the next number from the array, and it's corresponding line from the old file and so on.

2)Read in the first 100,000 or so lines, randomize them and write them to a temp file. Read the second 100,000, randomize, write and so on. When all 3.5 million are in 35 new files, select a random number between 1 and 35 and then read the next line of that file and put it on the end of your final file. This way they are not in random order in 100,000 line chunks, but are more evenly distributed.

The first method is just as 'random' as your normal method, because it is the same, but much slower. There are no memory problems though.

It think could be shown that the second method is also rather 'random' but it is different and perhaps not as 'acceptable' until you did a formal proof.


Comment on Re: Randomize lines with limited memory

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://303902]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (7)
As of 2014-12-19 07:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (73 votes), past polls