Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re: Randomize lines with limited memory

by scmason (Monk)
on Nov 02, 2003 at 10:44 UTC ( #303902=note: print w/ replies, xml ) Need Help??

in reply to Randomize lines with limited memory

This is simple. Two methods:

1)Create an array with values 1 to 3.5 million. Randomize array. Then start at the first element of the array, if it is (say) 1376, read the 1367th line from the file and write it to the new file. Then read the next number from the array, and it's corresponding line from the old file and so on.

2)Read in the first 100,000 or so lines, randomize them and write them to a temp file. Read the second 100,000, randomize, write and so on. When all 3.5 million are in 35 new files, select a random number between 1 and 35 and then read the next line of that file and put it on the end of your final file. This way they are not in random order in 100,000 line chunks, but are more evenly distributed.

The first method is just as 'random' as your normal method, because it is the same, but much slower. There are no memory problems though.

It think could be shown that the second method is also rather 'random' but it is different and perhaps not as 'acceptable' until you did a formal proof.

  • Comment on Re: Randomize lines with limited memory

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://303902]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2016-08-24 06:27 GMT
Find Nodes?
    Voting Booth?
    The best thing I ever won in a lottery was:

    Results (338 votes). Check out past polls.