Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: Randomize lines with limited memory

by Roger (Parson)
on Nov 01, 2003 at 22:57 UTC ( #303850=note: print w/replies, xml ) Need Help??

in reply to Randomize lines with limited memory

I have written something similar to what you are trying to achieve, I rewrote an old script that had a very inefficient algorithm, with a new script with an efficient algorithm. The algorithm I used was similar to Corion's but different in detail. The algorithm is as follows:

1. 1st pass through the file, build an array structure that records the position of the line number in the file, with the line number as the array index.
2. randomize the line numbers, generate a new randomized list with line numbers in the new sequence.
3. for each line number in the new sequence, look up the @linepos->[$line_number] => position in the array structure, seek to the desired position in the source file, read the line, and output the line in the new file.

This algorithm is very fast, I once have reduced the time of processing of a file from 15+ minutes using the old and clumsy algorithm, to just under 3 seconds using the new efficient algorithm.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://303850]
[shmem]: hmm module takeover... any hints? I'm being lazy wrt searches
[choroba]: Contact the author, ask other to help contacting the author, write to pause admins
[shmem]: ah perlfaq7
[shmem]: thanks choroba, found it. 'Tis about MJD Devel::Trace
[shmem]: I'd like to upload a v0.13
[choroba]: Very nice!

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (9)
As of 2018-02-23 14:52 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (302 votes). Check out past polls.