Beefy Boxes and Bandwidth Generously Provided by pair Networks Cowboy Neal with Hat
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
I have written something similar to what you are trying to achieve, I rewrote an old script that had a very inefficient algorithm, with a new script with an efficient algorithm. The algorithm I used was similar to Corion's but different in detail. The algorithm is as follows:

1. 1st pass through the file, build an array structure that records the position of the line number in the file, with the line number as the array index.
2. randomize the line numbers, generate a new randomized list with line numbers in the new sequence.
3. for each line number in the new sequence, look up the @linepos->[$line_number] => position in the array structure, seek to the desired position in the source file, read the line, and output the line in the new file.

This algorithm is very fast, I once have reduced the time of processing of a file from 15+ minutes using the old and clumsy algorithm, to just under 3 seconds using the new efficient algorithm.


In reply to Re: Randomize lines with limited memory by Roger
in thread Randomize lines with limited memory by natch

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others lurking in the Monastery: (13)
    As of 2014-04-18 13:12 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      April first is:







      Results (468 votes), past polls