Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

... thus confirming that, indeed, this is an I/O-bound process that benefits quite substantially from parallelism at (at least...) a factor of x4.   As would be expected.

And, meaning to take absolutely no thunder away from this most-excellent example, I would like to mention here that I’ve seen quite a few shop-written scripts written (at various past-life engagements) which routinely used two command-line parameters:   -s start_percent -e end_percent.   For exactly this purpose.

Each of the scripts which supported these two parameters started by determining the size of the file, and from this, the byte-position represented by each of these two percentage-numbers.   They then did a random-access seek to the starting position, then (if not zero%) read one line of text (presumed to be a fragment) and threw it away.   Then, they processed lines until, after processing a particular line, they noticed that the file-position was now beyond the specified ending position.   (Or end-of-file, whichever came first.)   Then, they ended normally.   In this way, each utility could be told to process a segment of the file.

The bash-scripts (or what have you) that executed these programs launched a number of parallel copies (with non-overlapping percentages as appropriate), as shell jobs, then did a wait for all of them to finish.   It’s exactly the concept that BrowserUK demonstrates here, but implemented (for better or for worse) in the design of the programs and of the shell-scripts that invoked them.

Some of these programs weren’t Perl, and they always used operating system calls to advise the OS that they were going to do “sequential-only” reads of those files, thereby requesting big read-ahead buffers.   They also explicitly said that the files were read-only and that they were shared, thereby encouraging shared use of common buffers.   None of the programs were threads or processes, yet all of them were thus prepared to be child processes ... of the shell.

Great post.   Thanks.

In reply to Re^2: Optimise file line by line parsing, substitute SPLIT by sundialsvc4
in thread Optimise file line by line parsing, substitute SPLIT by go9kata

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    [Discipulus]: good morning community!

    How do I use this? | Other CB clients
    Other Users?
    Others musing on the Monastery: (7)
    As of 2018-06-19 07:06 GMT
    Find Nodes?
      Voting Booth?
      Should cpanminus be part of the standard Perl release?

      Results (111 votes). Check out past polls.