Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^2: Optimise file line by line parsing, substitute SPLIT

by sundialsvc4 (Monsignor)
on Jun 04, 2013 at 23:53 UTC ( #1037073=note: print w/ replies, xml ) Need Help??


in reply to Re: Optimise file line by line parsing, substitute SPLIT
in thread Optimise file line by line parsing, substitute SPLIT

... thus confirming that, indeed, this is an I/O-bound process that benefits quite substantially from parallelism at (at least...) a factor of x4.   As would be expected.

And, meaning to take absolutely no thunder away from this most-excellent example, I would like to mention here that I’ve seen quite a few shop-written scripts written (at various past-life engagements) which routinely used two command-line parameters:   -s start_percent -e end_percent.   For exactly this purpose.

Each of the scripts which supported these two parameters started by determining the size of the file, and from this, the byte-position represented by each of these two percentage-numbers.   They then did a random-access seek to the starting position, then (if not zero%) read one line of text (presumed to be a fragment) and threw it away.   Then, they processed lines until, after processing a particular line, they noticed that the file-position was now beyond the specified ending position.   (Or end-of-file, whichever came first.)   Then, they ended normally.   In this way, each utility could be told to process a segment of the file.

The bash-scripts (or what have you) that executed these programs launched a number of parallel copies (with non-overlapping percentages as appropriate), as shell jobs, then did a wait for all of them to finish.   It’s exactly the concept that BrowserUK demonstrates here, but implemented (for better or for worse) in the design of the programs and of the shell-scripts that invoked them.

Some of these programs weren’t Perl, and they always used operating system calls to advise the OS that they were going to do “sequential-only” reads of those files, thereby requesting big read-ahead buffers.   They also explicitly said that the files were read-only and that they were shared, thereby encouraging shared use of common buffers.   None of the programs were threads or processes, yet all of them were thus prepared to be child processes ... of the shell.

Great post.   Thanks.


Comment on Re^2: Optimise file line by line parsing, substitute SPLIT
Re^3: Optimise file line by line parsing, substitute SPLIT
by Anonymous Monk on Jun 05, 2013 at 00:01 UTC

    ... thus confirming that, indeed, this is an I/O-bound process that benefits quite substantially from parallelism at (at least...) a factor of x4. As would be expected.

    No. Throwing memory at a IO speeds up processing. Throwing threads at memory, speeds splitting.

Re^3: Optimise file line by line parsing, substitute SPLIT
by BrowserUk (Pope) on Jun 05, 2013 at 00:14 UTC
    ... thus confirming that, indeed, this is an I/O-bound process that benefits quite substantially from parallelism at (at least...) a factor of x4. As would be expected.

    Even with the code and evidence right there for your inspection, you draw completely the wrong conclusion.

    {blah} In this way, each utility could be told to process a segment of the file.

    Prove it by posting the code. You won't (because you can't). Talk is cheap (and in your case invariably wrong!).

    What makes my code work is that all the IO is sequential. And that requires shared state.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1037073]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (9)
As of 2014-08-22 20:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (164 votes), past polls