Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

Personally, I'd default to the same as mbethke: no threading. And forking might not be any better.

If I were to even consider something like this, the file would have to be split. Across multiple physical disks. Because your CPU is so much faster than the disk access that there's no way for the disk to overwhelm a single CPU, nevermind multiple CPUs. Only with multiple physical disks all spinning and sending back data simultaneously do you have a chance at even keeping a single CPU busy. And even then, I suspect that the bus would still be a limiting factor.

If, however, you were to be doing analysis of each word to group similar-sounding words together (see soundex), then you might have enough to do to make the disks start to wait for the CPU. But I doubt it.

Assuming we completely change your requirements so that there would be a point to this, that your file was actually large files each on separate physical disks, and that you were doing some significant processing, even then, I'd probably avoid threading or forking if at all possible. To this end, I'd consider using IO::AIO (which uses threads under the covers, but not perl threads) to read the files.

If that ends up not being fast enough where you actually need to process on more than one CPU at a time, I would suggest as little sharing as possible. Having different threads be responsible for words starting with a different letter just doesn't scale easily (26 threads may not be optimal for your CPU configuration, and what if you start dealing with non-English languages, especially Russian or Greek or Chinese, among others?), nor uniformly (more words start with "s" in English than any other letter, for example, while the thread dealing with words starting with "x" will be awfully idle). Instead, one thread/process per file, and each thread/process doing exactly the same thing: populating a hash. Then the challenge is that in the parent process, you need to pull all this together and add them together. That's not really too hard:

for my $word (keys %child_results) { $total{$word} += $child_results{$ +word} }
The challenge is getting that data back to the parent atomically. That is, the entire hash, not a partial result. And doing so easily. One way is to share the %total hash in each thread, and then use a mutal exclusion semaphore on it that blocks concurrent access. Another way is to use AnyEvent::Util::fork_call to transfer the data back from the subprocess (not thread) to the parent process. In this model, the parent is not using threads at all, and so will not be able to get partial results.

But, for your original requirements, there's just no way that threading, forking, or any of that will add anything other than complexity. :-)

In reply to Re: to thread or fork or ? by Tanktalus
in thread to thread or fork or ? by Anonymous Monk

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others chanting in the Monastery: (4)
    As of 2014-08-02 03:26 GMT
    Find Nodes?
      Voting Booth?

      Who would be the most fun to work for?

      Results (54 votes), past polls