Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Parallel processing with ForkManager

by sundialsvc4 (Abbot)
on Sep 20, 2012 at 13:34 UTC ( #994664=note: print w/ replies, xml ) Need Help??


in reply to Parallel processing with ForkManager

I tend to agree that it probably would be stoppered-up by the capacity of the database server.   And let’s face it ... neither a 100MB text-file nor a 5GB database is, by today’s standards, that large.   Maybe you could make some read-only copies of the database at various places.   Maybe you could optimize the search process in the database in some useful way.   In general, I just think that trying to cluster this thing is going to be a lot of trouble, for doubtful benefit.

Clustering works really well when the workload is primarily CPU-bound and when there are no resource-contentions.   Here, both of these are not-the-case.

Edit:   BrowserUK’s subsequent recommendation to use temporary tables and a join-query, below, is in my view unquestionably the best approach to take in this case.   Now, nothing but the bulk move-in and the bulk move-out is “happening over the wire.”   The computer gets the essential job done in one step, and strictly within its own optimized world.


Comment on Re: Parallel processing with ForkManager

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://994664]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (11)
As of 2014-09-22 20:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (201 votes), past polls