Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: Design advice: Classic boss/worker program memory consumption

by oiskuu (Pilgrim)
on May 21, 2014 at 16:42 UTC ( #1086999=note: print w/ replies, xml ) Need Help??


in reply to Design advice: Classic boss/worker program memory consumption

Basically, you want a database that's not on the parent heap. How about this:

use GDBM_File; tie(my %dataset, 'GDBM_File', 'tmp.db', &GDBM_NEWDB, 0600) or die; ...

I've assumed your dataset needs no serialization.


Comment on Re: Design advice: Classic boss/worker program memory consumption
Download Code
Re^2: Design advice: Classic boss/worker program memory consumption
by sundialsvc4 (Abbot) on May 21, 2014 at 17:08 UTC

    A potential issue with that design is that now you have the issue of multiple read/write access to a single file ... locking, concurrency and all of that.   Which could become quite messy.

    I know that it is somewhat counter-intuitive to have a parent that does nothing but mind the kids, but I actually find it easier to do it that way because of the issue of separation-of-concerns, or in this case, of one-thing-to-wait-on.   The parent owns the kids, minds the kids, wipes their little noses, and that’s it.   The kids connect to their service-provider older brother, and send results to their service-consumer older sister.   Exactly how you IPC that stuff together is really up to you ... “tim toady.”   But, now, each process (including the parent) basically has only one concern, one thing to worry about at any particular time, and a very uncomplicated internal structure.   The processing will scale-up or scale-down easily, and be quite reliable.

Re^2: Design advice: Classic boss/worker program memory consumption
by shadrack (Acolyte) on May 22, 2014 at 02:43 UTC
    I've assumed your dataset needs no serialization.

    Unfortunately, it would. The work units are moderately complex data structures. Assembling them directly to a tied hash isn't really practical

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1086999]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (5)
As of 2014-09-23 02:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (210 votes), past polls