Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Re^2: To Fork or Not to Fork (bottle necks)

by oiskuu (Hermit)
on Jul 01, 2014 at 17:08 UTC ( #1091876=note: print w/replies, xml ) Need Help??

in reply to Re: To Fork or Not to Fork (bottle necks)
in thread To Fork or Not to Fork. Tis a question for PerlMonks

And a related question: Design advice: Classic boss/worker program memory consumption.

Did you also consider/bench a threaded version?

Might there be a practical way to manage multiple malloc areas in one perl process? A separate arena (mmap) per package? This could improve locality, and also segment the data so that CoW benefits are subdivided.

  • Comment on Re^2: To Fork or Not to Fork (bottle necks)

Replies are listed 'Best First'.
Re^3: To Fork or Not to Fork (!threads)
by tye (Sage) on Jul 01, 2014 at 19:44 UTC

    No, we didn't consider using threads (nor what Perl 5 calls "threads"). If using threads offered any performance benefits for our use case, the benefits would be truly trivial.

    The cost of using threads in terms of complexity would not be trivial. In some cases, you can use Perl threads with the Perl code mostly untouched and the added complexity is mostly just hiding inside of the perl executable. Though it will also add operational complexity when it comes to trouble shooting and similar tasks. But that complexity inside the perl executable very often doesn't stay completely hidden (go read the p5p thread on why Perl threads have been "depreciated", if you aren't familiar with the typical experience of trying to use Perl threads).

    But I don't think this is even one of those cases. Since the workers spawn other executables to do parts of the transcoding and spawning jobs involves waiting for SIGHCLD and mixing signals and threads is often pointed out as a bad idea, I suspect that the Perl code would actually have to get significantly more complex.

    So I didn't even consider adding significant complexity in at least 2 or 3 layers for a possible but at-most-trivial performance gain and maybe a (likely trivial) performance decrease.

    - tye        

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1091876]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (2)
As of 2018-05-25 23:04 GMT
Find Nodes?
    Voting Booth?