Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: Trouble with fork()

by phonybone_monk (Initiate)
on May 22, 2017 at 22:03 UTC ( [id://1190898]=note: print w/replies, xml ) Need Help??


in reply to Re: Trouble with fork()
in thread Trouble with fork()

I glossed over the initial processing of the file because I didn't think it was relevant to the problem at hand for two reasons, but I'll expand.

The input file is processed by open-software bio-informatics tools called "bowtie2" and "samtools". They are invoked by use of a system() call, and they write temporary files. All of that seems to work fine.

After those programs are finished, my program (which I inheritied, btw) does its forking and further processing.

So here's the funny thing. This program is running successfully on a MacPro with 48Gb of RAM and 24 CPU cores. On my linux box, with 128Gb of RAM and 40 CPU cores, the program fails as I've described. I'm assuming, because of the way that it's failing (some of the child processes die immediately with an error code of 11 (EAGAIN), that I'm running out of some critial resource and my fork is failing. What I can't figure out, and the answer that I'm seeking, is which resource? Is it memory? Swap space? Open file descriptors? Something else? How do I figure this out, other than trial-and-error raising of each limit in turn, which is what I've been trying to do.

So am I entirely confident my algorithms are correct? Given that they're running just fine on a smaller machine, I'm reasonably confident that they are, and that my problem is somehow related to the number of forks I'm running (one per logical CPU core).

Replies are listed 'Best First'.
Re^3: Trouble with fork()
by dave_the_m (Monsignor) on May 22, 2017 at 22:20 UTC
    (some of the child processes die immediately with an error code of 11 (EAGAIN)
    The bottom 7 bits of $? contain the signal number the process died with. 11 is usually a SEGV.

    Dave.

Re^3: Trouble with fork()
by Anonymous Monk on May 25, 2017 at 18:07 UTC

    You should be roughly aware of the per-process memory requirements (size of work). Do you have enough swap to cover it all?

    Is memory overcommit enabled on your linux box? Check it with sysctl vm.overcommit_memory

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1190898]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having a coffee break in the Monastery: (8)
As of 2024-03-29 15:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found