Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
on a qrsh ~18G it takes about 20 minutes. When I can't get a node with that kind of memory its a few hours.

Figures. When you fail to get a node with sufficient memory you are moving into swapping, and that will always kill performance big time. You need to avoid that at all costs.

When I run your code here, the fan out varies widely depending upon the random patterns:

c:\test>912999 I:1 L:1 I:2 L:14 I:3 L:34 I:4 L:182 I:5 L:977 I:6 L:6578 I:7 L:58659 Terminating on signal SIGINT(2) c:\test>912999 I:1 L:1 I:2 L:13 I:3 L:142 I:4 L:784 I:5 L:6166 I:6 L:49483 I:7 L:299369 Terminating on signal SIGINT(2) c:\test>912999 I:1 L:1 I:2 L:1 I:3 L:2 I:4 L:13 I:5 L:95 I:6 L:138 I:7 L:624 I:8 L:2923 Terminating on signal SIGINT(2) c:\test>912999 I:1 L:1 I:2 L:2 I:3 L:5 I:4 L:29 I:5 L:53 I:6 L:294 I:7 L:1935 Terminating on signal SIGINT(2) c:\test>912999 I:1 L:1 I:2 L:2 I:3 L:14 I:4 L:57 I:5 L:157 I:6 L:1467 I:7 L:3200 I:8 L:23871 I:9 L:81714 Terminating on signal SIGINT(2)

I'm going to assume that qrsh and qsub are GRID apis?

Whilst there is much that can be done to improve the performance of your posted code, given these statistics, it seems likely that the main constraint for your program is memory usage. When your program moves into swapping, any titivations done to save a few microseconds here and there will just get drowned in the noise of disk(memory) thrashing.

My suggestion would be to modify your script to monitor the size of the %allgen hash and when it reaches a size that is likely to push the minimum size node on your GRID into swapping, split the generations of that hash into (say) four files and qsub four nodes to read those files and pick up the algorithm from that point.

So, (say) you run 20 iterations and generate 1 million mutations. You split those 1 million into 4 files and start four nodes to pick up from that point with 1/4 million candidates. When each of those nodes approaches 1 million mutations, you repeat the split. And so on.

You'll need to judge the split points in the light of your knowledge of the systems available to you. On my (currently only 2GB) system, I've never managed to run your code past 10 iterations before the process moved into swapping.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re^7: Increasing the efficiency of a viral clonal expansion model by BrowserUk
in thread Increasing the efficiency of a viral clonal expansion model by ZWcarp

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (5)
As of 2024-03-29 10:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found