Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
the code took:67358 wallclock secs (29935.47 usr 60.38 sys + 0.00 cusr 0.00 csys = 29995.85 CPU) in considering each entry from PeopleFirst extract ..

That's interesing. So 29935/67358=44% of your time was spent on user CPU. That is significant, and you might want to look into profiling the app's CPU usage (using Devel::Profile and Devel::DProf).

Of course, it also means that 56% of your time is spent doing other things. If that is net latency then you'd do well to look at bulk import/export instead.

Your timestamp logging appeared to show ~6secs for one request, is that right? That can't be representative, since as noted elsewhere in this thread, you'd never manage to do 50k updates in 18 hours if each takes 6s.

Lastly, if you do profile the app, then it will probably benefit you to produce a cut-down version which runs more quickly. This is useful because the profilers slow things down and generate large amounts of data - they'll probably break on such a big run.

Also, having a more quickly repeatable test case (e.g. ~10mins) will greatly accelerate your ability to test ideas on code and algorithm changes.

However, the hard part is knowing if your cut-down test case has the same performance profile as your main job run.

Another thought: if the 'missing' 56% of your time is overnight you might be sharing a network with a backup job, or something else which saturates the net and makes your network response times go very slowly.


In reply to Re^3: Optimize my code with Hashes by jbert
in thread Optimize my code with Hashes by sukhicool

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others romping around the Monastery: (4)
    As of 2014-09-21 14:55 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      How do you remember the number of days in each month?











      Results (172 votes), past polls