Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Optimize my code with Hashes

by sukhicool (Initiate)
on Aug 27, 2008 at 09:51 UTC ( #707115=perlquestion: print w/replies, xml ) Need Help??
sukhicool has asked for the wisdom of the Perl Monks concerning the following question:

I am assigned the task of optimizing one PERL code which takes approx. 18 hours to execute. I have read the entire code and found that the code reads around 55000 to 60000 entries from LDAP Server and then fetch them into a HASH ed_en and then updates/creates entries in LDAP Server.

Hash structure is like this:
%ed_en = { '069836' => 'p1', '025939' => 'p2', ... }
#with p1 a pointer (ie address like 0x123454) towards the corresponding entry received from ldap_search #with p2 a pointer (ie address like 0x123456) towards the corresponding entry received from ldap_search

%p1 = { 'dn' => 'st-eduid=ed988766,ou=People,dc=xyz,dc=com } %p2 = { 'dn' => 'st-eduid=ed123456,ou=People,dc=xyz,dc=com }
Further, few hashes are created from this HASH for easier processing of the subroutines.

Systems uses version PERL 5.005_03 built for sun4-solaris.

Any help would be appreciated.

Thanks & Regards, Sukhi

Replies are listed 'Best First'.
Re: Optimize my code with Hashes
by jbert (Priest) on Aug 27, 2008 at 10:46 UTC

    Your process takes 18 hours to run. You want it to be less.

    You need to work out what the computer spends most of it's time doing in those 18 hours, so that you can try and reduce the most costly elements first (and get the biggest improvements).

    Prime possibilities are:

    • Waiting for a network response (latency). A lot of network protocols are request/response. If you need to do 50k requests and a single request takes 1s to process remotely, you'll end up waiting 50k secs == 14hours.
    • Waiting to swap stuff in+out. If your app working set is bigger than the RAM available, performance will plummet as it waits for the disk to access virtual memory.
    • CPU. If your app isn't waiting for anything else, it's trying to crunch through code. This is the case where profiling etc can help the most.
    • Other disk I/O. If your process is reading and writing to storage, that can be slow. Particularly if you are on a slow device and/or are doing a lot of flushing after small writes (but sometimes that is that you need to do).

    So you need to run some monitoring tools ('top' is a good first start, but solaris has many others - check out vmstat and iostat too). These will tell you which of the above issues is the problem. (Well, if it's not CPU, swapping or other disk I/O, then it's probably network latency.

    And after all that about measuring first?

    My best guess is that you're doing 50K LDAP operations and network latency is killing you (i.e. your box isn't busy when you're doing this, but perhaps your LDAP server (or network) is).

    Look for a bulk import/export tool for your LDAP server and use that instead.

      Total code took : 68258 secs

      While LDAP Update/Add entry took:

      the code took:67358 wallclock secs (29935.47 usr 60.38 sys + 0.00 cusr 0.00 csys = 29995.85 CPU) in considering each entry from PeopleFirst extract ...

      1. Do older versions of PERL has bad algorithms for handling of HASHES ?
      2. Will it help if we use Arrays instead of Hashes?

      If upgrading the PERL version will help, I will try to convince the management if you can help me in locating some url which quotes this.
        the code took:67358 wallclock secs (29935.47 usr 60.38 sys + 0.00 cusr 0.00 csys = 29995.85 CPU) in considering each entry from PeopleFirst extract ..

        That's interesing. So 29935/67358=44% of your time was spent on user CPU. That is significant, and you might want to look into profiling the app's CPU usage (using Devel::Profile and Devel::DProf).

        Of course, it also means that 56% of your time is spent doing other things. If that is net latency then you'd do well to look at bulk import/export instead.

        Your timestamp logging appeared to show ~6secs for one request, is that right? That can't be representative, since as noted elsewhere in this thread, you'd never manage to do 50k updates in 18 hours if each takes 6s.

        Lastly, if you do profile the app, then it will probably benefit you to produce a cut-down version which runs more quickly. This is useful because the profilers slow things down and generate large amounts of data - they'll probably break on such a big run.

        Also, having a more quickly repeatable test case (e.g. ~10mins) will greatly accelerate your ability to test ideas on code and algorithm changes.

        However, the hard part is knowing if your cut-down test case has the same performance profile as your main job run.

        Another thought: if the 'missing' 56% of your time is overnight you might be sharing a network with a backup job, or something else which saturates the net and makes your network response times go very slowly.

        You are asking the wrong questions. You already blame the hashes without really knowing what wastes all this time. Everyone above told you to do some profiling first and that is a really good idea. Often it is the algorithm used that kills the time.

        For example: If your program often makes a copy of your hash then that will really thrash your memory and cost time. But you won't get it faster by changing to an array then.

        Also you didn't show us what the code does with the hash you blame. Then how should we be able to tell you if an array ist better

        So either post some code here or do some profiling or both.

Re: Optimize my code with Hashes
by moritz (Cardinal) on Aug 27, 2008 at 12:27 UTC
    %p1 = { 'dn' => 'st-eduid=ed988766,ou=People,dc=xyz,dc=com }

    That's already wrong, use parenthesis instead:

    %p1 = ( 'dn' => 'st-eduid=ed988766,ou=People,dc=xyz,dc=com )
Re: Optimize my code with Hashes
by GrandFather (Sage) on Aug 27, 2008 at 10:02 UTC

    I presume you have profiled the code so that you know where the major portion of the time is being spent. How about you show us a small working sample that demonstrates the actual issue? What you have shown so far doesn't really illustrate any problematic Perl code.

    Perl reduces RSI - it saves typing
Re: Optimize my code with Hashes
by Anonymous Monk on Aug 27, 2008 at 09:59 UTC
    Without seeing code or something more specific than this, we won't be able to help you. 55k entries aren't much at all, so I think that the problem might be with LDAP? Also, upgrade your perl version, it's ancient.

      I'd have to agree with AM here ... sounds like you're blaming your hash structure when the LDAP server and/or network latency is your real issue. After upgrading your perl, take a look at Devel::Profile or Devel::NYTProf to try and pinpoint exactly where the code is spending all its time.

Re: Optimize my code with Hashes
by Bloodnok (Vicar) on Aug 27, 2008 at 10:03 UTC
    I wish to be associated with the remarks of the last speaker :-))
    LDAP is not always the speediest of protocols - esp. when there's firewall(s) in the way.

    You could try printing timestamps immediately before and after the LDAP access - this would help to confirm the location of the (suspected) bottleneck...

    A user level that continues to overstate my experience :-))
      I have already printed timestamps in the log files. We may see from here:
      [09:20:29.260258 ]: 144248 (BATALLONES,Jayson) looks like a creation [09:20:29.260564 ]: Creating entry 144248 (BATALLONES,Jayson) in ED... [09:20:35.192125 ]: createEDentry: Successfully created ED entry 'st-e +duid=ed953127,ou=people,dc=xyz,dc=com' 144248 (BATALLON ES,Jayson) for location 000194
      The above log shows that LDAP Server is not taking too much time in creating a entry. I have also used Benchmark for it and the result is as below:
      the code took: 1 wallclock secs ( 0.01 usr 0.00 sys + 0.00 cusr 0.0 +0 csys = 0.01 CPU) in Creating entry 144248(BATALLONES,Jayson) in ED +...
        Mmmm, if I read correctly, a single lookup takes, on average, 6 secs - if it takes 6 secs/entry, it should take about 330,000 secs i.e. a lot longer than 18 hrs, for 55,000 entries, so I think you're getting off lightly!

        It strikes me that the best answer to your problem is, as jbert suggests, to find a bulk LDAP download tool and use that - which will inevitably require knowledge of LDIF file processing.

        A user level that continues to overstate my experience :-))
        6s is quite slow. One question: Do you setup a new LDAP connection for each transaction? If yes, try to reuse an already opened LDAP connection (better: a pool of connections) since it is not necessary to bind and authenticate 50k times.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://707115]
Approved by GrandFather
and the leaves swirl about...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (8)
As of 2017-12-18 07:48 GMT
Find Nodes?
    Voting Booth?
    What programming language do you hate the most?

    Results (473 votes). Check out past polls.