Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

With 40ms ping common sense is that it will take a bit more to fetch data. So let's say 50ms to fetch one record. With 45.000 records multiplied with 50ms per record it comes to total of 2.250.000 ms = 2.250 sec = 37.5 minutes. Which matches your 'half an hour' description. Same calculation for 5000 records comes to 4.1 minute.

First thing that I would do is to place print "Point X at: " . time(); on a couple of places (change X with 1, 2, 3 ... so you know what point in code it is) and try to figure out what exactly is going on - kind of simple code profiling. Just to make sure something isn't wrong in the prepare() stage. Though it looks like you already checked that.

Second thing would be to actually not fetch data that you don't need (as someone wrote, filter those that you don't need with something like WHERE whatever IS NOT NULL - or whatever you are doing in get_data and Oracle/PlSQL uses) and not bind $valreal and $colorder that I don't see you are using.

Actually I wouldn't bind the things at all - I would just do my $raResults = $sth2->fetchall_arrayref(); which will give you back exactly the same data structure you are building in that while(fetch){} loop (though depends on what get_data is returning) and it should fetch data much faster then one by one. If you need to do some transformation with each row (which I don't see you are doing) - just do it in the get_data itself.


Have you tried freelancing/outsourcing? Check out Scriptlance - I work there since 2003. For more info about Scriptlance and freelancing in general check out my home node.

In reply to Re^3: latency and round-trips by techcode
in thread DBI Queries take half an hour by joec_

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chanting in the Monastery: (3)
As of 2024-04-24 18:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found