Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re^2: Processing ~1 Trillion records

by lo_tech (Scribe)
on Oct 25, 2012 at 21:49 UTC ( #1000953=note: print w/ replies, xml ) Need Help??


in reply to Re: Processing ~1 Trillion records
in thread Processing ~1 Trillion records

Off the top of my head:

1) You're sql query is doing an dynamic hash inner join. You may get better results making sure the joins (as well as your selection criteria) are indexed fields.

2) You're essentially slurping in db records to markup the fields and dump it to files. If there is any way to can get around reading a trillion records into a hash (i.e., ORDER BY in the database... and the ordered fields are indexed) then you can read/markup/write the records retail without thrashing ram/swap.

Well, that's my $.02 worth. No, for refunds you'll have to check our customer service department.


Comment on Re^2: Processing ~1 Trillion records

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1000953]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2014-08-01 08:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (257 votes), past polls