Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Processing ~1 Trillion records

by dHarry (Abbot)
on Oct 25, 2012 at 10:06 UTC ( #1000798=note: print w/ replies, xml ) Need Help??


in reply to Processing ~1 Trillion records

Can you give some background information of the problem at hand?

What database do you use? What does the query look like?

Restructuring a massive array and writing CSV files sounds costly. What pattern? How many CSV files?

Profiling a (small) subset of the data might give some indication. Did you time how long the various steps take?

In short: could you show some code please?


Comment on Re: Processing ~1 Trillion records

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1000798]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (15)
As of 2015-07-28 13:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (255 votes), past polls