Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re^4: Processing ~1 Trillion records

by sundialsvc4 (Abbot)
on Oct 25, 2012 at 16:06 UTC ( #1000896=note: print w/ replies, xml ) Need Help??

in reply to Re^3: Processing ~1 Trillion records
in thread Processing ~1 Trillion records

Indeed, if you can extract a list of what the so-called “markers” and “lines” are (which of course you can), then the problem decomposes itself quite-nicely into partitions.   All that you really need are indexes on the columns in question.

It might also be advantageous to define a view that corresponds to this query, so that you can examine and then optimize-the-hades-out-of that view.   In some DBMSes, defining a view serves to introduce the query as one that it now knows to be frequently-used, and it certainly puts that “6 days long” requirement squarely into the DBA’s management field-of-view.

Comment on Re^4: Processing ~1 Trillion records

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1000896]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (5)
As of 2015-11-25 23:50 GMT
Find Nodes?
    Voting Booth?

    What would be the most significant thing to happen if a rope (or wire) tied the Earth and the Moon together?

    Results (693 votes), past polls