Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re^4: Processing ~1 Trillion records

by sundialsvc4 (Abbot)
on Oct 25, 2012 at 16:06 UTC ( #1000896=note: print w/ replies, xml ) Need Help??

in reply to Re^3: Processing ~1 Trillion records
in thread Processing ~1 Trillion records

Indeed, if you can extract a list of what the so-called “markers” and “lines” are (which of course you can), then the problem decomposes itself quite-nicely into partitions.   All that you really need are indexes on the columns in question.

It might also be advantageous to define a view that corresponds to this query, so that you can examine and then optimize-the-hades-out-of that view.   In some DBMSes, defining a view serves to introduce the query as one that it now knows to be frequently-used, and it certainly puts that “6 days long” requirement squarely into the DBA’s management field-of-view.

Comment on Re^4: Processing ~1 Trillion records

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1000896]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (11)
As of 2015-10-06 19:45 GMT
Find Nodes?
    Voting Booth?

    Does Humor Belong in Programming?

    Results (160 votes), past polls