Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re^4: Processing ~1 Trillion records

by sundialsvc4 (Abbot)
on Oct 25, 2012 at 16:06 UTC ( #1000896=note: print w/replies, xml ) Need Help??

in reply to Re^3: Processing ~1 Trillion records
in thread Processing ~1 Trillion records

Indeed, if you can extract a list of what the so-called “markers” and “lines” are (which of course you can), then the problem decomposes itself quite-nicely into partitions.   All that you really need are indexes on the columns in question.

It might also be advantageous to define a view that corresponds to this query, so that you can examine and then optimize-the-hades-out-of that view.   In some DBMSes, defining a view serves to introduce the query as one that it now knows to be frequently-used, and it certainly puts that “6 days long” requirement squarely into the DBA’s management field-of-view.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1000896]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (16)
As of 2016-10-21 15:56 GMT
Find Nodes?
    Voting Booth?
    How many different varieties (color, size, etc) of socks do you have in your sock drawer?

    Results (289 votes). Check out past polls.