in reply to Re^3: Processing ~1 Trillion records
in thread Processing ~1 Trillion records
Indeed, if you can extract a list of what the so-called “markers” and “lines” are (which of course you can), then the problem decomposes itself quite-nicely into partitions. All that you really need are indexes on the columns in question.
It might also be advantageous to define a view that corresponds to this query, so that you can examine and then optimize-the-hades-out-of that view. In some DBMSes, defining a view serves to introduce the query as one that it now knows to be frequently-used, and it certainly puts that “6 days long” requirement squarely into the DBA’s management field-of-view.