Re^2: find difference in dates/items in same column

[zerocred]

oopps error in the earlier code I accidentally poseted while not logged in.

I tried the techniques here:
http://www.onlamp.com/pub/a/onlamp/excerpt/mysqlckbk/index3.html?page=2
it looks like this:

SELECT a.brand,
a.date AS newdate,
b.date AS olddate,  
DATEDIFF(a.date,b.date) AS diff
FROM(
    SELECT brand, date,
        (
              SELECT count(*)
            FROM test
            WHERE brand=T.brand
            AND date<T.date
        )+1
    AS rank1
    FROM test
    AS T
    )
AS a INNER JOIN
    (
        SELECT brand, date,
        (
            SELECT count(*)
            FROM test
            WHERE brand=T1.brand
            AND date<T1.date
        )
        AS rank2
        FROM test2
        AS T1
    )
AS b ON (a.rank1=b.rank2)
AND (a.brand = b.brand)
ORDER BY a.brand,
a.date DESC
;

This does work like you say by creating the T and T1 alias/copy tables then you line up the sequence numbers (rank1 and rank2) and process the columns of interest.

The problem is it takes about 1 hour to return the query on about 120,000 records... Even merely sequencing the brands alone (the inner Select count(*) part) takes about 7 minutes). I think teh SQL is slow because the whole table is searched each time to find the next in sequence. By using my perl program and executing the Select to return the table in the sorted order you know the next record is the one you are looking for. It runs in about 30secs -1min onthe same table.

When I started out looking to solve this problem I thought it would be a relatively easy piece of SQL to solve a very common problem, e.g. calculate the time between time stamps in a log file for example. But in SQL it seems not;)

[dragonchild]

Add indices to your columns. Also, why do you need the COUNT(*)? Removing that if you don't need it will speed you significantly.

[zerocred]

Indexes is a good idea but I already have the relevant columns already indexed. The count is there to sequence the same brands like:

id brand rank=(count*)
1  A     1
2  B     1
3  A     2
4  A     3
5  B     2
...

etc. That way the dates from consecutive brand appearances can be compared by the rank=rank+1 for the datediff() comparison. There already is an autoincrement id column but it can't be used for comparing consecutive dates of the same brand.

[dragonchild]

Ok - I'm interested now. I'm going to assume that the following is the table definition:

CREATE TABLE `test` (
    `id`    INT NOT NULL AUTO_INCREMENT PRIMARY KEY
   ,`brand` VARCHAR(255) NOT NULL
   ,`date`  DATE NOT NULL
   ,INDEX (`brand`,`date`)
   ,INDEX (`date`,`brand`)
);

Given that, you can do something assuming that the number of reads is at least 20x the number of writes (otherwise it's really not that performant). Basically, the idea is to maintain the `rank` column on each INSERT, UPDATE, or DELETE. Your writes get slow, but your reads get fast. This, essentially, is a cache. You can do this maintenance either in code or using a trigger. MySQL has triggers, so that works out nicely.

[ccn]

How fast would be the following query?

select brand,
       id,
       date,
       datediff(day, (select max(T2.date)
                        from test as T2
                       where T1.brand = T2.brand
                             and T2.date < T1.date) ,date)
  from test as T1
 order by brand, id

I am not sure if that syntax works for MySQL. May be it must be translated to INNER JOIN