Re: Specialized data compression

[salva]

The x,y,z acceleration values in your sample file are highly correlated on time, so using delta values for them should also improve their compressibility.

Then, you can divide the x,y,z deltas by the time deltas in order to get the derivate of the acceleration vector in order to get even more normalized values. The problem is that at this point you are using floats instead of integers, but unless you want absolute precision this should be a minor issue. Just select a resolution that fits your problem and discretize them accordingly.

Another possibility is to express the x,y,z deltas (or derivates) in the coordinate system defined by the Frenet-Serret frame. That would probably produce numbers that are nearer to zero and so, more compressible.

[wanna_code_perl]

These suggestions were helpful. Thank you! Using deltas for x,y,z did help quite a bit.

The time delta derivative approach was interesting to think about, but would basically break the lossless compression I have going. At the moment, at least, I need to be able to reconstruct the original input verbatim, and the only way to do that is with fixed point arithmetic.

The one concession I was able to safely (verifiably) make is quantizing the time deltas that are close to the sensor frequency. The clock has a large (+/- 15%!) but predictable jitter pattern I was able to isolate and then basically RLE the result.

With the latest optimizations, I'm sitting at about 9:1 compression, which is acceptable. I was shooting for 10:1, but that last 33 KiB won't come easy.