Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Re: Specialized data compression

by salva (Abbot)
on Sep 11, 2012 at 09:28 UTC ( #992967=note: print w/replies, xml ) Need Help??

in reply to Specialized data compression

The x,y,z acceleration values in your sample file are highly correlated on time, so using delta values for them should also improve their compressibility.

Then, you can divide the x,y,z deltas by the time deltas in order to get the derivate of the acceleration vector in order to get even more normalized values. The problem is that at this point you are using floats instead of integers, but unless you want absolute precision this should be a minor issue. Just select a resolution that fits your problem and discretize them accordingly.

Another possibility is to express the x,y,z deltas (or derivates) in the coordinate system defined by the Frenet-Serret frame. That would probably produce numbers that are nearer to zero and so, more compressible.

Replies are listed 'Best First'.
Re^2: Specialized data compression
by wanna_code_perl (Pilgrim) on Sep 16, 2012 at 16:37 UTC

    These suggestions were helpful. Thank you! Using deltas for x,y,z did help quite a bit.

    The time delta derivative approach was interesting to think about, but would basically break the lossless compression I have going. At the moment, at least, I need to be able to reconstruct the original input verbatim, and the only way to do that is with fixed point arithmetic.

    The one concession I was able to safely (verifiably) make is quantizing the time deltas that are close to the sensor frequency. The clock has a large (+/- 15%!) but predictable jitter pattern I was able to isolate and then basically RLE the result.

    With the latest optimizations, I'm sitting at about 9:1 compression, which is acceptable. I was shooting for 10:1, but that last 33 KiB won't come easy.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://992967]
[MidLifeXis]: Beer o'clock. Have a good weekend all. o/

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (10)
As of 2017-02-24 22:45 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (364 votes). Check out past polls.