Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re: Specialized data compression

by salva (Abbot)
on Sep 11, 2012 at 09:28 UTC ( #992967=note: print w/ replies, xml ) Need Help??

in reply to Specialized data compression

The x,y,z acceleration values in your sample file are highly correlated on time, so using delta values for them should also improve their compressibility.

Then, you can divide the x,y,z deltas by the time deltas in order to get the derivate of the acceleration vector in order to get even more normalized values. The problem is that at this point you are using floats instead of integers, but unless you want absolute precision this should be a minor issue. Just select a resolution that fits your problem and discretize them accordingly.

Another possibility is to express the x,y,z deltas (or derivates) in the coordinate system defined by the Frenet-Serret frame. That would probably produce numbers that are nearer to zero and so, more compressible.

Comment on Re: Specialized data compression
Replies are listed 'Best First'.
Re^2: Specialized data compression
by wanna_code_perl (Pilgrim) on Sep 16, 2012 at 16:37 UTC

    These suggestions were helpful. Thank you! Using deltas for x,y,z did help quite a bit.

    The time delta derivative approach was interesting to think about, but would basically break the lossless compression I have going. At the moment, at least, I need to be able to reconstruct the original input verbatim, and the only way to do that is with fixed point arithmetic.

    The one concession I was able to safely (verifiably) make is quantizing the time deltas that are close to the sensor frequency. The clock has a large (+/- 15%!) but predictable jitter pattern I was able to isolate and then basically RLE the result.

    With the latest optimizations, I'm sitting at about 9:1 compression, which is acceptable. I was shooting for 10:1, but that last 33 KiB won't come easy.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://992967]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (10)
As of 2015-12-01 10:37 GMT
Find Nodes?
    Voting Booth?

    What would be the most significant thing to happen if a rope (or wire) tied the Earth and the Moon together?

    Results (798 votes), past polls