Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: Specialized data compression

by Kenosis (Priest)
on Sep 11, 2012 at 19:36 UTC ( #993045=note: print w/ replies, xml ) Need Help??


in reply to Specialized data compression

Would it help to convert your decimal values to a higher base, e.g., base 64? Consider the following:

use Modern::Perl; use Math::BaseCnv; my $original = <<END; 17077.161362,-595,2619,-15202 17077.174405,601,2115,-14859 17077.187416,-816,1894,-15539 17077.200402,210,1790,-15284 17077.213445,1560,1721,-15457 17077.226432,1408,1483,-15834 17077.239474,947,1161,-17531 17077.252517,689,1396,-17515 17077.265504,-861,1206,-18238 17077.278547,-493,1369,-17951 17077.291589,850,1294,-17360 17077.304576,1110,1215,-17271 17077.317619,326,1055,-13948 17077.330662,1980,1083,-16239 17077.343648,1295,1057,-16154 17077.356691,-1302,1207,-16277 17077.369734,-1986,1216,-16537 17077.382721,-134,1108,-17409 17077.395763,-739,1059,-16734 17077.408774,-2262,1006,-16690 17077.421761,-807,1174,-17046 17077.434803,-591,1081,-17615 17077.447846,-2080,-5757,-21463 17077.460833,-209,-17608,-19409 END my @original = split ' ', $original; say for @{ b10to64( \@original ) }; sub b10to64 { my $original = $_[0]; chomp @$original; for ( my $i = 0 ; $i < @$original ; $i++ ) { my @entries = split ',', $$original[$i]; my ( $whole, $decimal ) = split '\.', $entries[0]; $entries[0] = cnv( $whole, 10, 64 ) . '.' . cnv( $decimal, 10, + 64 ); $entries[$_] = cnv( $entries[$_], 10, 64 ) for 1 .. 3; $$original[$i] = ( join ',', @entries ); } $original; }

Output:

4Ar.dPI,-9J,ex,-3jY 4Ar.gb5,9P,X3,-3eB 4Ar.jmO,-Cm,Tc,-3op 4Ar.mxI,3I,R.,-3kq 4Ar.q75,OO,Qv,-3nX 4Ar.tI0,M0,NB,-3tQ 4Ar.wTo,Ep,I9,-4Hx 4Ar.zfb,An,Lq,-4Hh 4Ar.10qW,-DT,Is,-4S. 4Ar.140J,-7j,LP,-4OV 4Ar.17C5,DI,KE,-4FG 4Ar.1AN0,HM,I_,-4Dt 4Ar.1DYp,56,GV,-3Py 4Ar.1Gkc,Uy,Gx,-3zl 4Ar.1JvW,KF,GX,-3yQ 4Ar.1N5J,-KM,It,-3.L 4Ar.1QH6,-V2,J0,-42P 4Ar.1TS1,-26,HK,-4G1 4Ar.1Wdp,-BZ,GZ,-45U 4Ar.1Zp6,-ZM,Fk,-44o 4Ar.1c.1,-Cd,IM,-4AM 4Ar.1g9p,-9F,Gv,-4JF 4Ar.1jLc,-WW,-1Pz,-5FN 4Ar.1mWX,-3H,-4J8,-4lH

Conversion artifact:

4Ar.1c.1,-Cd,IM,-4AM ^ | + - split on first '.' when converting back to base 10

This yielded a 30% reduction size of your data in accel.csv, which may help with your final data size after packing and compression.

Update: Evidently, this would not help...


Comment on Re: Specialized data compression
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://993045]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (6)
As of 2014-08-31 04:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls