Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Specialized data compression

by Kenosis (Priest)
on Sep 11, 2012 at 19:36 UTC ( #993045=note: print w/ replies, xml ) Need Help??


in reply to Specialized data compression

Would it help to convert your decimal values to a higher base, e.g., base 64? Consider the following:

use Modern::Perl; use Math::BaseCnv; my $original = <<END; 17077.161362,-595,2619,-15202 17077.174405,601,2115,-14859 17077.187416,-816,1894,-15539 17077.200402,210,1790,-15284 17077.213445,1560,1721,-15457 17077.226432,1408,1483,-15834 17077.239474,947,1161,-17531 17077.252517,689,1396,-17515 17077.265504,-861,1206,-18238 17077.278547,-493,1369,-17951 17077.291589,850,1294,-17360 17077.304576,1110,1215,-17271 17077.317619,326,1055,-13948 17077.330662,1980,1083,-16239 17077.343648,1295,1057,-16154 17077.356691,-1302,1207,-16277 17077.369734,-1986,1216,-16537 17077.382721,-134,1108,-17409 17077.395763,-739,1059,-16734 17077.408774,-2262,1006,-16690 17077.421761,-807,1174,-17046 17077.434803,-591,1081,-17615 17077.447846,-2080,-5757,-21463 17077.460833,-209,-17608,-19409 END my @original = split ' ', $original; say for @{ b10to64( \@original ) }; sub b10to64 { my $original = $_[0]; chomp @$original; for ( my $i = 0 ; $i < @$original ; $i++ ) { my @entries = split ',', $$original[$i]; my ( $whole, $decimal ) = split '\.', $entries[0]; $entries[0] = cnv( $whole, 10, 64 ) . '.' . cnv( $decimal, 10, + 64 ); $entries[$_] = cnv( $entries[$_], 10, 64 ) for 1 .. 3; $$original[$i] = ( join ',', @entries ); } $original; }

Output:

4Ar.dPI,-9J,ex,-3jY 4Ar.gb5,9P,X3,-3eB 4Ar.jmO,-Cm,Tc,-3op 4Ar.mxI,3I,R.,-3kq 4Ar.q75,OO,Qv,-3nX 4Ar.tI0,M0,NB,-3tQ 4Ar.wTo,Ep,I9,-4Hx 4Ar.zfb,An,Lq,-4Hh 4Ar.10qW,-DT,Is,-4S. 4Ar.140J,-7j,LP,-4OV 4Ar.17C5,DI,KE,-4FG 4Ar.1AN0,HM,I_,-4Dt 4Ar.1DYp,56,GV,-3Py 4Ar.1Gkc,Uy,Gx,-3zl 4Ar.1JvW,KF,GX,-3yQ 4Ar.1N5J,-KM,It,-3.L 4Ar.1QH6,-V2,J0,-42P 4Ar.1TS1,-26,HK,-4G1 4Ar.1Wdp,-BZ,GZ,-45U 4Ar.1Zp6,-ZM,Fk,-44o 4Ar.1c.1,-Cd,IM,-4AM 4Ar.1g9p,-9F,Gv,-4JF 4Ar.1jLc,-WW,-1Pz,-5FN 4Ar.1mWX,-3H,-4J8,-4lH

Conversion artifact:

4Ar.1c.1,-Cd,IM,-4AM ^ | + - split on first '.' when converting back to base 10

This yielded a 30% reduction size of your data in accel.csv, which may help with your final data size after packing and compression.

Update: Evidently, this would not help...


Comment on Re: Specialized data compression
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://993045]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (4)
As of 2015-07-08 02:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (93 votes), past polls