Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Storable: Byte order is not compatible

by moseley (Acolyte)
on Oct 02, 2003 at 01:37 UTC ( #295788=perlquestion: print w/ replies, xml ) Need Help??
moseley has asked for the wisdom of the Perl Monks concerning the following question:

On Solaris w/ Perl 5.00503 I have a Berkeley DB database. The "values" in the database are structures saved with Storable::nfreeze.

I need to move that to Linux running 5.8.0 and a newer version of Berkeley DB.

I moved the file and then run:

$ db_dump foo.old | db4.0_load foo.new
But when I try to thaw the data I'm still getting:

Byte order is not compatible at ../../lib/Storable.pm (autosplit into +../../lib/auto/Storable/thaw.al) line 358.
I was able to move another Berkeley DB table without problems (that table's values were plain text).

I suppose I can figure out another way to move the data, but I'm also wondering what I'm doing wrong. Well, for one thing I was not able to easily find the Changelog for Storable to see if there was something broken with nfreeze. And my googling for that error message just turned up suggestions to use nfreeze or nstore (but I'm already using nfreeze).

Comment on Storable: Byte order is not compatible
Select or Download Code
Re: Storable: Byte order is not compatible
by PodMaster (Abbot) on Oct 02, 2003 at 02:28 UTC
    Hmm, seems to work for me (perl 5.6.1 and 5.8.0)
    $Storable::VERSION = 2.08 $DB_File::VERSION = 1.805 $DB_File::db_ver = 4.001025

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

Re: Storable: Byte order is not compatible
by athomason (Curate) on Oct 02, 2003 at 02:29 UTC
    Quoth Storable:
    What this means is that if you have data written by Storable 1.x running on perl 5.6.0 or 5.6.1 configured with 64 bit integers on Unix or Linux then by default this Storable will refuse to read it, giving the error Byte order is not compatible. If you have such data then you you should set $Storable::interwork_56_64bit to a true value to make this Storable read and write files with the old header. You should also migrate your data, or any older perl you are communicating with, to this current version of Storable.
    See if setting that flag helps.
      I saw $Storable::interwork_56_64bit in the docs, but didn't think it would apply since I have 4 byte integers on both platforms. Not to mention it didn't solve the problem.

      I'm able to read the entire Berkeley DB table into a hash and write it with nstore (nice to have lots of RAM). Then I can read that on the other machine.

      BTW - I can't just tie %hash to the Berkeley DB and nstore( \%hash, 'file' ), though. That might haved saved some memory.

      So perhaps it is either an issue with nfreeze on the old machine or somehow converting the berkeley db table to a newer version trashed the data (which seem less likely).

Re: Storable: Byte order is not compatible
by Aristotle (Chancellor) on Oct 02, 2003 at 07:19 UTC
    You could try re-serializing your data using a different mechanism before you convert the database. FreezeThaw and Data::Dumper come to mind. Using the latter, you could even see if anything went awry during the conversion.

    Makeshifts last the longest.

      I ran into the following problem: in our database, we have some frozen Perl hashes that were created with an old 32-bit version of Storable. While moving our application to a newer version of Linux and Perl and Storable, we discovered that the newer version of Storable, 64-bit, couldn't thaw the old frozen hashes.

      I developed the following Perl solution, which works for the data that we have. The code can take something that an old, 32-bit Storable freeze created and convert it into something that a new, 64-bit Storable can thaw.

      sub fix_frozen { my $frozen= shift; my $new= join( '', (qw/04 08 08 31 32 33 34 35 36 37 38 04 08 08 08/, (map {sprintf("%02x", ord($_))} (split //, substr($frozen, 11))))); $fixed_frozen=~ s/0{7}b/0{15}b/sg; join('', map {chr hex $_} ($fixed_frozen =~ /[0-9a-z]{2}/gi)); }

      Keep in mind that this code will not work for all cases and that I've only ever tested it with the data that we have.

        in our database, we have some frozen Perl hashes that were created with [] Storable

        That right there is your problem. Storable makes no promises about binary compatibility across different releases or different perls. You do not want to use it as a serialisation mechanism. It is not a good idea for anything except intra- or interprocess communication between forks of the same process.

        Most likely you want JSON. That will also decouple your data from perl. JSON::XS even happens to be faster than Storable, and by no small margin (which is where the simplicity of the JSON data model pays off).

        Makeshifts last the longest.

        The above function doesn't look correct or useful. I wrote one to convert Storable 2.30 32-bit x86 data to 64-bit architecture. It seemed to work correctly for my data but was not thoroughly tested. Hope someone finds it useful. You should be able to tweak it as needed for other systems.

        # Convert Linux (little-endian) 32-bit Storable format to 64-bit. # Doesn't yet handle a few Storable types. # Tweak xxx lines for your architectures. # # Greg Ubben, 18 June 2012 # sub fix_frozen { local $data = shift; local $pos = 15; # xxx length of new header below local $lev = 0; # version, byte order, sizes (int, long, ptr, double) $data =~ s{^\x04\x07\x041234\x04\x04\x04\x08} # xxx {\x04\x08\x0812345678\x04\x08\x08\x08} # xxx or die "not a 32-bit x86 Storable"; object(); die "length error" if $pos != length( $data ); return $data; } sub object { local $_ = chr( 64 + byte() ); local $lev = $lev + 1; #printf "%8d %*s\n", $pos, $lev * 2, $_; # DEBUG return if /E|N|O|P/; return $pos += 1 if /H/; return $pos += 4 if /@|I/; return $pos += byte() if /J|W/; return $pos += len() if /A|X/; return object() if /D|T|K|L|M/; return object( $pos += vnum()) if /Q/; return object( vnum() ) if /R/; return fix_integer() if /F/; return fix_double() if /G/; if (/B|C|Y/) { # array or hash my $n = len(); $pos++ if /Y/; while ($n--) { object(); $pos++ if /Y/; $pos += len() if not /B/; } return; } die sprintf "Type %d unknown at pos %d\n", ord()-64, $pos-1; } sub byte { return ord( substr( $data, $pos++, 1 )); } sub len { my $len = unpack 'L', substr( $data, $pos, 4 ); # xxx 'V' or 'N' + ? $pos += 4; return $len; } sub vnum { my $n = byte(); return ($n < 128 ? $n : len()); } sub fix_integer { # xxx my $n = unpack 'l', substr( $data, $pos, 4 ); # was 32-bit substr( $data, $pos, 4 ) = pack 'q', $n; # now 64-bit $pos += 8; } sub fix_double { # xxx $pos += 8; # assume same floating-point format }

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://295788]
Approved by bart
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2014-09-22 02:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (177 votes), past polls