fokat has asked for the wisdom of the Perl Monks concerning the following question:
Dear monks:
I have a (very) large archive of processed reports stored in the filesystem. Those reports are actually Perl objects that have been stored using code resembling this (heavy editing for simplification):
use strict; use warnings; use Storable qw/nstore/; my $rep = bless { phrase => 'All your base belong to us' }, 'SomeClass +'; my $file = 'my_file_name'; eval { nstore($rep, $file) || warn "Storable::nstore failed with $ +!\n"; }; if ($@) { warn "Failed to store the object: $@\n"; }
By using gzip/Compress::Zlib/IO::Zlib we could be saving, on average, 24% - 30% of the disk space according to the testing we've done. Then I thought that it would be very simple to ask Storable to use IO::Zlib to read the compressed, serialized objects.
So, here I am writing this Template::Plugin that can read and present those objects... And banging my head against this code:
if ($path =~ m/\.gz$/) { $fh = new IO::Zlib } else { $fh = new IO::File } $ctx->throw('Abuse.open', "Problem opening Abuse report: $!") unless $fh->open($path, "r"); $rep = fd_retrieve($fh); close $fh;
This code is throwing an exception that reads: Not a GLOB reference at /usr/lib/perl5/site_perl/5.8.5/IO/Zlib.pm line 566. when taking the IO::Zlib path, but works beautifully when taking the IO::File branch.
This is the list of things that I've tested, without success:
- Rewriting the $fh in fd_retrieve in various ways (\*{$fh}, *{$fh}, ...). The IO::File branch always work, the IO::Zlib always throws the same exception, although sometimes it happens within Storable, likely due to the crazy things I've tried so far.
- Using IO::Scalar to first read the serialized object to a scalar and then, feeding fd_retrieve.
I can't believe that I am the only one trying to use those two modules together, although I am ready to admit that I am the only one failing :)
I know that likely, something along these lines...
my $fh = IO::File->new("gunzip $file |");
...would work, but I want to avoid calling external programs and all the issues that may come out of it. Specially when all the required code is already within Perl's module library. (Yes, I know that IO::Zlib will fallback to more or less that when no Compress::Zlib is around, but then I don't have to support IO::Zlib).
For the record, these are the relevant versions of what I am using:
$ find /usr/lib/perl* -type f -name Zlib.pm -o -name Storable.pm | xar +gs egrep -i '\$version = ' /usr/lib/perl5/5.8.5/i386-linux-thread-multi/Storable.pm:$VERSION = '2 +.15'; /usr/lib/perl5/5.8.5/Memoize/Storable.pm:$VERSION = 0.65; /usr/lib/perl5/site_perl/5.8.5/IO/Zlib.pm:$VERSION = "1.04"; /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Compress/Zlib.p +m:$VERSION = "1.41" ;
Update: Although I failed to mention this originally, I did look the code of both IO::File and Storable as suggested by frodo72 and samtregar...
This code:
sub AUTOLOAD { my $self = shift; $AUTOLOAD =~ s/.*:://; $AUTOLOAD =~ tr/a-z/A-Z/; return tied(*{$self})->$AUTOLOAD(@_); # line 566 }
Is calling ->READ() when ->read() is called. That call is coming from C code, part of Storable (which I'll be looking at shortly, although I'm very weak with XS.
Update: Prompted by frodo72 and samtregar, I probed deeper in the IO::Zlib and Storable code...
Adding a flag, I found this:
IO::Zlib AUTOLOAD=IO::Zlib::FILENO self=IO::Zlib=HASH(0x9d3c8a0) at /usr/lib/perl5/site_perl/5.8.5/IO/Zlib.pm line 567The call that is causing the exception, is happening within fd_retrieve and is looking to verify that it was passed a filehandle.
sub fd_retrieve { my ($file) = @_; my $fd = fileno($file); # <== THIS IS THE CALL logcroak "not a valid file descriptor" unless defined $fd; my $self; my $da = $@; # Could be from exception +handler eval { $self = pretrieve($file) }; # Call C routine logcroak $@ if $@ =~ s/\.?\n$/,/; $@ = $da; return $self; }
Adding this to IO::Zlib...
sub FILENO { 1 }...gets rid of the exception, but causes the check of the version number of the serialized file to fail, so there's something else.
Update: Following advice from sauoq, I dropped IO::Zlib in favor of PerlIO::gzip. The now fully working code looks similar to this...
$ctx->throw('Abuse.open', "Problem opening Abuse report: $!") unless $fh->open($path, ($path =~ /\.gz$/? "<:gzip" : "<"));
Best regards
-lem, but some call me fokat
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: Do Storable and IO::Zlib like to play together?
by sauoq (Abbot) on Nov 03, 2005 at 19:32 UTC | |
by fokat (Deacon) on Nov 03, 2005 at 19:52 UTC | |
by sauoq (Abbot) on Nov 03, 2005 at 19:58 UTC | |
Re: Do Storable and IO::Zlib like to play together? (one solution)
by sauoq (Abbot) on Nov 03, 2005 at 20:58 UTC | |
by fokat (Deacon) on Nov 03, 2005 at 21:19 UTC | |
Re: Do Storable and IO::Zlib like to play together?
by polettix (Vicar) on Nov 03, 2005 at 17:41 UTC | |
by fokat (Deacon) on Nov 03, 2005 at 18:22 UTC | |
Re: Do Storable and IO::Zlib like to play together?
by samtregar (Abbot) on Nov 03, 2005 at 17:49 UTC | |
by fokat (Deacon) on Nov 03, 2005 at 18:23 UTC |