Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Never use Storable to save XS objects

by Jouke (Curate)
on May 11, 2004 at 17:31 UTC ( #352519=perlmeditation: print w/replies, xml ) Need Help??

The past 6 months I've been struggling with a weird bug I had in pVoice. The application itself ran fine, but when the application closed, 99% of the time I got an Access Violation in Wx::Bitmap, which tried to free the memory used by some Bitmap I had obviously created...or so I thought

Yesterday evening I used Data::Dumper to dump the datafile I use for pVoice, which showed me lots of entries like this:
[ 'wassen', 'wassen', '1', 'C:\\Mijn documenten\\PCS\\Verbs\\Gr +ooming\\wash 1.jpg', bless( do{\(my $o = 12451544)}, 'Wx: +:Bitmap' ) ],
I had seen entries like that before and never really worried about it. Indeed, once I tried to save the actual Bitmap in the Storable file, which didn't work, but I didn't care about having that entry in my datafile. I didn't use it, did I?

But yesterday it struck me. Was it just a coincidence that my Access Violations happened in Wx::Bitmap and that I had those lines in my datafile? Obviously not. Removing all references to those Wx::Bitmap objects, and saving the datastructure again solved the problem. So what has been going on here?

My theory is this: the code references are only valid for the time the application runs. When the datastructure is read in another run, the data the Wx::Bitmap entry refers to doesn't exist, but when the application closes, it will still try to free the memory, which is obviously not in use by my application, so an Access Violation is generated.

Moral of the story: never use Storable to save objects that aren't pure Perl, but consist of XS code.

Jouke Visser
Speaking at the 2004 O'Reilly Open Source Convention about pVoice

Replies are listed 'Best First'.
Re: Never use Storable to save XS objects
by ysth (Canon) on May 11, 2004 at 17:44 UTC
    It depends; ideally modules mucking about as Wx::Bitmap seems to would provide Storable hooks to be properly serializable. (See Storable)

      ysth says:

      It depends; ideally modules mucking about as Wx::Bitmap seems to would provide Storable hooks to be properly serializable

      Nice, I'd missed/forgotten about those hooks. I don't think it'll work for Wx objects that are subclasses of Wx::Window. But it'd probably work fine for Wx::Bitmap. Here's some untested code for (de)?serialising Wx::Bitmaps...

      my $bitmap_data = $wx_bitmap->GetImage->GetData; # serialise bitmap data # later... $wx_bitmap = new Wx::Image()->SetData($bitmap_data)->GetBitmap;

      -- simonflk

Re: Never use Storable to save XS objects
by diotalevi (Canon) on May 11, 2004 at 20:23 UTC

    Also be sure not to use Storable on closures. The lexical environment of the closure is not captured in the Storable object.

    Functions are deparsed into approximately equivalent source code so beware that you get the same functional thing back.

Re: Never use Storable to save XS objects
by simonflk (Pilgrim) on May 11, 2004 at 17:51 UTC

    I've hit this in the past. In my case I was storing objects that sometimes contained DBI handles. It's a mistake I hope not to repeat.

    -- simonflk

      You shouldn't by any chance have some documentation about how you solved that hurdle?

      I have the DBI problem myself. My objects contain DBI handles, and I need to serialize them, but without the DBI code.

      Is there maybe a module in the DBI or Class namespace that can help with this? Any pointers appreciated...

        I think you need to answer a couple of questions first:
        • What do you expect to happen when you serialize the DB connection? Should the DB connection be closed? What should be stored in the serialized data structure?
        • What do you expect to happen when you de-serialize the DB connection? Should the connection happen immediately? What about waiting until the first query because you may not actually use the DB connection at every given de-serialization.
        There are some further concerns:
        • Should passwords be stored? If so, how?
        • How should DBI connections using DBD::Excel or DBD::CSV work?
        • What should happen if you de-serialize the thing and attempt to use the DB connection on a box that doesn't have the right things installed?
        And I'm sure that I'm missing some concerns. In other words, serializing stuff that uses external resources is non-trivial. I've run into the same problem with DBM::Deep and I don't have a really good way to solve it as of yet.

        My criteria for good software:
        1. Does it work?
        2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
Re: Never use Storable to save XS objects
by demerphq (Chancellor) on May 12, 2004 at 15:04 UTC

    You should be looking at exploiting the freezer/toaster methods in Data::Dumper or Storable. Data::Dump::Streamer has even more sophisticated mechanisms for handling this so you may also want to look there.


      First they ignore you, then they laugh at you, then they fight you, then you win.
      -- Gandhi

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://352519]
Approved by Steve_p
Front-paged by grinder
[davido]: The code seems pretty sane, and relatively foolproof. I guess I can answer my own question; move() should work across filesystems so long as they're mounted, rw, and not filled up.

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (3)
As of 2016-12-10 03:07 GMT
Find Nodes?
    Voting Booth?
    On a regular basis, I'm most likely to spy upon:

    Results (159 votes). Check out past polls.