Re: Serialise to binary?

[davido]

BSON implements BSON - Binary JSON, which is "...a bin­ary-en­coded seri­al­iz­a­tion of JSON-like doc­u­ments. Like JSON, BSON sup­ports the em­bed­ding of doc­u­ments and ar­rays with­in oth­er doc­u­ments and ar­rays. BSON also con­tains ex­ten­sions that al­low rep­res­ent­a­tion of data types that are not part of the JSON spec. For ex­ample, BSON has a Date type and a BinData type."

Sounds like that could be a decent fit, particularly since it is probably more portable than Storable.

[sectokia]

I tried BSON, I found that it would throw a lot of warnings on basic structures, especially it seems to miss identify some scalars as floats and attempts to pack them as floats causing "argument isn't numeric in pack". I also found floats having widly inconsistent values when encoded then decoded.

Because its so heavily tied to MongoDB, they don't see to really care about having it being able to encode arbitrary data structures (evidence by the fact that you have to pass a hash ref, no array ref allowed). They just want to decode their own binary data as they use it in Mongo and re-encode structures setup the same way.

So I wouldn't recommend it.

[xdg]

they don't see to really care about having it being able to encode arbitrary data structures

I think that's an unkind assumption about intent. (N.B. I am the current maintainer.)

But like JSON, BSON is document-oriented, so is not designed to store raw arrays or scalars the way Storable or Sereal will. So in that sense, it might not be the right choice for your needs.

Beyond that however, the goal of BSON is to handle whatever you can throw at it as best as possible given the ambiguities mapping data between a dynamic, largely typeless language like Perl and a typed data format like BSON. Knowing that some Perl scalar is binary data and not an arbitrary string is impossible without some hints from the programmer.

The MongoDB::BSON implementation is in XS and has been part of the MongoDB driver distribution. We hope to eventually split it out so that it can be used independently where warranted.

The BSON.pm implementation is pure Perl and was originally developed outside MongoDB (but has since been adopted by the company). There are still some areas where it is not yet as good as MongoDB::BSON.

Even if BSON is not right for this particular problem, if anyone experiences bugs using either implementation, I encourage you to report them or at least email us about them so we can fix them.

-xdg

[davido]

Thanks for jumping into the discussion! :) It's always great to hear from a maintainer.

I'm curious why BSON may not be the right choice. If passing a raw scalar or array is problematic, wouldn't this be a reasonable solution?

encode({data => $some_scalar});

Whether $some_scalar contains an array ref, a hash ref, or a flat scalar, encode would still see it as a hash based document.

But I don't have a lot of experience with BSON. I remember seeing it presented a few years ago at a PerlMongers meeting in So. Cal, and it stuck in my head as a more portable solution for data serialization when the data might contain binary elements.

I was a little surprised to see that there is no (documented) "TO_BSON" means of providing serialization procedures for objects. Is there any plan for that sort of hook in the future?

(Sorry to hijack the thread!)