Persistence and handling large amounts of multi-level self-referential data

[premchai21]

I am writing an OO program. It being OO (or some close approximation), it deals with its data in the form of objects. These objects are 2-3 levels deep in references, and refer to each other in a mad, free-form manner which would normally not be a problem, except...

... that this data may grow to be too large for memory to hold. Not only that, but also it may, nay, will, need to be saved to disk. Thus, I need persistence of some sort. My first instinct was to use a tied hash to store all the objects and use something like DB_File. However, DB_File stringifies everything, which is unacceptable. My next thought was to use MLDBM or similar (or write my own), which might work, except I'm not sure how well it handles circular structures. I think it dumps the whole structure into each slot.

Any thoughts?

[princepawn]

The new kid on the block in data freeze-thawing is Data::Denter and it does handle circular references perfectly.

I'm not sure why stringification is bad for you, but it does use a readable (indented) format for data dumping.

[premchai21]

I am aware of Data::Denter and in fact use it for simpler persistence situations.. By stringification I mean stringification of the reference, rather than the frozen version. Stringification of the frozen version is fine. However, I see no elegant way to use Data::Denter to do what I'm trying to do; that is, something like DB_File, with caching etc. If I just Denter the hash and dump that into a file (DB or otherwise), I run into the problem of possibly not having enough memory to read it back. It might be possible to write my own module that uses DB_File as a backend, but stores Dentered data to the DB_File, and Undents it when FETCHing. But then my circular references get clobbered.

[perrin]

MLDBM is a good approach. You can use MLDBM::Sync for multi-process concurrent access. You could also look at Cache::Cache which provides various storage options. Finally, you might consider using SPOPS, which supports dbm files but would also allow you to move your objects to an RDBMS pretty simply later on.

[Arguile]

I'm not familiar enough with OO to be confident in my reply, but I ran across something a while back that might interest you:

What is Tangram?

Tangram is an object-relational mapper for Perl objects. It makes objects persist in relational databases, and provides powerful facilities for retrieving and filtering them. Tangram fully supports object-oriented programming, including polymorphism, multiple inheritance and collections. It does so in an orthogonal fashion, that is, it doesn't require your classes to implement support functions nor inherit from a utility class.

See Also

• Tanagram Home
• WEBTechniques Column on Tanagram

[rob_au]

I looked into this collection of modules for a similar project where I had already built much of my object models and while the conversion of data types was fairly direct and straight-forward, documented examples proved to be incorrect and outdated and the whole experience left me software jaded with Tangram.

In short, while Tangram is certainly an excellent modules for establishing and storing data relations, its documentation is horrid and examples erroneous in points. Approach with plenty of time to experiment ...

 

Ooohhh, Rob no beer function well without!

[premchai21]

Update: Tested MLDBM, and it does work with multi-level circular structures. I think I might use that. I'm still looking for any suggestions, comments, etc. though.

[lachoy]

You should definitely check out The POOP page for a rundown of different persistence schemes. This is definitely an area of lots of work, including by me :-)

Chris
M-x auto-bs-mode