Fastest way to store and retrieve configuration information?

[Booger]

Dear Monks:

I'm currently working with an Apache/mod_perl Maypole-based web application that depends upon a fairly complex set of configuration directives.

These settings are currently stored in a number of XML files. I'm parsing them with XML simple and turning the resulting data structure over to the web application.

What I need is a more efficient (i.e. faster, less memory & hopefully less processor-intensive) way to store and serialize this data and cache it for the rest of the of the Apache children to read from (so that Apache child processes don't have to re-read the configuration every time they start).

Whatever format the configuration is stored in needs to be fairly easily read for my configuration application so that I can edit it as need be. Basically what we have here is a large configuration engine that understands the configuration data structure and causes the web application to act accordingly. Think a much smaller implementation of Siebel for Perl.

I'm looking for your suggestions for a fast (i.e. very fast) ways to store and read a complex configuration. I guess I'm essentially dependant upon some sort of serialization process that will convert my data structure in memory to disk so that I can read it back later.

As mentioned above I'd ideally like to do this in a much more efficient way than I am now (with my configuration in XML files and the data being converted to a HASH via XML::Simple).

Any suggestions or ideas would be greatly appreciated.

Thanks!

Matt

[GrandFather]

Take a look at Storable.

[Booger]

GrandFather:

Thank you for your suggestion.

From what I can tell, Storable is at least twice as fast as the XML-method I was using and I don't have to import any more large XML modules into my web app namespace, either. I like it.

Only one question now remains: is storable safe in terms of cross-platform (i.e. Linux to Mac OS X) and cross-architecture (i.e. 32-bit and 64-bit)?

I didn't see anything in the Storable POD that would indicate otherwise but I was just wondering if you knew off the top of your head.

Thing is, I'm going to want to generate the "storable" file on a 32-bit Linux server and use it on 64-bit Linux servers and Mac OS X machines.

Any thoughts?

Thanks again,

Matt

[GrandFather]

The following is of interest:

You can also store data in network order to allow easy sharing across multiple platforms, or when storing on a socket known to be remotely connected. The routines to call have an initial n prefix for network, as in nstore and nstore_fd. At retrieval time, your data will be correctly restored so you don't have to know whether you're restoring from native or network ordered data. Double values are stored stringified to ensure portability as well, at the slight risk of loosing some precision in the last decimals.

so if you use the n prefixed functions the data is serialised in network order so should be fine across architectures.

[Zaxo]

The combination of Data::Dumper and do is very handy for that.

use Fcntl qw/:flock/;
use Data::Dumper;

# Get stored config
my $config = do( '/path/to/my_config.cfg') || {};
# . . .
{   # Serialize current config
    open my $fh, '>', '/path/to/my_config.cfg' or die $!;
    flock $fh, LOCK_EX;
    print $fh Dumper($config);
}

[NetWallah]

Looks like a candidate for Cache::Cache.

Another approach would be Memoize or a variant therof.

[Booger]

NetWallah:

Thank you for your suggestion... I will have to check these options out when I have a few spare minutes!

Matt

[pajout]

Just only my opinion:
1/ Parse xml files at server startup (for instance, using XML::Simple) and store it in global variable
2/ It is chance to avoid the memory consumption if you will not change this global variable in child processes - hopefully it will stay in memory page shared across httpd's.
3/ If you really need to store and load this confs quickly, use Data::Dumper as Zaxo recommends...

[Perl Mouse]

It's hard to say what is the fastest, as it will depend on what data you have, how your datastructure looks like and how much data we are talking about. Possibilities include:
• Use a database. Probably not going to be the fastest, but it might. But it's very reliable.
• Use one of the serialize modules available: Data::Dumper, Storable, YAML. If you want something that's human editable, YAML may be your best choice (it would be mine if editability is an issue).
• Use *dbm files. Less suitable if the datastructure is complex.
• Use a "generic" format such as XML, Config::General or one of the other Config::* modules. Perhaps not the fastest, but a smart choice here could score big points in the editability requirement.
• Handroll something. Requires the most work, but (in theory) gives you the best runtime/editability requirement trade-off as you can tailor it to your problem domain.