Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Fastest way to store and retrieve configuration information?

by Booger (Pilgrim)
on Nov 30, 2005 at 03:22 UTC ( #512835=perlquestion: print w/ replies, xml ) Need Help??
Booger has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks:

I'm currently working with an Apache/mod_perl Maypole-based web application that depends upon a fairly complex set of configuration directives.

These settings are currently stored in a number of XML files. I'm parsing them with XML simple and turning the resulting data structure over to the web application.

What I need is a more efficient (i.e. faster, less memory & hopefully less processor-intensive) way to store and serialize this data and cache it for the rest of the of the Apache children to read from (so that Apache child processes don't have to re-read the configuration every time they start).

Whatever format the configuration is stored in needs to be fairly easily read for my configuration application so that I can edit it as need be. Basically what we have here is a large configuration engine that understands the configuration data structure and causes the web application to act accordingly. Think a much smaller implementation of Siebel for Perl.

I'm looking for your suggestions for a fast (i.e. very fast) ways to store and read a complex configuration. I guess I'm essentially dependant upon some sort of serialization process that will convert my data structure in memory to disk so that I can read it back later.

As mentioned above I'd ideally like to do this in a much more efficient way than I am now (with my configuration in XML files and the data being converted to a HASH via XML::Simple).

Any suggestions or ideas would be greatly appreciated.

Thanks!

Matt

Comment on Fastest way to store and retrieve configuration information?
Re: Fastest way to store and retrieve configuration information?
by GrandFather (Cardinal) on Nov 30, 2005 at 03:31 UTC

    Take a look at Storable.


    DWIM is Perl's answer to Gödel
      GrandFather:

      Thank you for your suggestion.

      From what I can tell, Storable is at least twice as fast as the XML-method I was using and I don't have to import any more large XML modules into my web app namespace, either. I like it.

      Only one question now remains: is storable safe in terms of cross-platform (i.e. Linux to Mac OS X) and cross-architecture (i.e. 32-bit and 64-bit)?

      I didn't see anything in the Storable POD that would indicate otherwise but I was just wondering if you knew off the top of your head.

      Thing is, I'm going to want to generate the "storable" file on a 32-bit Linux server and use it on 64-bit Linux servers and Mac OS X machines.

      Any thoughts?

      Thanks again,

      Matt

        The following is of interest:

        You can also store data in network order to allow easy sharing across multiple platforms, or when storing on a socket known to be remotely connected. The routines to call have an initial n prefix for network, as in nstore and nstore_fd. At retrieval time, your data will be correctly restored so you don't have to know whether you're restoring from native or network ordered data. Double values are stored stringified to ensure portability as well, at the slight risk of loosing some precision in the last decimals.

        so if you use the n prefixed functions the data is serialised in network order so should be fine across architectures.


        DWIM is Perl's answer to Gödel
Re: Fastest way to store and retrieve configuration information?
by Zaxo (Archbishop) on Nov 30, 2005 at 03:36 UTC

    The combination of Data::Dumper and do is very handy for that.

    use Fcntl qw/:flock/; use Data::Dumper; # Get stored config my $config = do( '/path/to/my_config.cfg') || {}; # . . . { # Serialize current config open my $fh, '>', '/path/to/my_config.cfg' or die $!; flock $fh, LOCK_EX; print $fh Dumper($config); }

    After Compline,
    Zaxo

Re: Fastest way to store and retrieve configuration information?
by NetWallah (Abbot) on Nov 30, 2005 at 04:19 UTC
    Looks like a candidate for Cache::Cache.

    Another approach would be Memoize or a variant therof.

         You're just jealous cause the voices are only talking to me.

         No trees were killed in the sending of this message.    However, a large number of electrons were terribly inconvenienced.

      NetWallah:

      Thank you for your suggestion... I will have to check these options out when I have a few spare minutes!

      Matt

Re: Fastest way to store and retrieve configuration information?
by pajout (Curate) on Nov 30, 2005 at 10:45 UTC
    Just only my opinion:
    1/ Parse xml files at server startup (for instance, using XML::Simple) and store it in global variable
    2/ It is chance to avoid the memory consumption if you will not change this global variable in child processes - hopefully it will stay in memory page shared across httpd's.
    3/ If you really need to store and load this confs quickly, use Data::Dumper as Zaxo recommends...
Re: Fastest way to store and retrieve configuration information?
by Perl Mouse (Chaplain) on Nov 30, 2005 at 11:33 UTC
    It's hard to say what is the fastest, as it will depend on what data you have, how your datastructure looks like and how much data we are talking about. Possibilities include:
    • Use a database. Probably not going to be the fastest, but it might. But it's very reliable.
    • Use one of the serialize modules available: Data::Dumper, Storable, YAML. If you want something that's human editable, YAML may be your best choice (it would be mine if editability is an issue).
    • Use *dbm files. Less suitable if the datastructure is complex.
    • Use a "generic" format such as XML, Config::General or one of the other Config::* modules. Perhaps not the fastest, but a smart choice here could score big points in the editability requirement.
    • Handroll something. Requires the most work, but (in theory) gives you the best runtime/editability requirement trade-off as you can tailor it to your problem domain.
    Perl --((8:>*

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://512835]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (5)
As of 2014-07-28 23:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (210 votes), past polls