Beefy Boxes and Bandwidth Generously Provided by pair Networks vroom
Problems? Is your data what you think it is?
 
PerlMonks  

Re^2: AUTOLOAD for variables?

by dpuu (Chaplain)
on Jul 08, 2004 at 04:19 UTC ( #372668=note: print w/ replies, xml ) Need Help??


in reply to Re: AUTOLOAD for variables?
in thread AUTOLOAD for variables?

I was trying not to get into too many details -- they aren't terribly important -- except for finding alternative solutions that don't involve the AUTOVIVIFY method.

Yes, the loads of users of perl-config files is kinda scary -- especially when you see how some people take advantage of having perl. The context has nothing to do with web. It's actually the test environment for our hardware (chip) development. People configure the test scripts by grabing configuration from a database of chip-specific information -- and that database has grown somewhat big over the years (along with the number of scripts that access it).

You are right that there are some serious design issues -- the sort that take a fair amount of time to sort out. We basically have to trawl through a deeply tested directory structure, over nfs, to find about 50K files in the leaves -- then parse those files into a flat namespace for the config file (its not a real database -- which may be part of the problem. Its not quite as bad as it sounds, because we can reduce the scanning with some caching) Then a config file uses only a small number of the values we supply. Multiply this onto a farm with multiple thousand CPUs, and soon you're talking real money!

It really is tremendously ugly (and it may be impossible to reverse-engineer a spec). I was hoping to find a quick fix to use as a stop-gap measure while sorting out the real problems. Perhaps if there is no quick fix, then there's a greater incentive to apply the resources needed to clean it up. I guess this is what happens when a startup becomes big, quickly.

--Dave
Opinions my own; statements of fact may be in error, or may be deliberately obscured.


Comment on Re^2: AUTOLOAD for variables?
Re^3: AUTOLOAD for variables?
by tachyon (Chancellor) on Jul 08, 2004 at 04:58 UTC

    flat namespace for the config file (its not a real database -- which may be part of the problem

    It really depends on how it actually works in practice but it sounds like your end data structure could work better. Depending on all sorts of factors an RDBMS or using a flatfile that stores a hash data structure, serialised with say Storable may help. With the Storable suggestion you would parse the data into a hash (very fast access) then store it into a flatfile with Storable. Anything that needs access just loads the file and gets the whole config hash ready to go. The net result is you effectively spend memory to gain speed.

    cheers

    tachyon

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://372668]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (10)
As of 2014-04-18 23:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (472 votes), past polls