Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Data Structure Design

by Tuna (Friar)
on Aug 17, 2001 at 04:42 UTC ( [id://105588]=perlquestion: print w/replies, xml ) Need Help??

Tuna has asked for the wisdom of the Perl Monks concerning the following question:

Ok. I'll try to articulate this the best that I can. I have a series of 5 clusters, each comprised of 7 machines each. I need to "take action" on all of these machines, in the following order:
Cluster_1 Host 1 Host 2 Host 3 Host 4 Host 5 Host 6 Host 7

...and so on, acting on a set of hosts for each cluster, before moving on to the next cluster.

Here's a catch: I need to act on hosts 1-6 differently than host 7 and before host 7. I have my routines pretty much finished, however, I can't for the life of me figure out how to construct, then iterate sequentially through an appropriate data structure.

My idea, although I'm not sure exactly how to implement it, is to use a HoHoL.

my %clusters = ( cluster1_TS => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6'], cluster1_SA => ['h7'], cluster2_TS => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6'], cluster2_SA => ['h7'], cluster3_TS => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6'], cluster3_SA => ['h7'], cluster4_TS => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6'], cluster4_SA => ['h7'], cluster5_TS => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6'], cluster5_SA => ['h7'], );

Not sure if the above even makes sense, but I'll continue.

My questions:

1. Does the above make sense, or is there another (better way) to accomplish what I need to accomplish?
2. What I need to accomplish, is to take action in EXACTLY the order portrayed in the data structure above. How can I create a looping statement for the above?
3. Assuming that the above data structure is doable, how would one populate it?

I know that this is a very (well, for me) complicated question, but any assistance would be appreciated, as my new employer needs to get this tool to QA tomorrow!

Replies are listed 'Best First'.
(Ovid - plan for the future) Re: Data Structure Design
by Ovid (Cardinal) on Aug 17, 2001 at 05:03 UTC

    Since you need to do these in order, I'd probably start with an array. Further, I like to try to make things flexible so that I can easily adjust the behavior in the future. The following quick hack will iterate over the clusters. It defines the total number of machines per cluster (in case they change), and if any machine in the cluster has a different behavior, allows you to assign a subref to that machine. In the future, you can change the number of machines and change the behavior for any machine for any cluster to anything you want. Further, the data structure is very intuitive. This should be simple to maintain.

    Note how easy it is to change the behavior for the fourth cluster.

    use strict; use warnings; my @clusters = ( { total => 7, 7 => \&two }, { total => 7, 7 => \&two }, { total => 7, 7 => \&two }, { total => 16, 7 => \&two, 4 => \&three }, { total => 7, 7 => \&two }, { total => 7, 7 => \&two }, { total => 7, 7 => \&two } ); foreach my $cluster ( @clusters ) { # decrement count by one to account for arrays starting at zero my $machine_count = $cluster->{ total } - 1; for my $machine ( 0 .. $machine_count ) { if ( ! exists $cluster->{ $machine + 1 } ) { &one; } else { &{ $cluster->{ $machine + 1 } }; } } } sub one { print "First sub\n"; } sub two { print "Second sub\n"; } sub three { print "Third sub\n"; }

    You'll probably want to play with that for loop as the increments and decrements look weird, but it's a start.


    Vote for paco!

    Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

Re: Data Structure Design
by kjherron (Pilgrim) on Aug 17, 2001 at 08:11 UTC
    First of all, let me commend you for being concerned with your data structures. Good data structures can make a program much simpler to understand and extend.

    Anyway, I'd be inclined to do something like this:

    my @clusters = ( { Name => 'cluster1', Workers => [ qw(h1 h2 h3 h4 h5 h6) ], Queen => 'h7' }, { ... } ); foreach my $cluster (@clusters) { foreach my $worker (@{$cluster->{Workers}}) { handle_worker($worker, ...); } handle_queen($cluster->{Queen}, ...); }
    This is obviously very abstract. If you want to do all the workers in parallel you could. I also don't know if "workers" and "queen" correctly describes the relationship between these servers; you could probably think of better terms.

    But this kind of arrangement gives you a lot of flexibility. For example, if another kind of server gets added to the mix later, it's trivial to add processing for it.

Re: Data Structure Design
by jepri (Parson) on Aug 17, 2001 at 04:54 UTC
    foreach my $num (1..5) { foreach my $letters ( ('TS','SA') ) { my $name = "cluster".$num."_".$letters; foreach ( @{$cluster{$name}}) { do_action($_); } } }

    I feel that's missing something, but I can't think what.. Update: It's missing a 2-D array.
    Double Update: Looking at Ovid solution, another idea presents itself - if all the clusters have the same number of computers, just do a modulo on the array index to find the one you have to treat specially. Consider:

    my @clusters = ( ['h1', 'h2', ... ], ['h8', 'h9', ... ], ['h16', 'h17', ... ], ['h24', 'h25', ... ], ['h32', 'h33', ... ], );

    and now iterate over the array. Or even better

    my @clusters = ( 'h1', 'h2', ... , 'h8', 'h9', ... , 'h16', 'h17', ... , 'h24', 'h25', ... , 'h32', 'h33', ... , );

    Of course you have to go to the effort of making each of the 'h's do something. It depends on the rest of your code how you want to attack it.

    I didn't believe in evil until I dated it.

Re: Data Structure Design
by Tuna (Friar) on Aug 17, 2001 at 05:24 UTC
    Follow up:

    Here's an example of the data, and routines that I am working with:

    $config_files{ $cluster } = [ $config_file ]; This contains: Key = Cluster_1 Value: remap.config, origin.db, local_cluster.db remap.config.hosts, origin.db.hosts, local_cluster.db.hosts

    I get my hostnames by parsing the above ".hosts" files. "TS" hostnames are contained in remap.config.hosts. "SA" hostnames (zone files) are contained in the two other ".hosts" files.

    TS servers must be reactivated first, then we attempt to reactivate the SA server. Then, we move on to the next cluster, assuming that the first succeeds.

    So, ultimately, what I need to do, is foreach host, in order:

    #####################7 # # # PROCESS SEQUENCES # # # ##################### @taskSequence = ( 'establishSession~~connecting to $clusterHost', 'local_checksum~~calculating local checksums', 'transferFilesToStage~~transferring files to $clusterHost: $cluster +StageDirectory', 'verifyPerms~~verifying file permissions', ); @activateSequence = ( 'activateTS~~$clusterHost: restarting traffic server with new confi +guration', 'activateSADNS~~$clusterHost: restarting SADNS with new configurati +on' ); @rollBackSequence = ( 'revertToExistingFiles~~$clusterHost: restarting using existing con +fig files', 'terminateSession~~$clusterHost: disconnecting from host' ); @offLineSequence = ( 'logBogusFiles~~$clusterHost: encountered corrupted files moved to +$errorLogDir', 'transferFilesToWorking~~$clusterHost: transferring previous files +to $clusterWorkingDirectory', 'takeClusterOffLine~~ $clusterHost is offline.'# < nsctl stop > ); @transferFile = ( 'copyFile~~copying file from remote host' ); #################################### # # MAIN # "foreach cluster" "foreach host, beginning with TA, then SA" @cleanUpTaskSequence = ('terminateSession~~disconnecting from host'); $executionStatus = &executeTaskSequence($clusterHost, @taskSequenc +e) ; $activationStatus = &executeTaskSequence($clusterHost, @activateSe +quence); $cleanUpStatus = &executeTaskSequence ($clusterHost, @cleanUpTaskS +equence);

      Using your existing data structure (a hash of array refs) and assuming that an alphabetic key sort gives the desired activation order (OK for cluster1-9 but breaks at cluster 10) this is quick and dirty:

      my %config_files = ( cluster1 => ['1h1', 'h2', 'h3', 'h4', 'h5', 'h6', '1h7'], cluster2 => ['2h1', 'h2', 'h3', 'h4', 'h5', 'h6', '2h7'], cluster3 => ['3h1', 'h2', 'h3', 'h4', 'h5', 'h6', '3h7'], cluster4 => ['4h1', 'h2', 'h3', 'h4', 'h5', 'h6', '4h7'], cluster5 => ['5h1', 'h2', 'h3', 'h4', 'h*5', 'h6', '5h7'], ); my @order = sort keys %config_files; print "Order will be @order\n"; for my $cluster (@order) { print "Cluster is $cluster\n"; my @machines = @{$config_files{$cluster}}; my $last = pop @machines; for my $box (@machines) { &start_up($box) or &failed($cluster, $box); } &special($last); } print "Done\n"; exit; sub start_up { my $box = shift; return 0 if $box eq 'h*5'; # test failed routine # do stuff, return 1 for success 0 or undef for failure print "\tStarted $box\n"; 1; } sub special { # do box 7 stuff my $box = shift; print "\tSpecial $box\n"; 1; } sub failed { my ($cluster, $box) = @_; # do whatever you want like retry until sucess # then you return to loop and continue my $tries = 5; my $delay = 2; while ($tries) { $tries--; sleep $delay; print "Retrying $box in $cluster\n"; return if &start_up($box); } die "Unable to start $box in $cluster\n"; }




Re: Data Structure Design
by MZSanford (Curate) on Aug 17, 2001 at 13:04 UTC
    in the spirit of KISS, here is my easy solution :
    my %clusters = ( cluster1 => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'h7'], cluster2 => ['h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'h7'], # ... ); foreach my $clust (sort keys %clusters) { foreach my $ind (0..5) { &process_1_6($clusters{$clust}[$ind]); } &process_7($clusters{$clust}[6]); }

    always more than one way to skin an amoebae
    -- MZS
Re: Data Structure Design
by jwest (Friar) on Aug 17, 2001 at 17:50 UTC
    Why not:
    my @machines = qw(c1h1 c1h2 ... c5h7); for my $machine (@machines) { if ($machine =~ /h7$/) { process_special($machine) } else { process_normal($machine) } }

    This way, you know without even the slightest doubt precisely what order every machine will be processed in. And adding a new set of elements is no more (and arguably less) complicated than expanding the hash structure.

    Hope this helps!


    -><- -><- -><- -><- -><-
    All things are Perfect
        To every last Flaw
        And bound in accord
             With Eris's Law
     - HBT; The Book of Advice, 1:7
Here's what I did:
by Tuna (Friar) on Aug 18, 2001 at 22:38 UTC
    First, I can't say how amazing that all of you made time to answer my questions. Without you guys/gals, I probably wouldn't have a job, right now.

    Here's how I solved it:

    - Each config file has an accompanying ".hosts" file, whose contents are the hostnames of each machine in the cluster.
    - Hosts 1-6 are contained on config_file.hosts
    - Host 7 is contained in dnsfile_file.hosts

    I have two hashes that I refer to in the code.

    - %config_files - Keys = cluster directory, Value = an array ref to a list of each config file associated with that cluster.

    - %config_params - Keys = config_file, Value = an array ref to a list containing config_file, config_file.hosts, user|, user|, and a file "weight"

    foreach $componentDir ( keys %config_files ) { $abs_cluster_dir = join("/", "$configdir", "$componentDir"); @cluster_files = @{ $config_files{ $componentDir } }; $href_cluster_files = \@cluster_files; &process_cluster( $abs_cluster_dir, \@cluster_files, \%config_params, \$err_msg ); } sub process_cluster { # subroutine parameters my $cluster_dir = $_[ 0 ]; my $aref_cluster_files = $_[ 1 ]; my $href_config_params = $_[ 2 ]; my $sref_err_msg = $_[ 3 ]; my $clusterHost; my $config_file; my $counter; my $file; my @files_to_process; my $hosts_file; my @target_hosts; $counter = 0; foreach $file ( @{ $aref_cluster_files } ) { if (defined ( $href_config_params->{ $file } )) { $files_to_process[ $href_config_params->{ $file }[ 3 ]] = $f +ile; } } foreach $config_file ( @files_to_process ) { $hosts_file = new IO::File ( "$abs_cluster_dir/$ +s" );print "Hosts file = $hosts_file\n"; unless ( defined ($hosts_file )) { $$sref_err_msg = $!; return 0; } $counter = 0; while ( ! $hosts_file->eof() ) { $target_hosts[ $counter ] = ( $hosts_file->getline( ) ) ; chomp @target_hosts; $counter++; } $hosts_file->close( ); foreach $clusterHost ( @target_hosts ) { ####DO STUFF#### } }
    The trick here, is to assign a numerical weight to each file that I need to process, so that I can populate an array of config files in the order that I need to process them!!!

    Again, thanks to you all, I began to consider alternatives.


Re: Data Structure Design
by Nitsuj (Hermit) on Aug 18, 2001 at 20:48 UTC
    Make cluster a class with 7 elements (could also be classes containing info like hostnames, so forth). Or perhaps an array of the first 6 or a marker for the 7th or something. Then instantiate this class for all of the clusters, perhaps in an array of some sort. You could just walk down this structure executing your actions in a separate script taking the parameters specific to each node.

    Just Another Perl Backpacker

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://105588]
Approved by root
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (3)
As of 2024-05-23 17:35 GMT
Find Nodes?
    Voting Booth?

    No recent polls found