http://www.perlmonks.org?node_id=1031118

newperluser2013 has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks I am very new to perl and was hoping you could help me with how to do the following. I am trying to recursively traverse a rather large complex data structure which is a hash of hashes. While I am traversing this data structure I want to search for keys with a certain name and then once I have all the keys which make up one line of output I then want to print that line with all the corresponding values. I want to then keep traversing through this data structure and then find the next line of keys that match the keylist I am looking for and then print out the whole line of output. I want to keep doing this until I reach the bottom of the data structure. So I have the following hash of hashes with the keys %export_name, %filer_device, %filer_volume, %mount_acl, %mount_group, %mount_opts, and %mount_user. I am able to parse this and grab all these keys and put them into a list called @keylist.

"nfs" => { "%comment" => "Properties for an NFS mount definition", "%export_name" => { "%type" => "path" }, "%filer_device" => { "%type" => "string" }, "%filer_volume" => { "%type" => "path" }, "%mount_acl" => { "%type" => "integer" }, "%mount_group" => { "%type" => "string" }, "%mount_opts" => { "%type" => "string" }, "%mount_user" => { "%type" => "string" } }

Now what I have is this really big complex data structure which I want to recursively parse and look for the existance of all the keys in @keylist and print the values. I need to find all of the keys; %export_name, %filer_device, %filer_volume, %mount_acl, %mount_group, %mount_opts, and %mount_user before I can print a line of output as all these are needed to fully define an nfs mount point. There are hundreds of nfs mount points defined in this data structure and I need to print out a line of output in the following format for each and every one thats found: %export_name:%filer_device:%filer_volume:%mount_acl:%mount_group:%mount_user:%mount_opts This is a small sample of this very large data structure I am trying to parse:

{ "%name": "Global Central Configuration file", "%description": "A representation of nfs mount points for dev mach +ines", "%schema": "schema.conf", "@dev": { "home_nfs": { "%default": { "%mount_opts": "nfsvers=3,timeo=600,retrans=2", "%mount_user": "root", "%mount_group": "root", "%mount_acl": "0755" }, "%comment": "----Home NFS Directories----", "home-lnk-mpt": { "%export_name": "/links", "%filer_device": { "@west": "nydevnfs_links", "@ridge": "rnap7750-s" }, "%filer_volume": { "@west": "/vol/links", "@ridge": "/vol/links_c" } }, "home7_mpt": { "%export_name": "/home7", "%filer_device": { "@west": "nydevnfs_home7", "@ridge": "rnap7751-s" }, "%filer_volume": { "@west": "/vol/home7", "@ridge": "/vol/home7_c" } }, "home8_mpt": { "%export_name": "/home8", "%filer_device": { "@west": "nydevnfs_home2", "@ridge": "rnap2114-s" }, "%filer_volume": { "@west": "/vol/home2", "@ridge": "/vol/home2_c" } },

The following is my attempt of writing a recursive sub routine to parse this large data structure but I just cant work out how to get it to do what I want

sub recurse_hash { my($hash,@findkeys) = @_; foreach (sort keys %{$hash}) { my $value = $hash->{$_}; if (ref($value) eq 'HASH') { recurse_hash($value,@findkeys); } for my $key (@findkeys) { if ($key eq $_) { print "$_ = $value\n"; } } } }

I am calling this subroutine like this but I can't get it to accept passing @keylist to it.

recurse_hash(\%$decoded_json_obj,"@keylist");

If I hardcode the key names like so it doesn't error and prints all all the corresponding values but they are all on different lines and don't match up.

recurse_hash(\%$decoded_json_obj,"%export_name","%filer_device","%file +r_volume","%mount_acl","%mount_group","%mount_opts");

I am not allowed to hardcode the key names as args to this subroutine. I need to pass in a list of keys defined in @keylist

Using the small sample of this large data structure I have shown I am trying to print lines in the following format.

/links:nydevnfs_links:/vol/links:0755:root:root:nfsvers=3,timeo=600,re +trans=2 /links:rnap7750-s:/vol/links_c:0755:root:root:nfsvers=3,timeo=600,retr +ans=2 /home7:nydevnfs_home7:/vol/home7:0755:root:root:nfsvers=3,timeo=600,re +trans=2 /home7:rnap7751-s:/vol/home7_c:0755:root:root:nfsvers=3,timeo=600,retr +ans=2 /home8:nydevnfs_home2:/vol/home2:0755:root:root:nfsvers=3,timeo=600,re +trans=2 /home8:rnap2114-s:/vol/home2_c:0755:root:root:nfsvers=3,timeo=600,retr +ans=2

I mustn't hardcode any references to any mount point names, filer device/volumes names etc. I purely have to parse this data structure and at each level look to see if any of the keys in @keylist exist if they do then store the value until I have values for each key in that list and can then print a line of output, and then continue doing the same throughout parsing the rest of the data structure printing a line of output for each mount point.

Please can you help me with the above? I have spent so much time trying to do this but I just cant seem to get it to work

Replies are listed 'Best First'.
Re: Traversing a complex data structure searching for certain keys and printing their values
by LanX (Saint) on Apr 28, 2013 at 23:38 UTC
    Hi

    > I am calling this subroutine like this but I can't get it to accept passing @keylist to it.

    changing

    recurse_hash(\%$decoded_json_obj,"@keylist");

    to

    recurse_hash(\%$decoded_json_obj,@keylist);

    should do.

    And please, Perl has variable interpolation, so never put variables into doublequotes if you don't them to be expanded.Use single quotes!

    HTH =)

    Cheers Rolf

    ( addicted to the Perl Programming Language)

    UPDATE

    > If I hardcode the key names like so it doesn't error and prints all all the corresponding values but they are all on different lines and don't match up.

    I don't understand, could you show us the problem with your input?

Re: Traversing a complex data structure searching for certain keys and printing their values
by choroba (Cardinal) on Apr 29, 2013 at 00:38 UTC
    To get the output you requested, I had to make several assumptions:
    1. The %filer_device and %filer_volume type of subhashes are of depth 1 (i.e. @west and @ridge are not structures).
    2. %defaults are special and their name must be hardcoded.

    #!/usr/bin/perl use warnings; use strict; use v5.14; # Hashrefs for keys and values. use JSON; use Data::Dumper; sub recurse_hash { my ($hash, $found, @findkeys) = @_; for my $key (sort { # Defaults go first. local $a = q() if '%default' eq $a; local $b = q() if '%default' eq $b; $a cmp $b; } keys %{$hash}) { my $value = $hash->{$key}; if ('HASH' eq ref $value) { $found = {%$found, %{ recurse_hash($value, $found, @findke +ys) }}; } for my $find (@findkeys) { if ($key eq $find) { if ('HASH' eq ref $value) { $found->{$key} = [ map $value->{$_}, sort keys $va +lue ]; } else { $found->{$key} = $value; } } } } if (@findkeys == keys $found) { my $count; $count = @{ (grep 'ARRAY' eq ref $_, values $found)[0] // [] } +; $count ||= 1; for my $i (0 .. $count - 1) { print join(':', map { 'ARRAY' eq ref $_ ? $_->[$i] : $_ } @{$found}{@findkeys}), "\n"; } print "\n"; $found = {}; } return $found; } my @keylist = qw(%export_name %filer_device %filer_volume %mount_acl %mount_group %mount_user %mount_opts); my $string = do { local $/; <DATA> }; my $struct = decode_json($string); recurse_hash($struct, {}, @keylist); __DATA__ { "%name": "Global Central Configuration file", "%description": "A representation of nfs mount points for dev mach +ines", "%schema": "schema.conf", "@dev": { "home_nfs": { "%default": { "%mount_opts": "nfsvers=3,timeo=600,retrans=2", "%mount_user": "root", "%mount_group": "root", "%mount_acl": "0755" }, "%comment": "----Home NFS Directories----", "home-lnk-mpt": { "%export_name": "/links", "%filer_device": { "@west": "nydevnfs_links", "@ridge": "rnap7750-s" }, "%filer_volume": { "@west": "/vol/links", "@ridge": "/vol/links_c" } }, "home7_mpt": { "%export_name": "/home7", "%filer_device": { "@west": "nydevnfs_home7", "@ridge": "rnap7751-s" }, "%filer_volume": { "@west": "/vol/home7", "@ridge": "/vol/home7_c" } }, "home8_mpt": { "%export_name": "/home8", "%filer_device": { "@west": "nydevnfs_home2", "@ridge": "rnap2114-s" }, "%filer_volume": { "@west": "/vol/home2", "@ridge": "/vol/home2_c" } } } } }
    لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

      Thanks for your help. This looks really good. Unfortunately I only have perl 5.12 so this fails with these errors

      Type of arg 1 to keys must be hash or array (not private variable) at +./test.pl line 25, near "$value ]" Type of arg 1 to keys must be hash or array (not private variable) at +./test.pl line 32, near "$found) " Type of arg 1 to values must be hash or array (not private variable) a +t ./test.pl line 34, near "$found)"

      Also as regards %filer_device and %filer_volume sometimes in this data structure these are not hashes, sometimes both are, and sometimes one is and the other isn't I am really not too sure how to go about dealing with that :-( Also regarding the %defaults section I am not allowed to hardcode that. In the small sample of this data structure I have shown this is tied in to the tag "@dev". So any mountpoint defined under here gets those permissions. In the rest of the data structure I have other tags like @build and @software and under each one of these they have their own set of multiple mount point definitions and their own set of default permissions that need to be set. I ultimately need to print the strings out in the following format without hardcoding any key names in the subroutine I use to parse the data struture. All I can pass to the subroutine is the data structure and the @keylist:

      @dev:/links:nydevnfs_links:/vol/links:0755:root:root:nfsvers=3,timeo=6 +00,re +trans=2 @dev:/links:rnap7750-s:/vol/links_c:0755:root:root:nfsvers=3,timeo=600 +,retr +ans=2 @dev:/home7:nydevnfs_home7:/vol/home7:0755:root:root:nfsvers=3,timeo=6 +00,re +trans=2 @dev:/home7:rnap7751-s:/vol/home7_c:0755:root:root:nfsvers=3,timeo=600 +,retr +ans=2 @dev:/home8:nydevnfs_home2:/vol/home2:0755:root:root:nfsvers=3,timeo=6 +00,re +trans=2 @dev:/home8:rnap2114-s:/vol/home2_c:0755:root:root:nfsvers=3,timeo=600 +,retr +ans=2
        To fix the first problem, just prepend % to the indicated reference variables.

        To the rest, I cannot say more than "Good luck".

        لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
Re: Traversing a complex data structure searching for certain keys and printing their values
by LanX (Saint) on Apr 29, 2013 at 00:12 UTC
    OK. I translated your JSON manually to a Perl hash and added a missing else-block to your routine.

    Now I got the following output:

    %mount_acl = 0755 %mount_group = root %mount_opts = nfsvers=3,timeo=600,retrans=2 %export_name = /links %export_name = /home7 %export_name = /home8

    apparently most elements you wanted, just in different lines because you finished your prints with "\n".

    IMHO you still need to store the defaults and print them if there are no other details for a path.

    Here my code. this should give you a start to continue:

    Cheers Rolf

    ( addicted to the Perl Programming Language)

      Thanks for your help Rolf I really appreciate it. I am still stuck on printing out the lines like this:

      /links:nydevnfs_links:/vol/links:0755:root:root:nfsvers=3,timeo=600,re +trans=2 /links:rnap7750-s:/vol/links_c:0755:root:root:nfsvers=3,timeo=600,retr +ans=2 /home7:nydevnfs_home7:/vol/home7:0755:root:root:nfsvers=3,timeo=600,re +trans=2 /home7:rnap7751-s:/vol/home7_c:0755:root:root:nfsvers=3,timeo=600,retr +ans=2 /home8:nydevnfs_home2:/vol/home2:0755:root:root:nfsvers=3,timeo=600,re +trans=2 /home8:rnap2114-s:/vol/home2_c:0755:root:root:nfsvers=3,timeo=600,retr +ans=2

      please can you help me?

        > please can you help me?

        Sorry, we are giving assistance to learn Perl and not in doing their job without understanding.

        If you really wrote the code you provided, you should be also able to (or at least learn to) modify it!

        Cheers Rolf

        ( addicted to the Perl Programming Language)