Obtain the full path of a json key

[eny]

Hello all,

I receive such json file as input. As you can see it is an array of objects where each object is an array of sub-objects. Brief it is a collection of nested objects and arrays. I don't know in advance the structure, the name of the keys and the levels of nesting. Or even it can also be the other way around, a collection of objects with arrays nested on it. The only thing I should know is the object identifier "key name" found at level 2 (in my example "Obj11Id", "Obj12Id", etc). My goal is for each pair of key name and value, to have the "full path" of the key name and the key value. By "full path" I understand the name of the above keys in the structure.

Example: for the pair "Obj11AttributesObj1Key1": "1", the "full path" would be: "Obj1.Obj11Attributes.Obj11AttributesObj1.Obj11AttributesObj1Key1" (dot "." being the separator of the above keys).

I'm new in Perl, I've tried many ways to obtain this info but failed. I'm using the library JSON::PP and function decode_json.

Thank you

The input json file:

[
    {
        "Obj1": [
            {
                "Obj11Id": "Id11",
                "Obj11Version": "v1",
                "Obj11Attributes": {
                    "Obj11AttributesObj1": {
                        "Obj11AttributesObj1Key1": "1",
                        "Obj11AttributesObj1Key2": "3"
                    },
                    "Obj11AttributesObj2": {
                        "Obj11AttributesObj2Key1": "9"
                    }
                }
            },
            {
                "Obj12Id": "Id12",
                "Obj12Version": "v1",
                "Obj12Attributes": {
                    "Obj12AttributesObj1": {
                        "Obj12AttributesObj1Key1": "2",
                        "Obj12AttributesObj1Key2": "4"
                    },
                    "Obj12AttributesObj2": {
                        "Obj12AttributesObj2Key1": "8"
                    },
                    "Obj12AttributesArray1": [
                        {
                            "Obj12AttributesArray1Obj1Key1": 1
                        }
                    ]
                }
            }
        ]
    },
    {
        "Obj2": [
            {
                "Obj21Id": "Id21",
                "Obj21Version": "v1",
                "Obj21Attributes": {
                    "Obj21AttributesObj1": {
                        "Obj21AttributesObj1Key1": "3",
                        "Obj21AttributesObj1Key2": "5"
                    },
                    "Obj21AttributesObj2": {
                        "Obj21AttributesObj2Key1": "7"
                    }
                }
            },
            {
                "Obj22Id": "Id22",
                "Obj22Version": "v1",
                "Obj22Attributes": {
                    "Obj22AttributesObj1": {
                        "Obj22AttributesObj1Key1": "4",
                        "Obj22AttributesObj1Key2": "6"
                    },
                    "Obj22AttributesObj2": {
                        "Obj22AttributesObj2Key1": "0"
                    },
                    "Obj22AttributesArray1": [
                        {
                            "Obj22AttributesArray1Obj1Key1": 1
                        }
                    ]
                }
            }
        ]
    }
]

[haukex]

I've tried many ways to obtain this info but failed. I'm using the library JSON::PP and function decode_json.

It would be best if you showed us what you tried and how it failed (SSCCE), because otherwise we can't tell if you're just asking us do your homework for you or (ab)using us as a code writing service. But since this was a nice little excercise, here's a freebie...

use warnings;
use strict;
use JSON::PP qw/decode_json/;

# search a data structure recursively for a specific hash key
# and return its "path" as a string if it is found
sub get_path_by_key {
    my ($data, $search_key) = @_;
    # an array of the data items left to search, each being a hash
    # with keys "d" for the data item and "p" for the item's path:
    my @to_search = ( { d=>$data, p=>[] } );
    while (@to_search) {
        # get the item we're currently inspecting:
        my $cur = shift @to_search;
        if ( ref $$cur{d} eq 'ARRAY' ) {
            # if the data item is an array, go through its indices
            for my $i ( 0 .. $#{$$cur{d}} ) {
                # skip any array values that aren't nested data:
                next unless ref $cur->{d}[$i];
                # put each of the array's values on the search list
                push @to_search, { d=>$cur->{d}[$i],
                    # this means the output won't include indices:
                    p=>$$cur{p} };
                    # this would be one way to include indices:
                    #p=>[ @{$$cur{p}}, "[$i]" ] };
            }
        }
        elsif ( ref $$cur{d} eq 'HASH' ) {
            # if the current data item is a hash,
            # first check if it contains the search key:
            if ( exists $cur->{d}{$search_key} ) {
                # this builds the return value string:
                return join '.', @{$$cur{p}}, $search_key;
            }
            # otherwise, go through the hash's keys:
            for my $k ( sort keys %{$$cur{d}} ) {
                # skip any hash values that aren't nested data:
                next unless ref $cur->{d}{$k};
                # put each of the hash's values on the search list
                push @to_search, { d=>$cur->{d}{$k},
                    p=>[ @{$$cur{p}}, $k ] };
            }
        }
    }
    return  # return nothing in case the key isn't found
}

# this is just an idiom to read the whole __DATA__ section:
my $json = do { local $/; <DATA> };
# decode the JSON string into a Perl data structure:
my $data = decode_json($json);
# use our function:
my $path = get_path_by_key($data, 'Obj11AttributesObj1Key1');
print "path=$path\n";

__DATA__
[ {
    "Obj1": [ {
        "Obj11Id": "Id11",
        "Obj11Version": "v1",
        "Obj11Attributes": {
            "Obj11AttributesObj1": {
                "Obj11AttributesObj1Key1": "1",
                "Obj11AttributesObj1Key2": "3"
            },
            "Obj11AttributesObj2": {
                "Obj11AttributesObj2Key1": "9"
            }
        }
    }, {
        "Obj12Id": "Id12",
        "Obj12Version": "v1",
        "Obj12Attributes": {
            "Obj12AttributesObj1": {
                "Obj12AttributesObj1Key1": "2",
                "Obj12AttributesObj1Key2": "4"
            },
            "Obj12AttributesObj2": {
                "Obj12AttributesObj2Key1": "8"
            },
            "Obj12AttributesArray1": [ {
                "Obj12AttributesArray1Obj1Key1": 1
            } ]
        }
    } ]
}, {
    "Obj2": [ {
        "Obj21Id": "Id21",
        "Obj21Version": "v1",
        "Obj21Attributes": {
            "Obj21AttributesObj1": {
                "Obj21AttributesObj1Key1": "3",
                "Obj21AttributesObj1Key2": "5"
            },
            "Obj21AttributesObj2": {
                "Obj21AttributesObj2Key1": "7"
            }
        }
    }, {
        "Obj22Id": "Id22",
        "Obj22Version": "v1",
        "Obj22Attributes": {
            "Obj22AttributesObj1": {
                "Obj22AttributesObj1Key1": "4",
                "Obj22AttributesObj1Key2": "6"
            },
            "Obj22AttributesObj2": {
                "Obj22AttributesObj2Key1": "0"
            },
            "Obj22AttributesArray1": [ {
                "Obj22AttributesArray1Obj1Key1": 1
            } ]
        }
    } ]
} ]

If you're new to Perl you'll probably need perlreftut to understand much of the code.

[Arunbear]

There is a tool called gron which makes JSON greppable. So with your example,

% gron 11165436.json | head               
json = [];
json[0] = {};
json[0].Obj1 = [];
json[0].Obj1[0] = {};
json[0].Obj1[0].Obj11Attributes = {};
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj1 = {};
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj1.Obj11AttributesObj
+1Key1 = "1";
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj1.Obj11AttributesObj
+1Key2 = "3";
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj2 = {};
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj2.Obj11AttributesObj
+2Key1 = "9";

Then, if you just wanted a list of keys, you could strip away the values,

% gron 11165436.json | perl -pE 's/ =.+//' | head
json
json[0]
json[0].Obj1
json[0].Obj1[0]
json[0].Obj1[0].Obj11Attributes
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj1
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj1.Obj11AttributesObj
+1Key1
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj1.Obj11AttributesObj
+1Key2
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj2
json[0].Obj1[0].Obj11Attributes.Obj11AttributesObj2.Obj11AttributesObj
+2Key1

[Corion]

The easiest way is to enumerate all paths and then only return those that match your criteria:

#!perl
use 5.020;
use feature 'signatures', 'postderef';
no warnings 'experimental::signatures';

sub all_json_paths( $obj, $prefix = '' ) {
    #say "$prefix|$obj";
    if( !ref $obj ) {
        return $prefix

    } elsif( ref $obj eq 'ARRAY' ) {
        my @res;
        my $idx = 1; # 1-based or 0-based indexing?
        for my $el ($obj->@*) {
            push @res, all_json_paths( $el, $prefix . "[$idx]" );
            $idx++;
        }
        return @res

    } elsif( ref $obj eq 'HASH' ) {
        my @res;
        for my $k (sort keys $obj->%*) {
            push @res, all_json_paths( $obj->{$k}, $prefix . ".$k" );
        }
        return @res
    }

}

sub path_to_key( $key, $obj ) {
    my @p = all_json_paths( $obj );
    return grep /\Q$key\E$/, all_json_paths( $obj )
}

my $struct = [
    {
        "Obj1"=> [
            {
                "Obj11Id"=> "Id11",
                "Obj11Version"=> "v1",
                "Obj11Attributes"=> {
                    "Obj11AttributesObj1"=> {
                        "Obj11AttributesObj1Key1"=> "1",
                        "Obj11AttributesObj1Key2"=> "3"
                    },
                    "Obj11AttributesObj2"=> {
                        "Obj11AttributesObj2Key1"=> "9"
                    }
                }
            },
            {
                "Obj12Id"=> "Id12",
                "Obj12Version"=> "v1",
                "Obj12Attributes"=> {
                    "Obj12AttributesObj1"=> {
                        "Obj12AttributesObj1Key1"=> "2",
                        "Obj12AttributesObj1Key2"=> "4"
                    },
                    "Obj12AttributesObj2"=> {
                        "Obj12AttributesObj2Key1"=> "8"
                    },
                    "Obj12AttributesArray1"=> [
                        {
                            "Obj12AttributesArray1Obj1Key1"=> 1
                        }
                    ]
                }
            }
        ]
    },
    {
        "Obj2"=> [
            {
                "Obj21Id"=> "Id21",
                "Obj21Version"=> "v1",
                "Obj21Attributes"=> {
                    "Obj21AttributesObj1"=> {
                        "Obj21AttributesObj1Key1"=> "3",
                        "Obj21AttributesObj1Key2"=> "5"
                    },
                    "Obj21AttributesObj2"=> {
                        "Obj21AttributesObj2Key1"=> "7"
                    }
                }
            },
            {
                "Obj22Id"=> "Id22",
                "Obj22Version"=> "v1",
                "Obj22Attributes"=> {
                    "Obj22AttributesObj1"=> {
                        "Obj22AttributesObj1Key1"=> "4",
                        "Obj22AttributesObj1Key2"=> "6"
                    },
                    "Obj22AttributesObj2"=> {
                        "Obj22AttributesObj2Key1"=> "0"
                    },
                    "Obj22AttributesArray1"=> [
                        {
                            "Obj22AttributesArray1Obj1Key1"=> 1
                        }
                    ]
                }
            }
        ]
    }
];

say path_to_key('Obj21Id', $struct);

I've omitted reading in the JSON file into a Perl structure. Use JSON::Tiny or Mojo::JSON or CPANEL::JSON::XS for that.

[tybalt89]

Short and sweet :)

#!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11165436
use warnings;
use Path::Tiny;
use JSON::PP;

sub names
  {
  my ($data) = @_;
  if( 'ARRAY' eq ref $data )
    {
    map names($_), @$data;
    }
  elsif( 'HASH' eq ref $data )
    {
    map { my $id = $_; map "$id.$_", names($data->{$_}) } keys %$data;
    }
  else { "\n" }
  }

print s/\.$//r for names decode_json path('d.11165436')->slurp; # FIXM
+E filename

Outputs:

Obj1.Obj11Id
Obj1.Obj11Attributes.Obj11AttributesObj2.Obj11AttributesObj2Key1
Obj1.Obj11Attributes.Obj11AttributesObj1.Obj11AttributesObj1Key1
Obj1.Obj11Attributes.Obj11AttributesObj1.Obj11AttributesObj1Key2
Obj1.Obj11Version
Obj1.Obj12Attributes.Obj12AttributesArray1.Obj12AttributesArray1Obj1Ke
+y1
Obj1.Obj12Attributes.Obj12AttributesObj2.Obj12AttributesObj2Key1
Obj1.Obj12Attributes.Obj12AttributesObj1.Obj12AttributesObj1Key2
Obj1.Obj12Attributes.Obj12AttributesObj1.Obj12AttributesObj1Key1
Obj1.Obj12Version
Obj1.Obj12Id
Obj2.Obj21Attributes.Obj21AttributesObj2.Obj21AttributesObj2Key1
Obj2.Obj21Attributes.Obj21AttributesObj1.Obj21AttributesObj1Key2
Obj2.Obj21Attributes.Obj21AttributesObj1.Obj21AttributesObj1Key1
Obj2.Obj21Version
Obj2.Obj21Id
Obj2.Obj22Version
Obj2.Obj22Attributes.Obj22AttributesArray1.Obj22AttributesArray1Obj1Ke
+y1
Obj2.Obj22Attributes.Obj22AttributesObj2.Obj22AttributesObj2Key1
Obj2.Obj22Attributes.Obj22AttributesObj1.Obj22AttributesObj1Key2
Obj2.Obj22Attributes.Obj22AttributesObj1.Obj22AttributesObj1Key1
Obj2.Obj22Id

[Anonymous Monk]

Thank you all for the contributions, I'm installing the missing package Path::Tiny to try all them :)

[tybalt89]

You don't really need Path::Tiny. If I didn't have it, I would have done something like this using just standard perl:

#!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11165436
use warnings;
use JSON::PP;

sub names
  {
  my ($data) = @_;
  if( 'ARRAY' eq ref $data )
    {
    map names($_), @$data;
    }
  elsif( 'HASH' eq ref $data )
    {
    map { my $id = $_; map "$id.$_", names($data->{$_}) } keys %$data;
    }
  else { "\n" }
  }

@ARGV or @ARGV = 'd.11165436.json';                   # FIXME filename
print map s/\.$//r, names decode_json join '', <>;

This would allow you to put the JSON file name on the command line, or if you didn't put any arguments on the command line, would run the program against a test file.