Merging multiple JSON files into one using JSON::XS

[walkingthecow]

Evening monks. Sorry to be asking such a simple question, but it seems that this one has my brain in a knot. I have a directory with a bunch of JSON files that look like this:

[
   {
      "name" : "bob",
      "title" : "janitor",
      "email" : "",
      "iq" : "180",
      "favorite_food" : "wagyu steak"
   },
   {
      "name" : "joe",
      "title" : "software engineer",
      "email" : "",
      "iq" : "80",
      "favorite_food" : "raw hamburger"
   }
]

I am attempting to merge these files together so that they look like this:

{
   "People" : [
      {
         "name" : "bob",
         "title" : "janitor",
         "email" : "",
         "iq" : "180",
         "favorite_food" : "wagyu steak"
      },
      {
         "name" : "joe",
         "title" : "software engineer",
         "email" : "",
         "iq" : "80",
         "favorite_food" : "raw hamburger"
      },
      {
         "name" : "sandy",
         "title" : "dishwasher",
         "email" : "",
         "iq" : "240",
         "favorite_food" : "filet mignon"
      },
      {
         "name" : "george",
         "title" : "software engineer",
         "email" : "",
         "iq" : "14",
         "favorite_food" : "tacos"
      }
   ]   
}

The code I am using to do this is here:

#!/usr/local/perl5/bin/perl
use strict;
use warnings;
use JSON::XS;
use File::Slurp qw( read_file );

open(my $fh, '<', '/tmp/test') or die $!; # contains list of file name
+s

my @fields;
my $uuts = {};
while(<$fh>) {
    chomp;
    next if !-e "/tmp/files/$_.json";
    my $decoded = decode_json( read_file("/tmp/files/$_.json") );
    push @fields, $decoded;
}
$uuts->{People} = [ @fields ];
my $coder = JSON::XS->new->ascii->pretty->allow_nonref;
my $pretty_printed_unencoded = $coder->encode ($uuts);
print $pretty_printed_unencoded;

Now, the problem I am facing is that the output looks like this:

{
   "People" : [
      [
         {
            "name" : "bob",
            "title" : "janitor",
            "email" : "",
            "iq" : "180",
            "favorite_food" : "wagyu steak"
         },
         {
            "name" : "joe",
            "title" : "software engineer",
            "email" : "",
            "iq" : "80",
            "favorite_food" : "raw hamburger"
         }
      ],
      [
         {
            "name" : "sandy",
            "title" : "dishwasher",
            "email" : "",
            "iq" : "240",
            "favorite_food" : "filet mignon"
         },
         {
            "name" : "george",
            "title" : "software engineer",
            "email" : "",
            "iq" : "14",
            "favorite_food" : "tacos"
         }
      ]
   ]
}

I don't want the content of each file to be its own array. I want People to be an array whose contents is the content of each file. Yet, for the life of me I cannot seem to accomplish this.

Any help is greatly appreciated. Thank you Monks!

[kcott]

G'day walkingthecow,

You were almost there. I suspect the unnecessary @fields is possibly what threw you (as that does seem to have complicated the code somewhat). To achieve what you want, change all of this:

my @fields;
my $uuts = {};
while(<$fh>) {
    chomp;
    next if !-e "/tmp/files/$_.json";
    my $decoded = decode_json( read_file("/tmp/files/$_.json") );
    push @fields, $decoded;
}
$uuts->{People} = [ @fields ];

to just this:

my $uuts;
while (<$fh>) {
    chomp;
    next if ! -e "/tmp/files/$_.json";
    push @{$uuts->{People}}, @{decode_json(read_file("/tmp/files/$_.js
+on"))};
}

I'm not really in a position to test that; however, here's a small test that shows the technique involved:

$ perl -Mstrict -Mwarnings -le '
    use Data::Dumper;
    my @jsons = ([{a=>1}, {b=>2}], [{c=>3}, {d=>4}]);
    my $combo;
    for (@jsons) {
        print Dumper $_;
        push @{$combo->{People}} => @$_;
    }
    print Dumper $combo;
'
$VAR1 = [
          {
            'a' => 1
          },
          {
            'b' => 2
          }
        ];

$VAR1 = [
          {
            'c' => 3
          },
          {
            'd' => 4
          }
        ];

$VAR1 = {
          'People' => [
                        {
                          'a' => 1
                        },
                        {
                          'b' => 2
                        },
                        {
                          'c' => 3
                        },
                        {
                          'd' => 4
                        }
                      ]
        };

[walkingthecow]

I tested it and it works perfectly! Thank you. I had solved the issue by not printing pretty (i.e., printing one big line), and doing some replacing. Your solution is much, much better than what I had.

[NetWallah]

This looks like a weird hack, but it works to produce the desired output:

my $pretty_printed_unencoded = $coder->encode ({People =>  @fields });

I have no idea how/why - it makes no sense to me. Would love to be enlightened on what the "right" way is, and why this works.

Update:: After seeing kcott's(++) post below, I see the error of my ways.

His solution is more elegant, but I made mine work by coding:

my $pretty_printed_unencoded = $coder->encode ({People =>  [map {@$_} 
+@fields] });

The previous version at top of this post worked for me for ONE file.

[walkingthecow]

This did not seem to work for me for some reason, I'd either get this error:

Odd number of elements in anonymous hash

or I'd end up with this:

{
   "ARRAY(0x834963c)" : [
      {

That "ARRAY..." is supposed to be People. Here's the code I used.

#!/usr/local/perl5/bin/perl
use strict;
use warnings;
use JSON::XS;
use File::Slurp qw( read_file );
use Data::Dumper;


open(my $fh, '<', '/tmp/test') or die $!; # contains list of file name
+s

my @fields;
while(<$fh>) {
    chomp;
    next if !-e "/tmp/files/$_.json";
    my $decoded = decode_json( read_file("/tmp/files/$_.json") );
    push @fields, $decoded;
}
close $fh;
my $coder = JSON::XS->new->ascii->pretty->allow_nonref;
my $encoded = $coder->encode ({People =>  @fields });
print $encoded;

If I added a \ before fields (i.e., \@fields) then I'd get People as expected, but I'd also get the same output I was dealing with before (each file being its own array). Also, the slash in front of fields made it so I would no longer get the error about having an odd number of elements.

Though it didn't quite work I really appreciate you taking the time to try and figure it out with me :)

[frozenwithjoy]

I've never used this module but what happens if you substitute this:

$uuts->{People} = [ @fields ];

with this (array ref):

$uuts->{People} = \@fields;

or this:

$uuts->{People} = @fields;

[walkingthecow]

$uuts->{People} = \@fields; yields the same thing, and $uuts->{People} = @fields; yields what you see below:

{
   "People" : 24
}

:)

[Anonymous Monk]

JSON is array of stuff  [ ... ] or hash (aka object) of stuff  { ... }

So dereference the arrays of objects you have (aohash)

push @fields, @{ $decoded }; ######<<<<<<<<

references quick reference

Later you make  $hashof{Purple} = \@stuff;