Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Merging multiple JSON files into one using JSON::XS

by walkingthecow (Friar)
on Aug 12, 2013 at 02:33 UTC ( #1049047=perlquestion: print w/ replies, xml ) Need Help??
walkingthecow has asked for the wisdom of the Perl Monks concerning the following question:

Evening monks. Sorry to be asking such a simple question, but it seems that this one has my brain in a knot. I have a directory with a bunch of JSON files that look like this:
[ { "name" : "bob", "title" : "janitor", "email" : "", "iq" : "180", "favorite_food" : "wagyu steak" }, { "name" : "joe", "title" : "software engineer", "email" : "", "iq" : "80", "favorite_food" : "raw hamburger" } ]
I am attempting to merge these files together so that they look like this:
{ "People" : [ { "name" : "bob", "title" : "janitor", "email" : "", "iq" : "180", "favorite_food" : "wagyu steak" }, { "name" : "joe", "title" : "software engineer", "email" : "", "iq" : "80", "favorite_food" : "raw hamburger" }, { "name" : "sandy", "title" : "dishwasher", "email" : "", "iq" : "240", "favorite_food" : "filet mignon" }, { "name" : "george", "title" : "software engineer", "email" : "", "iq" : "14", "favorite_food" : "tacos" } ] }
The code I am using to do this is here:
#!/usr/local/perl5/bin/perl use strict; use warnings; use JSON::XS; use File::Slurp qw( read_file ); open(my $fh, '<', '/tmp/test') or die $!; # contains list of file name +s my @fields; my $uuts = {}; while(<$fh>) { chomp; next if !-e "/tmp/files/$_.json"; my $decoded = decode_json( read_file("/tmp/files/$_.json") ); push @fields, $decoded; } $uuts->{People} = [ @fields ]; my $coder = JSON::XS->new->ascii->pretty->allow_nonref; my $pretty_printed_unencoded = $coder->encode ($uuts); print $pretty_printed_unencoded;
Now, the problem I am facing is that the output looks like this:
{ "People" : [ [ { "name" : "bob", "title" : "janitor", "email" : "", "iq" : "180", "favorite_food" : "wagyu steak" }, { "name" : "joe", "title" : "software engineer", "email" : "", "iq" : "80", "favorite_food" : "raw hamburger" } ], [ { "name" : "sandy", "title" : "dishwasher", "email" : "", "iq" : "240", "favorite_food" : "filet mignon" }, { "name" : "george", "title" : "software engineer", "email" : "", "iq" : "14", "favorite_food" : "tacos" } ] ] }

I don't want the content of each file to be its own array. I want People to be an array whose contents is the content of each file. Yet, for the life of me I cannot seem to accomplish this.

Any help is greatly appreciated. Thank you Monks!

Comment on Merging multiple JSON files into one using JSON::XS
Select or Download Code
Re: Merging multiple JSON files into one using JSON::XS
by frozenwithjoy (Curate) on Aug 12, 2013 at 02:47 UTC
    I've never used this module but what happens if you substitute this:
    $uuts->{People} = [ @fields ];

    with this (array ref):

    $uuts->{People} = \@fields;

    or this:

    $uuts->{People} = @fields;
      $uuts->{People} = \@fields; yields the same thing, and $uuts->{People} = @fields; yields what you see below:
      { "People" : 24 }
      :)
Re: Merging multiple JSON files into one using JSON::XS
by NetWallah (Abbot) on Aug 12, 2013 at 03:54 UTC
    This looks like a weird hack, but it works to produce the desired output:
    my $pretty_printed_unencoded = $coder->encode ({People => @fields }); +
    I have no idea how/why - it makes no sense to me. Would love to be enlightened on what the "right" way is, and why this works.

    Update:: After seeing kcott's(++) post below, I see the error of my ways.

    His solution is more elegant, but I made mine work by coding:

    my $pretty_printed_unencoded = $coder->encode ({People => [map {@$_} +@fields] });
    The previous version at top of this post worked for me for ONE file.

                 My goal ... to kill off the slow brain cells that are holding me back from synergizing my knowledge of vertically integrated mobile platforms in local cloud-based content management system datafication.

      This did not seem to work for me for some reason, I'd either get this error:
      Odd number of elements in anonymous hash
      or I'd end up with this:
      { "ARRAY(0x834963c)" : [ {
      That "ARRAY..." is supposed to be People. Here's the code I used.
      #!/usr/local/perl5/bin/perl use strict; use warnings; use JSON::XS; use File::Slurp qw( read_file ); use Data::Dumper; open(my $fh, '<', '/tmp/test') or die $!; # contains list of file name +s my @fields; while(<$fh>) { chomp; next if !-e "/tmp/files/$_.json"; my $decoded = decode_json( read_file("/tmp/files/$_.json") ); push @fields, $decoded; } close $fh; my $coder = JSON::XS->new->ascii->pretty->allow_nonref; my $encoded = $coder->encode ({People => @fields }); print $encoded;

      If I added a \ before fields (i.e., \@fields) then I'd get People as expected, but I'd also get the same output I was dealing with before (each file being its own array). Also, the slash in front of fields made it so I would no longer get the error about having an odd number of elements.

      Though it didn't quite work I really appreciate you taking the time to try and figure it out with me :)
Re: Merging multiple JSON files into one using JSON::XS (array or hash or object)
by Anonymous Monk on Aug 12, 2013 at 07:02 UTC
    JSON is array of stuff  [ ... ] or hash (aka object) of stuff  { ... }

    So dereference the arrays of objects you have (aohash)

    push @fields, @{ $decoded }; ######<<<<<<<<

    references quick reference

    Later you make  $hashof{Purple} = \@stuff;

Re: Merging multiple JSON files into one using JSON::XS
by kcott (Abbot) on Aug 12, 2013 at 08:59 UTC

    G'day walkingthecow,

    You were almost there. I suspect the unnecessary @fields is possibly what threw you (as that does seem to have complicated the code somewhat). To achieve what you want, change all of this:

    my @fields; my $uuts = {}; while(<$fh>) { chomp; next if !-e "/tmp/files/$_.json"; my $decoded = decode_json( read_file("/tmp/files/$_.json") ); push @fields, $decoded; } $uuts->{People} = [ @fields ];

    to just this:

    my $uuts; while (<$fh>) { chomp; next if ! -e "/tmp/files/$_.json"; push @{$uuts->{People}}, @{decode_json(read_file("/tmp/files/$_.js +on"))}; }

    I'm not really in a position to test that; however, here's a small test that shows the technique involved:

    $ perl -Mstrict -Mwarnings -le ' use Data::Dumper; my @jsons = ([{a=>1}, {b=>2}], [{c=>3}, {d=>4}]); my $combo; for (@jsons) { print Dumper $_; push @{$combo->{People}} => @$_; } print Dumper $combo; ' $VAR1 = [ { 'a' => 1 }, { 'b' => 2 } ]; $VAR1 = [ { 'c' => 3 }, { 'd' => 4 } ]; $VAR1 = { 'People' => [ { 'a' => 1 }, { 'b' => 2 }, { 'c' => 3 }, { 'd' => 4 } ] };

    -- Ken

      I tested it and it works perfectly! Thank you. I had solved the issue by not printing pretty (i.e., printing one big line), and doing some replacing. Your solution is much, much better than what I had.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1049047]
Approved by frozenwithjoy
Front-paged by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (11)
As of 2014-10-20 23:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (93 votes), past polls