Re: Config file to tree format with id, parent, key, value

[choroba]

If you can't find a module to parse your config format, write your own parser. Marpa::R2 can help you in the task:

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

use Marpa::R2;

my $input = << '__INPUT__';
# This is a comment
feed_realtime_processor_pool = ( 11, 12 ) ;
dropout_detection_time_start = "17:00";
# Sometimes the config can have sub-structures
named_clients = (
{
  name = "thread1";
  user_threads = (
     { name = "realtime1"; cpu = 11; } # more comments
     { name = "realtime2"; cpu = 12; } # more comments
    );
}
);
__INPUT__

my $dsl = << '__DSL__';

lexeme default = latm => 1
:default ::= action => ::first

Config    ::= Elements
Elements  ::= Element+                              action => grep_def
+ined
Element   ::= (Comment)                             action => empty
            | Name (s eq s) Value                   action => [values]
            | Name (s eq s) Value (semicolon s)     action => [values]
Comment   ::= (hash nonnl nl)                       action => empty
Name      ::= alpha
Value     ::= List
            | String
            | Num
            | Struct
List      ::= (lpar) Nums (rpar s semicolon s)
Nums      ::= Num+  separator => comma              action => listify
Num       ::= (s) digits (s)
            | (s) digits
            | digits (s)
            | digits
String    ::= (qq) nqq (qq semicolon s)             action => quote
Struct    ::= (lpar s) InStructs (rpar semicolon s)
InStructs ::= InStruct+                             action => grep_def
+ined
InStruct  ::= (lcurl s) Elements (rcurl s)
            | (Comment s)                           action => empty
            | Element

s         ~ [\s]*
eq        ~ '='
hash      ~ '#'
nonnl     ~ [^\n]*
nl        ~ [\n]
alpha     ~ [a-z_]+
lpar      ~ '('
rpar      ~ ')'
lcurl     ~ '{'
rcurl     ~ '}'
semicolon ~ ';'
comma     ~ ','
digits    ~ [\d]+
qq        ~ '"'
nqq       ~ [^"]+

__DSL__

sub listify      { shift; [ @_ ] }
sub quote        { qq("$_[1]") }
sub empty        {}
sub grep_defined { shift; [ grep defined, @_ ] }


my $id = 1;
sub show {
    my ($parent, $name, $elems) = @_;
    if (ref $elems->[0]) {
        show($parent, $name, $_) for @$elems;
    } elsif (ref $elems->[1]) {
        if (ref $elems->[1][0]) {
            say join ', ', $id, $parent, $elems->[0], 'null';
            show($id++, $elems->[0], $elems->[1]);
        } else {
            for my $e (@{ $elems->[1] }) {
                say join ', ', $id++, $parent, $elems->[0], $e;
            }
        }
    } else {
        say join ', ', $id++, $parent, @$elems;
    }
}


my $grammar = 'Marpa::R2::Scanless::G'->new({ source => \$dsl });
show('null', q(), ${ $grammar->parse(\$input, 'main') });

Output:

1, null, feed_realtime_processor_pool, 11
2, null, feed_realtime_processor_pool, 12
3, null, dropout_detection_time_start, "17:00"
4, null, named_clients, null
5, 4, name, "thread1"
6, 4, user_threads, null
7, 6, name, "realtime1"
8, 6, cpu, 11
9, 6, name, "realtime2"
10, 6, cpu, 12

Note that the id of the last line is 10, not 9 as in your sample. Adding leading zeroes and aligning the columns left as an exercise for the OP.

[arthur_accioly]

Wow, it's working!!!! Thank you a lot. Just an addendum, to install this module I had to install a lot of extra dependencies that were not declared:

sudo cpan IPC::Cmd
sudo cpan Module::Build
sudo cpan Time::Piece
sudo cpan Marpa::R2

Thanks again!

[hippo]

IPC::Cmd and Time::Piece have been in core since 5.9.5 which is very nearly a decade ago. Many dists don't mention core modules as dependencies, perhaps because the authors (erroneously) assume that everyone has the core installed by default. Module::Build used to be in core and it's possible that the author of Marpa::R2 did not notice when it was dropped from core.

Which version of Perl are you running and on which operating system?

[arthur_accioly]

Hi, this is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi. I'm using centos7 but can use 6 as well. It's not a problem to install those extra modules. Thanks.

[ron.savage]

See also DBIx::Tree for ideas.

[arthur_accioly]

Hi Choroba, can I ask you one more thing? Those config files can come in many different formats. I have two questions: 1. The order where the entries are coming is important? For example, if "dropout_detection_time_start" comes before "feed_realtime_processor_pool", is this going to break my parser? 2. If we have situations where some of the rules are not present in the config file, is this going to break the parser? For example, if I have "another_realtime_processor_pool", is this going to be discarded? Obviously I'll create a grammar for those extra parameters, but there are situations where the configs can have different entries depending of what the user wants to put in the config. Thanks.

[choroba]

1. The order shouldn't be important in this case. You can see that both "dropout_detection_time_start" and "feed_realtime_processor_pool" are instances of Element, that combine to Elements in any order ( Element+ ).

2. Have you noticed that the parser is universal? It doesn't look for "realtime_processor" anywhere, it searches for Name , so you can define anything you like (however, at the same time, it means the parser doesn't catch typos - if you have the exhaustive list of possible Names available, replace the general alpha by something more specific).