(RFC) XML::Rules - yet another XML parser

[Jenda]

This is a continuation to my previous (RFC) XML::TransformRules node. The first version of the module, for now without support for namespaces and processing instructions (and comments - you might want to process even those in some cases) may be found at my site.

A few weeks ago someone compared XML to Lisp. I can't find the node now, but it was something about how you could just transform the XML to Lisp and let it execute to produce the result you need. This forced me to think ... if we can transform the XML to Lisp, we can just as well transform it to Perl:

<root>
 <foo x="5">
   <bar>hello</bar>
   <baz>world</baz>
 </foo>
</root>

=>

root(
 foo( x => 5,
  bar("hello"),
  baz("world")
 )
)

but the question is whether we would gain anything. Of course it would be silly to convert the whole XML to Perl code and then eval("") it, we can instead execute the subroutines as we parse the closing tag and just remember the results so that we could pass them to the subroutine for the parent tag.

I tried to come up with a few examples of things I might want to do with a XML and tried to implement them in this style and I like the results. Maybe it's my functional programming affected brain, but I find this style convenient.

$xml = <<'*END*';
<doc>
 <person>
  <fname>Jane</fname>
  <lname>Luser</lname>
  <email>[e-mail]</email>
  <address>
   <street>Washington st.</street>
   <city>Old Creek</city>
   <country>The US</country>
   <bogus>bleargh</bogus>
  </address>
  <phones>
   <phone type="home">123-456-7890</phone>
   <phone type="office">663-486-7890</phone>
   <phone type="fax">663-486-7000</phone>
  </phones>
 </person>
 <person>
  <fname>John</fname>
  <lname>Other</lname>
  <email>[e-mail]</email>
  <address>
   <street>Grant's st.</street>
   <city>New Creek</city>
   <country>Canada</country>
   <bogus>sdrysdfgtyh degtrhy <foo>degtrhy werthy</foo>werthy drthyu</
+bogus>
  </address>
  <phones>
   <phone type="office">663-486-7891</phone>
  </phones>
 </person>
</doc>
*END*

%rules = (
    _default => 'content',
    bogus => undef, # means "ignore"
    address => sub {address => "$_[1]->{street}, $_[1]->{city} ($_[1]-
+>{country})"},
    person => sub {
        '@person' => "$_[1]->{lname}, $_[1]->{fname}\n<$_[1]->{email}>
+\n$_[1]->{address}"
    },
    doc => sub { join "\n\n", @{$_[1]->{person}} },
);
# $parser->parse() will return a single string containin the addresses
+ 
# in plain text format

%rules = (
    _default => 'content',
#    bogus => sub {}, # means "returns no value. The subtags ARE proce
+ssed.
    bogus => undef, # means "ignore". The subtags ARE NOT processed.
    address => 'no content',
    person => 'no content array',
    doc => sub {$_[1]->{person}}, #'pass no content',
    foo => sub {print "FOOOOOOOO\n"},
);
# returns a simplified data structure kinda similar to XML::Simple

my $parser = new XML::Rules (
    rules => [
        _default => sub {$_[0] => $_[1]->{_content}},
        'fname,lname' => sub {$_[0] => $_[1]->{_content}},
        bogus => undef,
        address => sub {address => "$_[1]->{street}, $_[1]->{city} ($_
+[1]->{country})"},
        phone => sub {$_[1]->{type} => $_[1]->{_content}},
            # let's use the "type" attribute as the key and the conten
+t as the value
        phones => sub {delete $_[1]->{_content}; %{$_[1]}},
            # remove the text content and pass along the type => conte
+nt from the child nodes
        person => sub { # lets print the values, all the data is readi
+ly available in the attributes
            print "$_[1]->{lname}, $_[1]->{fname} <$_[1]->{email}>\n";
            print "Home phone: $_[1]->{home}\n" if $_[1]->{home};
            print "Office phone: $_[1]->{office}\n" if $_[1]->{office}
+;
            print "Fax: $_[1]->{fax}\n" if $_[1]->{fax};
            print "$_[1]->{address}\n\n";
            return; # the <person> tag is processed, no need to rememb
+er what it contained
        },
    ]
);
# prints the addresses, returns nothing

As you can see the rules applied to the parsed tags are basicaly of two types. Either they specify what data gets passed to the parent tag's rule and how or they do something with the attributes of the tag and the data returned by the rules of subtags. You can of course do both in your rules. For example if the XML looked like this:

<doc>
 <person>
  <fname>Jane</fname>
  <lname>Luser</lname>
  <email>[e-mail]</email>
  <address>
   <street>Washington st.</street>
   <city>Old Creek</city>
   <country>The US</country>
   <bogus>bleargh</bogus>
  </address>
  <phones>
   <phone type="home">123-456-7890</phone>
   <phone type="office">663-486-7890</phone>
   <phone type="fax">663-486-7000</phone>
  </phones>
 </person>
 <person>
  <fname>John</fname>
  <lname>Other</lname>
  <email>[e-mail]</email>
  <address id="12345"/>
  <phones>
   <phone type="office">663-486-7891</phone>
  </phones>
 </person>
</doc>

You might use a subroutine like this for the <address> tag:

sub {
 if (exists $_[1]->{id} and $_[1]->{id}+0 > 0) {
  $get_addr->execute($_[1]->{id});
  my $result = $get_addr->fetchall_arrayref();
  my ($street, $sity, $country) = ( $result->[0][0], $result->[0][1], 
+$result->[0][2]);
  return address => "$_[1]->{street}, $_[1]->{city} ($_[1]->{country})
+"
 } else {
  return address => "$_[1]->{street}, $_[1]->{city} ($_[1]->{country})
+"
 }
}

and proceed as if the data was directly in the XML in all cases.

Let me know please what you think. I'd also be grateful for any suggestions regarding the support for XML namespaces.

Update 2006-11-07: I just uploaded an updated version of the module, with more tests and "start tag" rules allowing you to skip branches of XML if you can decide based on the tag's attribute that you do not need them. I also uploaded the module to CPAN.

[Jenda]

Regarding the namespace support ... I think the easiest solution is to allow the module users to use namespace prefixes when specifying the rules, but at the same time let (or even force) them to specify what prefix belongs to what namespace. So that if the XML producer decides to use a different prefix for the same namespace the module will convert it behind the scenes.

$parser = XML::Rules->new(
 rules => [
  'SOAP-ENV:Envelope' => ...,
  'job:Location' => ...
  ...
 ],
 namespaces => {
  'http://schemas.xmlsoap.org/soap/envelope/' => 'SOAP-ENV',
  'http://www.foo.com/schemas/job.xsd' => 'job',
  ...
 },
);

And then no matter what prefix there is in the parsed XML, the rules starting with 'job:' will be used for tags in namespace 'http://www.foo.com/schemas/job.xsd'. Does this make sense?

[dewey]

I also think that changing XML into functional code like this is quite natural, kudos for writing the module. I would post a more constructive comment, but right now I don't have a project that needs this kind of processing, and that's really the only way I'll get an idea of the strengths and weaknesses of this approach.

[wazoox]

About XML as an equivalent to LISP : I guess you're thinking of The emacs problem. A fantastic article, that unfortunately looks unavailable right now.

[Jenda]

I'm sure it was something here on PerlMonks. But it's quite likely that even the author of that node would not know I mean his node. They were probably discussing something completely unrelated and this was just an unimportant sidenote.

Anyway, there are a few more things i want to add to the module before I release it to CPAN, but generaly I think this particular apple found the right head to hit ;-)

[wazoox]

Well, I actually found it from http://plasmasturm.org, which is the weblog from a well known and esteemed Monk... Aristotle. BTW, the fantastic article about XML as Lisp is back online here : The Emacs problem

[trwww]

Hi Jenda

I've done stuff like this very often.

When I see this, I see a generic state maintainence mechanism for SAX.

SAX is great. It is sometimes the only option when you have documents that are too big to fit in to RAM. But because it provides nothing more than a dispatch mechanism for document particles, maintaining state between the different callbacks can get tricky, boring, and error prone.

As an example, refer to Kip Hampton's excellent xml.com article High-Performance XML Parsing With SAX.

In it, he uses an XML document that represents a series of emails that need sent as the sample data. He uses SAX to build up an argument list for Mail::Sendmail. When the end_element callback is fired for the record, he has Mail::Sendmail send the email.

The relevant part here is note how much code he has to write to extract the data from the individual record.

According to the POD in your module:

Or you could view it as yet another event based XML parser that differs from all the others only in two things. First that it only let's you hook your callbacks to the closing tags. And that stores the data for you so that you do not have to use globals or closures and wonder where to attach the snippet of data you just received onto the structure you are building

what you have would help quite a bit in maintaining the state of an XML record.

If you dont mind, what would the rules for your module look like to perform the same task Hampton did with SAX?

Your module would be very useful implemented as a SAX handler so users could take advantage of the many features of SAX (swappable parsers, chained filters/handlers, document writers, standardized interface). Imagine the process you describe aboive as a web serice. You could use a SAX writer in the same pipeline to build up the response for the client request.

Regardless of how you proceed, I'll definitely keep an eye on it. I know I'll use it.

trwww

[Jenda]

The code would look like this:

my ($message_count, $sent_count);
my $parser = new XML::Rules (
    rules => [
    _default => 'content',
    message => sub {
        $message_count++;
        Mail::Sendmail::sendmail(
            from => $_[1]->{from},
            to => $_[1]->{to},
            subject => $_[1]->{subject},
            body => $_[1]->{body},
        )
            or warn "Mail Error: $Mail::Sendmail::error";
        $sent_count++ unless $Mail::Sendmail::error;
        return;
    },
    messages => sub {
        print "SAX Mailer Finished\n$sent_count of $message_count mess
+age(s) sent\n";
        return;
    }
]);
$parser->parsefile($file);

or, if you did not want to use any external variables:

my $parser = new XML::Rules (
    rules => [
    _default => 'content',
    message => sub {
        $_[3][-1]->{message_count}++;
        Mail::Sendmail::sendmail(
            from => $_[1]->{from},
            to => $_[1]->{to},
            subject => $_[1]->{subject},
            body => $_[1]->{body},
        )
            or warn "Mail Error: $Mail::Sendmail::error";
        $_[3][-1]->{sent_count}++ unless $Mail::Sendmail::error;
        return;
    },
    messages => sub {
        print "SAX Mailer Finished\n$_[1]->{sent_count} of $_[1]->{mes
+sage_count} message(s) sent\n";
        return;
    }
]);
$parser->parsefile($file);

or

my $parser = new XML::Rules (
    rules => [
    _default => 'content',
    message => sub {
        my ($tag_name, $tag_hash, $context, $parent_data) = @_;
        $parent_data->[-1]->{message_count}++;
        Mail::Sendmail::sendmail(
            from => $tag_hash->{from},
            to => $tag_hash->{to},
            subject => $tag_hash->{subject},
            body => $tag_hash->{body},
        )
            or warn "Mail Error: $Mail::Sendmail::error";
        $parent_data->[-1]->{sent_count}++ unless $Mail::Sendmail::err
+or;
        return;
    },
    messages => sub {
        print "SAX Mailer Finished\n$_[1]->{sent_count} of $_[1]->{mes
+sage_count} message(s) sent\n";
        return;
    }
]);
$parser->parsefile($file);

I guess I should add a few more ways to add data to the parent node. Apart from 'attributename' that sets (and if needed overwrites) the attribute and '@attributename' that appends the value to the array I should also allow '+attributename' and '.attributename'. '-attributename' is not needed, but I wonder whether to add '*attributename'.

With the '+attributename' the code would look like this:

my $parser = new XML::Rules (
    rules => [
    _default => 'content',
    message => sub {
        Mail::Sendmail::sendmail(
            from => $_[1]->{from},
            to => $_[1]->{to},
            subject => $_[1]->{subject},
            body => $_[1]->{body},
        )
            or warn "Mail Error: $Mail::Sendmail::error";
        return '+message_count' => 1, '+sent_count' => ($Mail::Sendmai
+l::error ? 0 : 1);
    },
    messages => sub {
        print "SAX Mailer Finished\n$_[1]->{sent_count} of $_[1]->{mes
+sage_count} message(s) sent\n";
        return;
    }
]);
$parser->parsefile($file);

All code except the last snippet is tested :-)

I'll have a look at SAX, currently the module sits on top of XML::Parser:Expat, but I think it should not be a big deal to change that. Or to change the code so that you can choose what to use.