Re^4: Parsing HTML/XML with Regular Expressions (XML::Twig)

[haukex]

Interesting post, thank you! I tested it and except that I had to strip non-word characters out of the values, it mostly works - it doesn't pick up the id of the Sunday Saturday entry, and it also picks up the values "bbbdddeeeggg", but overall it's a very interesting start. Regexes are a fine tool for lexing, and by adding some logic around them keeping track of the nested tags etc., it's basically like building a simple parser.

[RonW]

I tried it and got no output. Did you fix something in my code?

I did add a statement to output the list of elements from the shallow parse regex. As far as I can tell, it split out the elements correctly, but it left the embedded newlines in the mark-up elements.

For example, the following:

</div
>

became </div\n>

In the case of the Sunday div:

<div title=" class='data' id='Foo'>Bar"
id="Seven" class="data">&#xA0;Sunda&#121;</div>

became:

<div title=" class='data' id='Foo'>Bar"\nid="Seven" class="data">
&#xA0;Sunda&#121;
</div>

So, I added tr/\n/ / for (@elements); to get rid of the embedded newlines. Still no output (other than the dump of the elements list).

I did encounter an unexpected error: Variable "$XML_SPE" is not imported at extractor.pl line 46. So, I changed:

my @elements = $xml =~ /$XML_SPE/g;

to:

my @elements = $xml =~ /$::XML_SPE/g;

I don't have time to try to debug my code, now. Will try, later.

Current code:

Read more... (6 kB)

And the output:

Read more... (3 kB)

[haukex]

Here's the code I ran, other than adding the necessary stuff to get it to compile and read the external file, the only difference to your code is the addition of s/\W+//g;. The output I get is the following. <update> You were right, it does pick up the wrong id for Sunday, it was the id of Saturday that was missing, my mistake. </update>

Zero=, One=Monday, Two=Tuesday, Three=Wednesday, Four=Thursday, Five=F
+ridaySaturday, Foo=xA0Sunda121bbbdddeeeggg

Read more... (4 kB)

[RonW]

Ignoring entity decoding and handling of quoted strings (the attribute values), I've gotten as close to the expected output as I care to pursue.

The extra output of the Sunday division ("bbbdddeeeggg") was a side effect of incorrectly handling the empty division. The interesting divisions after the empty one were handled correctly because my code allows any division to contain an interesting division. The provided input does have that, but once the empty division handling was fixed, the extra output was eliminated and the rest of the output was correct (other than not decoding the entities and the incorrect id of the Sunday division as mentioned above).

The embedded newlines (that the shallow parsing regex leaves as-is) have no general "solution". For mark-up elements, converting them to spaces provides clear enough syntax to reliably parse the attributes (at least for this challenge). The content elements need case by case handling. For this challenge, removing leading and trailing newlines gave the desired results.

The shallow parsing regex is interesting and might be useful for some projects, but most projects will be better served by using one of the better XML modules from CPAN.

Thanks, again, to [id://haukex] for the challenge and contributing to this little regex adventure.

My (probably) final code for this:

Read more... (4 kB)

[RonW]

I ran your version of my code and got the same output you did.

Since I already discovered the embedded newlines in the elements list, I added tr/\n//d; at the top of the for loop:

for (@elements)
{
    tr/\n//d;

After doing that, the id for Saturday picked up correctly. Also, out of curiosity, I removed the s/\W+//g; you added. The result was:

Zero=, One=Monday, Two=Tuesday, Three=Wednesday, Four=Thursday, Five=F
+riday, Six=Saturday, Foo= Sundaybbbdddeeeggg

So, Saturday is cleaned up.

I know why the id for Sunday is Foo, but still not sure why the "bbbdddeeeggg" is picked up. I will have to step through the code to see what's happening.

As for the &#xA0;, that's encoding dependent. Not sure why it would get excluded other than by explicitly filtering out non-ASCII characters.

The &#121; is the y in Sunday. Just requires entity decoding.

[RonW]

Thanks. Also, you've got me curious. I still haven't tested it, but I'm guessing the interesting title attribute for the Sunday division is part of the problem. You said it didn't pick up the id. I would have thought my code would have picked up id='Foo'. About the bbbdddeeeggg I'm thinking my code had trouble finding the correct </div>.

I will try it and look at the list of elements generated by the shallow parsing regex.