vagabonding electron has asked for the wisdom of the Perl Monks concerning the following question:
Dear Monks,
I parse a certain amount of HTML pages ( > 400 ) which have a structure as shown in the <DATA> part of the script below.
The relevant part of the page begins with <div id="bodyContent"> so that I put this part only in the script.
What I need is the text between the certain <h2>-tags.
I used HTML:TreeBuilder:XPath but I did not find how I could formulate an intersection there (e.g. following of <h2>[1] and preceding of <h2>[2] at the same time).
As a workaround I take the preceding-sibling in sequence of <h2>[i] tags, stringify the output and use substr to subtract the preceding chunks of text.
This works (after some clean up) but the code looks no fun to me.
Please give me a hint how I could make it better.
Thank you!
VE
I parse a certain amount of HTML pages ( > 400 ) which have a structure as shown in the <DATA> part of the script below.
The relevant part of the page begins with <div id="bodyContent"> so that I put this part only in the script.
What I need is the text between the certain <h2>-tags.
I used HTML:TreeBuilder:XPath but I did not find how I could formulate an intersection there (e.g. following of <h2>[1] and preceding of <h2>[2] at the same time).
As a workaround I take the preceding-sibling in sequence of <h2>[i] tags, stringify the output and use substr to subtract the preceding chunks of text.
This works (after some clean up) but the code looks no fun to me.
Please give me a hint how I could make it better.
Thank you!
VE
#!/usr/bin/perl use strict; use warnings; use LWP::Simple; use HTML::TreeBuilder::XPath; my $page; $page .= $_ while <DATA>; my $p = HTML::TreeBuilder::XPath->new_from_content( $page ); my @page_content =$p->findnodes( '//div[@id="bodyContent"]' ); for my $content ( @page_content ) { my @preface = $content->findvalues( './h2[1]/preceding-sibling::*' + ); my $preface_text; my ( $keyword, $actualised ); for my $pref ( @preface ) { # $pref =~ s/^\s*(\S+)/$1/; $preface_text .= $pref; # print $preface_text, "--\n"; ( undef, $keyword ) = split /:\s*?/, $pref, 2 if $pref =~ +/^\s*?Key words/; ( undef, $actualised ) = split /:\s*?/, $pref, 2 if $pref +=~ /^Actualised/; } print $keyword, "\n"; print $actualised, "\n"; my @problems = $content->findvalues( './h2[2]/preceding-siblin +g::*' ); my $probl; $probl .= $_ for @problems; $probl = substr( $probl, length( $preface_text) ); print $probl, "\n"; my @solution_1 = $content->findvalues( './h2[3]/preceding-sibl +ing::*' ); my $sol; $sol .= $_ for @solution_1; $sol = substr( $sol, length( $preface_text ) + length( $probl +) ); print $sol, "\n"; my @solution_2 = $content->findvalues( './h2[4]/preceding-sibl +ing::*' ); my $sol_2; $sol_2 .= $_ for @solution_2; $sol_2 = substr( $sol_2, length( $preface_text ) + length( $pr +obl ) + length( $sol ) ); print $sol_2 , "\n"; } __DATA__ <head> </head> <body> <div id="bodyContent"> <!-- start content --> <p>Key words: Some words. </p><p>Date: 2012-01-16 </p><p>Actualised: 2008-01-08 </p><p>Commented: 05.06.2007 </p><p>Encoded: Some code. </p> <h2> <span class="mw-headline" id="Problem"> Problem </span></h2> <p>Problem description. </p><p>Another description. </p> <h2> <span class="mw-headline" id="Solution1"> Solution 1 </span></h2> <p>Solution description. </p> <h2> <span class="mw-headline" id="Solution2"> Solution 2 </span></h2> <p>Solution description. </p> <h2> <span class="mw-headline" id="Comment"> Comment. </span></h2> <p>Text of the comment. </p><p><br /> </p> </div> <hr /> </body>
|
---|
Back to
Seekers of Perl Wisdom