Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re: Parsing HTML files to recover data...

by GrandFather (Sage)
on Nov 21, 2006 at 21:13 UTC ( #585357=note: print w/replies, xml ) Need Help??

in reply to Parsing HTML files to recover data...

For this sort of 'Parse and extract' from HTML problem I reach for HTML::TreeBuilder. A first cut solution might look like:

use strict; use warnings; use HTML::TreeBuilder; my $str = do {local $/; <DATA>}; my $tree = HTML::TreeBuilder->new; $tree->parse ($str); my @jobs; for my $para ($tree->find ('p')) { my @class = $para->look_down ('class', 'jobname'); my @names = $para->look_down ('name', 'em'); my @offices = $para->look_down ('name', 'offices'); next unless @class && @names && @offices; my $job = $class[0]->as_text ();; $job .= ': ' . join '; ', map {$_->as_text ()} @names; $job .= ' (' . join (', ', map {$_->as_text ()} @offices) . ')'; push @jobs, $job; } print join "\n", @jobs; __DATA__ <p><b><span class="jobname">Sandbagger, Level 2</span> <span class="jobserial">(19000)</span><br /> Current members:<br /> <span name='em'>Fred</span><span name='em'>Wilma</span><br /> <span name='offices'>Erewon</span> </p> <p><b><span class="jobname">Accounting Assistant, Level 2</span> <span class="jobserial">(19203)</span><br /> Current members:<br /> <span name='em'>Plow, Elliot</span><span name='em'>Wang, Susan</span>< +br /> <span name='offices'>Huston</span> </p> <blockquote> Job descriptions here. This block quoted text contains a job description and it what I am rea +lly looking to recover. </blockquote> <blockquote><a href="#top">Go to the top of this page</a>.</blockquote +> <blockquote><a href='companyHR.html'>Check for open positions now!</a> +</blockquote>


Sandbagger, Level 2: Fred; Wilma (Erewon) Accounting Assistant, Level 2: Plow, Elliot; Wang, Susan (Huston)

I imagine that there might be lists of people fillin a particular role by office. This code will munge all of those entries together and list the offices, thus losing the association of people with offices. That can be fixed by using a $para->look_down ('name', qr/em|offices/) then iterating over the list spitting out employee names and office names as appropriate.

DWIM is Perl's answer to Gödel

Replies are listed 'Best First'.
Re^2: Parsing HTML files to recover data...
by UrbanHick (Sexton) on Nov 22, 2006 at 02:02 UTC

    Thank you very much for your detailed response. The example code is most instructive. I will likely use this code as a frame work to fashion several programs to each parse a specific document for the data that the Powers That Be want extracted from each old webpage.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://585357]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2017-01-17 21:56 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (158 votes). Check out past polls.