Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Parsing HTML files to recover data...

by GrandFather (Cardinal)
on Nov 21, 2006 at 21:13 UTC ( #585357=note: print w/ replies, xml ) Need Help??


in reply to Parsing HTML files to recover data...

For this sort of 'Parse and extract' from HTML problem I reach for HTML::TreeBuilder. A first cut solution might look like:

use strict; use warnings; use HTML::TreeBuilder; my $str = do {local $/; <DATA>}; my $tree = HTML::TreeBuilder->new; $tree->parse ($str); my @jobs; for my $para ($tree->find ('p')) { my @class = $para->look_down ('class', 'jobname'); my @names = $para->look_down ('name', 'em'); my @offices = $para->look_down ('name', 'offices'); next unless @class && @names && @offices; my $job = $class[0]->as_text ();; $job .= ': ' . join '; ', map {$_->as_text ()} @names; $job .= ' (' . join (', ', map {$_->as_text ()} @offices) . ')'; push @jobs, $job; } print join "\n", @jobs; __DATA__ <p><b><span class="jobname">Sandbagger, Level 2</span> <span class="jobserial">(19000)</span><br /> Current members:<br /> <span name='em'>Fred</span><span name='em'>Wilma</span><br /> <span name='offices'>Erewon</span> </p> <p><b><span class="jobname">Accounting Assistant, Level 2</span> <span class="jobserial">(19203)</span><br /> Current members:<br /> <span name='em'>Plow, Elliot</span><span name='em'>Wang, Susan</span>< +br /> <span name='offices'>Huston</span> </p> <blockquote> Job descriptions here. This block quoted text contains a job description and it what I am rea +lly looking to recover. </blockquote> <blockquote><a href="#top">Go to the top of this page</a>.</blockquote +> <blockquote><a href='companyHR.html'>Check for open positions now!</a> +</blockquote>

Prints:

Sandbagger, Level 2: Fred; Wilma (Erewon) Accounting Assistant, Level 2: Plow, Elliot; Wang, Susan (Huston)

I imagine that there might be lists of people fillin a particular role by office. This code will munge all of those entries together and list the offices, thus losing the association of people with offices. That can be fixed by using a $para->look_down ('name', qr/em|offices/) then iterating over the list spitting out employee names and office names as appropriate.


DWIM is Perl's answer to Gödel


Comment on Re: Parsing HTML files to recover data...
Select or Download Code
Re^2: Parsing HTML files to recover data...
by UrbanHick (Sexton) on Nov 22, 2006 at 02:02 UTC

    Thank you very much for your detailed response. The example code is most instructive. I will likely use this code as a frame work to fashion several programs to each parse a specific document for the data that the Powers That Be want extracted from each old webpage.

    -UH

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://585357]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (16)
As of 2014-07-11 15:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (230 votes), past polls