Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Validating HTML structures

by b374 (Initiate)
on Oct 31, 2005 at 00:27 UTC ( #504092=note: print w/replies, xml ) Need Help??


in reply to Validating HTML structures

HTML::Tree is your friend. I used it very extensive in converting old pages to new formats, pulling out content, and even XML parsing.

As for identifying various structures, the easiest approach I found was the following:

Presuming you have like 10,000 pages of somehow similar structure but have suspicions there might be some different. Build the parse tree for each page and eliminate anything which isn't structural from the tree. Dump the html with no spaces, no comments to a string and make a hash out of that string. For each hash in this stage keep a list of files that matched it and the "blueprint" in some structures of your choice (mine are files for example)... After this step you'll get some classes of pages, each matching one blueprint.

After you get the structures based on the blocks on the pages, repeat the same procedure for each block and contained information.

The case in which this method got the best results was extracting information out of about 18,000 pages which contained something like 3 tables of key-values pairs, some free form content (descriptions, reviews, comments), images and some other blocks. The layout was a mess, part of the pages being initially generated through 3-4 different scripts and then updated manually over a few years with anything from text editors to *cough* FrontPage... the result was pretty impressive... in just about a day (actually a night), over 99% of the pages were in a nice database.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://504092]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (2)
As of 2020-06-05 01:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?



    Results (35 votes). Check out past polls.

    Notices?