Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^4: how to quickly parse 50000 html documents? (Updated: 50,000 pages in 3 minutes!)

by aquarium (Curate)
on Nov 28, 2010 at 23:18 UTC ( #874145=note: print w/ replies, xml ) Need Help??


in reply to Re^3: how to quickly parse 50000 html documents? (Updated: 50,000 pages in 3 minutes!)
in thread how to quickly parse 50000 html documents?

i take the criticism for using strong words.
the html as provided is reminiscent of 80's websites. the html provides no data structure elements, merely look/feel html elements/attributes.
hence as the tags are superficial in terms of data, it would be easier and less likely to break if the scrape is based on the html converted to text. basing the regexes on the well defined terms followed by a collon. basing the regexes instead on largely irrelevant (look/feel) html, I see as not an effective design.
i'm not criticising the html just for the sake of being critical...but i believe in basing programs on best variant of input data. so if one has no control over the website, then at least making best attempt at getting non-breaking data is better (imho) to just scraping the worst and hoping for the best. so instead of merely criticising, it's basing decisions on likely factors...deciding to use text form of data instead of anchoring on outdated or likely-to-be-not-well-formed html. that's all.

the hardest line to type correctly is: stty erase ^H


Comment on Re^4: how to quickly parse 50000 html documents? (Updated: 50,000 pages in 3 minutes!)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://874145]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (11)
As of 2014-08-22 18:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (163 votes), past polls