Re^2: Screen scraping complex tables and divs (updated)

[parser]

Rolf,

I am confused now too. Are you saying WWW::Mechanize supports CSS selector and XPath? Or that WWW::Mechanize::Firefox does? If the latter, I also read it was very difficult to build.

Query syntax is not a Perl question, but there are plenty of good tutorials online.

I agree. However, determining how best to query HTML source via Perl is.

The option of mirroring the DOM into a Perl/XML data structure and using the query API sounds quite good. I'll give that a go and see how it works. Anything is better than parsing table tags with TokParser.

[Corion]

As a very simplicistic application of combining WWW::Mechanize with HTML::TreeBuilder and HTML::Selector::XPath, I wrote App::scrape. This module encapsulates extracting data from HTML either via CSS selectors or XPath queries. Maybe you can use that as a starting point.

[LanX]

WWW::Mechanize::Firefox does and I took it as an example out of many because I worked with it in the past.

But it really depends if you need JS or not, so I don't want to go into details.

Querying Html was your question, something like xpath or css selector is mostly the solution.

Regarding the Perl backend: it depends.

Sorry there is no generic answer for TIMTOWTDI .

[LanX]

PS:

> > Look out for browser features/addons allowing to play around with queries.

I had very good experience using Firepath to find the right CSS selectors / XPath expressions inside Firefox.

You can copy an auto-generated explicit expression by right clicking on a DOM-element and change them interactively.

Simply copy the final path and/or selector into your Perl code then.

HTH! :)

[parser]

Good catch! Firepath is saving me much time!