Re^2: WWW::Mechanize::Chrome VERY slow on xpath obtaining TDs of a TR

[LanX]

> If you don't need JavaScript

Even if ...

supposing communication overhead or an implementation loop are causing a bottleneck ...

... he could also try to fetch the whole table as html once using WWW::Mechanize::Chrome and do the parsing with Mojo::UserAgent

[marto]

" ... he could also try to fetch the whole table as html once using WWW::Mechanize::Chrome and do the parsing with Mojo::UserAgent"

I've used this work around in the past for things that need special sign in or bounce back things that aren't being detected as a 'real' browser, purely so I don't have to do a lot of code changes :) As the location of the bottleneck is not yet understood this may not resolve the issue of performance.

[LanX]

> As the location of the bottleneck is not yet understood this may not resolve the issue of performance.

but may help narrowing down the underlying problem.

[ait]

I will try this, thank you!

I noticed that fetching the TRs of the table seems pretty fast with WWW::Mechanize::Chrome and xpath. What's seems absurd is that fetching the TDs relative to a single TR takes so long, and the time is proportional to the number of total TRs. That doesn't make any sense unless there's a bug somewhere in WWW::Mechanize::Chrome xpath implementation.

[LanX]

I can't look into it now, so some general advice
• Try the Xpath inside the browser's dev console
• Try logging what mechanize does under the hood.

Back in the time when I used W:M:FF I was able (and sometimes needed) to send and eval JS and fetch the result as JSON.

All this will help you specifying a feature request (if needed) for W:M:C

HTH :)

[ait]

Thank you!
I'm on a tight schedule to deliver so I think I'm going with extracting the HTML of the table with W::M::Chrome and parsing it with HTML-Tree
I'll post my solution in the OP so that can help others in the future.

[ait]

If I get a good payday from this small gig, I'll look into the W::M::Chrome code myself and see if I can at least contribute in some way...