Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Re^3: Building a Spidering Application

by Your Mother (Bishop)
on Jul 09, 2012 at 15:25 UTC ( #980705=note: print w/replies, xml ) Need Help??

in reply to Re^2: Building a Spidering Application
in thread Building a Spidering Application

You don't need URI::ImpliedBase. WWW::Mechanize::Link objects that Mech uses/returns have a method, url_abs, to cover this. Of course then it's up to the spider to decide if query params are relevant or duplicates or no-ops and, in the hacky world of HTML4.9, if fragments are meaningful (but only JS aware Mech would be able to care here).

Replies are listed 'Best First'.
Re^4: Building a Spidering Application
by pemungkah (Priest) on Jul 09, 2012 at 16:17 UTC
    Thanks! I didn't know about that one. Last time I wrote a spider was maybe six years ago, and as I recall it wasn't there then - though I may have just missed it at the time. Handy for LWP folks still, I guess.

      I'm pretty sure you're right about it not being there at that time.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://980705]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (5)
As of 2018-10-21 18:29 GMT
Find Nodes?
    Voting Booth?
    When I need money for a bigger acquisition, I usually ...

    Results (119 votes). Check out past polls.