Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re^3: Building a Spidering Application

by Your Mother (Chancellor)
on Jul 09, 2012 at 15:25 UTC ( #980705=note: print w/replies, xml ) Need Help??


in reply to Re^2: Building a Spidering Application
in thread Building a Spidering Application

You don't need URI::ImpliedBase. WWW::Mechanize::Link objects that Mech uses/returns have a method, url_abs, to cover this. Of course then it's up to the spider to decide if query params are relevant or duplicates or no-ops and, in the hacky world of HTML4.9, if fragments are meaningful (but only JS aware Mech would be able to care here).

Replies are listed 'Best First'.
Re^4: Building a Spidering Application
by pemungkah (Priest) on Jul 09, 2012 at 16:17 UTC
    Thanks! I didn't know about that one. Last time I wrote a spider was maybe six years ago, and as I recall it wasn't there then - though I may have just missed it at the time. Handy for LWP folks still, I guess.

      I'm pretty sure you're right about it not being there at that time.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://980705]
help
Chatterbox?
[LanX]: well you can warn about missing features...
[LanX]: ... i.e. avoid headaches about bug reports...
[Corion]: Yeah, I've updated the documentation to mention just that :)

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (12)
As of 2017-12-12 20:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What programming language do you hate the most?




















    Results (336 votes). Check out past polls.

    Notices?