Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^3: Building a Spidering Application

by Your Mother (Chancellor)
on Jul 09, 2012 at 15:25 UTC ( #980705=note: print w/replies, xml ) Need Help??


in reply to Re^2: Building a Spidering Application
in thread Building a Spidering Application

You don't need URI::ImpliedBase. WWW::Mechanize::Link objects that Mech uses/returns have a method, url_abs, to cover this. Of course then it's up to the spider to decide if query params are relevant or duplicates or no-ops and, in the hacky world of HTML4.9, if fragments are meaningful (but only JS aware Mech would be able to care here).

Replies are listed 'Best First'.
Re^4: Building a Spidering Application
by pemungkah (Priest) on Jul 09, 2012 at 16:17 UTC
    Thanks! I didn't know about that one. Last time I wrote a spider was maybe six years ago, and as I recall it wasn't there then - though I may have just missed it at the time. Handy for LWP folks still, I guess.

      I'm pretty sure you're right about it not being there at that time.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://980705]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (3)
As of 2016-12-08 11:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    On a regular basis, I'm most likely to spy upon:













    Results (140 votes). Check out past polls.