Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Re^2: Normalizing URLs

by Anonymous Monk
on Jul 21, 2005 at 14:46 UTC ( #476858=note: print w/replies, xml ) Need Help??

in reply to Re: Normalizing URLs
in thread Normalizing URLs

Some background - I am developing a scraper and it needs to know if it has scraped the page already - hence the normalization. It needn't be perfect, just good enough for all but the most arcane. Needs only work with http.

Replies are listed 'Best First'.
Re^3: Normalizing URLs
by spiritway (Vicar) on Jul 22, 2005 at 03:18 UTC

    What about using the "last_modified" method in LWP? Keep track of it locally. When you access the page again, check the time it was modified and skip it if that time is not newer than what you've saved.

    This idea is from "Spidering Hacks" (hack #16).

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://476858]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (4)
As of 2020-06-02 18:34 GMT
Find Nodes?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?

    Results (19 votes). Check out past polls.