Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

WWW::Robot no longer compatible with LWP::RobotUA. Now what?

by nysus (Deacon)
on Nov 13, 2005 at 04:49 UTC ( #508069=perlquestion: print w/ replies, xml ) Need Help??
nysus has asked for the wisdom of the Perl Monks concerning the following question:

I want to create a robot that spiders a handful of small sites a couple of times a day and sends me an e-mail notification when it finds an updated page. I tried to creating a program that used the WWW::Robot module but can't because WWW::Robot is badly out of date. This has already been discussed at a WWW::Robot robot ?.

So I'm trying to figure out where to go from here. I'd really appreciate some ideas that will save me from reinventing the wheel.

$PM = "Perl Monk's";
$MCF = "Most Clueless Friar Abbot Bishop Pontiff";
$nysus = $PM . $MCF;
Click here if you love Perl Monks

Edit: g0n - fixed link

Comment on WWW::Robot no longer compatible with LWP::RobotUA. Now what?
Re: WWW::Robot no longer compatible with LWP::RobotUA. Now what?
by fokat (Deacon) on Nov 13, 2005 at 05:23 UTC

    Dear nysus:

    I want to create a robot that spiders a handful of small sites a couple of times a day (...) I'd really appreciate some ideas that will save me from reinventing the wheel

    Have you seen WWW::Mechanize? I have used it a few times, with excellent results.

    Best regards

    -lem, but some call me fokat

      No, I have not heard of that module before. Thanks. I will look into it.

      $PM = "Perl Monk's";
      $MCF = "Most Clueless Friar Abbot Bishop Pontiff";
      $nysus = $PM . $MCF;
      Click here if you love Perl Monks

Re: WWW::Robot no longer compatible with LWP::RobotUA. Now what?
by ghenry (Vicar) on Nov 13, 2005 at 10:04 UTC

    I've looked at LWP::RobotUA, but I can't see that it needs WWW::Robot anywhere? Maybe I've missed something?

    If I have, apologies,
    Gavin.

    Walking the road to enlightenment... I found a penguin and a camel on the way.....
    Fancy a yourname@perl.me.uk? Just ask!!!
      WWW::Robot does not need RobotUA. It's the other way around. WWW::Robot had some features that would have saved me some time, however.

      $PM = "Perl Monk's";
      $MCF = "Most Clueless Friar Abbot Bishop Pontiff";
      $nysus = $PM . $MCF;
      Click here if you love Perl Monks

Re: WWW::Robot no longer compatible with LWP::RobotUA. Now what?
by Cody Pendant (Prior) on Nov 14, 2005 at 02:56 UTC
    I just found WWW::CheckSite::Spider which looks very useful and recently updated.


    ($_='kkvvttuu bbooppuuiiffss qqffssmm iibbddllffss')
    =~y~b-v~a-z~s; print
Re: WWW::Robot no longer compatible with LWP::RobotUA. Now what?
by spiritway (Vicar) on Nov 14, 2005 at 05:31 UTC

    If you want some more ideas, you might check out Spidering Hacks. It has some great ideas in it, and the examples are in Perl. You can download the code from here.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://508069]
Approved by spiritway
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (9)
As of 2014-08-22 08:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (150 votes), past polls