Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Re: Local::SiteRobot - a simple web crawling module

by merlyn (Sage)
on Nov 24, 2001 at 21:26 UTC ( #127278=note: print w/replies, xml ) Need Help??

in reply to Local::SiteRobot - a simple web crawling module

This was where my journey hit a stumbling block ... WWW::SimpleRobot didn't work!
Rather than partially reinvent a wheel, did you try to work with the author to figure out why it "didn't work"? And can you please explain what "didn't work" for you? I've used the module successfully, as have many hundreds of others.

"forking" a codebase should be considered carefully, otherwise the open source users as a community all lose.

-- Randal L. Schwartz, Perl hacker

  • Comment on Re: Local::SiteRobot - a simple web crawling module

Replies are listed 'Best First'.
Re: Re: Local::SiteRobot - a simple web crawling module
by rob_au (Abbot) on Nov 25, 2001 at 05:17 UTC
    The problems which I encountered with WWW::SimpleRobot related to the traverse function not returning traversal results via the $object->pages and $object->urls methods - The problem appeared to relate to the shift method by which the author was iterating through the constructed queue undefining the @pages results array before it was returned at the end of the function.

    Better than just reporting this to the author, I have submitted a fix patch which corrects this behaviour by pushing results into a separate array to the queue.

    115a116 > my @results; 150a152 > push (@results, $page); 165,166c167,168 < $self->{pages} = \@pages; < $self->{urls} = [ map { $_->{url} } @pages ]; --- > $self->{pages} = \@results; > $self->{urls} = [ map { $_->{url} } @results ];

    Note that I never meant for my little piece of code to be viewed as a code fork from WWW::SimpleRobot but rather just an additional available option.


    Ooohhh, Rob no beer function well without!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://127278]
[Tanktalus]: talexb I saw that one, I was going to apply anyway, but never got around to it.

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2018-01-22 02:43 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (230 votes). Check out past polls.