Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Scrappy Module

by sankarg (Initiate)
on May 12, 2011 at 11:12 UTC ( #904357=perlquestion: print w/ replies, xml ) Need Help??
sankarg has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

Any information on Scrappy Module. I just use that module and found results for some of the sites. But mostly it may not be work for some kind of tags in the content. So, help me in scrappy module. Any latest news also update.

Thanks Sankar G.

Comment on Scrappy Module
Re: Scrappy Module
by marto (Chancellor) on May 12, 2011 at 11:22 UTC
      Thanks for your immediate reply marto. I already worked with scrappy module. I can able to get the content when scrapping a website. My question is in the latest version of scrappy
      use Scrappy; my $scraper = Scrappy->new; $scraper->crawl('http://search.cpan.org/recent', '/recent' => { '#cpansearch li a' => sub { print $_[1]->{href}, "\n"; } } );
      you can find that this the url 'http://search.cpan.org/recent' means we need to give the 'recent' tag. it is working only for this cpan site. And it is not working for other sites. That is my question. How we could use the tags and get scrape a website. Can you able to understand.

        I've never used this module, if you look at the source for the URL you provide in your example along with the documentation (see the crawl method), unless other sites contain the same '/recent' link and element with and id of 'cpansearch' etc, it's not going to work. In other words, you need to write your own code to work with your own sites. Other parsing modules are available, see WWW::Mechanize::Firefox among others.

Re: Scrappy Module
by Anonymous Monk on May 12, 2011 at 11:38 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://904357]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2014-09-23 10:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (217 votes), past polls