Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Fetching URL

by Shaveta_Chawla (Acolyte)
on Oct 03, 2011 at 09:32 UTC ( #929298=perlquestion: print w/ replies, xml ) Need Help??
Shaveta_Chawla has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to fetch a url from a site. i have used form method to fetch URL and successfully fetched first page URL.i have just submitted basic URL where search page is available. The code used is::

my @forms = HTML::Form->parse($res); @forms = grep{$_->attr('name') eq 'adv_search_form'} @forms; my $form = shift @forms; my $req = $form->make_request;
the URL generates dynamic value. when i am trying to fetch second page URL, means next page request the code used is::
my $referer = $res->request->uri; my $uri = $res->request->uri; my $headers = $res->request->headers; my %link_tags = ( a => 'href', ); my $parser = HTML::TokeParser->new(\$res->decoded_content()); while ( my $token = $parser->get_tag( keys %link_tags ) ) { my $attrs = $token->[1]; my $text = $parser->get_trimmed_text("/a"); my $url = $attrs->{$link_tags{'a'}}; if($text eq 'Next page') { $url2 = "abc.com" . $url; } $headers->header(referer => $referer); my $req = HTTP::Request->new('GET', $url2, $headers); return $req;
i have got the exact URL but that URL is not working. Please help me out...

Comment on Fetching URL
Select or Download Code
Re: Fetching URL
by Priti24 (Novice) on Oct 03, 2011 at 09:40 UTC
    I m not able to understand your problem. So Plz Tell me in brief..thnks!!!
Re: Fetching URL
by Corion (Pope) on Oct 03, 2011 at 09:42 UTC

    What do you mean by "not working"? Please describe exactly how it "does not work", and what you expect instead.

    Have you looked at WWW::Mechanize?

Re: Fetching URL
by ym_chaitu (Initiate) on Oct 04, 2011 at 06:11 UTC
    didn't got a clarity on what you are saying but if you like to build a webcrawler then i have written one small code on that which you might be interested in. Webcraler
      I am trying to fetch a url which contains dynamic value, i have used form method, follow_link method but couldn't succeed. The URL which i have fetched is same that the source is generating but when i am opening that URL on browser,it is giving me wrong page information. May be some cookie based issue.. I am unable to understand the cause as the url is same.. can you guide me what can be the issue..

        You will need to send exactly the same information that your browser sends. Find out what the browser sends, for example using the Mozilla Live Headers extension or using Wireshark, and then replicate that data from your script.

Re: Fetching URL
by Shaveta_Chawla (Acolyte) on Oct 04, 2011 at 11:02 UTC
    Actually the Content length, content header and cookies all the values changes with every request.. I am not able to generate the same header information. Can you guide me on this.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://929298]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (4)
As of 2014-09-21 04:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (166 votes), past polls