Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re: How to scraper ASP websites

by Gangabass (Priest)
on Sep 05, 2012 at 15:04 UTC ( #991875=note: print w/ replies, xml ) Need Help??

in reply to How to scraper ASP websites

Always use strict; and use warnings;. You don't define $__EVENTVALIDATION and $__VIEWSTATE that's why your code send empty values to the target site.

These values are located in the HTML form you receive from the target site and WWW::Mechanize will send these values automatically for you.

So usually you just need to provide login/password (and ParentUID in your example) to process auth page. May be you need to provide button name for the click()... Or may be this is some kind of Javascript magic (try to login in your browser with Javascript disabled)...

Also I'm sure that WWW::Mechanize::Firefox will work for such site (of course you must don't touch __VIEWSTATE and __VIEWSTATE).

Comment on Re: How to scraper ASP websites
Select or Download Code
Re^2: How to scraper ASP websites
by Anonymous Monk on Sep 05, 2012 at 20:07 UTC
    #!/usr/bin/perl use WWW::Mechanize; my $mech = WWW::Mechanize->new(); my $url = (' +--afbudsrejser/'); $mech->get($url); my $hsh={}; $links = $mech->find_all_links(url_regex=>qr/templates\/textPage\.aspx +\?id/i, text_regex=>qr/Afbudsrejser/i); foreach my $link (@$links) { $url = $link->url_abs(); $mech->get($url); my $content = $mech->content(); while ($content=~/tr class="bgrow1"><td>(.*?)<\/td><td cla +ss="countryValue">(.*?)<\/td><td class="destnameValue">(.*?)<\/td><td + class="hotelNameValue">(.*?)<\/td><td class="durationValue">(.*?)<\/ +td><td align="RIGHT" class="priceValue"><a target="_blank" href="(.*? +)">(.*?)<\/a><\/td>/gisxm) { $hsh->{'url'} = $6; $hsh->{'crap_id'} = ''; $hsh->{'date'} = $1; $hsh->{'country'} = $2; $hsh->{'destination'} = $3; $hsh->{'trip_type'} = $4; $hsh->{'trip_length'} = $5; $hsh->{'price'}=$7; print "$hsh->{'date'}, $hsh->{'country'}, $hsh->{'destina +tion'}, $hsh->{'trip_type'}, $hsh->{'trip_length'}, $hsh->{'price'}, +$hsh->{'crap_id'}, $hsh->{'url'}, $airport\n\n"; } }
    Please have a look in this code and also check the link and tell me how can I scrape the details from here. Regards

      This is less likely to get help than the node you messily copied and pasted it from.

      My recommendation is to think up an actual programming question relating to the code you are presenting. Something along the lines of:

      I'm trying to scrape a website. The following minimal code snippet is failing to produce the output I was expecting. I was expecting xyz, but instead I'm getting abc, plus an explosion of shards of solidified lava. I think the problem is with the pdq statement, but when I tried lmnop I got hot molten lava instead. How should I rewrite the thingamagizzer so that it would produce xyz rather than abc and hot lava?

      (Fill in the variables and problem description as necessary to reflect the current situation)


      Please check the code:
      use WWW::Mechanize; my $mech = WWW::Mechanize->new(); my @urls = (' +---afbudsrejser/'); foreach my $url (@urls) { $mech->get($url); my $hsh={}; $links = $mech->find_all_links(url_regex=>qr/templates\/textPage\. +aspx\?id/i, text_regex=>qr/Afbudsrejser/i); foreach my $link (@$links) { $url = $link->url_abs(); print "\n\n\n".$url."\n\n"; $mech->get($url); my $content = $mech->content(); print $content; while ($content=~/tr class="bgrow1"><td>(.*?)<\/td><td clas +s="countryValue">(.*?)<\/td><td class="destnameValue">(.*?)<\/td><td +class="hotelNameValue">(.*?)<\/td><td class="durationValue">(.*?)<\/t +d><td align="RIGHT" class="priceValue"><a target="_blank" href="(.*?) +">(.*?)<\/a><\/td>/gisxm) { $hsh->{'url'} = $6; $hsh->{'crap_id'} = ''; $hsh->{'date'} = $1; $hsh->{'country'} = $2; $hsh->{'destination'} = $3; $hsh->{'trip_type'} = $4; $hsh->{'trip_length'} = $5; $hsh->{'price'}=$7; print "$hsh->{'date'}, $hsh->{'country'}, $hsh->{'destinat +ion'}, $hsh->{'trip_type'}, $hsh->{'trip_length'}, $hsh->{'price'}, $ +hsh->{'crap_id'}, $hsh->{'url'}, $airport\n\n"; } } }
      The site is developed in asp , so the source contents are not exact HTML format. That's why I am facing lots of problem in fetching data from this site.

        When added to a regex, the x modifier tells the regex engine to ignore whitespace — that is, to omit the spaces, etc., in the regex from the pattern to be matched. So, if you are trying to match something like:

        <td class="countryValue"> # ^ note the space

        and your regex has an x modifier, you must specify the space(s) to be matched explicitly. For example:

        <td \s+ class="countryValue">

        That said, when I run your code with this fix applied:

        while ($content =~ m! tr \s+ class="bgrow1"> <td> (.*?) + # $1 </td> <td \s+ class="countryValue"> (.*?) + # $2 country </td> <td \s+ class="destnameValue"> (.*?) + # $3 destination </td> <td \s+ class="hotelNameValue"> (.*?) + # $4 </td> <td \s+ class="durationValue"> (.*?) + # $5 trip_length </td> <td \s+ align="RIGHT" \s+ class="priceValue"> <a \s+ target="_blank" \s+ href="(.*?)"> + # $6 url (.*?) + # $7 </a> </td> !gisxm)

        the regex still gets no matches, so there is more wrong than just the missing whitespace. (Or, there is more whitespace lurking in the target webpages than I have allowed for.) For further help from the monks, please follow the advice given above by davido, and reduce your problem to a minimal code snippet demonstrating the problem and complete with representative data.

        BTW, the variable $airport is accessed in the final print statement, but never initialized. You would have seen this if you had begun the script with

        use strict; use warnings;

        as Gangabass advised in Re: How to scraper ASP websites.

        Athanasius <°(((><contra mundum

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://991875]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (7)
As of 2014-09-21 20:57 GMT
Find Nodes?
    Voting Booth?

    How do you remember the number of days in each month?

    Results (176 votes), past polls