Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Bad scheme in LWP::UserAgent

by alini76 (Initiate)
on Oct 21, 2008 at 22:29 UTC ( #718603=perlquestion: print w/ replies, xml ) Need Help??
alini76 has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I wrote the following piece of code:

#!/usr/bin/perl -w use lib qw( ..); use LWP::Debug qw(+ -cons); use LWP::UserAgent; use HTML::Form; use HTTP::Request::Common; my $url = "https://www.infoteldistributors.com/secure/b2blogin.asp"; my $login = "anemail\@yahoo.com"; my $passwd = "apassword"; my $ua = LWP::UserAgent->new; my $response = $ua->get($url); #login if (defined $response) { my @forms = HTML::Form->parse($response); my $form = @forms2; $form->value("EM",$login); $form->value("PW",$passwd); my $afterlogin = $ua->request( $form->click ); #infoteldistributors.com has a redirect immediately after login if($afterlogin->content =~ /<meta http-equiv='refresh' content='0;url= +(^'*)'/) { $redirurl = $1; print "redirect: $redirurl\n"; } my $categpages = $ua->get( "https://www.infoteldistributors.com/$redir +url" ); print "before foreach loop\n"; #extract all categories URLs foreach $1 ( $categpages->content =~ /<a title="^"*" href="http:\/\/(^ +"*)"/g ) { print "$1\n"; #my $uri = URI->new( $str => "$1", $scheme => "http" ); #my $req = HTTP::Request->new( GET => "http://$1" ); #$req->header( 'Accept' => 'text/html' ); ########### HERE IT CRASH ############# my $categResp = $ua->get( "https://$1" ); ########### HERE IT CRASH ############# print $categResp->content; #get category name $categResp->content =~ /<span style="text-transform: uppercase;">(^<*) +</; </span> my $categ = $1; #get products foreach $1 ($categResp->content =~ /(^<*) /g){ </span> my $title = $1; #get description $categResp->content =~ /(^<*)</g; </span> my $desc = $1; print "$categ | $title | $desc\n"; } } } else { #If an error occurs then $content will not be defined. print "Cannot get login page.\n"; }
Unfortunately I get the following output around the line marked in the code above:
LWP::UserAgent::request: () LWP::UserAgent::send_request: GET https://www.infoteldistributors.com/ +applications/category/category_tlc.asp?CatId=35&name=Wireless-Network +ing LWP::UserAgent::_need_proxy: Not proxied LWP::UserAgent::request: Simple response: Not Implemented 501 Protocol scheme 'www.infoteldistributors.com/applications/category +/category_tlc.asp?catid=35&name=wireless-networking' is not supported +

Comment on Bad scheme in LWP::UserAgent
Select or Download Code
Re: Bad scheme in LWP::UserAgent
by Anonymous Monk on Oct 21, 2008 at 23:34 UTC
    • Surround your code with <C> code here </C> tags (<code> code here </code>)
    • Use use LWP::Debug qw(+);
    • I doubt that's all the code, are you sure you're not using a proxy?
      Its hard to believe LWP would ever think 'www.infoteldistributors.com/applications/category/category_tlc.asp?catid=35&name=wireless-networking' is a a scheme.
    • Do you have Crypt::SSLeay installed?
Re: Bad scheme in LWP::UserAgent
by thezip (Vicar) on Oct 21, 2008 at 23:40 UTC

    Is there any cookie traffic occurring? My guess is that the server is setting a client-side cookie after the authentication, and you're not mimicking that particular behavior.

    You could try deleting the applicable cookie from your browser, and then manually logging into the site. Then, inspect the cookie to see what behavior needs to be implemented.

    You might also use Wireshark to do some sniffing to see what traffic is being transmitted.

    BTW, you really ought not use HTML tags to format your code. It causes your output to be messed up like it is, and it causes problems for other viewers of your node. The correct way to do it is by using code tags, as in:

    <code> ... your code goes here ... etc. </code>
    Update: Changed comment re: formatting code with <pre></pre> tags

    What can be asserted without proof can be dismissed without proof. - Christopher Hitchens

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://718603]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (8)
As of 2014-12-19 02:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (70 votes), past polls