Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Bad scheme in LWP::UserAgent

by alini76 (Initiate)
on Oct 21, 2008 at 22:29 UTC ( #718603=perlquestion: print w/ replies, xml ) Need Help??
alini76 has asked for the wisdom of the Perl Monks concerning the following question:


I wrote the following piece of code:

#!/usr/bin/perl -w use lib qw( ..); use LWP::Debug qw(+ -cons); use LWP::UserAgent; use HTML::Form; use HTTP::Request::Common; my $url = ""; my $login = "anemail\"; my $passwd = "apassword"; my $ua = LWP::UserAgent->new; my $response = $ua->get($url); #login if (defined $response) { my @forms = HTML::Form->parse($response); my $form = @forms2; $form->value("EM",$login); $form->value("PW",$passwd); my $afterlogin = $ua->request( $form->click ); has a redirect immediately after login if($afterlogin->content =~ /<meta http-equiv='refresh' content='0;url= +(^'*)'/) { $redirurl = $1; print "redirect: $redirurl\n"; } my $categpages = $ua->get( "$redir +url" ); print "before foreach loop\n"; #extract all categories URLs foreach $1 ( $categpages->content =~ /<a title="^"*" href="http:\/\/(^ +"*)"/g ) { print "$1\n"; #my $uri = URI->new( $str => "$1", $scheme => "http" ); #my $req = HTTP::Request->new( GET => "http://$1" ); #$req->header( 'Accept' => 'text/html' ); ########### HERE IT CRASH ############# my $categResp = $ua->get( "https://$1" ); ########### HERE IT CRASH ############# print $categResp->content; #get category name $categResp->content =~ /<span style="text-transform: uppercase;">(^<*) +</; </span> my $categ = $1; #get products foreach $1 ($categResp->content =~ /(^<*) /g){ </span> my $title = $1; #get description $categResp->content =~ /(^<*)</g; </span> my $desc = $1; print "$categ | $title | $desc\n"; } } } else { #If an error occurs then $content will not be defined. print "Cannot get login page.\n"; }
Unfortunately I get the following output around the line marked in the code above:
LWP::UserAgent::request: () LWP::UserAgent::send_request: GET +applications/category/category_tlc.asp?CatId=35&name=Wireless-Network +ing LWP::UserAgent::_need_proxy: Not proxied LWP::UserAgent::request: Simple response: Not Implemented 501 Protocol scheme ' +/category_tlc.asp?catid=35&name=wireless-networking' is not supported +

Comment on Bad scheme in LWP::UserAgent
Select or Download Code
Re: Bad scheme in LWP::UserAgent
by Anonymous Monk on Oct 21, 2008 at 23:34 UTC
    • Surround your code with <C> code here </C> tags (<code> code here </code>)
    • Use use LWP::Debug qw(+);
    • I doubt that's all the code, are you sure you're not using a proxy?
      Its hard to believe LWP would ever think '' is a a scheme.
    • Do you have Crypt::SSLeay installed?
Re: Bad scheme in LWP::UserAgent
by thezip (Vicar) on Oct 21, 2008 at 23:40 UTC

    Is there any cookie traffic occurring? My guess is that the server is setting a client-side cookie after the authentication, and you're not mimicking that particular behavior.

    You could try deleting the applicable cookie from your browser, and then manually logging into the site. Then, inspect the cookie to see what behavior needs to be implemented.

    You might also use Wireshark to do some sniffing to see what traffic is being transmitted.

    BTW, you really ought not use HTML tags to format your code. It causes your output to be messed up like it is, and it causes problems for other viewers of your node. The correct way to do it is by using code tags, as in:

    <code> ... your code goes here ... etc. </code>
    Update: Changed comment re: formatting code with <pre></pre> tags

    What can be asserted without proof can be dismissed without proof. - Christopher Hitchens

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://718603]
Approved by GrandFather
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2015-01-25 15:19 GMT
Find Nodes?
    Voting Booth?

    My top resolution in 2015 is:

    Results (183 votes), past polls