Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Bad scheme in LWP::UserAgent

by alini76 (Initiate)
on Oct 21, 2008 at 22:29 UTC ( #718603=perlquestion: print w/replies, xml ) Need Help??
alini76 has asked for the wisdom of the Perl Monks concerning the following question:


I wrote the following piece of code:

#!/usr/bin/perl -w use lib qw( ..); use LWP::Debug qw(+ -cons); use LWP::UserAgent; use HTML::Form; use HTTP::Request::Common; my $url = ""; my $login = "anemail\"; my $passwd = "apassword"; my $ua = LWP::UserAgent->new; my $response = $ua->get($url); #login if (defined $response) { my @forms = HTML::Form->parse($response); my $form = @forms2; $form->value("EM",$login); $form->value("PW",$passwd); my $afterlogin = $ua->request( $form->click ); has a redirect immediately after login if($afterlogin->content =~ /<meta http-equiv='refresh' content='0;url= +(^'*)'/) { $redirurl = $1; print "redirect: $redirurl\n"; } my $categpages = $ua->get( "$redir +url" ); print "before foreach loop\n"; #extract all categories URLs foreach $1 ( $categpages->content =~ /<a title="^"*" href="http:\/\/(^ +"*)"/g ) { print "$1\n"; #my $uri = URI->new( $str => "$1", $scheme => "http" ); #my $req = HTTP::Request->new( GET => "http://$1" ); #$req->header( 'Accept' => 'text/html' ); ########### HERE IT CRASH ############# my $categResp = $ua->get( "https://$1" ); ########### HERE IT CRASH ############# print $categResp->content; #get category name $categResp->content =~ /<span style="text-transform: uppercase;">(^<*) +</; </span> my $categ = $1; #get products foreach $1 ($categResp->content =~ /(^<*) /g){ </span> my $title = $1; #get description $categResp->content =~ /(^<*)</g; </span> my $desc = $1; print "$categ | $title | $desc\n"; } } } else { #If an error occurs then $content will not be defined. print "Cannot get login page.\n"; }
Unfortunately I get the following output around the line marked in the code above:
LWP::UserAgent::request: () LWP::UserAgent::send_request: GET +applications/category/category_tlc.asp?CatId=35&name=Wireless-Network +ing LWP::UserAgent::_need_proxy: Not proxied LWP::UserAgent::request: Simple response: Not Implemented 501 Protocol scheme ' +/category_tlc.asp?catid=35&name=wireless-networking' is not supported +

Replies are listed 'Best First'.
Re: Bad scheme in LWP::UserAgent
by thezip (Vicar) on Oct 21, 2008 at 23:40 UTC

    Is there any cookie traffic occurring? My guess is that the server is setting a client-side cookie after the authentication, and you're not mimicking that particular behavior.

    You could try deleting the applicable cookie from your browser, and then manually logging into the site. Then, inspect the cookie to see what behavior needs to be implemented.

    You might also use Wireshark to do some sniffing to see what traffic is being transmitted.

    BTW, you really ought not use HTML tags to format your code. It causes your output to be messed up like it is, and it causes problems for other viewers of your node. The correct way to do it is by using code tags, as in:

    <code> ... your code goes here ... etc. </code>
    Update: Changed comment re: formatting code with <pre></pre> tags

    What can be asserted without proof can be dismissed without proof. - Christopher Hitchens
Re: Bad scheme in LWP::UserAgent
by Anonymous Monk on Oct 21, 2008 at 23:34 UTC
    • Surround your code with <C> code here </C> tags (<code> code here </code>)
    • Use use LWP::Debug qw(+);
    • I doubt that's all the code, are you sure you're not using a proxy?
      Its hard to believe LWP would ever think '' is a a scheme.
    • Do you have Crypt::SSLeay installed?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://718603]
Approved by GrandFather
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (8)
As of 2017-06-29 17:16 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (673 votes). Check out past polls.