Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Use LWP::UserAgent to go to a password protected ASP page

by Puma_perl (Acolyte)
on Aug 09, 2005 at 18:46 UTC ( [id://482366] : perlquestion . print w/replies, xml ) Need Help??

Puma_perl has asked for the wisdom of the Perl Monks concerning the following question:

Dear perl gurus:

I am humbly asking you for your assistance. I was trying to use LWP::UserAgent to go to a password protected ASP page (my login/password works fine with IE browser). The code I have written is like this:

my $h = new HTTP::Headers Content_Base => '$url'; $h->authorization_basic($uname, $password); my $request = HTTP::Request->new( 'GET', $url, $h ) ; my $ua = LWP::UserAgent->new; my $response = $ua->request($request); ### get content of this HTML $content = $response->content; #print $content;

These lines work well with other web pages written in CGI but it did not work for a link written in ASP. Here is the error I got from the run:
"HTTP 401.2 - Unauthorized: Logon failed due to server configuration Internet Information Services".

The returned HTML also suggested:
"This is usually caused by a server-side script not sending the proper WWW-Authenticate header field. Using Active Server Pages scripting this is done by using the AddHeader method of the Response object to request that the client use a certain authentication method to access the resource."

So, it hinted that something might need to be done on the server side. However, since I don't have control/contract from the server hosting this link, is there any ways that I can solve this problem from my end? Any suggestions or tips will be highly appreciated!



Replies are listed 'Best First'.
Re: Use LWP::UserAgent to go to a password protected ASP page
by shiza (Hermit) on Aug 09, 2005 at 19:10 UTC
    The code below was actually taken from a script I wrote that accessed an IIS server. It has been modified a bit, but should work:
    #!/usr/bin/perl use strict; use LWP; use LWP::UserAgent; my $URI = ''; my $user = 'foo'; my $pass = 'bar'; # define user agent my $ua = LWP::UserAgent->new(); $ua->agent("USER/AGENT/IDENTIFICATION"); # make request my $request = HTTP::Request->new(GET => $URI); # authenticate $request->authorization_basic($user, $pass); # except response my $response = $ua->request($request); # get content of response my $content = $response->content(); # do whatever you need to do with the content here print $content; exit;
Re: Use LWP::UserAgent to go to a password protected ASP page
by davidrw (Prior) on Aug 09, 2005 at 19:14 UTC
    You can set the WWW-Authenticate (or any other) header in your HTTP::Headers object:
    $h->header('WWW-Authenticate' => 'foo');
    To see what values you need, try using the Live HTTP Headers plugin for firefox..

    (Side note: WWW::Mechanize may make some of whatever you're trying to do a lot easier)
Re: Use LWP::UserAgent to go to a password protected ASP page
by Puma_perl (Acolyte) on Aug 09, 2005 at 19:34 UTC
    Dear shiza and davidrw:

    I am thrilled by such a quick response from you two. Both solutions are running fine. However, they did not solve the problem, i.e still could not return the link of ASP page. Any other suggestions?

    Thanks in advance!


      You could put up a http proxy server using HTTP::Proxy. This would allow you to see all http headers (and the message if you wish) before they are sent.
      In IE6 change you LAN Settings to use the proxy. Add the host name and port you are using. If you use the example below you would use port 8080.
      use warnings; use strict; use Data::Dumper; use HTTP::Proxy; use HTTP::Proxy::HeaderFilter::simple; use IO::File; my $proxy = HTTP::Proxy->new(port => 8080); $proxy->host(undef); my $filter = HTTP::Proxy::HeaderFilter::simple->new(\&myfilter); $proxy->push_filter(request => $filter); $proxy->start(); sub myfilter { my ($this, $headers, $message) = @_; print '=' x 70, "\n"; print $headers->as_string(); STDOUT->flush(); }
      Using this proxy you can compare the http headers being send when you submit you page via a browser versus lwp useragent.
      Just use a packet sniffer.
      1. Download and install packet sniffer.
      2. Turn it on.
      3. Check the page successfully with Internet Explorer.
      4. Check the page unsuccessfully with the script.
      5. Turn off packet sniffer and inspect the two requests.

      The differences in the headers of the two HTTP request-responses will tell you what is going wrong.

      The one I use on a Windows client machine is called Smart Sniff.


      Andrew Tomazos  |  |
        Thanks for your tips Andrew! Where can I get Smart Sniff?


Re: Use LWP::UserAgent to go to a password protected ASP page
by sanPerl (Friar) on Aug 10, 2005 at 09:13 UTC
    PLease check if this can help you
    use strict;<br> use WWW::Mechanize;<br> my $username = "USER"; # Login<br> my $password = "PASSWORD"; # Password<br> my $url = ""; # Your URL Name<br> my $mech = WWW::Mechanize->new();<br> $mech->get($url);<br> ## Browse the HTML source of the ASP page to get these parameters <br> $mech->form("form_Name"); # The Login Form Name<br> $mech->field("txt_id", $username); # Username Texbox name<br> $mech->field("txt_pwd", $password); # Password Textbox name<br> $mech->click("cmd_ok"); # Button name<br> my $current_site = $mech->{uri}; #The $current_site would give you the address of next page...
    - SanPerl
Re: Use LWP::UserAgent to go to a password protected ASP page
by ranjan_jajodia (Monk) on Aug 10, 2005 at 05:09 UTC
    You can also use Charles while accessing the asp page to see what exactly goes to the http server and what comes back. This has helped me a number of times in pin pointing the errors.

    --Let Logic Rule--
      Dear Ranjan:

      I did download and run Charles to monitor my internet traffic. However, Charles only records internet activities through proxy server. When I start to run Charles, it automatically set my IE to use proxy server. The url that I have problem with are only accessible through "Automatically detect settings" with LAN settings for IE. Therefore, with Charles is running, I got the same error message from IE browser as I got from my perl script.

      So, my guess is that the web server for my problematic url doesn't recognize connection of client through proxy. Am I right? If so, what's the solution for this?

      Thanks again!


        It is going to be difficult then. I can think of two things (I may be wrong totally):

        1) See if adding cookies to the user agent solves it. You can use HTTP::Cookies
        2)Try to make your useragent appear as IE or Firefox. I am not sure if it legal. Sincere apologies if it is illegal. Don't try until you figure that out. Regarding how to do it it should be something like,
        $ua->agent("Mozilla/4.76 [en] (Windows NT 5.0; U)");

        That is all that comes to my mind. Monks can any one help us on this.


        --Let Logic Rule--