Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
Monks,

I return from my quest with success!! It took me a couple of days and many ponderings on Monks' responses to my previous postings, but I now have code that will enter into my selected user/pass protected https webpage and cruise around internally to do almost whatever I want or need. I thought I would post and share as there were no definitive answers during my quest on how to get this done.

Comments for improvement are welcome.

#!/usr/bin/perl use strict; use Crypt::SSLeay; use LWP::UserAgent; use WWW::Mechanize; use HTTP::Cookies; use HTTP::Request; ##The first two lines of code are used because ##I was sending my results to a web browser. use CGI::Carp qw/ fatalsToBrowser /; print "Content-type: text/html\n\n"; my $user = 'MyUser'; my $pass = 'MyPass'; my $base_address = 'https://wwws.stocksRus.com'; my $output = ''; my $dv_data = ''; my $webpage = ''; my $url = 'https://wwws.stocksRus.com/cgi-bin/LogIn'; my $agent = WWW::Mechanize->new( autocheck => 1); # Set up cookie jar $agent->cookie_jar(HTTP::Cookies->new); ##Go through login to get appropriate cookies ##so that I can then move onto ##the following pages with my cookie jar properly loaded. $agent->get($url); die $agent->response->status_line unless $agent->success; #In order to login properly it was necessary to send #a hidden input called DV_DATA along with user/pass that #appeared to be time based and was assigned on entry to #the main login page. I extracted this value and #assigned it as one of the inputs for the form. #This is a critical component of the login. $output = $agent->content; for ($output =~ /name=\"DV_DATA\" type=\"hidden\" VALUE=\"(.*?)\">/smi +){ $dv_data = $1; } $agent->form_name( 'LoginFormName' ); $agent->set_fields( USERID => $user, PASSWORD => $pass, DV_DATA => $dv_data ); $agent -> submit(); ###END Login ##I now have the appropriate cookies and may continue. #On the main page there is some javascript that codes for #the pull down menus. I could not access these using #WWW::Mechanize and one of them had the link I needed. #To solve, I logged into the webpage in a normal browser #and found the url of the page that I wanted and plugged #it in below. ##Accessing this page directly does not work as there ##is some Authorization procedure that requires going ##through login first. $agent->get('https://wwws.stocksRus.com/cgi-bin/Quotes'); die $agent->response->status_line unless $agent->success; #This page has a form that holds a symbol that I can #submit on to get data on that symbol. $agent -> field ('ticker','AMGN'); $agent -> submit(); #At first the webpage wouldn't come up in my web browser #(which is where the output from this run was sent), #because I didn't have the base URL. All the addresses #were similar to "/cgi-bin/menu/equity". #I substitute the base URL in below. $webpage = $agent -> content; $webpage =~ s/\"\//\"$base_address\//smig; $webpage =~ s/\'\//\"$base_address\//smig; #Prints out my webpage of interest. print $webpage; exit;
Chris Herold

In reply to WWW::Mechanize to Access HTTPS with Cookies by cdherold

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (5)
As of 2024-03-28 12:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found