Win32::OLE VBA to Perl Translation Question about Internet Explorer

[sailortailorson]

For various reasons, I am trying to use Perl to drive Internet Explorer as a user agent for testing a webserver. I am using this in apposition (note: apposition, not opposition) to LWP::UserAgent for several reasons. I am trying to build up a toolkit of overlapping technologies. It has been suggested to me that this might be a good thing to do if there is a lot of Javascript involved (and there is), moreover I am finding that driving IE this way has gotten me around whatever problems are slowing me down in LWP::UserAgent. Sure, I should learn how to use LWP::UserAgent better, but this is a tactical situation.

Microsoft describes their interface for a method called Navigate:

object.Navigate( _
    url As String, _
    [Flags As Variant,] _
    [TargetFrameName As Variant,] _
    [PostData As Variant,] _
    [Headers As Variant])

I have used this joyfully so far, but only with the first param, which is the URL. So, now I now need to use the PostData and Headers, and am having only sorrow in using anything beyond the first param. I thought it might have been some kind of lost-in-translation from VBA to Perl. I looked at
perldoc Win32::OLE
which says:

$ex->Amethod("arg")->Bmethod->{'Property'} = "foo";
$ex->Cmethod(undef,undef,$Arg3);
$ex->Dmethod($RequiredArg1, {NamedArg1 => $Value1, NamedArg2 => $Value
+2});

The program below illustrates my problem. When I try to use the first method, I believe I am passing everything as a GET. This does work, and takes me to the socks search results to dwell there 5 seconds. The other two instances seem like they should work too, one via positional parameters and the other via named parameter, for 5 seconds each. Alas, they take me to our 404 handling page, where a pained-looking woman expresses her displeasure at my attempt.

use Win32::OLE qw(EVENTS);
use strict;

my $IE  = Win32::OLE->new("InternetExplorer.Application")
    || die "Could not start Internet Explorer.Application\n";
 

$IE->{visible} = 1;

$IE->Navigate("http://amos.shop.com/amos/cc/main/ccn_search/ccsyn/150/
+stid/6150608?search_form=&st=socks&sy=products&search_button.x=39&sea
+rch_button.y=12");
sleep 5;
$IE->Navigate("http://amos.shop.com/amos/cc/main/ccn_search/ccsyn/150/
+stid/6150608", undef, undef, "search_form=&st=socks&sy=products&searc
+h_button.x=39&search_button.y=12", undef);
sleep 5;;

$IE->Navigate("http://amos.shop.com/amos/cc/main/ccn_search/ccsyn/150/
+stid/6150608", {PostData=>"search_form=&st=socks&sy=products&search_b
+utton.x=39&search_button.y=12"});
sleep 5;

[Joost]

I don't know if this is helpful, but have you considered using Win32::IE::Mechanize? It's basically a port of LWP::UserAgent, but it uses Internet Explorer as the driver, so you can go all out on the JavaScript without having to learn yet another API.

Also available: Mozilla::Mechanize and Mozilla::Mechanize::GUITester (the latter appears to be dependent on X windows)

[sailortailorson]

Yes, some of this discussion went on in the Chatterbox before I took it to SOPW, and I think bart had mentioned Win32::IE::Mechanize. I did not know about it, nor did I know about the Mozilla packages you mention, and they might be very useful to us too, from the point of view of having overlapping technologies.

However, there is either a serious hole in my understanding of calling methods in Win32::OLE or a problem there. Either way, I need to understand and fix that hole. Thank you for your comment. I am making a note of the libraries you mentioned.

[ikegami]

All three version produce a GET request.

Some searching revealed

The post data specified by PostData is passed as a SAFEARRAY Data Type structure. The VARIANT should be of type VT_ARRAY and point to a SAFEARRAY Data Type. The SAFEARRAY Data Type should be of element type VT_UI1, dimension one, and have an element count equal to the number of bytes of post data.

Armed with that knowledge, I tried

$IE->Navigate("http://...", undef, undef, [ ord('A'), ord('='), ord('B
+') ]);

Yay! It produced a POST request! Unforunately, the data is kinda weird. I get the following 48 bytes of data (shown in hex).

03 00 00 00 00 00 00 00
41 00 00 00 00 00 00 00
03 00 00 00 00 00 00 00
3D 00 00 00 00 00 00 00
03 00 00 00 00 00 00 00
42 00 00 00 00 00 00 00

In python, the solution is to use buffer("...post data...") as the argument, but I don't know the Perl equivalent.

This page provides a workaround.

I wish I could solve this (using Win32::OLE::Variant?), but I need to go. I'll try tomorrow. [ Bingo! See my followup for details. ]

[sailortailorson]

Well, thanks for making something work, confirming that it is not a piece of cake, and for explaining how it works elsewhere and giving pointers to other resources.

This is great!

I would effuse more, but I have some late night testing to do. Thank you Ikegami.

[pemungkah]

Also, not Perl, but Selenium (http://www.openqa.org/selenium/) is pretty darn nice; it's a Javascript-based automation framework, and works well with Javascript-heavy sites. It can be driven from Perl with Test::WWW::Selenium.

[sailortailorson]

O.K. I'll make a note of it. Right now, however, I am looking for a Perl solution.

[ikegami]

[ This post is the a continuation of my previous work on the subject. ]

Solved!!

Right from the start, I was dismayed at the lack of documentation of PostData's type. (Saying Variant is totally meaningless.) After lots of research, I found how to format the data to IE's liking.

use strict;
use warnings;

use Win32::OLE             qw( );
use Win32::OLE::Variant; # qw( Variant VT_* );

my $IE = Win32::OLE->new("InternetExplorer.Application")
   or die("Could not start Internet Explorer.Application\n");
 
$IE->{visible} = 1;

my $url       = "http://amos.shop.com/amos/cc/main/ccn_search/ccsyn/15
+0/stid/6150608";
my $post_data = "search_form=&st=socks&sy=products&search_button.x=39&
+search_button.y=12";

$IE->Navigate(
   $url,
   undef,
   undef,
   Variant(VT_UI1, $post_data),
   "Content-Type: application/x-www-form-urlencoded\r\n",
);

# Alternative Syntax
# ==================
#
# $IE->Navigate($url, {
#    PostData => Variant(VT_UI1, $post_data),
#    Headers  => "Content-Type: application/x-www-form-urlencoded\r\n"
+,
# });

The CGI script I used for debugging returned:

REQUEST_METHOD=POST
CONTENT_LENGTH=71
CONTENT_TYPE=application/x-www-form-urlencoded

Content length: 71
Content (hex):  7365617263685f666f726d3d2673743d736f636b732673793d7072
+6f6475637473267365617263685f627574746f6e2e783d3339267365617263685f627
+574746f6e2e793d3132
Content:        search_form=&st=socks&sy=products&search_button.x=39&s
+earch_button.y=12

[sailortailorson]

Ikegami,

Thank you very, very much. You are a very kind monk. This will help other people beside me too.

Also, thanks to bart, tye, corion, jdporter and wfsp.

[pKai]

From the docs of the Navigate method:

Headers

Optional. A value that specifies additional HTTP headers to send to the server. These headers are added to the default Microsoft Internet Explorer headers. The headers can specify things like the action required of the server, the type of data being passed to the server, or a status code. This parameter is ignored if URL is not an HTTPURL.

This probably implies, that a

Content-Type: application/x-www-form-urlencoded

has to be passed to make it a POST.

This is also noted in the the article ikegami refered to as providing a workaround, which then says that "It will work on some sites, but on many others you'll get an error."

/me scratches head

[ikegami]

I tried that, but that's not the determinine factor. The request is a POST request if and only if PostData is an array ref.

It's still a good idea to include that header, however. Some sites rely on Content-Type being present. While Content-Length is set automatically, Content-Type is omitted unless specified.

[sailortailorson]

Point taken. Any knowledge or even opinion about the importance and impact of any piece of the request is important to me at this point.