LWP::UserAgent does not send Cookies

[CodeVixen]

I am trying to automate some web browsing for testing and I need to include cookies for a particular URL (I'm obfuscating the URLs intentionally - this code will not run as is). I have researched the use of LWP::UserAgent and HTTP::Cookies and it seems rather straightforward. I am directing my script to a proxy to grab the output via Fiddler (127.0.0.1:8888). When I view my request, I do not see the cookie information included. Thoughts?

#!c:\perl\bin -w

# Uses
use strict;
use HTTP::Request::Common qw(GET POST);
use HTTP::Cookies;
use LWP::UserAgent;

#Globals
my $response;

# URL
my $url = 'http://www.someurl.com';

# User Agent Object
my $ua = LWP::UserAgent->new;
$ua->proxy(['http', 'https'], 'http://127.0.0.1:8888');
$ua->agent('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KH
+TML, like Gecko) Chrome/37.0.2062.124 Safari/537.36');

# Set cookies
my $cookie_jar = HTTP::Cookies->new( autosave=>'true', ignore_discard=
+>'true' );
$cookie_jar->set_cookie(0,'__ibxt','dXRrXzU0MjRiMjc4NWQ2MjYxLjA5NjQyMj
+A0','/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'__uvst','5409ce8fb5a7a5787a65c95ca76ee4b9',
+'/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'__uvstr','d9b4984a7ad2bfee5fabe70a9c25819a'
+,'/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'__utmc','1','/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'__uvt','','/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'_bt_id.BT-22382467-3.d389','b5822f6f5d11758
+4.1411691130.1.1411691169.1411691130.','/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'_mkto_trk','id:273-CKQ-053&token:_mch.com-1
[phone]-21192','/','',80,0,0,86400,0); 
$cookie_jar->set_cookie(0,'access_token','4e92a468ae3d022d72d177f4b018
+333131a904ab','/','',80,0,0,86400,0); 
$cookie_jar->set_cookie(0,'jwplayer.volume','90','/','',80,0,0,86400,0
+); 
$cookie_jar->set_cookie(0,'ki_r','','/','',80,0,0,86400,0); 
$cookie_jar->set_cookie(0,'lastLoggedInDate','09%2F25%2F14','/','',80,
+0,0,86400,0);
$cookie_jar->set_cookie(0,'locale','en_US','/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'optimizelyEndUserId','oeu1411691125629r0.67
+0045307604596','/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'optimizelySegments','%7B%22306102099%22%3A%
+22gc%22%2C%22306155008%22%3A%22direct%22%2C%22306161004%22%3A%22false
+%22%7D','/','',80,0,0,86400,0);
$cookie_jar->set_cookie(0,'PHPSESSID','dtdfb3b4v8vm16uml1ro7lf38ktu6ot
+e','/','',80,0,0,86400,0);

$ua->cookie_jar($cookie_jar);

# Request
my $request = GET $url;
$request->referer('https://www.xxxx.com/blah/blah/');
#$cookie_jar->add_cookie_header( $request );
$response = $ua->request($request);

open OUTPUT, '>c:\output.txt' or die "Couldn't open file $!";
    
if ($response->is_success) {
    print $response->decoded_content;
    print OUTPUT $response->decoded_content;
}
else {
    print STDERR $response->status_line, "\n";
}

close OUTPUT;

[Corion]

Did you set the cookies as-is in your program or did you remove the hostname?

[CodeVixen]

I set them as is. I have browsed the site using Fiddler and I'm capturing the output from Fiddler. When I review the headers, this is what I see in the cookie header:

Cookie: optimizelyEndUserId=oeu1411691125629r0.670045307604596; access
+_token=4e92a468ae3d022d72d177f4b018333131a904ab; lastLoggedInDate=09%
+2F26%2F14; PHPSESSID=dtdfb3b4v8vm16uml1ro7lf38ktu6ote; jwplayer.volum
+e=90; locale=en_US; optimizelySegments=%7B%22306102099%22%3A%22gc%22%
+2C%22306155008%22%3A%22direct%22%2C%22306161004%22%3A%22false%22%7D; 
+optimizelyBuckets=%7B%7D; __ssid=883bcda6-f5e9-41de-b9fb-576453cc4c6c
+; optimizelyPendingLogEvents=%5B%5D; _mkto_trk=id:273-CKQ-053&token:_
+mch-.com-1411691127927-21192; __utma=1.1403134398.1411759895.14117598
+95.1411759895.1; __utmb=1.1.10.1411759895; __utmc=1; __utmz=1.1411759
+895.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __uvt=; bounce
+ClientVisit858v={"r":"https%3A%2F%2Fwww.com%2Fhome%2Fmy-courses%2F","
+lp":"https%3A%2F%2Fwww.com%2Fserver-2012-exam-70-410-tutorial%2F%23%2
+Flecture%2F1249780"}; uvts=282anRcS5hWaWQdg; _bt_id.BT-22382467-3.d38
+9=b5822f6f5d117584.1411691130.2.1411759896.1411691169.; _bt_ses.BT-22
+382467-3.d389=*; __ibxt=dXRrXzU0MjRiMjc4NWQ2MjYxLjA5NjQyMjA0; bounceC
+lientVisit858={"v":{"ever_logged_in":true,"language":"en-US","em_conv
+_bool":"1"},"fvt":1411759894,"vid":1411759894484985,"ao":0,"as":0,"vp
+v":1,"d":"d","lp":"https%3A%2F%2Fwww.com%2Fserver-2012-exam-70-410-tu
+torial%2F%23%2Flecture%2F1249780","r":"https%253A%252F%252Fwww.com%25
+2Fhome%252Fmy-courses%252F","cvt":1411759894,"gcr":45,"m":0,"lvt":141
+1759894}; ki_t=1411759896169%3B1411759896169%3B1411759896169%3B1%3B1;
+ ki_r=; __uvstr=d9b4984a7ad2bfee5fabe70a9c25819a; __uvst=5409ce8fb5a7
+a5787a65c95ca76ee4b9

So they appear to be $key/$value pairs and I'm attempting to reconstruct that. If i can find a way to save the cookie file and then just import it into the jar, that would work too. But when I tried that, the LWP module complained that the cookie wasn't formatted properly.

[Corion]

I think you will actually need to supply the proper hostname when you call ->add_cookie in your code. See HTTP::Cookies on the parameter order.

Personally, I would first set up an empty cookie jar, visit the site with (say) WWW::Mechanize, save the cookie jar to a file, and then reuse that file. This removes all the hassle of writing and debugging code just to get the correct cookies in place.