Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Submit from browser works but LWP does not

by Plankton (Priest)
on Jul 12, 2005 at 19:49 UTC ( #474371=perlquestion: print w/ replies, xml ) Need Help??
Plankton has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I am having one hell of a time trying to get my LWP script to work with this aggravating web server using tons of javascript, jsp, and one of those awful oracle web/database solution Larry Ellison fools people into buying.

My script does this ...

. . . $UA->cookie_jar->add_cookie_header(); $resp = $UA->post ( $actionUrl, @ns_headers, 'Content_type' => 'application/x-www-form-urlencoded', 'Content' => [ <... a bunch of stuff ...> ] ); . . .

What that is suppose to do is create add some data to a database, but when I check the database there is nothing to be found.

When I submit the form via a browser everthing works fine, so I decided to do a tcpdump when I run my script and when I submit the form via the browser and look for differnces. And there are! When I run my script I get this tcpdump output:

POST /ditor/ditctl.do?method=save HTTP/1.1 Connection: keep-alive Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9 +,text/plain;q=0.8;,image/png,*/*;q=0.5 Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Encoding: gzip,deflate Accept-Language: en-us,en;q=0.5 Host: somejunk.js.jps.com:8888 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20 +050512 Red Hat/1.7.8-1.1.3.1 Content-Length: 336 Content-Type: application/x-www-form-urlencoded Cookie: JSESSIONID=1545454aae552555a48773717374620 Cookie2: $Version="1" Keep-Alive: 300 !pu` !pu` content=things&blah=balhahah&morejunk=&nonsense=xyz !pu` HTTP/1.1 100 Continue Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE Date: Tue, 12 Jul 2005 18:00:00 GMT !pu` !pu` HTTP/1/1 200 OK Date: Tue, 12 Jul 2005 18:00:00 GMT Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE Connection: Close Content-Type: text/html Transfer-Encoding:chunked <EOT>

But when I submit the form via the browser I get this output from tcpdump ...

POST /ditor/ditctl.do?method=save HTTP/1.1 Host: somejunk.js.jps.com:8888 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20 +050512 Red Hat/1.7.8-1.1.3.1 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9 +,text/plain;q=0.8;,image/png,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Referer: http://somejunk.js.jps.com:8888/ditor/ditctl.do?stuff=stuff Cookie: JSESSIONID=9989887adf879da987982jkd11 !pu` !pu` Content-Type: application/x-www-form-urlencoded Content-Length: 384 content=things&blah=balhahah&morejunk=&nonsense=xyz !pu` HTTP/1.1 100 Continue Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE Date: Tue, 12 Jul 2005 17:29:04 GMT !pu` !pu` HTTP/1/1 200 OK Date: Tue, 12 Jul 2005 17:29:04 GMT Server: Oracle9iAS (9.0.3.0.0) Containers for J2EE Content-Location: http://somejunk.js.jps.com:8888/ditor/Some.jps Connection: Keep-Alive Keep-Alive: tomeout=15, max=100 Content-Type: text/html Transfer-Encoding:chunked ... a bunch html ...

What is going on after the form is submitted? What's this Content-Location header? How can I make my script do what the browser does??? Thanks!

READMORE tags added by Arunbear

Comment on Submit from browser works but LWP does not
Select or Download Code
Re: Submit from browser works but LWP does not
by ikegami (Pope) on Jul 12, 2005 at 20:12 UTC

    Ignoring the "!pu`" and ignoring the missing blank lines in both outputs, the differences are:

    script: Connection: keep-alive
    brower: [nothing]
    This shouldn't matter.

    script: Content-Length: 336
    browser: Content-Length: 384
    You're not sending the same data. That could definitely cause this problem.

    script: Cookie: JSESSIONID=1545454aae552555a48773717374620
    browser: Cookie: JSESSIONID=9989887adf879da987982jkd11
    Differences are to be expected here, but there's no way to tell if the session is still valid. If the session is no longer valid, it could explain your problem. The difference in length of the session ids is odd, but not necessarily meaningful.

    script: Cookie2: $Version="1"
    brower: [nothing]
    This shouldn't matter.

    script: [nothing]
    browser: Referer: http://somejunk.js.jps.com:8888/ditor/ditctl.do?stuff=stuff
    This is could be important. Add this header to the script's request.

    There are differences in the order of the headers, but that shouldn't matter.

    You might have an easier time using WWW::Mechanize. It should handle just about everything for you. There are two downsides: 1) You have to install it, and 2) it won't help if the page requires JavaScript to submit.

Re: Submit from browser works but LWP does not
by BaldPenguin (Friar) on Jul 12, 2005 at 20:19 UTC
    Not that I can shed too much light, but the difference in Content-length is the Content-Location header, or at least they are the same length. This is most likely a custom header for the OraWeb server to track some workspace for the java work it uses

    Don
    WHITEPAGES.COM | INC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://474371]
Approved by Joost
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (11)
As of 2014-10-01 14:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (22 votes), past polls