Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

[SOLVED] Basic HTTP problems

by Bolemo (Acolyte)
on Feb 05, 2021 at 08:47 UTC ( #11127927=perlquestion: print w/replies, xml ) Need Help??

Bolemo has asked for the wisdom of the Perl Monks concerning the following question:

Greetings fellow monks,

I am unable to reproduce a very simple Python script into Perl. Basically, it just do a few HTTP requests. The Python script is successful and gets the data needed. The Perl script fails (no errors, but the HTTP sent does not convince the server; I do get the sid fine.). I compared the tcpdump produced by both scripts, and they are different (8 packets total for Python and 18 for Perl).

Python script:
#!/usr/bin/env python3 import requests # User vars ont_host = "192.168.18.1" ont_username = USERNAME ont_password = BASE64PASS # Build URL prefix ont_urlprefix = "http://" + ont_host # Start http session ont_session = requests.Session() # Get sid req = ont_session.post(ont_urlprefix + "/asp/GetRandCount.asp") sid = req.text[-32:] # Session authentication auth_data = {"UserName": ont_username, "PassWord": ont_password, "x.X_ +HW_Token": sid} req = ont_session.post(ont_urlprefix + "/login.cgi", data=auth_data) # Retrieve info req = ont_session.get(ont_urlprefix + "/html/amp/opticinfo/opticinfo.a +sp") print(req.text) # Logout ont_session.post(ont_urlprefix + "/logout.cgi?RequestFile=html/logout. +html") exit(0)
My Perl script attempt:
#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use HTTP::Request::Common; # User vars my $ont_host = "192.168.18.1"; my $ont_username = USERNAME; my $ont_password = BASE64PASS; # Build URL prefix my $url = "http://" . $ont_host; # Start http session my $ua = LWP::UserAgent->new(keep_alive => 1); # Get sid my $res = $ua->post($url . '/asp/GetRandCount.asp'); my $sid = $res->content; # Session authentication $res = $ua->post($url . '/login.cgi', [UserName => "$ont_username", Pa +ssWord => "$ont_password", 'x.X_HW_Token' => "$sid"]); # Retrieve info $res = $ua->get($url . '/html/amp/opticinfo/opticinfo.asp'); print($res->content); $res = $ua->get($url . '/logout.cgi?RequestFile=html/logout.html');
Python script tcpdump:
root@HERMES:~$ tcpdump -ttnnvvS 'tcp and net 192.168.18.0/24' tcpdump: listening on ethwan, link-type EN10MB (Ethernet), capture siz +e 262144 bytes 1612530239.807502 IP (tos 0x0, ttl 63, id 6338, offset 0, flags [DF], +proto TCP (6), length 52) 192.168.18.2.54798 > 192.168.18.1.80: Flags [S], cksum 0xb418 (cor +rect), seq 4024652913, win 64240, options [mss 1460,nop,nop,sackOK,no +p,wscale 7], length 0 1612530239.808064 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro +to TCP (6), length 52) 192.168.18.1.80 > 192.168.18.2.54798: Flags [S.], cksum 0x90f7 (co +rrect), seq 1577355000, ack 4024652914, win 14600, options [mss 1460, +nop,nop,sackOK,nop,wscale 3], length 0 1612530239.808814 IP (tos 0x0, ttl 63, id 6339, offset 0, flags [DF], +proto TCP (6), length 40) 192.168.18.2.54798 > 192.168.18.1.80: Flags [.], cksum 0x08d8 (cor +rect), seq 4024652914, ack 1577355001, win 502, length 0 1612530239.808876 IP (tos 0x0, ttl 63, id 6340, offset 0, flags [DF], +proto TCP (6), length 223) 192.168.18.2.54798 > 192.168.18.1.80: Flags [P.], cksum 0x7f17 (co +rrect), seq 4024652914:4024653097, ack 1577355001, win 502, length 18 +3: HTTP, length: 183 POST /asp/GetRandCount.asp HTTP/1.1 Host: 192.168.18.1 User-Agent: python-requests/2.25.1 Accept-Encoding: gzip, deflate Accept: */* Connection: keep-alive Content-Length: 0 1612530240.359236 IP (tos 0x0, ttl 64, id 26662, offset 0, flags [DF], + proto TCP (6), length 522) 192.168.18.1.80 > 192.168.18.2.54798: Flags [FP.], cksum 0xa1b0 (c +orrect), seq 1577390430:1577390912, ack 4024653989, win 2361, length +482: HTTP 1612530240.361735 IP (tos 0x0, ttl 63, id 6369, offset 0, flags [DF], +proto TCP (6), length 40) 192.168.18.2.54798 > 192.168.18.1.80: Flags [.], cksum 0x785d (cor +rect), seq 4024653989, ack 1577390913, win 501, length 0 1612530240.390570 IP (tos 0x0, ttl 63, id 6370, offset 0, flags [DF], +proto TCP (6), length 40) 192.168.18.2.54798 > 192.168.18.1.80: Flags [F.], cksum 0x785c (co +rrect), seq 4024653989, ack 1577390913, win 501, length 0 1612530240.391007 IP (tos 0x0, ttl 64, id 21626, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.1.80 > 192.168.18.2.54798: Flags [.], cksum 0x7118 (cor +rect), seq 1577390913, ack 4024653990, win 2361, length 0 ^C 8 packets captured 8 packets received by filter 0 packets dropped by kernel
Perl script tcpdump;
root@HERMES:~$ tcpdump -ttnnvvS 'tcp and net 192.168.18.0/24' tcpdump: listening on ethwan, link-type EN10MB (Ethernet), capture siz +e 262144 bytes 1612532228.397661 IP (tos 0x0, ttl 63, id 13929, offset 0, flags [DF], + proto TCP (6), length 52) 192.168.18.2.54818 > 192.168.18.1.80: Flags [S], cksum 0xe5dc (cor +rect), seq 3899533838, win 64240, options [mss 1460,nop,nop,sackOK,no +p,wscale 7], length 0 1612532228.398192 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro +to TCP (6), length 52) 192.168.18.1.80 > 192.168.18.2.54818: Flags [S.], cksum 0x6b28 (co +rrect), seq 3067119039, ack 3899533839, win 14600, options [mss 1460, +nop,nop,sackOK,nop,wscale 3], length 0 1612532228.398911 IP (tos 0x0, ttl 63, id 13930, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.2.54818 > 192.168.18.1.80: Flags [.], cksum 0xe308 (cor +rect), seq 3899533839, ack 3067119040, win 502, length 0 1612532228.591851 IP (tos 0x0, ttl 64, id 27663, offset 0, flags [DF], + proto TCP (6), length 507) 192.168.18.1.80 > 192.168.18.2.54818: Flags [FP.], cksum 0x52f1 (c +orrect), seq 3067119452:3067119919, ack 3899534340, win 2093, length +467: HTTP 1612532228.596974 IP (tos 0x0, ttl 63, id 13936, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.2.54818 > 192.168.18.1.80: Flags [F.], cksum 0xdda3 (co +rrect), seq 3899534340, ack 3067119920, win 501, length 0 1612532228.597411 IP (tos 0x0, ttl 64, id 27664, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.1.80 > 192.168.18.2.54818: Flags [.], cksum 0xd76b (cor +rect), seq 3067119920, ack 3899534341, win 2093, length 0 1612532228.598786 IP (tos 0x0, ttl 63, id 64322, offset 0, flags [DF], + proto TCP (6), length 52) 192.168.18.2.54820 > 192.168.18.1.80: Flags [S], cksum 0x7b93 (cor +rect), seq 4176643025, win 64240, options [mss 1460,nop,nop,sackOK,no +p,wscale 7], length 0 1612532228.599223 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro +to TCP (6), length 52) 192.168.18.1.80 > 192.168.18.2.54820: Flags [S.], cksum 0x7611 (co +rrect), seq 274904827, ack 4176643026, win 14600, options [mss 1460,n +op,nop,sackOK,nop,wscale 3], length 0 1612532228.599942 IP (tos 0x0, ttl 63, id 64323, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.2.54820 > 192.168.18.1.80: Flags [.], cksum 0xedf1 (cor +rect), seq 4176643026, ack 274904828, win 502, length 0 1612532228.604128 IP (tos 0x0, ttl 64, id 24356, offset 0, flags [DF], + proto TCP (6), length 507) 192.168.18.1.80 > 192.168.18.2.54820: Flags [FP.], cksum 0x6117 (c +orrect), seq 274905011:274905478, ack 4176643195, win 1825, length 46 +7: HTTP 1612532228.604628 IP (tos 0x0, ttl 64, id 24357, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.1.80 > 192.168.18.2.54820: Flags [R.], cksum 0xe58e (co +rrect), seq 274905479, ack 4176643195, win 1825, length 0 1612532228.611532 IP (tos 0x0, ttl 63, id 44109, offset 0, flags [DF], + proto TCP (6), length 52) 192.168.18.2.54822 > 192.168.18.1.80: Flags [S], cksum 0x3645 (cor +rect), seq 3432379770, win 64240, options [mss 1460,nop,nop,sackOK,no +p,wscale 7], length 0 1612532228.612001 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], pro +to TCP (6), length 52) 192.168.18.1.80 > 192.168.18.2.54822: Flags [S.], cksum 0xd064 (co +rrect), seq 2196415697, ack 3432379771, win 14600, options [mss 1460, +nop,nop,sackOK,nop,wscale 3], length 0 1612532228.612563 IP (tos 0x0, ttl 63, id 44110, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.2.54822 > 192.168.18.1.80: Flags [.], cksum 0x4845 (cor +rect), seq 3432379771, ack 2196415698, win 502, length 0 1612532228.618717 IP (tos 0x0, ttl 64, id 19728, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.1.80 > 192.168.18.2.54822: Flags [F.], cksum 0x3f59 (co +rrect), seq 2196416348, ack 3432379947, win 1959, length 0 1612532228.618842 IP (tos 0x0, ttl 63, id 44113, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.2.54822 > 192.168.18.1.80: Flags [.], cksum 0x450c (cor +rect), seq 3432379947, ack 2196416348, win 501, length 0 1612532228.618873 IP (tos 0x0, ttl 64, id 19729, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.1.80 > 192.168.18.2.54822: Flags [R.], cksum 0x3f55 (co +rrect), seq 2196416349, ack 3432379947, win 1959, length 0 1612532228.619373 IP (tos 0x0, ttl 64, id 54882, offset 0, flags [DF], + proto TCP (6), length 40) 192.168.18.1.80 > 192.168.18.2.54822: Flags [R], cksum 0x09cf (cor +rect), seq 2196416348, win 0, length 0 ^C 18 packets captured 18 packets received by filter 0 packets dropped by kernel

Any ideas on what I am doing wrong?

Replies are listed 'Best First'.
Re: Basic HTTP problems
by marto (Cardinal) on Feb 05, 2021 at 10:32 UTC
      Actually, you are right: there are 3 invisible control characters before the 32 sid string. So substring 32 is indeed different, However, that does not solve the issue...
Re: Basic HTTP problems
by hippo (Chancellor) on Feb 05, 2021 at 09:06 UTC
    sid = req.text[-32:]

    I don't speak python so don't know what this is doing but as there's no reference to "32" anywhere in your Perl code perhaps it is something which needs to be addressed.

    See the Basic Debugging Checklist for more info. Print data sent before every request (particularly that auth - seems an obvious place for something to go wrong) and $res->code for every response until you nail it down.


    🦛

      I believe that's a python-y array slice pulling off the last 32 characters of the (in this case) string so OP'd need something like $sid = substr( $req->content, -32 ) to get the equivalent value (although blindly pulling out the last 32 characters rather than actually parsing the returned value may be living dangerously regardless of the implementation language . . .).

      The cake is a lie.
      The cake is a lie.
      The cake is a lie.

      It corresponds to
      my $sid = substr $res->content, -32;
      map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
Re: Basic HTTP problems
by Bolemo (Acolyte) on Feb 05, 2021 at 09:42 UTC

    The sid returned is a 32 characters long string. I believe that Python might add a prefix so the programmer made sure only the returned 32 chars are taken. Anyway, with or without substring in Perl, the sid string is exactly the same (len of 32), and the result is the same. The issue is not there.

    $res->code is 200 for each request. There is no HTTP error.

    With the Python script, the server is sending the right information.
    With the Perl script, the server is acting like it is not authenticated.

    The problem seems to rely more on what an how $ua->post is actually sending the data. The tcpdump shows it is different, where it should be the same.

    Also, maybe each request is not considered in the same HTTP session in Perl?

      Consider inspecting the actual data that goes over the wire. The tcpdump output does not show the data.

      You want to print the request from both programs and compare that. Either create the POST request separately and dump it (using HTTP::Request::Common maybe):

      print $req->as_string;

      ... or add dumping of sent and received data:

      $ua->add_handler("request_send", sub { shift->dump; return }); $ua->add_handler("response_done", sub { shift->dump; return });

      I'm sure that Python also has ways to show the data that goes out and comes back in.

        The handlers are so precious advice, thank you!

        It allowed me to discover that the sid has hidden control chars before the 32 chars, so I fixed that.
        BUT that does not fix the issue.

        However after sending the first sid for authentication, the server is sending a cookie with another sid, so I need to probably deal with that. Also, I need to do the same handler trick to the python code, and that will definitively help to solve the problem.

        EDIT: it was the cookies. Adding a cookie jar did the trick!


        Thank you!
        When troubleshooting issues like this, always look at server logs also. Looks like this was an authentication-related issue due to the absence of cookies, and the server will have logged it as such. You really can't tell much from TCPDUMP.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11127927]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (1)
As of 2021-06-19 01:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (91 votes). Check out past polls.

    Notices?