Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

How to remove HTTP Keep-Alive 300 header from LWP::UserAgent request

by Veltro (Hermit)
on Apr 24, 2018 at 13:23 UTC ( #1213475=perlquestion: print w/replies, xml ) Need Help??

Veltro has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I'd like to remove the Keep-Alive 300 header from the request but I can't figure out how.

I am instantiating the LWP::UserAgent with the intention to use HTTP 1.1 and keeping the connection alive as follows:

my $ua = LWP::UserAgent->new(keep_alive => 1, send_te => 0) ;

send_te => 0 removes the TE header and the header information that I want to send contains tags like:
my @ns_headers = ( 'ACCEPT' => '..', 'ACCEPT_ENCODING' => '...', 'ACCEPT-LANGUAGE => '..', 'UPGRADE_INSECURE_REQUESTS' => '1', 'USER-AGENT' => '...', ) ;
I have tried several methods creating the request but each and one of them sends the Keep-Alive 300 header.

my $response = $ua->get( $link, @ns_headers ) ;

my $response = $ua->request(GET $link) ; # This does not use @ns_headers
my $getReq = HTTP::Request->new( GET => $link, HTTP::Headers->new( @ns +_headers ) ) ; my $response = $ua->request( $getReq ) ;
Tryig to get rid of it by specifying this in the @ns_headers does not work:

'KEEP-ALIVE' => undef,

Anyone knows how to do this? Thanks

Replies are listed 'Best First'.
Re: How to remove HTTP Keep-Alive 300 header from LWP::UserAgent request
by jeffenstein (Friar) on Apr 26, 2018 at 09:55 UTC

    Since you've asked for keep-alives in the constructor: my $ua = LWP::UserAgent->new(keep_alive => 1, send_te => 0) ; then LWP will add both the Connection: keep-alive and the Keep-Alive: 300, as the standard specifies

    According to the LWP::UserAgent docs, the argument to keep_alive is the maximum number of connections to cache, so if you do need keep-alives, then 1 is probably not the right number.

      Sorry but that does not sound right to me. In think that the implementation is incorrect. There is no standard that says the Keep-Alive: 300 header is mandatory. E.g. from the 5 Internet Browsers I have looked at, none of them sends this header.

      https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive

      quote: The Keep-Alive general header allows the sender to hint about how the connection and may be used to set a timeout and a maximum amount of requests.

      https://tools.ietf.org/html/rfc2068#section-19.7.1.1

      quote: When the Keep-Alive connection-token has been transmitted with a request or a response, a Keep-Alive header field MAY also be included

      Another thing is that to me this header looks parameterized in form in both request and response (according to each of the specifications mentioned, including the one you have provided). Thus not of the form 'Header : value'

      Regards the keep_alive argument I think you are mixing it up. The keep_alive argument is in the 'new' of the LWP::UserAgent and is not related to the "Keep-Alive: 300" header. Quote from LWP::UserAgent: "The keep_alive value is passed on as the total_capacity for the connection cache".

        The actual code that adds the header is in Net::HTTP::Methods, at line 180 in the current version.

        if ($self->keep_alive) { if ($peer_ver eq "1.0") { # from looking at Netscape's headers push(@h2, "Keep-Alive: 300"); unshift(@connection, "Keep-Alive"); } }

        So, I would guess that Net::HTTP believes your connection is http/1.0, and is therefore sending the http/1.0 header. If it's not, maybe you can make a bug report against the module?

        E.g. from the 5 Internet Browsers I have looked at, none of them sends this header.

        Are any of these browsers from 1995? First came keep-alive, then came the RFC, thats history for you

Re: How to remove HTTP Keep-Alive 300 header from LWP::UserAgent request
by Anonymous Monk on Apr 27, 2018 at 02:18 UTC

    Hi

    Well, since you responded to jeffenstein, here you go Veltro

    LWP header -> Re: LWP is there any way to get "real" outgoing headers?

    #!/usr/bin/perl -- use strict; use warnings; use LWP; use Data::Dump(); sub LWP::Protocol::http::SocketMethods::format_request { package LWP::Protocol::http::SocketMethods; my( $socket, $method, $fullpath, @h ) = @_; #~ my $req_buf = $socket->Net::HTTP::Methods::format_request($meth +od, $fullpath, @h, 'Keep-Alive',666); ## fail, adds another header my $req_buf = $socket->Net::HTTP::Methods::format_request($method, + $fullpath, @h ); $req_buf =~ s{Keep-Alive: 300}{Keep-Alive: 30}g; ## win, removes e +xisting header Data::Dump::dd( $req_buf ); return $req_buf; } LWP::UserAgent->new( keep_alive=>1 )->get( q{http://127.0.0.1:80/} )-> +dump; __END__ "GET / HTTP/1.1\r\nTE: deflate,gzip;q=0.3\r\nKeep-Alive: 30\r\nConnect +ion: Keep-Alive, TE\r\nHost: 127.0.0.1:80\r\nUser- Agent: libwww-perl/6.15\r\n\r\n" HTTP/0.9 200 Assumed OK Client-Date: Tue, 24 Apr 2018 23:25:20 GMT Client-Peer: 127.0.0.1:80 Client-Response-Num: 1 Echo: GET / HTTP/1.1\r Echo: TE: deflate,gzip;q=0.3\r Echo: Keep-Alive: 30\r Echo: Connection: Keep-Alive, TE\r Echo: Host: 127.0.0.1:80\r Echo: User-Agent: libwww-perl/6.15\r Echo: \r\n
      Thanks for showing how to override format_request. Very useful for introspection. I have also posted my reply to jeffenstein with different solution here for more details
        Neat, thanks. I hope I can count on your vote, and I hope you enjoy the pastrami on rye.
Re: How to remove HTTP Keep-Alive 300 header from LWP::UserAgent request
by Anonymous Monk on Apr 24, 2018 at 21:56 UTC

    Why?

    You tell LWP you want keep_alive but then you dont want the header? Those goals are contradictory

      I'll clarify. There are two different things:
      1. Connection: keep-alive
      2. Keep-Alive 300
      I don't want 2

        I'll clarify. There are two different things: 1. Connection: keep-alive 2. Keep-Alive 300 I don't want 2

        :)

        That doesn't explain why "Keep-Alive: 300" need to be removed.

        Maybe you can teach LWP something about HTTP.

        So please explain why, thats my price for showing you how.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1213475]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (3)
As of 2021-11-29 18:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?