double underscore in http-header

[morgon]

Hi,

I am try to configure a modem that has a http-api.

When I capture what the browser-based GUI does I can see that it sends http-headers that look like this:

__RequestVerificationToken: acP92nszWSeNIAeL/Al9Rj24scVQXt+o

This is what I try to replicate in a perl-script.

What I do is this:

my $mech = WWW::Mechanize->new;
my $r = HTTP::Request->new("POST", $some_url);
$r->header("__RequestVerificationToken" => $key);
# set content and some other headers here
$mech->request($r);

But when I capture what the script sends I find that the leading underscores seem to get replaced with minuses, i.e.

--RequestVerificationToken: whatever

So my question is: Why is that and how can I turn it off so I can send headers with leading double-underscores?

Many thanks!

[kcott]

G'day morgon,

"... how can I turn it off so I can send headers with leading double-underscores?"

The short answer first. Somewhere near the start of your code you'll have:

use HTTP::Request;

Change that line to:

BEGIN {
    use HTTP::Request;
    $HTTP::Headers::TRANSLATE_UNDERSCORE = 0;
}

That's the fix. Read on for details ...

My test:

#!/usr/bin/env perl -l

use strict;
use warnings;

BEGIN {
    use HTTP::Request;
    $HTTP::Headers::TRANSLATE_UNDERSCORE = 0;
}

use Data::Dump;

my $r = HTTP::Request->new;

my $header = $r->header('__X_Y__', 1);

dd $r;

With just "use HTTP::Request;", instead of that BEGIN block, I get:

bless({
  _content => "",
  _headers => bless({ "--x-y--" => 1, "::std_case" => { "--x-y--" => "
+--X-Y--" } }, "HTTP::Headers"),
  _method  => undef,
  _uri     => undef,
}, "HTTP::Request")

With the BEGIN block, as shown, I get:

bless({
  _content => "",
  _headers => bless({ "::std_case" => { __x_y__ => "__X_Y__" }, "__x_y
+__" => 1 }, "HTTP::Headers"),
  _method  => undef,
  _uri     => undef,
}, "HTTP::Request")

"Why is that ..."

A little investigation, using the published documentation, provided the answer.

In HTTP::Request, under "$r->header( $field => $value )", you'll see:

"This is used to get/set header values and it is inherited from HTTP::Headers ..."

In HTTP::Headers, under "$h->header( $field => $value )", you'll see:

"To make the life easier for perl users who wants to avoid quoting before the => operator, you can use '_' as a replacement for '-' in header names."

You'll also see similar comments for other methods.

Following the Source link (at the top of the page), shows:

...
# The $TRANSLATE_UNDERSCORE variable controls whether '_' can be used
# as a replacement for '-' in header field names.
our $TRANSLATE_UNDERSCORE = 1 unless defined $TRANSLATE_UNDERSCORE;
...
$field =~ tr/_/-/ if $TRANSLATE_UNDERSCORE;
...

I searched the documentation of both HTTP::Request and HTTP::Headers for "TRANSLATE_UNDERSCORE": no matches were found. Perhaps you'd care to raise a bug report for a documentation enhancement.

[morgon]

Exhaustive treatment, many thanks.

I actually consider the default behavior a bug (convenience features that change data should be explicitely turned on, rather than turned off), but of course it's much too late to change that now I suppose.

[kcott]

I saw kennethk's reply after posting my own. I followed the first link provided and found a much easier fix, without the need for my BEGIN block, at the end of that thread (Re^2: Underscores changed to hyphens by HTTP::Proxy header filter):

$r->header(':__X_Y__', 1);

It also mentions why $TRANSLATE_UNDERSCORE has been undocumented, so there's probably little point in requesting a "documentation enhancement" such as I suggested.

There's no mention of using the colon prefix in the documentation for the header() methods of either HTTP::Request or HTTP::Headers. The latter could probably at least supply a link to the short section, right at the end of its doco, which explains it: "NON-CANONICALIZED FIELD NAMES".

"... (convenience features that change data should be explicitely turned on, rather than turned off) ..."

In general, I'd agree with that statement. In this specific case, I'm not so sure: that particular feature is, I suspect, exactly what the vast majority of users want; I've certainly never needed it to work differently. Having said that, I do think such features need to be well documented: in this instance, it probably just needs a short sentence with a L<...> added to the POD.

Update (fix): Oops! I left out a word which completely changed the intended meaning: s{I'm so sure}{I'm not so sure}.

[kennethk]

A lot of CGI.pm default behavior is undesirable, but is the way it is because it was a fidelitous implementation of the CGI spec. That's why they stopped shipping it as CORE.

[kennethk]

This is a result of underscore-to-hyphen translation as a default in CGI. It looks like there is a solution in Underscores changed to hyphens by HTTP::Proxy header filter which I found with HTTP::Request underscore (though not the first hit).

[morgon]

Ok, you probably want to reprimand me for not searching enough (fair enough) and the link you provide seems to answer my second question (how to turn this off) but does anybody know why this feature exists at all?

[kennethk]

Why underscores are forbidden in HTTP header names addresses it sort of, but really doesn't get into the initial design rationale. I didn't know about this "feature" until this thread.