Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Wget works but not HTTP::Request

by perltastica (Initiate)
on Jan 14, 2020 at 00:36 UTC ( #11111384=perlquestion: print w/replies, xml ) Need Help??

perltastica has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, Thought this would be easy after got it going with wget + jq, but in perl it always returns a 403 error The wget example below works fine, although I have removed the actual API key ;) -- but it works no problems at all, returns a correctly parsed json object. As you can see the API key must be passed in the header as 'hibp-api-key: <API KEY>'

# This works fine wget --quiet -O- '' --header='hibp-api-key: + SAMPLEKEY' | jq -r '.[] | [.Name, .DataClasses[]] | @csv'

This perl code does NOT work, returns a 403 error.

#!/usr/bin/perl require LWP::UserAgent; require HTTP::Request; my $address = shift or die "Enter an email address to check\n"; my $APIKEY = 'SAMPLEKEY'; my $ua = LWP::UserAgent->new; my $url = ''; my $header = [ 'hibp-api-key' => $APIKEY ]; $request = HTTP::Request->new( 'GET', $url . $address . '?truncateResponse=false', $header ); print $request->as_string; my $resp = $ua->request($request); print $resp->as_string;

Output of that perl:

# ./ GET Hibp-Api-Key: <SAMPLEKEY> HTTP/1.1 403 Forbidden Cache-Control: max-age=10 Connection: close Date: Tue, 14 Jan 2020 00:33:08 GMT Server: cloudflare Content-Type: text/plain; charset=UTF-8 Expires: Tue, 14 Jan 2020 00:33:18 GMT CF-RAY: 554b844d8c61f4b2-YVR Client-Date: Tue, 14 Jan 2020 00:33:08 GMT Client-Peer: Client-Response-Num: 1 Client-SSL-Cert-Issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO + CA Limited/CN=COMODO ECC Domain Validation Secure Server CA 2 Client-SSL-Cert-Subject: /OU=Domain Control Validated/OU=PositiveSSL M +ulti-Domain/ Client-SSL-Cipher: ECDHE-ECDSA-AES128-GCM-SHA256 Client-SSL-Socket-Class: IO::Socket::SSL Client-Transfer-Encoding: chunked Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.c +om/cdn-cgi/beacon/expect-ct" Set-Cookie: __cfduid=da914544ec647170750f29b53abed85cf1578961988; expi +res=Thu, 13-Feb-20 00:33:08 GMT; path=/;; +HttpOnly; SameSite=Lax Strict-Transport-Security: max-age=31536000; includeSubDomains; preloa +d X-Content-Type-Options: nosniff error code: 1010

Seems like the header is there as it should be in the output, so I have no idea why this wouldn't work... Please enlighten me monks! Thank-you JS

Replies are listed 'Best First'.
Re: Wget works but not HTTP::Request
by hippo (Chancellor) on Jan 14, 2020 at 10:29 UTC

    The documentation is really quite explicit about the reason for returning a 403 code:

    Forbidden — no user agent has been specified in the request

    You will need to address that.

Re: Wget works but not HTTP::Request
by tobyink (Canon) on Jan 14, 2020 at 10:27 UTC

    Maybe they're blocking LWP::UserAgent? Try setting a different User-Agent header.

Re: Wget works but not HTTP::Request (wget --quiet)
by Anonymous Monk on Jan 14, 2020 at 01:42 UTC
    # This works fine wget --quiet

    Why are you using --quiet ?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11111384]
Approved by kcott
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (6)
As of 2020-01-27 18:00 GMT
Find Nodes?
    Voting Booth?