perlquestion
hacker
<p>I've run into an interesting problem while testing a new piece of code I'm writing.
<p>During testing, I pointed my code at various dozens of websites; static content, dynamic content, images, pdfs, etc. and it all worked great. I was checking the remote end's Content-Type header and their Content-Length header using HEAD, to see if I should fetch it or not.
<p>Basically if the size reported in Content-Length was too large, I'd ignore the fetch.
<p><code>
my $req = HTTP::Request->new(HEAD => $url);
my $resp = $ua->request($req);
my $type = $resp->header('Content-Type');
my $content = $resp->content;
my $content_len = $resp->header('Content-Length');
</code>
<p>This was working great, until I realized that a lot of servers don't send a Content-Length header. DOH! Even sites serving static, flat text or html content, are not sending a Content-Length header.
<p>In the above snippet, I'm using HEAD, so as to avoid using a GET request on larger files, and then ignore the processing of them after I'd already fetched them.
<p>So I started trying to figure out a way to determine the length of the remote content, without actually fetching the content itself, and this is where I'm stuck.
<p>I could do this:
<p><code>
my $req = HTTP::Request->new(GET => $url);
my $content = $resp->content;
my $content_len = length($content);</code>
<p>But now I'm doing a GET, and if someone decides to point that to a 20-gigabyte file, or a DVD iso or something like that, it'll drown my bandwidth, and DDoS my tool for other users.
<p>Is there some other way to do this, without doing a full fetch of the remote resource?
<p><strong>Update</strong>: This sort-of works, but for sites without a Content-Length header, I do a double-hit, HEAD first, then GET second. Is there a better way?
<p><code>
my $req = HTTP::Request->new(HEAD => $pl_url);
my $resp = $ua->request($req);
my $type = $resp->header('Content-Type');
my $status_line = $resp->status_line;
my ($content, $content_len);
if ($resp->header('Content-Length')) {
$content_len = $resp->header('Content-Length');
} else {
$req = HTTP::Request->new(GET => $pl_url);
$resp = $ua->request($req);
$content = $resp->content;
$content_len = length($content);
}</code>