LWP Byte Limit

[vbrtrmn]

Hey All, I'm dealing with some large XML files, which I only need the first few lines. I don't really want to get the whole file, is there a nice way, using LWP to only get the first say 5k? I'm using the really basic HTTP::Request...

use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
   $ua->agent("Faux Browser/1.0");
my $req = HTTP::Request->new(GET=>$url);
   $req->content_type('application/x-www-form-urlencoded');
   $req->content('query=libwww-perl&mode=dist');
my $res = $ua->request($req);
if ($res->is_success) {
   # do stuff
} else {
   # error out
}

[Aristotle]

As I posted in LWP: Downloading First 2KB of an HTML File:

my $data;
my $res = $ua->request( $req, sub { $data = shift; die }, 5120 );

Note that $res->content will be empty.

[Fletch]

Theoretically you should be able to set the Content-Range header to the first chunk you want, but keep in mind that servers aren't required to implement handling this header (and may not be able to for things like dynamically generated documents). See section 14.16 of RFC 2616.

[merlyn]

You can set a max_size during the new for your useragent. The default value of undef sucks down the entire response.