Re: LWP::UserAgent downloading zip files

[1nickt]

Hi, welcome to the Monastery.

Your code seems to have a number of problems.

• You are trying to fetch a file but are using a POST request. The server seems to expect a GET.
• You are using content() to apparently try to set a header (content-type?).
• You are trying to set the content-type to CSV but you are requesting a zip file.

Here's how I would do it (using the more modern, lighter-weight, simpler HTTP::Tiny to make the http request):

use strict; use warnings;
use HTTP::Tiny;
use IO::Uncompress::Unzip qw(unzip $UnzipError);

my $url = 'https://nse-india.com/content/historical/EQUITIES/2017/AUG/
+cm14AUG2017bhav.csv.zip';
my $filename = '/path/to/your/folder/cm14AUG2017bhav.csv';

my $res = HTTP::Tiny->new->get( $url );
die 'Download failed' if not $res->{'success'};
unzip \$res->{'content'}, $filename or die "unzip died: $UnzipError";

__END__

After you get your CSV file successfully onto disk, make sure to use a proper module for reading it, such as Text::CSV.

Hope this helps!

[Gtforce]

Funnily enough, the POST works just fine and gets me the file I'm looking for. I've seen other snippets of code all over the place and using POST to download files. I've got to sit down and look at the difference between these two. Thanks, 1nickt.

[1nickt]

You are welcome. Trust me, there is a difference between GET and POST requests. It is quite possible that a server could be configured to accept GET and POST requests for the same URL, but your program should not assume that.

As far as seeing snippets of code: Try to stick to the official documentation for the modules you use (and a good place to look for implementation examples is in the module distribution's tests) and build up your code following that so you know what you are doing. Copying chunks of code you spot in various places in the wild can be useful but should be accompanied with working hard to understand how the tools you are using really work.