downloading images from a webpage

Aldebaran has asked for the wisdom of the Perl Monks concerning the following question:

Greetings good people

I've been trying everything I can to download the images off a webpage. Although the content is ideological, I can assure you that I am not a fascist.

I redirected the output of this to get a list of the images: $ cat hitler1.pl

 
#!/usr/bin/perl -w
use strict;

use WWW::Mechanize;

my $domain = 'http://www.nobeliefs.com/nazis.htm';
my $m = WWW::Mechanize->new;
$m->get( $domain);

my @list = $m->dump_images();
print "@list \n";
$
[download]

my list doesn't do anything there (why not?) and I redirected output to text1.txt

then I try to download these images with getstore and got only html as output and then this gives me jpg of zero size. What gives? $ cat hitler5.pl

 

#!/usr/bin/perl -w
use strict;


use LWP::Simple;  

open FILE, "text1.txt" or die $!;

my $data;
my $url;
my $text;
my %params;

while (<FILE>)
{
$text = $_;

$url = 'http://www.nobeliefs.com/nazis/' . $text;
$data = LWP::Simple::get $params{URL};
$text =~ s#images/##;
print "$url\n";
print "$text\n";
open (FH, ">$text");
binmode (FH);
print FH $data;
close (FH);

}

$
[download]

Thx for your comment and happy easter.

Comment on downloading images from a webpage Select or Download Code

Replies are listed 'Best First'.
Re: downloading images from a webpage by blakew (Monk) on Apr 06, 2012 at 21:47 UTC
You don't call getstore you call get.	[reply]
Re^2: downloading images from a webpage by Aldebaran (Curate) on Apr 07, 2012 at 18:33 UTC
that was my first tack, and I thought I was on the right track, but what I ended up with using getstore() was files that kind of thought they were jpg's and kind of thought they were html docs. Here's the script I used: `#!/usr/bin/perl -w use strict; use LWP::Simple; open FILE, "text1.txt" or die $!; my $url; my $text; while (<FILE>) { $text = $_; $url = 'http://www.nobeliefs.com/nazis/' . $text; $text =~ s#images/##; print "$url\n"; print "$text\n"; getstore($url, $text) or die "Can't download: $@\n"; }` [download] an ls command shows question marks: `$ ls ... prayingHitler.jpg? PraysingCelebration.jpg? priests-salute.jpg? received.jpg reichchurch.gif? ...` [download] and when I open up a jpg it looks like this: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http:/ +/www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html> <head> <meta http-equiv="Content-type" content="text/html; charset=utf-8"> <title>Website Moved</title> <style type="text/css"> .statusBox { width: 80px; } .fb { width:43%; float:left; text-align:center; margin:5px 20px 5px 20px; padding:20px 0 20px 0px; background:#eef8fd; height:110px; border:solid 1px #dff4fe; } .fb2 { width:43%; float:right; text-align:center; margin:5px 20px 5px 20px; padding:20px 0 20px 0px; background:#eef8fd; height:110px; border:solid 1px #dff4fe; ... [download] I think the trick might be to find a way to define $params such that this works, but I haven't been able to do that yet. (I only get errors) `my $data = LWP::Simple::get $params{URL}; my $filename = "image.jpg"; open (FH, ">$filename"); binmode (FH); print FH $data; close (FH);` [download]	[reply] [d/l] [select]
Re^3: downloading images from a webpage by Aldebaran (Curate) on Apr 09, 2012 at 05:44 UTC
I finally got output, but it looks like a kid did it. I'd like to polish it up and be able to have a script that uses WWW::Mechanize more effectively. `#!/usr/bin/perl -w use strict; use LWP::Simple; open FILE, "text1.txt" or die $!; my $url; my $text; while (<FILE>) { $text = $_; $text =~ s/\s+//; $url = 'http://www.nobeliefs.com/' . $text; print qq[ '$url' ]; $text =~ s#images/##; print "$text\n"; getstore($url, $text) or die "Can't download: $@\n"; }` [download] How would I use chomp instead of $text =~ s/\s+//;? Nothing I tried worked. My failure with WWW::Mechanize was almost complete. The most I could get it to do was dump the names of the images to STDOUT. How could I re-write this to avoid all the nonsense with saving to a file which I then have to read? The syntax for $mech->images is: Lists all the images on the current page. Each image is a WWW::Mechanize::Image object. In list context, returns a list of all images. In scalar context, returns an array reference of all images. I tried a dozen different things, but I don't get why this is not list context: `#!/usr/bin/perl -w use strict; use WWW::Mechanize; open FILE, "text2.txt" or die $!; my $domain = 'http://www.nobeliefs.com/nazis.htm'; my $m = WWW::Mechanize->new; $m->get( $domain); my @list = $m->images(); print "@list \n"; #$m->text(); #$m->content( format => 'text2.txt' ); #print FILE $m; close FILE;` [download]	[reply] [d/l] [select]
Re^4: downloading images from a webpage by blakew (Monk) on Apr 09, 2012 at 15:54 UTC
Re^4: downloading images from a webpage by Aldebaran (Curate) on Apr 14, 2012 at 00:56 UTC


There's more than one way to do things
	PerlMonks