Parallel::ForkManager - machine way tooo slow - need a quad-core machine

[Perlbeginner1]

g day monks

tryin to get this to work


do i need a 4 Core processor!?`


why - well i am guessing that my machine is bit too solow hmmm i added Parallel::ForkManager -


But if you do so - you need a blisteringly fast machine it uses lots of mb of memeory and a bunch of the core running opensuse 12.2 and perl 5.14.2


my machine is too slow - whats if i want to run this with lemme say 200 URLS (or websites) do i need a 4 Core processor!?`

#!/usr/bin/perl 
 
use WWW::Mechanize::Firefox; 
use strict; 
use warnings; 
#use Parallel::ForkManager; 
 
 
 
#my $fork = Parallel::ForkManager->new(25); 
 
#sites 
my @urls = qw(http://www.google.com http://www.yahoo.com http://www.cn
+n.com http://www.bing.com http://www.nbcnews.com/... and thosusand 

others more ); 
#temp base dir 
my $temp = '/home/myspace/cgi-bin/'; 
 
for my $each (@urls){ 
    #$fork->start and next; 
    my $mech = WWW::Mechanize::Firefox->new(launch => 'firefox',create
+ => 1,); 
     
    $each =~ /www\.(\w+)\.com/; 
    my $name = $1; 
    print "creating $name.png\n"; 
     
     
    $mech->get($each); 
    my $png = $mech->content_as_png(undef, undef, {width => 240,height
+ => 240}); 
     
    my $dir_name = "$temp/$name".".png"; 
    open my $file, ">", "$dir_name" or die "couldnt create $temp/$1.pn
+g"; 
    binmode $file; 
    print {$file} $png; 
     
    close $file; 
    sleep 5; # sleep some to give it a little time to make sure things
+ compleated..... you'll need this more using fork 
     
    #$fork->finish; 
} 
print "Well All done!\n"; 
#$fork->wait_all_children;

[flexvault]

Perlbeginner1,

Only you know what size machine you need. I have never used 'Parallel::ForkManager', but I have used 'fork' on a lot of different *nix machines. So I'll give you some ideas to look at:

• What does 'top' tell you when you run (1..10) scripts.
• How are you connected to the Internet. If you don't have enough bandwidth you can add 1024 cores and it won't help.
• How often do you have to check the URLs? Too often and you may be bounced by the URL's admins.

I've found that adding more than 4 forks per core starts slowing things down. But only you will know that after testing current performance.

Now if you really want that new 4-core machine, and you want our premission, then I'm all for it :-)

Good Luck!

[zentara]

do i need a 4 Core processor!?`

.... if you can't code it correctly, throw more processing power at it. :-) I'm mean a quad core is probably cheaper than what it would cost to hire a competent programmer to do it. :-)

[Old_Gray_Bear]

The real question is not "do I need..." but "can my budget afford ...". Parkingson's law applyies to CPUs as well as paperwork.

[Perlbeginner1]

hello dear buddy many many thanks for the reply- guess that the quad core will bring much much more performance.

[Anonymous Monk]

You have no legitimate reason to scrape these sites, let alone thousands more, as per your post history. They have terms of use for a reason, you're not following them.