Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Parallel::ForkManager - machine way tooo slow - need a quad-core machine

by Perlbeginner1 (Scribe)
on Oct 26, 2012 at 13:27 UTC ( #1001078=perlquestion: print w/ replies, xml ) Need Help??
Perlbeginner1 has asked for the wisdom of the Perl Monks concerning the following question:

g day monks

tryin to get this to work


do i need a 4 Core processor!?`


why - well i am guessing that my machine is bit too solow hmmm i added Parallel::ForkManager -


But if you do so - you need a blisteringly fast machine it uses lots of mb of memeory and a bunch of the core running opensuse 12.2 and perl 5.14.2


my machine is too slow - whats if i want to run this with lemme say 200 URLS (or websites) do i need a 4 Core processor!?`
#!/usr/bin/perl use WWW::Mechanize::Firefox; use strict; use warnings; #use Parallel::ForkManager; #my $fork = Parallel::ForkManager->new(25); #sites my @urls = qw(http://www.google.com http://www.yahoo.com http://www.cn +n.com http://www.bing.com http://www.nbcnews.com/... and thosusand others more ); #temp base dir my $temp = '/home/myspace/cgi-bin/'; for my $each (@urls){ #$fork->start and next; my $mech = WWW::Mechanize::Firefox->new(launch => 'firefox',create + => 1,); $each =~ /www\.(\w+)\.com/; my $name = $1; print "creating $name.png\n"; $mech->get($each); my $png = $mech->content_as_png(undef, undef, {width => 240,height + => 240}); my $dir_name = "$temp/$name".".png"; open my $file, ">", "$dir_name" or die "couldnt create $temp/$1.pn +g"; binmode $file; print {$file} $png; close $file; sleep 5; # sleep some to give it a little time to make sure things + compleated..... you'll need this more using fork #$fork->finish; } print "Well All done!\n"; #$fork->wait_all_children;

Comment on Parallel::ForkManager - machine way tooo slow - need a quad-core machine
Download Code
Replies are listed 'Best First'.
Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by flexvault (Prior) on Oct 26, 2012 at 14:14 UTC

    Perlbeginner1,

    Only you know what size machine you need. I have never used 'Parallel::ForkManager', but I have used 'fork' on a lot of different *nix machines. So I'll give you some ideas to look at:

    • What does 'top' tell you when you run (1..10) scripts.
    • How are you connected to the Internet. If you don't have enough bandwidth you can add 1024 cores and it won't help.
    • How often do you have to check the URLs? Too often and you may be bounced by the URL's admins.

    I've found that adding more than 4 forks per core starts slowing things down. But only you will know that after testing current performance.

    Now if you really want that new 4-core machine, and you want our premission, then I'm all for it :-)

    Good Luck!

    "Well done is better than well said." - Benjamin Franklin

Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by zentara (Archbishop) on Oct 26, 2012 at 15:01 UTC
    do i need a 4 Core processor!?`

    .... if you can't code it correctly, throw more processing power at it. :-) I'm mean a quad core is probably cheaper than what it would cost to hire a competent programmer to do it. :-)


    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by Old_Gray_Bear (Bishop) on Oct 26, 2012 at 16:36 UTC
    The real question is not "do I need..." but "can my budget afford ...". Parkingson's law applyies to CPUs as well as paperwork.

    ----
    I Go Back to Sleep, Now.

    OGB

      hello dear buddy many many thanks for the reply- guess that the quad core will bring much much more performance.
Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by Anonymous Monk on Oct 26, 2012 at 17:38 UTC
    You have no legitimate reason to scrape these sites, let alone thousands more, as per your post history. They have terms of use for a reason, you're not following them.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1001078]
Approved by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (14)
As of 2015-07-31 07:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (275 votes), past polls