Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Parallel::ForkManager - machine way tooo slow - need a quad-core machine

by Perlbeginner1 (Scribe)
on Oct 26, 2012 at 13:27 UTC ( #1001078=perlquestion: print w/ replies, xml ) Need Help??
Perlbeginner1 has asked for the wisdom of the Perl Monks concerning the following question:

g day monks

tryin to get this to work


do i need a 4 Core processor!?`


why - well i am guessing that my machine is bit too solow hmmm i added Parallel::ForkManager -


But if you do so - you need a blisteringly fast machine it uses lots of mb of memeory and a bunch of the core running opensuse 12.2 and perl 5.14.2


my machine is too slow - whats if i want to run this with lemme say 200 URLS (or websites) do i need a 4 Core processor!?`
#!/usr/bin/perl use WWW::Mechanize::Firefox; use strict; use warnings; #use Parallel::ForkManager; #my $fork = Parallel::ForkManager->new(25); #sites my @urls = qw(http://www.google.com http://www.yahoo.com http://www.cn +n.com http://www.bing.com http://www.nbcnews.com/... and thosusand others more ); #temp base dir my $temp = '/home/myspace/cgi-bin/'; for my $each (@urls){ #$fork->start and next; my $mech = WWW::Mechanize::Firefox->new(launch => 'firefox',create + => 1,); $each =~ /www\.(\w+)\.com/; my $name = $1; print "creating $name.png\n"; $mech->get($each); my $png = $mech->content_as_png(undef, undef, {width => 240,height + => 240}); my $dir_name = "$temp/$name".".png"; open my $file, ">", "$dir_name" or die "couldnt create $temp/$1.pn +g"; binmode $file; print {$file} $png; close $file; sleep 5; # sleep some to give it a little time to make sure things + compleated..... you'll need this more using fork #$fork->finish; } print "Well All done!\n"; #$fork->wait_all_children;

Comment on Parallel::ForkManager - machine way tooo slow - need a quad-core machine
Download Code
Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by flexvault (Parson) on Oct 26, 2012 at 14:14 UTC

    Perlbeginner1,

    Only you know what size machine you need. I have never used 'Parallel::ForkManager', but I have used 'fork' on a lot of different *nix machines. So I'll give you some ideas to look at:

    • What does 'top' tell you when you run (1..10) scripts.
    • How are you connected to the Internet. If you don't have enough bandwidth you can add 1024 cores and it won't help.
    • How often do you have to check the URLs? Too often and you may be bounced by the URL's admins.

    I've found that adding more than 4 forks per core starts slowing things down. But only you will know that after testing current performance.

    Now if you really want that new 4-core machine, and you want our premission, then I'm all for it :-)

    Good Luck!

    "Well done is better than well said." - Benjamin Franklin

Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by zentara (Archbishop) on Oct 26, 2012 at 15:01 UTC
    do i need a 4 Core processor!?`

    .... if you can't code it correctly, throw more processing power at it. :-) I'm mean a quad core is probably cheaper than what it would cost to hire a competent programmer to do it. :-)


    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by Old_Gray_Bear (Bishop) on Oct 26, 2012 at 16:36 UTC
    The real question is not "do I need..." but "can my budget afford ...". Parkingson's law applyies to CPUs as well as paperwork.

    ----
    I Go Back to Sleep, Now.

    OGB

      hello dear buddy many many thanks for the reply- guess that the quad core will bring much much more performance.
Re: Parallel::ForkManager - machine way tooo slow - need a quad-core machine
by Anonymous Monk on Oct 26, 2012 at 17:38 UTC
    You have no legitimate reason to scrape these sites, let alone thousands more, as per your post history. They have terms of use for a reason, you're not following them.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1001078]
Approved by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2014-12-28 18:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (182 votes), past polls