Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

parallel web get with lwp::simple

by barathbr (Scribe)
on Jul 26, 2004 at 07:53 UTC ( #377373=perlquestion: print w/ replies, xml ) Need Help??
barathbr has asked for the wisdom of the Perl Monks concerning the following question:

Hello Folks,

My objective is to stress test a load balanced web server. I wrote a simple script using lwp::simple. The script in its current form is fetching the page. However, since it is initiating the connections seqentially I am actually not stressing the gateway. Could somebody point me in the right direction on how to go about spawning multiple instances. Code snippets would be really really appreciated.

Thanks a lot The current version is pasted below:
use LWP::Simple; use strict; my ($count, $max, $URL); unless (defined ($URL = shift)) { print "\n\nEnter the site address which you would like to access: +"; chomp($URL = <STDIN>); } $URL = "http://" . "$URL"; print "Site address is $URL \n"; unless (defined ($max = shift)) { print "Enter the number of times to hit the site: "; chomp($max = <STDIN>); } for ($count = 0; $count <= $max; $count++){ my $content; unless (defined ($content = get $URL)) { die "could not get $URL\n"; } if ($content =~ /Test1</font>/i) { print "."; } elsif ($content =~ /Test2</font>/i) { print "Fetched page from Server2 \n"; $count++; &result; } else { print "Page not retreived \n" } } sub result { print "page retrieved after $count tries \n"; exit; }

Edited by Chady -- replaced <pre> tags with <code> tags

Comment on parallel web get with lwp::simple
Download Code
Re: parallel web get with lwp::simple
by davido (Archbishop) on Jul 26, 2004 at 08:03 UTC
    To my knowlege, LWP::Simple doesn't internally provide for multiple instance running concurrently. The standard solution, from what I understand, is to fork, and then allow each forked process to independantly do its LWP::Simple get() request simultaneously.

    Dave

      Another very simple way to fork processes using LWP is by using Parallel::ForkManager. It's shockingly simple to make your code parallel in this way, here's how you could apply it to the loop in your code:
      use Parallel::ForkManager my $max_forks = 20; my $forkmanager = Parallel::ForkManager->new( $max_forks ); for ($count = 0; $count <= $max; $count++){ $forkmanager->start and next; my $content; unless (defined ($content = get $URL)) { die "could not get $URL\n"; } if ($content =~ /Test1/i) { print "."; } elsif ($content =~ /Test2/i) { print "Fetched page from Server2 \n"; $count++; &result; } else { print "Page not retreived \n" }; $forkmanager->finish; } $forkmanager->wait_all_children;
      This testing is also probably dependent on the maximum number of clients you support, this may not be high enough to stress your gateway either. Check on the MaxClients parameter in your httpd.conf for more information. Depending on what you're testing, it may be fruitful to use LWP to download larger files rather than making more requests.

      Best of luck. :)

Re: parallel web get with lwp::simple
by ccn (Vicar) on Jul 26, 2004 at 08:15 UTC
      Or use ab anyway, provided it's handy on a nearby machine, because after all, it speaks HTTP just like any webserver does. There's nothing Apache-specific to the tool as such.

      Makeshifts last the longest.

Re: parallel web get with lwp::simple
by davorg (Chancellor) on Jul 26, 2004 at 08:15 UTC
      thanks for the response guys, I guess the the lwp user agent would be just right for me to use.
Re: parallel web get with lwp::simple
by knoebi (Friar) on Jul 26, 2004 at 09:00 UTC
    Maybe you want to look at httperf? It does its job "httpd performance testing" very well, there you can als find an example.

    If you need to playback complicated requests, you could try out HTTP::Recorder to record your scripts (there was also a extension on the ml for recording httperf scripts, i never tested it tough).

    ciao, knoebi

Re: parallel web get with lwp::simple
by BrowserUk (Pope) on Jul 26, 2004 at 13:49 UTC

    You could take a look at Re: Simulating heavy traffic and see if it could be adapted to your needs.


    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon
Re: parallel web get with lwp::simple
by pg (Canon) on Jul 26, 2004 at 15:12 UTC
Re: parallel web get with lwp::simple
by mpeters (Chaplain) on Jul 26, 2004 at 15:53 UTC
Re: parallel web get with lwp::simple
by water (Chaplain) on Jul 26, 2004 at 22:54 UTC
    Check out Randal's column on POE for parallel link checking... easily adapted to other parallel web activities. Merlyn's written a series of Parallel Web columns over the years; interesting to read them as a bunch to see the evolution of his thinking and the evolution of useful CPAN support modules.
Re: parallel web get with lwp::simple
by sri (Vicar) on Jul 27, 2004 at 11:53 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://377373]
Approved by Corion
Front-paged by grinder
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (11)
As of 2014-08-20 18:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (121 votes), past polls