Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re^5: Async DNS with LWP

by BrowserUk (Pope)
on Oct 05, 2010 at 22:34 UTC ( #863696=note: print w/replies, xml ) Need Help??

in reply to Re^4: Async DNS with LWP
in thread Async DNS with LWP

Not sure I see what you mean by your point about Coro.

Coro isn't threaded! (Despite the blatant lies in the documentation!).

It is cooperative task-switching--like Windows 3.1--which means that if one of your coro instances is busy, none of the others will do anything at all until it either: finishes; goes into a wait for IO; or yields.

It also means that regardless of how many cores you have, it will only ever use one of them.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^6: Async DNS with LWP
by jc (Acolyte) on Oct 06, 2010 at 21:39 UTC


    so at this point I'm now thinking:

    * LWP and Mechanize are nice toys to make a quick proof of concept of a real web crawler but in practise not useful for anything more than low bandwidth automated tasks.

    * With AnyEvent::HTTP and Coro you can make a proof of concept which performs better but you're still not quite there

    * In order to build a real performing parallel web crawler that makes the best use of network resources performing parallel asynchronous DNS and parallel HTTP requests then I either need to use Perl's bloated thread model and directly use Perl's UDP and TCP interface or I need to give up on Perl and go ahead and build this in C

    It really seems a shame that there are so many Perl modules dedicated to crawling tasks and yet none of them really have proved up to the job of being the back end of a high performance crawler that makes best use of network resources. The fact that people have dedicated so much time to making such modules would seem to suggest that many Perl users have an interest in web crawling. I'm wondering (new to PerlMonks, please help me out here) if there's anything we can do to set up a team of Perl developers that can improve the situation and develop easy to use Perl modules that are up to the job?

      I either need to use Perl's bloated thread model and directly use Perl's UDP and TCP interface or

      If you can afford to pay for sufficient bandwidth to allow for serious web-crawling, then affording a box with sufficient memory to start enough threads to saturate that bandwidth, will be the least of your concerns.

      I have 4GB of ram and I can run hundreds, even thousands of threads without getting anywhere near to running out of memory. So the "bloat" of the ithreads model is neither here nor there.

      Personally, I'd forget about asynchronous DNS. I'd stick an LWP::Parallel::UserAgent instance in one thread per core, and and watch them totally saturate my bandwidth. No Matter how fat a pipe I can afford.

      This trivial demo is running 100 threads, each with a parallel user agent, on this box as I type, in just 1/2 GB of memory:

      #! perl -slw use strict; use threads ( stack_size => 4096 ); use Thread::Queue; use LWP::Parallel; sub worker { my $tid = threads->tid; my( $Qin, $Qout ) = @_; my $ua = LWP::Parallel::UserAgent->new; print "Thread: $tid ready to go"; while( defined( my $url = $Qin->dequeue ) ) { print $url; } } our $T //= 4; my( $Qin, $Qout ) = map Thread::Queue->new(), 1 ..2; my @workers = map async( \&worker, $Qin, $Qout ), 1 .. $T; sleep 100; ## Read your urls and feed the Q here.

      Not that running that many threads on my 4 cores, would be an effective strategy, but even if you're running on one of IBMs $250,000, 256-core, 1024 thread monsters, affording 5GB of memory so that you can run one parallel useragent on each core, is the least of your worries, but the 25 lines of code above will scale to it. AS-IS.

      And that's what you get with threads. Simplicity and scalability.

      But that bit is easy.

      The complicated part of a high throughput webcrawler is not saturating the bandwidth. The complicated parts are:

      1. respecting robots.txts;
      2. an efficient url extractor that can deal with not just html hyperlinks, but all the other kinds of absolute and relative links you need to discover and follow.
      3. scheduling urls so that you don't hit up any particular server with thousands of (concurrent or serial) requests in an effective DoS attack.
      4. having enough disk bandwidth to allow you to write the stuff out without holding everything up.
      5. having an efficient indexing/digesting mechanism to stop you chasing your tail in loops of cross-referenced pages.

        And that means indexing (digesting) the content, not just the urls, because the same content can hide behind many different urls.

      6. And the indexing/digesting mechanism will have to be disk-based--for both persistance and size-- but must not impact the to-disk throughput of your workers.

      Yes. Saturating your bandwidth is trivial, it is the rest that is hard. Worrying about asynchronous DNS at this point is premature and pointless.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.


        actually I'd already solved most of these problems and that's why I was hammering on with saturating bandwidth. I have a list of 90,000,000 registered .com domains and my breadth first policy ensures that I'm not hitting the same server repeatedly. In fact, I'm crawling so many domains that I don't think even the most rigorous analysis of server logs would flag that my browsing looks like an intensive crawl.

        I use the URI module to resolve all URIs to absolute URIs and so this is not a problem. No duplicate absolute URIs are added to my to do list.

        My disk I/O policy is also pretty stable. I cache results in memory in a hash table. When that hash table has grown to its user defined memory limit then the in memory hash table is used to update values in an on disk version of the hash table using MLDBM. The theory behind this strategy is that the I/O bottleneck is reduced by lumping writing into one big set of writes.

        As it stands I'm trying to build a list of links from the front pages of all registered .com domains. I don't want to have to wait 3 years to do this (my current estimate based on a serial version based on LWP). If I could reduce this to 3 days or 3 hours I would be a very happy chappy.

        Now, I've taken a quick look at your example code but notice that you are not actually doing anything with LWP. You've consumed 1/2 GB with only 100 threads that are not performing any TCP communication. The moment you start doing TCP the TCP/IP stack of whatever OS you are using will start consuming even more memory resources. (Note that the small webbook I am developing on has about 1/2 GB to work with).

        Now using a Async::HTTP event driven model I've managed to get an average of about 50 concurrent TCP connections working (on Windows 7, I'm still not sure about the internal limitations of Windows 7 TCP and how I can tweak these, the model has changed from XP and the old registry keys are no longer valid). Note that I am consuming no where near the amount of memory you quoted for your 100 threads.

        Now, I'm willing to accept that the bloat in your example is probably from loading fat LWP instances and not from the process like thread model used by Perl. So I'll experiment with Threads and Sockets before giving up on Perl for this job.

        In conclusion, I'm wondering at this point how difficult it would be to reimplement WWW::Mechanize as a thread safe wrapper around optimized C code. My current experiments seem to suggest that this is what it would take to put Perl in the running for a simple interface with a high performance back end. As it stands, LWP is fat, and Mechanize is even fatter. The easier the implementation the fatter and slower the web crawling application seems to be the current trend with Perl web crawling modules.
        LWP::Parallel::UserAgent doesn't work with newer versions of LWP, or did that change?

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://863696]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (5)
As of 2017-02-22 18:13 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (334 votes). Check out past polls.