Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re^2: multi threading DBI

by BrowserUk (Patriarch)
on Oct 29, 2013 at 19:13 UTC ( [id://1060203]=note: print w/replies, xml ) Need Help??


in reply to Re: multi threading DBI
in thread multi threading DBI

All of this is completely orthogonal to the OP's question.

Thus -- even if it is essentially correct, which is debatable -- makes it totally irrelevant drivel in the context of the question asked.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^3: multi threading DBI
by marinersk (Priest) on Oct 29, 2013 at 19:28 UTC
    As the OP didn't specify, we are left to presume what it is about the process that is deemed to be too slow. Experience does strongly suggest that -- given the scraper is in the same linear thread as the DBI operation upon which it must wait before proceeding to the next scrape -- the part which is troubling the OP is almost certainly the speed with which it proceeds from one scrape operation to the next.

    Thus BrowserUK is likely correct that relief is likely to be found in stacking up the updates guided by a multi-threaded scraper to a single-threaded DBI queue -- it moves the most likely point of contention and delay outside the loop which constrains the operation which is deemed to be, in the OP's words, "too slow".

    However, it is only prudent to point out that if this assumption were to prove to be incorrect, and the user is hoping to improve the speed of the SQLite updates and not merely improve the scraper hang time, that the scraper::queue::DBI model might not produce the gains the OP was hoping for. It's a corner case, almost certainly, but the corner is not imaginary.

    I agree that, absent information to the contrary, it is a side note and not the meat of the response. But I don't think sundialsvc4's response is necessarily out-of-band.

      I definitely agree with what you said, marinersk. From the context, I would think that what is slow is collecting the data on the Internet, therefore BrowserUK's solution to multithread the web scraping part is certainly the first solution that came to my mind and I believe most probably the right one. Having said that, sundialsvc4's response also makes an interesting point which is worth taking into consideration (and can very easily be tested with minimal code changes).

      The point is that we would really need profiling of the program to figure out what is really slow.

      Let's see what the OP says for sure, but given "scrape a sports news site and then enter details into sqlite", but:

      1. fetching pages across the internet;
      2. parsing html to extract data;
      3. loading the extracts into a DB;

      Which bit(s) are likely to be the bottleneck? Which bit(s) are likely to be helped by multi-threading?


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        You summarized my diatribe nicely.  :-)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1060203]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (5)
As of 2024-03-29 12:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found