Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re^3: multi threading DBI

by BrowserUk (Pope)
on Oct 30, 2013 at 07:55 UTC ( #1060282=note: print w/ replies, xml ) Need Help??


in reply to Re^2: multi threading DBI
in thread multi threading DBI

Thanks BrowserUK for the list of stuff, I'm still going thru that, haven't found some source code that I can use yet

If you post your single threaded code -- or a working, cut-down version of it -- I'll multi-thread it for you in the way I outlined; but I'm not going to try and re-create your code based upon your brief description.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re^3: multi threading DBI
Re^4: multi threading DBI
by Anonymous Monk on Oct 30, 2013 at 19:01 UTC
    Thanks for offering, I have cut out the main part here:
    my $dbh = DBI->connect("dbi:SQLite:dbname=event.db3", "", "", { RaiseE +rror => 1, AutoCommit => 0 }, ) or die $DBI::errstr; my @arr_uname = @{$dbh->selectcol_arrayref("select uname from event")} +; $dbh->disconnect(); foreach $uname(@arr_uname) { ### ### all the LWP::Simple and the parsing happens here ### ### [deleted] ### output result if ($result ne '') { $dbh->do("INSERT OR IGNORE INTO detail_info (id,uname,url,thumb, +txt) VALUES (?,?,?,?,?)", undef, $$result{'id'},$uname,$$result{'url' +},$$result{'thumb'},$$result{'txt'}); $dbh->do("INSERT OR IGNORE INTO score_info (id,uname,tid1,tid2,t +eam1,team2,) VALUES (?,?,?,?,?,?)", undef, $$result{'id'},$uname,$$re +sult{'tid1'},$$result{'tid2'},$$result{'team1'},$$result{'team2'}); $dbh->commit; } }

      Hm. That's not a "working, cut-down version" ...

      So this isn't working either, but it ought to be close once you've filled in the blanks:

      #! perl -slw use strict; use threads; use Threads::Queue; use LWP::Simple; use DBI; sub fetchNParse { my( $Qin, $Qout ) = shift; while( my $uname = $Qin->dequeue ) { my $content = get $uname; my @bits = $content =~ m[....]; $Qout->enqueue( join $;, $uname, @bits ); } $qOut->enqueue( undef ); } our $THREADS //= 16; my $Qunames = new Thread::Queue; my $Qrets = new Thread::Queue; threads->create( \&fetchNParse, $Qunames, $Qrets )->detach for 1 .. $T +HREADS; my $dbh = DBI->connect( ... ); my $unames = $dbh->selectcol_arrayref("select uname from event"); $Qunames->enqueue( @$unames ); for( 1 .. $THREADS ) { while( my $ret = $Qrets->dequeue ) { my( $uname, @bits ) = split $;, $ret. $dbh->do( insert stuff ); } } $dbh->disconnect;

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        Thank you, but I have a bunch of termination errors in "fetchNParse": Thread 1 terminated abnormally: Can't call method "enqueue" on an undefined value at test.cgi line 16. seems $Qout is causing problem?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1060282]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (8)
As of 2014-10-01 07:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (389 votes), past polls