Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^4: multi threading DBI

by Anonymous Monk
on Oct 30, 2013 at 19:01 UTC ( #1060416=note: print w/ replies, xml ) Need Help??


in reply to Re^3: multi threading DBI
in thread multi threading DBI

Thanks for offering, I have cut out the main part here:

my $dbh = DBI->connect("dbi:SQLite:dbname=event.db3", "", "", { RaiseE +rror => 1, AutoCommit => 0 }, ) or die $DBI::errstr; my @arr_uname = @{$dbh->selectcol_arrayref("select uname from event")} +; $dbh->disconnect(); foreach $uname(@arr_uname) { ### ### all the LWP::Simple and the parsing happens here ### ### [deleted] ### output result if ($result ne '') { $dbh->do("INSERT OR IGNORE INTO detail_info (id,uname,url,thumb, +txt) VALUES (?,?,?,?,?)", undef, $$result{'id'},$uname,$$result{'url' +},$$result{'thumb'},$$result{'txt'}); $dbh->do("INSERT OR IGNORE INTO score_info (id,uname,tid1,tid2,t +eam1,team2,) VALUES (?,?,?,?,?,?)", undef, $$result{'id'},$uname,$$re +sult{'tid1'},$$result{'tid2'},$$result{'team1'},$$result{'team2'}); $dbh->commit; } }


Comment on Re^4: multi threading DBI
Download Code
Re^5: multi threading DBI
by BrowserUk (Pope) on Oct 30, 2013 at 20:08 UTC

    Hm. That's not a "working, cut-down version" ...

    So this isn't working either, but it ought to be close once you've filled in the blanks:

    #! perl -slw use strict; use threads; use Threads::Queue; use LWP::Simple; use DBI; sub fetchNParse { my( $Qin, $Qout ) = shift; while( my $uname = $Qin->dequeue ) { my $content = get $uname; my @bits = $content =~ m[....]; $Qout->enqueue( join $;, $uname, @bits ); } $qOut->enqueue( undef ); } our $THREADS //= 16; my $Qunames = new Thread::Queue; my $Qrets = new Thread::Queue; threads->create( \&fetchNParse, $Qunames, $Qrets )->detach for 1 .. $T +HREADS; my $dbh = DBI->connect( ... ); my $unames = $dbh->selectcol_arrayref("select uname from event"); $Qunames->enqueue( @$unames ); for( 1 .. $THREADS ) { while( my $ret = $Qrets->dequeue ) { my( $uname, @bits ) = split $;, $ret. $dbh->do( insert stuff ); } } $dbh->disconnect;

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Thank you, but I have a bunch of termination errors in "fetchNParse": Thread 1 terminated abnormally: Can't call method "enqueue" on an undefined value at test.cgi line 16. seems $Qout is causing problem?

        I'm not psychic. Post the (exact) code that is giving you the errors.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1060416]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (16)
As of 2014-07-25 14:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (172 votes), past polls