Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Thread::Queue and WWW::Curl

by Anonymous Monk
on Sep 24, 2012 at 14:24 UTC ( #995397=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks I am taking baby steps in creating a Boss/Worker threading I have liberally used the example at Re: Perl Threads Boss/Worker Example to create the following:

my ($accessToken, $appId, $userId) = (shift, shift, shift); use threads; use Thread::Queue; use WWW::Curl::Easy; use WWW::Curl::Form; my $threadLimit = 1; my $testUsers=listTestUsersFull($accessToken, $appId); my $q = Thread::Queue->new(); # A new empty queue # Worker threads my @thr = map { threads->create(sub { while (my @item = $q->dequeue(2)) { # assuming undef isn't a valid value and so can be a marker. last unless defined $item[0]; last unless defined $item[1]; deleteUser($item[1], $appId, $item[0]); } } , $appId )->detach(); } 1..$threadLimit; # Send work to the threads @testUsers = @testUsers[1..5]; foreach (keys %{$testUsers}){ $q->enqueue($userId, $$testUsers{$userId}->{access_token}); } # send markers. $q->enqueue(undef) for (1..$threadLimit); # terminate. $_->join() for @thr;

the deleteUser subroutine calls another which contains

use WWW::Curl::Easy; use WWW::Curl::Form;
, thus sending a simple form. My problem is the subroutine with those WWW::Curl:: calls returns at the line that posts the form:  my $response = $curl->perform; however when called via a non-threaded subroutine, everything just works. The documentation for WWW::Curl states threading is hopefully possible.

I am hoping to gain some wisdom on my problem, I thank you!

Comment on Thread::Queue and WWW::Curl
Select or Download Code
Re: Thread::Queue and WWW::Curl
by Anonymous Monk on Sep 24, 2012 at 14:25 UTC
    Amending title.
Re: Thread::Queue and WWW::Curl
by Corion (Pope) on Sep 24, 2012 at 14:33 UTC

    You don't show where you create instances of WWW::Curl(::Easy). I recommend that you do so separately and only within each thread, because most likely the WWW::Curl objects are backed by corresponding C objects, and those may or may not like dancing between different threads and maybe are not even threadsafe unless precautions are taken.

    Personally, I would look at ditching threads and looking at the Curl-native routines in WWW::Curl::Multi. These should be able to run within a single worker thread and at least for the fetching part, you are most likely limited by network bandwidth/latency anyway.

Re: Thread::Queue and WWW::Curl
by BrowserUk (Pope) on Sep 24, 2012 at 14:42 UTC
    My problem is the subroutine with those WWW::Curl:: calls returns at the line that posts the form: my $response = $curl->perform;

    Is that a problem description? Because I cannot see where it says something fails, much less what; how; what error mesages/code you get.

    the deleteUser subroutine calls another which contains use WWW::Curl::Easy; use WWW::Curl::Form;

    That kind of implies that the code that is failing is in the subroutines that you haven't bothered to post. Is that the case?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

    network sites:

      Thanks for the speedy replies. The 2 missing subroutines are as follows:

      sub deleteUser { # in: app access token, app id, FB userId to delete # deletes user using app access token # out: nothing my ($accessToken, $appId, $userId) = (shift, shift, shift); my %form = ( method => 'delete', access_token => $accessToken ); my $url = 'https://graph.facebook.com/' . $userId; my $post = hitFbNew2('post', $url , \%form); return ($post); }
      and
      sub hitFbNew { my ($method, $endpoint, $form) = (shift, shift, shift); my $curl = WWW::Curl::Easy->new; my $curlf = WWW::Curl::Form->new; my $response_body; for my $key (keys %{$form}){ $curlf->formadd($key, $form->{$key}); } $curl->setopt(CURLOPT_WRITEDATA, \$response_body); $curl->setopt(CURLOPT_URL, $endpoint); $curl->setopt(CURLOPT_HTTPPOST, $curlf); print "\nHitting FB using the curl libraries..."; my $response = $curl->perform; print "\nresponse was $response"; my %reply = ( content => $response_body, responseCode => $curl->getinfo(CURLINFO_HTTP_CODE) ); print "\nreply code is $reply{responseCode} and content is $reply{co +ntent}"; return (\%reply); }
      I will look at WWW::Curl::Multi in the meantime.

        You still haven't said what fails? Or how you know something is failing?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://995397]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2014-07-11 06:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (219 votes), past polls