Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: using parallel processing to concatenate a string, where order of concatenation doesn't matter

by diotalevi (Canon)
on Oct 18, 2006 at 14:15 UTC ( #579080=note: print w/replies, xml ) Need Help??


in reply to using parallel processing to concatenate a string, where order of concatenation doesn't matter

If Parallel::Queue's threading worked for me I'd suggest the following.

use threads::shared; use Parallel::Queue; print concatenate_parallel( ['a' .. 'z'], 4 ) . "\n"; sub concatenate_parallel { my $result :shared; my @input = map { my $string = $_; sub { $result .= $string; }; } @{ shift @_ }; my $max_threads = shift @_; my $mgr = Parallel::Queue->construct( 'thread' ); $mgr->runqueue( $max_threads, @input ); return $result; }

⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

  • Comment on Re: using parallel processing to concatenate a string, where order of concatenation doesn't matter
  • Download Code

Replies are listed 'Best First'.
Re^2: using parallel processing to concatenate a string, where order of concatenation doesn't matter
by tphyahoo (Vicar) on Oct 18, 2006 at 14:26 UTC
    Thanks, that's intriguing, but my perl doesn't seem to like that code.
    hartman@ds0207:~/pfmArena> perl test2.pl Bogus Parallel::Queue: "fork" and "thread" are exclusive at test2.pl l +ine 17 hartman@ds0207:~/pfmArena> perl -e '@ARGV=("test2.pl"); while (<>) {di +e $_ if $. == 17}' my $mgr = Parallel::Queue->construct( 'thread' ); hartman@ds0207:~/pfmArena> hartman@ds0207:~/pfmArena> perl -V | grep -i thread osname=linux, osvers=2.6.16, archname=i586-linux-thread-multi config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusr +binperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=tr +ue -Doptimize=-O2 -march=i586 -mtune=i686 -fmessage-length=0 -Wall -D +_FORTIFY_SOURCE=2 -g -Wall -pipe' usethreads=define use5005threads=undef useithreads=define usemulti +plicity=define cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS +-DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement +-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGI +NG -fno-strict-aliasing -pipe -Wdeclaration-after-statement' libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E - +Wl,-rpath,/usr/lib/perl5/5.8.8/i586-linux-thread-multi/CORE' PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD +S /usr/lib/perl5/5.8.8/i586-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8/i586-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8/i586-linux-thread-multi hartman@ds0207:~/pfmArena> perl -V | grep -i ithread usethreads=define use5005threads=undef useithreads=define usemulti +plicity=define PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD +S hartman@ds0207:~/pfmArena>
    Any ideas?

      Tell it to use threading by default: use Parallel::Queue 'thread';.

      ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

        Can't... get it... to work...

        I tried as you suggested...

        hartman@ds0207:~/Desktop/misc/idealoSubversionArena/parallelListProces +sing> ./testpq.pl Can't locate object method "new" via package "Threads" (perhaps you fo +rgot to load "Threads"?) at /usr/lib/perl5/site_perl/5.8.8/Parallel/Q +ueue.pm line 197. hartman@ds0207:~/Desktop/misc/idealoSubversionArena/parallelListProces +sing> head testpq.pl #!/usr/bin/perl use threads::shared; use Parallel::Queue 'thread'; print concatenate_parallel( ['a' .. 'z'], 4 ) . "\n"; sub concatenate_parallel { my $result :shared; my @input = map { my $string = $_;

        Also tried other permutations with spelling variations, but I feel like I'm stumblinb around in the dark. Parallel::Queue indicates that forking works, but threading is "still in progress".

        Are you able to run this code?

        I am really keen to make progress on this, because I am hoping to use this as a basis for threadedreduce which would serve as an abstraction layer for all sorts of list processing that could potentially be speeded up by using threading, but without changing anything in the "meat" of the program (which would be a function builder that uses threadedreduce or nonthreadedreduce, each having the same effect).

        See

        Could there be ThreadedMapReduce (and/or ForkedMapReduce) instead of DistributedMapReduce?

        and my followup, with threadedreduce stub code, at Re^2: Could there be a ThreadedMapReduce (instead of DistributedMapReduce)?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://579080]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (1)
As of 2018-07-22 05:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    It has been suggested to rename Perl 6 in order to boost its marketing potential. Which name would you prefer?















    Results (451 votes). Check out past polls.

    Notices?