Re^2: using parallel processing to concatenate a string, where order of concatenation doesn't matter

[tphyahoo]

Thanks, that's intriguing, but my perl doesn't seem to like that code.

hartman@ds0207:~/pfmArena> perl test2.pl
Bogus Parallel::Queue: "fork" and "thread" are exclusive at test2.pl l
+ine 17
hartman@ds0207:~/pfmArena> perl -e '@ARGV=("test2.pl"); while (<>) {di
+e $_ if $. == 17}'
    my $mgr = Parallel::Queue->construct( 'thread' );
hartman@ds0207:~/pfmArena>

hartman@ds0207:~/pfmArena> perl -V | grep -i thread
    osname=linux, osvers=2.6.16, archname=i586-linux-thread-multi
    config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusr
+binperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=tr
+ue -Doptimize=-O2 -march=i586 -mtune=i686 -fmessage-length=0 -Wall -D
+_FORTIFY_SOURCE=2 -g -Wall -pipe'
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS 
+-DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement 
+-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGI
+NG -fno-strict-aliasing -pipe -Wdeclaration-after-statement'
    libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -
+Wl,-rpath,/usr/lib/perl5/5.8.8/i586-linux-thread-multi/CORE'
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
    /usr/lib/perl5/5.8.8/i586-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.8/i586-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.8/i586-linux-thread-multi
hartman@ds0207:~/pfmArena> perl -V | grep -i ithread
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
                        PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREAD
+S
hartman@ds0207:~/pfmArena>

Any ideas?

[diotalevi]

Tell it to use threading by default: use Parallel::Queue 'thread';.

[tphyahoo]

Can't... get it... to work...

I tried as you suggested...

hartman@ds0207:~/Desktop/misc/idealoSubversionArena/parallelListProces
+sing> ./testpq.pl
Can't locate object method "new" via package "Threads" (perhaps you fo
+rgot to load "Threads"?) at /usr/lib/perl5/site_perl/5.8.8/Parallel/Q
+ueue.pm line 197.
hartman@ds0207:~/Desktop/misc/idealoSubversionArena/parallelListProces
+sing> head testpq.pl
#!/usr/bin/perl
use threads::shared;
use Parallel::Queue 'thread';

print concatenate_parallel( ['a' .. 'z'], 4 ) . "\n";

sub concatenate_parallel {
    my $result :shared;
    my @input = map {
        my $string = $_;

Also tried other permutations with spelling variations, but I feel like I'm stumblinb around in the dark. Parallel::Queue indicates that forking works, but threading is "still in progress".

Are you able to run this code?

I am really keen to make progress on this, because I am hoping to use this as a basis for threadedreduce which would serve as an abstraction layer for all sorts of list processing that could potentially be speeded up by using threading, but without changing anything in the "meat" of the program (which would be a function builder that uses threadedreduce or nonthreadedreduce, each having the same effect).

See

Could there be ThreadedMapReduce (and/or ForkedMapReduce) instead of DistributedMapReduce?

and my followup, with threadedreduce stub code, at Re^2: Could there be a ThreadedMapReduce (instead of DistributedMapReduce)?

[ikegami]

Fix the module as follows:

• Line 197: Change Threads->new to Thread->new.

• Line 292: Insert require Thread; before require Thread::Semaphore;.

These fixes have not been tested, just like the module itself:

Threads are unstested. Forks work; threads are still in progress. The code may work but I have not tested it yet.

Update: Added spelling fix.

[diotalevi]

Well... take out your debugger, look at the source of the module and make it go. Simple.