Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

instead of managing all that file locking complexity, why not write the results of each element to its own file. then there's no need for dbm deep. you can persist to different files without worrying on stepping on other results. ( i used freeze/thaw to save complex data structures )

forked_map takes a dir where it temporarily persists results, a max number of workers ( forked children ) so you can tune it as you wish, the function to perform, and the data to perform it on. it uses the same pattern as •Re: Run N similar tasks in parallel
use strict; use warnings; use Data::Dumper; use File::Path; use File::Spec; use Storable qw(freeze thaw); use POSIX ":sys_wait_h"; my $slow_matches_b = sub { sleep 1; return unless $_[0]; return 1 if $_[0] =~ /b/; }; my $test_strings = [ ('blee','blah','bloo', 'qoo', 'fwee' ) ]; my @results = forked_map( 'c:\\testdir', 2, $slow_matches_b, @$test_st +rings ); print Dumper \@results; sub forked_map { my $tempdir = shift; my $worker_count = shift; my $function = shift; # # ensure tempdir is not some file # if ( -f $tempdir ) { die "$tempdir exists!"; } # # delete dir, then recreate ( destroy any previous results ) # unlink $tempdir; File::Path::mkpath( $tempdir ); # # assign keys to each data element # my $id = 0; my %hash_input = map { ( $id++, $_ )} @_; my %workers; # # loop block for assigning work to workers # { # # assign work to available workers # while ( keys %hash_input and keys %workers < $worker_count ) { $id = (keys %hash_input)[0]; my $data = $hash_input{$id}; delete $hash_input{$id}; # # fork the child process # my $pid = fork; if ( ! defined $pid ) { die "cannot fork: $!"; } if ( $pid ) { # # track the pid in the parent process # $workers{$pid} = 'ACTIVE'; } else { # # work in the child process # my $result = $function->($data); my $frozen_result = freeze(\$result); # # save the results # my $tempfile = File::Spec->catfile($tempdir,$id); open(my $fh, "> $tempfile") or die $!; print $fh $frozen_result; # # bye bye baby # exit 0; } } # # wait for any child to complete # my $pid = wait(); if ($pid == -1) { %workers = (); } else { delete $workers{$pid}; } # # loop if there is any work remaining # redo if ( ( keys %hash_input ) or ( keys %workers ) ) } # # read results from file # my %results; opendir(my $dh,$tempdir); for ( readdir($dh) ) { my $filepath = File::Spec->catfile($tempdir,$_); if ( -f $filepath ) { open(my $fh, $filepath ) or die $!; my $frozen_result = do {local $/; <$fh> }; $results{$_} = ${thaw($frozen_result)}; } } # # clean up after ourselves # unlink $tempdir; # # return results in order # return map { $results{$_} } sort keys %results; }

update: cleaned up freeze thaw usage

In reply to forked map by LanceDeeply
in thread Using DBM::Deep and Parallel::ForkManager for a generalized parallel hashmap function builder (followup to "reducing pain of parallelization with FP") by tphyahoo

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    [james28909]: politicians making jobs for themselves? that is absurd! we all know how hard those politicians have it, having to take millions from lobbiest while screwing over their citizens. have a little sympathy why dont ya!

    How do I use this? | Other CB clients
    Other Users?
    Others having an uproarious good time at the Monastery: (9)
    As of 2017-12-15 18:33 GMT
    Find Nodes?
      Voting Booth?
      What programming language do you hate the most?

      Results (442 votes). Check out past polls.