Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer

figuring out Parallel::ForkManager slots

by ramicio (Novice)
on Sep 06, 2019 at 19:40 UTC ( #11105742=perlquestion: print w/replies, xml ) Need Help??

ramicio has asked for the wisdom of the Perl Monks concerning the following question:

Hello. I have a script I have used to encode video for years now. All of a sudden, after an OS update, wine decides it's not going to behave nicely when running more than once. So I figured out that I needed to create a second WINEPREFIX to use. Now the trouble I am having is that I cannot figure out a way to know which ForkManager "slot" I am running under, to give a command as to which WINEPREFIX directory to use. The variable for the slot seems to change between execution time. This would all work if each video encoded literally the same amount of time, but they don't, so I can't do a simple flip-flop with that slot number. I'm not super proficient at coding. Never was. I can get by, but this is absolutely stumping me. Please help.
$slot = 1; my $pm = new Parallel::ForkManager( 2 ); foreach $key ( sort { lc $a cmp lc $b || $a cmp $b } keys %array ) { if( $slot == 2 ){ $slot = 1; } elsif( $slot == 1 ){ $slot = 2; } $pm->start and next; if( ! -e "/mnt/storage/videotemp/$key.hevc" ) { $cmd = 'WINEPREFIX=~/.wine64_'.$slot.' wine ... ; print "\n$cmd\n\n"; exec($cmd); } $pm->finish; } $pm->wait_all_children;
That is just the flip-flop version I thought would work until I ran it and the first video was a long one (6 hours) and the rest were 1 hour videos. I also have been tinkering around with a test file and doing a sleep timer to simulate things:
@data = ( 10,4,20,2,15,6 ); %slot = ( 'one' => 'unoccupied', 'two' => 'unoccupied' ); my $pm = new Parallel::ForkManager( 2 ); DATA_LOOP: foreach my $n (@data) { if ( $slot{ 'one' } eq 'unoccupied' ) { $slotnow = 'one'; $slot{ ' +one' } = 'occupied'; } elsif ( $slot{ 'two' } eq 'unoccupied' ) { $sl +otnow = 'two'; $slot{ 'two' } = 'occupied'; } $pm->start and next DATA_LOOP; print $slotnow." sleep($n)\n"; system( "sleep $n" ); #print "- $slot{$slotnow} - $n\n"; $slot{ $slotnow } = 'unoccupied'; #print Dumper( %slot ); $pm->finish; } $pm->wait_all_children;
I just can't get that test file to kick back to slot one when it is told to. It seems like it is being evaluated correctly at the if statement, but after than it loses what $slotnow and $slot{#} were told to become. I really don't understand what is going on. Is there nothing that is a part of ForkManager to tell which slot is being used? Thank you.

Replies are listed 'Best First'.
Re: figuring out Parallel::ForkManager slots
by tybalt89 (Parson) on Sep 06, 2019 at 22:24 UTC

    You set the slot "unoccupied" in a child, that has to be done in the parent after the child exits. I'm not sure ForkManager can do that.

    In the spirit of always trying to provide a solution. however, here's your "sleep" test case done in my Forking::Amazing (which I wrote because I didn't like Parallel::ForkManager)

    #!/usr/bin/perl # use strict; use warnings; use Forking::Amazing; my @data = ( 10,4,20,2,15,6 ); Forking::Amazing::run 2, sub # in child { my ($slotnow, $n) = @{ shift() }; print "started $slotnow sleep($n)\n"; [ sleep $n ]; }, sub # in parent { my ($slotnow, $n) = @{ shift() }; print " ended $slotnow - $n\n"; @data and push @Forking::Amazing::ids, [ $slotnow, shift @data ]; }, map [ $_, shift @data ], qw( one two );

    Which outputs:

    started one sleep(10) started two sleep(4) ended two - 4 started two sleep(20) ended one - 10 started one sleep(2) ended one - 2 started one sleep(15) ended two - 20 started two sleep(6) ended one - 15 ended two - 6

    And here's Forking::Amazing

      Any plans to CPANify it?

      map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
      I can't even figure out how to install that. I don't even know how I would apply that to my encoding script anyway. EDIT: I Apologize. I am impatient. I figured it out. That is some forking amazing work! Thank you.
Re: figuring out Parallel::ForkManager slots
by jcb (Hermit) on Sep 06, 2019 at 22:28 UTC

    As an "XY answer", why are you running an encoder under Wine? Could you use a native MEncoder build and avoid these problems?

    Returning to the question, the important step towards your solution is to note that $pm->start optionally accepts an indentifier, and the callback set with $pm->run_on_finish gets that identifier. An untested partial example:

    my @slot_in_use = (0, 0); my @slot_names = qw/one two/; sub find_slot () { for (my $i = 0; $i < @slot_in_use; $i++) { unless ($slot_in_use[$i]) { $slot_in_use[$i] = 1; return $i } } die "slot table full"; } $pm->run_on_finish(sub { my ($c_pid, $c_exit, $c_slot) = @_; $slot_in_use[$c_slot] = 0; # release slot }); foreach my $n (@data) { my $slot = find_slot; $pm->start($slot) and next; # ... $pm->finish; }

    Update: Remember how I said "untested"? Stupid variable name error fixed. Using strict does not help if you never run the code at all...

      The wine part is just a frame server. Things like mencoder and ffmpeg leave much to be desired. The x265 binary is compiled on the machine for the machine. Such architectural things like AVX-512 is why I use such a binary, instead of libraries through stuff like mencoder or ffmpeg. There is really no other way to get frames to such a program other than a pipe. x264 could be built to read video files directly, but processing options were garbage. x265 I don't believe will do anything. I couldn't even get that example to run. It just dies and says slots full without doing anything.

        The example has not been tested, since I write out pipe/fork/select/waitpid longhand when I need them (usually with child processes feeding reports into the Tk event loop through pipes) and do not have Parallel::ForkManager installed. It had a silly mistake that is now fixed, although it does not actually do anything because it does not define contents for the @data array nor is there any actual code to run in the child processes.

        A frame server? Some program that buffers video piecemeal while x265 works on it? Is there a native alternative you could use? Pulling in Wine here just seems ... unnecessary.

        Thank you for your time.
Re: figuring out Parallel::ForkManager slots
by holli (Monsignor) on Sep 07, 2019 at 11:46 UTC
    Did you consider installing Perl in the Wine environment and run your script "natively" in there?


    You can lead your users to water, but alas, you cannot drown them.
      If I ran a perl environment under wine, I couldn't be able to call back to Linux's x265. Video encoding is slow enough as it is. I don't feel like making it slower through OS emulation (running Windows encoders). The frame serving is fast. Granted wine will slow it down, but it still can serve frames many times faster than the encoder can process them. There are Linux avisynth-like ports, but the actual Avisynth has a huge ecosystem of plugins that won't work in those ports (they're compiled DLLs).
Re: figuring out Parallel::ForkManager slots
by karlgoethebier (Monsignor) on Sep 07, 2019 at 15:52 UTC

    As far as i remember your task isn't for low budget equipment. Don't you need a wide gamut monitor to even view these videos? Something like the ColorEdge series from Eizo up from model CG247X? Not to talk about a fat machine with many cores, big RAM and big fast SSD's.

    What about an investment in some pro software? On the Mac there is Final Cut Pro X and Compressor. About € 400 together. Little money compared to the hardware investment. I don't know what is available/similiar for the Linux or Win platform. Just some thoughts. Please correct me if i'm totally wrong. Regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

      How I encode works fine. It just happens to be that before I updated, wine somehow ran multiple instance out of the same prefix directory, and then when I updated it didn't. Some sources are 1080p 8-bit videos, some are 2160p 10-bit HDR videos. The stuff gets viewed on a 4k HDR TV anyway. You don't need anything other than a beefy CPU to simply encode video. That's all I'm doing here, and cropping, which is a simple task. Professional video editing requires fast disk access and lots of RAM.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11105742]
Approved by stevieb
Front-paged by Discipulus
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (11)
As of 2019-09-17 10:18 GMT
Find Nodes?
    Voting Booth?
    The room is dark, and your next move is ...

    Results (206 votes). Check out past polls.