Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Unlink not deleting, Threading issue?

by sundialsvc4 (Abbot)
on Jun 05, 2014 at 18:29 UTC ( [id://1088892]=note: print w/replies, xml ) Need Help??


in reply to Unlink not deleting, Threading issue?

Definitely suggest looking for what error-messages are being reported ... and, generally re-consider your design.   It superficially appears to me that you are attempting to launch one thread per-file.   I will guarantee that you will not succeed in launching 25,000 children.   Your code does not anticipate that the fork won’t succeed.

May I kindly now refer you, without further ado, to BrowserUK’s most-excellent first reply to the following recent thread: Proper undefine queue with multithreads.   Your situation is exactly the same.   His design is different, but easy to do, and his design works.

Incidentally ... in a Un*x/Linux environment ... this whole thing just might be already-done for you!   If the xargs command on your system (man xargs) supports the -P maxprocs parameter, then you can bypass all of this nonsense.   Simply write (in Perl) a command that expects to receive a filename as a parameter.   This command converts that one file, processes it, then deletes it, then ends.   Meanwhile, command-line-pipe the output of a find command into xargs which uses that parameter.   Now, the find command is posting the filenames to xargs, which farms-out the work to maxprocs identical, single-purpose (Perl ...) process instances.   The same business requirement has now been solved, in a “very Un*x-ish way,” and the complexity of your (Perl) script has been greatly simplified.   (Perhaps you decide that this alternate approach is “right for you,” or perhaps not, and either choice is up-to-you.   But you will see that it is functionally equivalent in its general approach to the problem.)

Replies are listed 'Best First'.
Re^2: Unlink not deleting, Threading issue?
by Anonymous Monk on Jun 05, 2014 at 21:35 UTC

    It superficially appears to me that you are attempting to launch one thread per-file.

    That's what I thought too at first. I didn't get the memo that fork on Win32 was ready for actual use so I got real curious about how Parallel::ForkManager dealt with this. Turns out, when you use $pm->start, if you've already used all your children (14 here), it waits for a child and finally launches a new one and continues the parent program. So yeah, it seems that's how you're "supposed" to do it.

    As good as Parallel::ForkManager seems to be, it still strikes me as odd to reach for forks on a platform that's natively threaded.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1088892]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2024-03-29 06:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found