Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Re^3: Proper undefine queue with multithreads

by sundialsvc4 (Abbot)
on Jun 03, 2014 at 18:47 UTC ( #1088477=note: print w/replies, xml ) Need Help??

in reply to Re^2: Proper undefine queue with multithreads
in thread Proper undefine queue with multithreads

In this case, apparently, it would seem that the processing consists of two distinct steps:   first, the discovery of the complete list of paths that need to be processed (which cannot be parallelized), followed by, second, the processing of files within the paths thus-discovered (which, apparently, can).

Therefore, I simply suggest that you should follow BrowserUK’s sage advice exactly as given.   The main-thread will first undertake, and fully complete, the file-path discovery process.   “That is step-one.”   (Since the underlying software does not let you be recursive, you will have to “push” directory-names onto some kind of pushdown-stack as-found, then continue until that stack is empty.)   Then, having done all of that as described, it will move forward to “step two.”   It will launch some “pool” of independent processes to deal with each path in parallel, because the “step two” process clearly is something that can(??) be parallelized.   The parent thread will feed path-names into the queue, then it will feed undefs into the same queue, then it will wait for all of the children to end.   Clearly, none of my suggestions would apply to this particular case.   Therefore, take BrowserUK’s excellent recommendations, and go.

Uhhhh... but first... Do be certain that omnidb can, in fact, support an arbitrary number of parallel requests that originate from the same process context!   (And when I say, “be certain,” I do mean ... “prove it!”)   The folks who designed omnidb, all those years ago, might have taken certain shortcuts.

And here’s a quick way to (maybe) find out, in a Un*x-like environment:   issue one omnidb-command in the shell, adding the “&” command to cause it to launch as a background-job, then immediately issue a second one in the foreground.   Prefix both of these with the time command, and be sure that both commands will sensibly take some time to complete.   (Enough time to actually allow you to issue both of the commands quickly-enough that both actually run in parallel.)   Verify that both of these actually do run in-parallel with one another, (a) without conflict, and (b) without one of them simply being blocked by the other one.   The designers of omnidb may or may not have anticipated parallelism (especially, within the context of a single process), and, if they did, they might have taken a shortcut.   Now is the proper time to find out.

  • Comment on Re^3: Proper undefine queue with multithreads

Replies are listed 'Best First'.
Re^4: Proper undefine queue with multithreads
by BrowserUk (Pope) on Jun 04, 2014 at 18:17 UTC
    The designers of omnidb may or may not have anticipated parallelism (especially, within the context of a single process),

    To clarify -- for those that can't see it -- my (main; there are several others) objection to this post:

    There is no "context of a single process". All the parallel commands run as separate instances of the omnidb executable. Which makes the post nothing but meaningless guff!

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1088477]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (10)
As of 2019-11-22 20:24 GMT
Find Nodes?
    Voting Booth?
    Strict and warnings: which comes first?

    Results (113 votes). Check out past polls.