Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
Sorry I wasn't clear enough here. I want one file per queried machine with all of the command outputs in it.

Then I do not understand your stated problem (from the OP): "The problem I am trying to solve is the printed order of the commands in the output file. ", or why you think you need all those darn semaphores in your code?

If you programmed a single threaded solution to this is might look something like:

for my $machine ( @machines ) { open my $out, '>', "$machine.dat" or die $!; for my $cmd ( @commands ) { my $content = get "$machine/$cmd"; print $out $content; } close $out; }

The outputs from the commands end up in the file in the same order as the commands are run, because that the order you print them in.

To turn that into a threaded solution, just make the body of the outer loop the thread:

for my $machine ( @machines ) { async { open my $out, '>', "$machine.dat" or die $!; for my $cmd ( @commands ) { my $content = get "$machine/$cmd"; print $out $content; } close $out; }->detach; }

And (essentially*) that's it! Each thread it using a different file, so no conflicts or ordering problems arise. No need for locking or semaphores or synchronisation.

*As shown, the above would start a new thread for every one of the 1000s of machines and run them concurrently which would blow your memory to hell and thrash your disc to death. But fixing that is very simple:

my $running :shared = 0; ## This tracks the number of conc +urrent threads for my $machine ( @machines ) { async { { lock $running; ++$running; } ## incr on start open my $out, '>', "$machine.dat" or die $!; for my $cmd ( @commands ) { my $content = get "$machine/$cmd"; print $out $content; } close $out; { lock $running; --$running; } ## decr on finish }->detach; sleep 1 until $running < 10; ## sleep a bit if more + than 10 are running } sleep 1 while $running; ## Make sure the main threads waits for th +e last few threads to finish

(That would be simpler still if the API allowed sleep 1 while threads->list( threads::detach ) > 10;; but it doesn't.)

It would also be more efficient of your machine resources (cpu & memory) to use a thread pool (NOT Thread::Pool!!!) solution; but as you're IO-bound; and limiting that for the sake of your proxy; you are unlikely to trouble the resources of even the least well specified machine with the above code.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re^3: Program Design Around Threads by BrowserUk
in thread Program Design Around Threads by aeaton1843

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others examining the Monastery: (5)
    As of 2018-06-24 08:11 GMT
    Find Nodes?
      Voting Booth?
      Should cpanminus be part of the standard Perl release?

      Results (126 votes). Check out past polls.