Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^2: Program Design Around Threads

by aeaton1843 (Acolyte)
on Mar 06, 2013 at 18:15 UTC ( #1022055=note: print w/ replies, xml ) Need Help??


in reply to Re: Program Design Around Threads
in thread Program Design Around Threads

I wanted to answer your questions. I will have to run through the proposed solution tonight.

1. It depends but 2-3k, maybe less if I can narrow to a specific data center.

2a. I am being careful with thread bombing the proxy. I have a thread creation limitter. Right now it is sitting at 10 threads.

2b. Worst case, I have to pull an entire configuration off an Arista. That process can take 30 seconds on a busy Arista device. Much less for most everything else.

3. I think the worst case is "show interface" @ 20ish*384/per machine The 20ish is variable based on what is configured on the interface. Traffic shaping for example.

4. Sorry I wasn't clear enough here. I want one file per queried machine with all of the command outputs in it.

5. The outputs of each command should be put into the output file in the order of the command list in the command file. IE if show running-configuration is first in the command file then its output should appear first in the output file. Not that it matters but right now the file's name is that of the machine name.


Comment on Re^2: Program Design Around Threads
Re^3: Program Design Around Threads
by BrowserUk (Pope) on Mar 06, 2013 at 18:48 UTC
    Sorry I wasn't clear enough here. I want one file per queried machine with all of the command outputs in it.

    Then I do not understand your stated problem (from the OP): "The problem I am trying to solve is the printed order of the commands in the output file. ", or why you think you need all those darn semaphores in your code?

    If you programmed a single threaded solution to this is might look something like:

    for my $machine ( @machines ) { open my $out, '>', "$machine.dat" or die $!; for my $cmd ( @commands ) { my $content = get "$machine/$cmd"; print $out $content; } close $out; }

    The outputs from the commands end up in the file in the same order as the commands are run, because that the order you print them in.

    To turn that into a threaded solution, just make the body of the outer loop the thread:

    for my $machine ( @machines ) { async { open my $out, '>', "$machine.dat" or die $!; for my $cmd ( @commands ) { my $content = get "$machine/$cmd"; print $out $content; } close $out; }->detach; }

    And (essentially*) that's it! Each thread it using a different file, so no conflicts or ordering problems arise. No need for locking or semaphores or synchronisation.

    *As shown, the above would start a new thread for every one of the 1000s of machines and run them concurrently which would blow your memory to hell and thrash your disc to death. But fixing that is very simple:

    my $running :shared = 0; ## This tracks the number of conc +urrent threads for my $machine ( @machines ) { async { { lock $running; ++$running; } ## incr on start open my $out, '>', "$machine.dat" or die $!; for my $cmd ( @commands ) { my $content = get "$machine/$cmd"; print $out $content; } close $out; { lock $running; --$running; } ## decr on finish }->detach; sleep 1 until $running < 10; ## sleep a bit if more + than 10 are running } sleep 1 while $running; ## Make sure the main threads waits for th +e last few threads to finish

    (That would be simpler still if the API allowed sleep 1 while threads->list( threads::detach ) > 10;; but it doesn't.)

    It would also be more efficient of your machine resources (cpu & memory) to use a thread pool (NOT Thread::Pool!!!) solution; but as you're IO-bound; and limiting that for the sake of your proxy; you are unlikely to trouble the resources of even the least well specified machine with the above code.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      That's just it though. This isn't true: "Each thread it using a different file, so no conflicts or ordering problems arise. No need for locking or semaphores or synchronisation." There are as many threads running as there are machines and commands. For Example:

      tid1 -> machine1 -> "show running-config" takes 30 seconds to get reply.

      tid2 -> machine2 -> "show running-config" takes 30 seconds to get reply

      tid3 -> machine1 -> "show vlan" takes 5 seconds to get reply.

      tid4 -> machine2 -> "show vlan" takes 5 seconds to get reply.

      Since tid 3&4 finish first, my 2 output files now have show vlan at the top instead of show running-config because those threads finished before thread 1&2 and wrote contents into file machine1.txt and machine2.txt. Maybe I am missing something?

        There are as many threads running as there are machines and commands

        Only if you run the different commands for each machine from different threads. That's the wrong way to do it.

        If you look at the code snippets I posted, one thread connect to one machine, opens one file, and runs all the commands for that machine, sequentially. That way, there is no possibility for ordering problems or file conflicts.

        Maybe I am missing something?

        Please read my last post slowly and thoroughly. Making multiple connections to each machine from different threads to run single commands is both inefficient and the source of your problems.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1022055]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (6)
As of 2014-12-21 10:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (104 votes), past polls