Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

How can I limit the number of Cores in multi-process programming

by milesjin (Initiate)
on Nov 17, 2015 at 02:23 UTC ( #1147869=perlquestion: print w/replies, xml ) Need Help??

milesjin has asked for the wisdom of the Perl Monks concerning the following question:

Hi expert,

I am using fork() to parallelize my perl script. I hope my perl script can run faster, however it should not eat up all the CPU resouce. I understand I can achieve it indirectly by limiting the number of child processes. Apart from this, do we have better way to limit my program to a number of CPU cores? Let's say my program should use less than 4 cores, but it may have totally 6 processes?

Thanks in advance.

  • Comment on How can I limit the number of Cores in multi-process programming

Replies are listed 'Best First'.
Re: How can I limit the number of Cores in multi-process programming
by BrowserUk (Patriarch) on Nov 17, 2015 at 02:50 UTC
    I hope my perl script can run faster, however it should not eat up all the CPU resouce.

    Why shouldn't it use all the CPU resource? What are you saving it for?

    Essentially what I'm saying here is that cpu resource is time; and time inexorably moves forward; you cannot save it for later; so unused CPU is essentially wasted cpu that you can never recover. Your processor should either be running as close to 100% as your workload permits; or it should be idle because you have nothing to do. Trying to restrain it to say 75% or whatever; doesn't achieve anything not even energy savings.

    Perhaps what you are concerned with is that you want to ensure that some other, sporadic processes will have an opportunity to run concurrently with this workload when they need to?

    If so, the proper way to deal with that is not to try an 'reserve' some cpu; but rather to run this workload at a lower priority than the sporadic workload(s) so that you make full use of the cpu for this workload when can; whilst ensuring that those sporadic workloads will get a look in when they need it.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Hi,

      Thanks a lot for your reply. In my environment, many users are sharing hosts, and each job has CPU quota. If you request more quota like occuping the host exclusively, pending time would be longer, so I prefer to limit my CPU quotas to a optimal size.

        In my environment, many users are sharing hosts, and each job has CPU quota.
        1. Is it a virtualised environment?

          Ie. Is your process running in a VM or container?

        2. What units are your CPU quotas specified in?

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
        In the absence of evidence, opinion is indistinguishable from prejudice.

        It sounds strange that you want to multithread for more speed, but then turn around and want to use less CPU % (so it takes longer again?).

        Do you have busy loops wasting CPU cycles or something? If so, use a sleep or a select(undef,undef,undef,0.01 to reduce the polling rate and drop the CPU usage to <1%.

        You can try a nap of a few milliseconds per iteration on your big long running loops as well if they're hogging resources, but if the loop is doing productive things, letting it finish is usually a good idea ;).

Re: How can I limit the number of Cores in multi-process programming
by ctilmes (Vicar) on Nov 17, 2015 at 02:59 UTC

    On a unix box you can play with BSD::Resource (see setpriority()) or play with the nice command.

    ("nice perl myscript.pl" is a good start..)

      Thanks a lot. It's helpful.
Re: How can I limit the number of Cores in multi-process programming
by Lennotoecom (Pilgrim) on Nov 17, 2015 at 09:32 UTC
    oi mate
    look into
    taskset -cp 3,4 1234
    where processors are 3 and 4
    and 1234 is the process id
    that's a power to control smp affinity mate

    also I believe this will launch your script with the correct affinity
    taskset -c 1,2 script.pl
Re: How can I limit the number of Cores in multi-process programming
by Anonymous Monk on Nov 17, 2015 at 02:35 UTC
      Thanks for the reply.
Re: How can I limit the number of Cores in multi-process programming
by oiskuu (Hermit) on Nov 17, 2015 at 17:30 UTC

    Not for the faint of heart, but there is a syscall in perl. To hack a sched_setaffinity call, one might:

    #! /usr/bin/perl sub setaffinity { use Config; die if "@Config{qw(ptrsize archname)}" !~ /8.*x86_64.*linux/; my $mask = pack "Q", shift; syscall 203, 0, 8, $mask; } setaffinity 0x22; # allow cores/threads 1 and 5 fork for 1..3; # 8 workers 1 while 1; # busy we are

    To easily find out the linux syscall numbers, I recommend looking at strace sources: there are syscallent.h headers for each architecture.

      Thanks for the suggestion!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1147869]
Approved by LanX
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2022-10-03 12:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My preferred way to holiday/vacation is:











    Results (13 votes). Check out past polls.

    Notices?