Re: How can I limit the number of Cores in multi-process programming

I hope my perl script can run faster, however it should not eat up all the CPU resouce.

Why shouldn't it use all the CPU resource? What are you saving it for?

Essentially what I'm saying here is that cpu resource is time; and time inexorably moves forward; you cannot save it for later; so unused CPU is essentially wasted cpu that you can never recover. Your processor should either be running as close to 100% as your workload permits; or it should be idle because you have nothing to do. Trying to restrain it to say 75% or whatever; doesn't achieve anything not even energy savings.

Perhaps what you are concerned with is that you want to ensure that some other, sporadic processes will have an opportunity to run concurrently with this workload when they need to?

If so, the proper way to deal with that is not to try an 'reserve' some cpu; but rather to run this workload at a lower priority than the sporadic workload(s) so that you make full use of the cpu for this workload when can; whilst ensuring that those sporadic workloads will get a look in when they need it.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: How can I limit the number of Cores in multi-process programming

Replies are listed 'Best First'.
Re^2: How can I limit the number of Cores in multi-process programming by milesjin (Initiate) on Nov 17, 2015 at 03:50 UTC
Hi, Thanks a lot for your reply. In my environment, many users are sharing hosts, and each job has CPU quota. If you request more quota like occuping the host exclusively, pending time would be longer, so I prefer to limit my CPU quotas to a optimal size.	[reply]
Re^3: How can I limit the number of Cores in multi-process programming by BrowserUk (Patriarch) on Nov 17, 2015 at 06:06 UTC
In my environment, many users are sharing hosts, and each job has CPU quota. Is it a virtualised environment? Ie. Is your process running in a VM or container? What units are your CPU quotas specified in? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^4: How can I limit the number of Cores in multi-process programming by milesjin (Initiate) on Nov 17, 2015 at 08:31 UTC
It's load balanced farm. User requests hosts with max_cpu specification, if your jobs use more cpu cores, would get warnings and the job would be killed. Thanks	[reply]
Re^5: How can I limit the number of Cores in multi-process programming by Corion (Patriarch) on Nov 17, 2015 at 08:34 UTC
Re^6: How can I limit the number of Cores in multi-process programming by milesjin (Initiate) on Nov 17, 2015 at 08:49 UTC
Re^5: How can I limit the number of Cores in multi-process programming by BrowserUk (Patriarch) on Nov 17, 2015 at 09:23 UTC
Re^6: How can I limit the number of Cores in multi-process programming by milesjin (Initiate) on Nov 18, 2015 at 07:44 UTC
Re^3: How can I limit the number of Cores in multi-process programming by SuicideJunkie (Vicar) on Nov 17, 2015 at 14:09 UTC
It sounds strange that you want to multithread for more speed, but then turn around and want to use less CPU % (so it takes longer again?). Do you have busy loops wasting CPU cycles or something? If so, use a sleep or a `select(undef,undef,undef,0.01` to reduce the polling rate and drop the CPU usage to <1%. You can try a nap of a few milliseconds per iteration on your big long running loops as well if they're hogging resources, but if the loop is doing productive things, letting it finish is usually a good idea ;).	[reply] [d/l]


Perl-Sensitive Sunglasses
	PerlMonks