Sure, if you have a CPU that can be run in multiple modes, the option with more cores will be slower than the one with 1 core. However we've hit intrinsic limits on how fast we can make CPUs. Therefore Moore's Law can only continue by increasing the number of cores. Therefore over time we expect to see clock speeds remain about where they are, but the number of cores will increase.

Now you're right when you say that you're at the mercy of the OS to decide whether your core is shared with other applications. But with 1 core you had to share that CPU anyways. And as you increase the number of cores, the odds that other applications need to be scheduled on your core goes down. Which is good. The bad, though, is that managing more cores takes OS time. So if you have too many cores, then with an SMP system the core you're on won't be doing full work.

However modulo that small effect, if you're happy with the current speed of your code, multiple cores is not a big deal either way. It is only an issue if your code needs to go faster than it does already.