The essential code-paths within the Perl compiler/interpreter itself are already well-identified and are optimized.
That may be true (it isn't, but it's at least plausible) if you take into account the current design of the Perl 5 VM. Unfortunately, the current design of the Perl 5 VM makes some assumptions which preclude performance.
It's enlightening to run some of the silly microbenchmarks people use for competitive benchmarking to see exactly why Perl 5 scales so badly on some of them. (See also my response on compact, typed arrays.)
Certainly, there are bona-fide edge cases, throughout the Perl system and its libraries, that are properly handled right now with “XS” code, in C or C++.
Cases also exist where writing XS will make your code slower—and I don't mean writing inefficient XS.