Judging by this and other threads the main problem for most people seems to be the cpan nodelet. I was having connection problems, including a lot of timeouts, so I turned off the cpan nodelet and everything sped up for a bit. When my connection times started getting bad again, I just turned off everything but the chatterbox and other users, and it's been working fine since then.
So what you probably want to do is try turning the cpan nodelet off, if that doesn't work just turn all of your nodelets off and progressively add them back again and figure out what's the problem.
Also I'm sure a visit to the Offering Plate wouldn't hurt the site's times either :)