Without seeing the "do miscellaneous tasks" part it's tough to be sure, but the first thing that springs to mind is that you're seeing the lag because of variable destruction. Either you've got some variables that have slow DESTROY methods (perhaps something's doing a DNS lookup for logging purposes and resolver issues are causing this) or you're letting very large hashes fall out of scope and the cleanup's taking forever because of pathological C library behaviour.
If it's a resolver issue, fix it. If the C library on the machine's an old-ish version of glibc, upgrade it. Otherwise you've presented insufficient details to diagnose the problem.
