Are you sure it's total memory used and not just in-core memory? You might be seeing some of your unused pages swapped out to disk. If so, that's usually a sign you've got garbage that's not getting cleaned up, either because of circular data structures, or you (or perl) isn't disposing of something used earlier on.
If you really are seeing a footprint shrink like that, it's likely you've got something being cached somewhere that gets reused after a while and shrinks on that reuse. (Perl does some aggressive caching of array structures in lexicals, for example)
Finally, it's possible you're running afoul of an almost-but-not-quite empty memory segment. Perl's asked for a 5 meg chunk at some point, uses it and then frees up all but a few bytes right at the beginning, so it has to hold on to the chunk until the last thing using a part of it dies or goes somewhere else. (And there's not much you can do about that, unfortunately)
