a) if the module is sloppy or not cool enough, and I could do better (rare)
b) if the module takes more than one CPAN command with dependancy scanning to install (frequent! Tk::Zinc as of late)
c) if I think the module is changing interfaces too fast or is otherwise unstable (frequent)
d) if the module is really really heavy for what I want to do and may cause maintaince problems due to the amount of new stuff I need to grok and remember (aka PAR) (frequent)
e) if I need to redistribute the script to a bunch of folks with just perl installed by default, PAR isn't good enough or makes too large of a binary, and the audience isn't Perl-educated enough to use CPAN (very frequent) -- this happens a lot when I'm writing quicky scripts that are little more than shell scripts, module use causes problems for those I need to give the script too, but Perl is still better than shell.

For my own coding, I use (good) modules all of the time. But there are plenty of situations when I can't use them, just like I feel I can't use any random C library I find on the internet -- it has to be stable, good, and easy for other people to compile/use, with a decent user base. This isn't exclusively a perl problem.