I think I agree with you, but when I use debuggers, it's usually not on my code. In fact, I rarely use the perl debugger, because when I debug perl, its usually my own code :) I've on rare occasions used the perl debugger to help track down problems in library modules, but usually looking at the source is good enough because I find perl source easy to find, and (mostly) easy to follow.
I've mostly used debuggers with monolithic 4gl code which would include libraries on top of libraries, and the only way to find a problem without inserting a zillion print statements was to run the debugger (and I don't want to fully comprehend all the code, at least not today, I just want to fix the problem). And I would inherit problems from other programmers who would work for days on a problem but couldn't use the debugger, and I'd solve it in a few minutes or hours. So IMO it just depends on the environment. I use the perl debugger so rarely that every time I have a notion to use it, I find I have to relearn it :-)
Update: Oh, and one more reason I use the debugger less in perl is that you don't have to go through that 'compile it again' phase of the debugging process that you have to endure in other languages.
Very late update: Recently, I found myself tracking down a problem with totally unfamiliar CPAN http/socket library code. I inserted a couple of $DB::single; statements rather than numerous print's (these were dynamically loaded modules, so this was the easiest way to set breakpoints) and learned something about HTTP...specifically about chunking in the responses.