|Pathologically Eclectic Rubbish Lister|
Rewriting vs. Refactoring
If you start from a blank source file, I'd consider that rewriting, even if you start transplanting subroutines from the original back into the new file. I feel that taking a working system and transforming it without breaking it is refactoring, while building a new thing (even when borrowing heavily) from the original is rewriting. Of course, you can do both in the same project, as you may refactor some bits and rewrite other bits.The problem I generally see with rewriting is that there's quite a bit of knowledge that's encoded into the system that's not immediately obvious. Things like:
Of course, refactoring has its own issues. If you can imagine a cleaner structure for your program and try to refactor towards it, you'll find that "you can't get there from here", or you have to first go to Timbuktu before you can get back home.
With my experience (quite a bit), I find that the less you know about the domain or the application, the more you should lean toward refactoring. Similarly, the closer you are to being a domain expert, the more sense rewriting can make. The problem is that it's often difficult to objectively judge just how much knowledge you have about the domain. My approach is normally to find some dividing lines in the system where I can break it apart with the fewest changes possible. Then I can choose to refactor some chunks and rewrite others.
Overdesign and customer needs
You've pretty much hit the nail on the head: If I'm paraphrasing your arguments correctly, the difference between overengineering or not is largely communication. If you're going off and doing anything beyond what you've discussed with the customer, you're overengineering things. If you think the system needs to do something specific, or that the architecture needs to go in a certain direction, you need to have a talk with the customer about anticipated future changes so you can shape things correctly. So if you discuss things with the customer and get buy-in, then you're doing your job correctly. If you have discussed things with the customer, and they're adamant about a particular direction, then you need to do what they want, or you're throwing their money away.
(Gasp - they have a business need for a framework architecture?)
I've heard it mentioned that "functionality is an asset, while code is a liability". Too often, programmers know they need some functionality, but build their own rather than buying it. (I, unfortunately, succumb to this temptation a bit too often, myself.) I've been trying to periodically take a break from design and/or coding so I can sit back and review the requirements so I can stop myself from going off into the rabbit holes.
When your only tool is a hammer, all problems look like your thumb.