|Welcome to the Monastery|
Balancing engineering with over-engineering is how we ended up with relatively safe business languages (which can often be good in Corporate America) and the expense of stiffling creativity.
So while one can say this is a distinction between a "programmer" and the (shudder as I use the word) "Software Engineer", one most also forget what is lost from the world of computer science. Simplicity. Elegance. Mathematical Purity.
I desire to write programs in this pure form, I don't always succeed, but sometimes you have to know when building a skateboard from scratch is better than building a freight train from scratch. Or when building a time machine (think Damian or Brian Ingerson) is better, just because, and the official management specifications for the freight train be darned to heck.
Anyhow, any time I hear the word "software engineer" (which I used to call myself), a small part of me dies. Software engineering is about maintainability, defects, and so on. That's all well and good, but there are reasons I did not become a car mechanic, and ironically one of the best "software engineers" here LOVES to work on cars. I like to dream.
As for seperation of concerns, it's all good and it's all important. use modules. Make things flexible. But stop when the building the layers of seperation make things un-fun. I've had MANY game projects in my earlier days killed because I over engineered them and never got them past the design phase, while if I just designed them functionality and let them grow as living programs, they would have come out successfully and probably been even cleaner in terms of source code.
For those who have read the SimCity 2000 strategy guide (ha!), the term is "Nucleated Expansion". It's a horrible way to design cities, but it allows needs to be treated as they arise, rather than pretending to have an omniscient view and building "architecture". Meanwhile, it is ALWAYS important to have modules and modularity, but the general concept is "meta-modularity" -- do not design modularity for specific things into the program -- design modularity for everything, knowing that if you need to change something -- anything -- you can change it, not just certain layers. And don't design modularity that you don't need immediately, just be able to add it later. Code is allowed to change.
Despite me not believing it for a long long time, I am finding my functional-programming endeavors result in code that is much more maintainable, flexible, and bug-free than my "software engineered" mega-OO systems (though I still use objects, that's beside the point -- the principle of functional programming is basically lego-like expansion). Naturally, though, you don't want crappy legos, and you don't want to build a horse that looks like a cow. Exercise skill with your legos, but have fun. Who says your design diagrams have to be in 2D with boxes and arrows, after all. Two dimensions and stack diagrams get tiring. Not everything should be a stack diagram of layers and plugins just because that's the way executive powerpoint shows make things appear. The much-revered LISP is essentially an organic blob when you finish a program, and it's writable top-down, and when you are done you really don't need a design diagram, because it just IS. So begins my process of unlearning, selecting and deselecting from my "Software Engineering" indoctrination just as Jefferson edited his Bible. Ironically Jefferonson's Bible and rejection of Calvinism's constructs is more heretical now as then, just as hacking and embracing "fun" programming is more heretical now than in the 70's and 80's. Yes, "goto is considered harmful", but all that we take as canon (maybe goto is warranted) carries a poison pill, and our programming loses soul, just like if we still had to make flowerbox comments (today we call this javadoc). Is there a point here? Of course not, there is no point. Or maybe there is...
As the Ruby creator (Matz) said, remember what it felt like to program when you were 14? Are you having as much fun now? Over-design is the mind-killer. This is, to some extent, the pitfall that "Software Engineering" has become. Maintaining code. Coding so that less-skilled developers can work equally well with your code. Dumbing down programming. Nothing is allowed to be shiny, shiny is evil. Nothing is allowed to be complex, complex is evil.