Code Conflation: Considered Harmful?

[rvosa]

In my spare time, I am working on a homegrown content management system. It's built around a fairly standard setup that consists of:

• perl, with CGI, XML::Twig, XML::RSS and DBI
• MySQL
• Apache (still have to figure out mod_perl)
• X(HT)ML templates, styled using CSS

So, on any page that this system serves up, when a user clicks on a link, the request goes to the server, it sends a perl script into action which asks XML::Twig (which I think is fantastic) to parse an XHTML layout template and deploy the twighandlers that populate the page with content from the database based on the conditions set by the CGI parameters. For example, when the parsing process comes along a <span class="timestamp"/> it inserts the 'last updated' timestamp for that page from the database as element text for the <span/>. Once the parsing is done, the twig is printed out, the server sends it back to the user, who is happy with the results.

As I was working on this I noticed that I work from the underlying idea that different languages (perl, SQL, XHTML, CSS) must not be combined within the same code block or file because they are meant to do different things (request processing, database interaction, layout and style, respectively). So that, for example, if you know a wizard in web design, she can work on the layout without having to work around perl code in the same file.

Most examples that demonstrate CGI programming show how a perl script might send out strings of HTML or text that are more or less hardcoded into the script, e.g.

print qq{Content-type: text/plain\n\nHello World!\n};

This strikes me as bad practice because two different things (the code that gets executed on the server side and whatever is sent back to the user) are combined in the same file, creating a maintainability nightmare.

Similarly, most examples on php start out by embedding PHP code into HTML files, and the same is true for embedded perl and for things like Mason (I think), as well as most other systems commonly used to generate dynamic web pages, such as ASP. Aren't these all bad approaches? Shouldn't different languages that are meant to do different things be separated one from another? Wasn't there a sage who spoke wise words about this?

What's your view?

[dragonchild]

You have hit upon something most experienced developers call "Separation of Concerns". It's the underlying principle behind MVC - Model / View / Controller. (It's called by other names, but that's the most popular.) It was also a driving force behind creation of OO methodologies.

This is usually seen in Perl with the use of templating systems, like HTML::Template or Template Toolkit, for separation of Perl (business logic and engine, or Model / Controller) from output generation (View). Other distributions, such as SQL::Catalog, can help with the separation of SQL (data logic, or Model) from Perl (business logic and engine, or Model / Controller).

Many people, including myself, tend to provide for a different separation between HTML, CSS, and Javascript. I look at the View layer as generally composed of three elements:

• Layout (HTML)
• Style (CSS)
• Client-side interaction (JS)

Perl tends to not be involved with those at all, except for ease of maintenance in the assembly of the pieces. So, I tend to have different directories for each (/templates, /javascript, and /css). So, if I were truly separating out the five languages, I would have the following directories:

• /lib (Perl modules)
• /sql (SQL catalog) (I tend to not use this one, preferring to embed my SQL within a separate Perl module designed specifically for database interaction.)
• /templates (H::T or TT templates that bring together the various View components, plus contain HTML)
• /javascript (Client-side interaction)
• /css (Style sheets)

Most examples you tend to see are for complete newbies, who just want something to get them up and running quickly. The reason for that is that newbies need to have instant gratication, or they give up. Later, if they are interested, they will do what you're doing - come to those with experience and ask for advice. At that point, you'll improve your code. :-)

[CountZero]

I can only speak from personal experience and what works for me does not necessarily work for someone else.

I have made a dynamic website (Apache2 + MySQL + (mod_)Perl). Template::Toolkit provides the templates; some additional "raw" perl left and right for the really tricky things and it all outputs XML, which is then send through an XSLT processor (part of Template::Toolkit), prettyfied by CSS and output as pure HTML to the user's browser.

The Template-layer builds the basics and is easy to modify; XSLT builds the bridge to HTML (if necessary this could be done "client"-side but not all browsers can do it and I don't like to send pure XML to the user. It is far too easy to parse that and "steal" the data); CSS is about the visual aspect.

I find this a nice and clean separation of concern.

With the same effort I could have Toolkit::Template output HTML, but that would be less "clean" as then HTML and logic would again be joined in one file.

With a little bit more effort I could perhaps have used a module to write the XML-output directly, well that is maybe something for the next release!

[FoxtrotUniform]

People seem to like the idea of separating code from presentation, and in general it's a good idea, but it can impose an extra burden of complexity. For a small project, jumping on the "reuse and generality" bandwagon isn't a particularly good idea, unless you want to turn it into a large project.

There's some pretty good discussion of code-separation in the context of HTML templating at Reusable template components with HTML::Template.

[Anonymous Monk]

It depends on the project. The worst approach, IMO, is deciding what your style will be without taking what kind of project it is into account. For smaller projects, or small teams of developers who are all good in the various languages used, it might be that separating things adds more complexity then is necessary.

Personally, I find separation on functionality far more important than separation on language. It shouldn't be needed to reshuffle files around if you rewrite part of your program in C. I do agree though that there shouldn't be SQL mixed with Perl (C, Java, Python, whatever). All SQL code should live in the database - and all the code is allowed to do is to call stored procedures. And the code calling the stored procedures should be in a layer itself as well. Hence, I wouldn't use MySQL, as that doesn't have stored procedures. (Or triggers. Or subselects.)

[dragonchild]

I do agree though that there shouldn't be SQL mixed with Perl (C, Java, Python, whatever).

So far, so good. I don't think anyone will disagree with that statement.

All SQL code should live in the database - and all the code is allowed to do is to call stored procedures.

In other words, there should be an API mapping business logic to the datamodel. Your statement goes a little farther in that you actually call for a specific architecture to do that. However, I am in agreement with you, on principle.

And the code calling the stored procedures should be in a layer itself as well.

So, what's the difference between having calls to stored procedures (which is SQL) and the actual SQL statements in this layer? You're just adding an additional layer between the code calling the business-logic DB functions and the actual SQL.

Now, it can make sense to do this. mpeppler is a big fan of this solution and he has a good reason for it - the data model is owned by the DBA, not just the database instance. APIs exist to allow people in one group to call functions owned by another group. Creating APIs for their own sake is a good way to get lost in a maze of Java-like passages, all alike.

Hence, I wouldn't use MySQL, as that doesn't have stored procedures. (Or triggers. Or subselects.)

Oh, really? 4.1.4 has subselects - I'm using them right now in a production web application. The 5.x line (currently under development) will have stored procedures, triggers, and updateable views. It'll also have usage of multiple indices per table in the same query. (A feature, I might add, that neither Oracle, SQL*Server, DB2, or Sybase have.) There is also work being done on clustered MySQL databases, using an architecture with zero single points of failure (unlike Oracle RAC, which has a single point of failure). NDB integration should be complete by Q2, 2005, AFAIK.

I would challenge you (or anyone else, for that matter) to actually find the facts before promulgating FUD about any opensource project. Do your projects stay stagnant for 3 years in terms of features? Mine don't, and neither does MySQL.

Update: added "single" when discussing the number of failure points in MySQL/NDB clustering. Thanks, sandfly!

[sandfly]

Zero points of failure?

I think you're decrementing when you should be incrementing. The question is: what's the minimum number of components that need to fail for the system to fail? You're saying that for Oracle RAC, it's one. I'm not clear what you're claiming for the MySQL cluster, but obviously if every CPU in the cluster burned out, that would do it, so there is an upper bound.

[exussum0]

As dragonchild wrote, use MVC. And you are right, it is bad. But this is what differentiates a programmer from a software engineer. I can take the most complex program in the world, and write it as assembler, or c with thousands of globals and no stored procedures, plus gotos. As a computer scientist, the programs are identical. But as a software engineer, it's not, as other factors come into play,those of which, are maintainability, scalability and sound design. Computer scientists don't necessarily write "bad code". A computer scientist is more concerned about ability, where a software engineer is worried about the possibilities.

[SpanishInquisition]

Balancing engineering with over-engineering is how we ended up with relatively safe business languages (which can often be good in Corporate America) and the expense of stiffling creativity.

So while one can say this is a distinction between a "programmer" and the (shudder as I use the word) "Software Engineer", one most also forget what is lost from the world of computer science. Simplicity. Elegance. Mathematical Purity.

I desire to write programs in this pure form, I don't always succeed, but sometimes you have to know when building a skateboard from scratch is better than building a freight train from scratch. Or when building a time machine (think Damian or Brian Ingerson) is better, just because, and the official management specifications for the freight train be darned to heck.

Anyhow, any time I hear the word "software engineer" (which I used to call myself), a small part of me dies. Software engineering is about maintainability, defects, and so on. That's all well and good, but there are reasons I did not become a car mechanic, and ironically one of the best "software engineers" here LOVES to work on cars. I like to dream.

As for seperation of concerns, it's all good and it's all important. use modules. Make things flexible. But stop when the building the layers of seperation make things un-fun. I've had MANY game projects in my earlier days killed because I over engineered them and never got them past the design phase, while if I just designed them functionality and let them grow as living programs, they would have come out successfully and probably been even cleaner in terms of source code.

For those who have read the SimCity 2000 strategy guide (ha!), the term is "Nucleated Expansion". It's a horrible way to design cities, but it allows needs to be treated as they arise, rather than pretending to have an omniscient view and building "architecture". Meanwhile, it is ALWAYS important to have modules and modularity, but the general concept is "meta-modularity" -- do not design modularity for specific things into the program -- design modularity for everything, knowing that if you need to change something -- anything -- you can change it, not just certain layers. And don't design modularity that you don't need immediately, just be able to add it later. Code is allowed to change.

Despite me not believing it for a long long time, I am finding my functional-programming endeavors result in code that is much more maintainable, flexible, and bug-free than my "software engineered" mega-OO systems (though I still use objects, that's beside the point -- the principle of functional programming is basically lego-like expansion). Naturally, though, you don't want crappy legos, and you don't want to build a horse that looks like a cow. Exercise skill with your legos, but have fun. Who says your design diagrams have to be in 2D with boxes and arrows, after all. Two dimensions and stack diagrams get tiring. Not everything should be a stack diagram of layers and plugins just because that's the way executive powerpoint shows make things appear. The much-revered LISP is essentially an organic blob when you finish a program, and it's writable top-down, and when you are done you really don't need a design diagram, because it just IS. So begins my process of unlearning, selecting and deselecting from my "Software Engineering" indoctrination just as Jefferson edited his Bible. Ironically Jefferonson's Bible and rejection of Calvinism's constructs is more heretical now as then, just as hacking and embracing "fun" programming is more heretical now than in the 70's and 80's. Yes, "goto is considered harmful", but all that we take as canon (maybe goto is warranted) carries a poison pill, and our programming loses soul, just like if we still had to make flowerbox comments (today we call this javadoc). Is there a point here? Of course not, there is no point. Or maybe there is...

As the Ruby creator (Matz) said, remember what it felt like to program when you were 14? Are you having as much fun now? Over-design is the mind-killer. This is, to some extent, the pitfall that "Software Engineering" has become. Maintaining code. Coding so that less-skilled developers can work equally well with your code. Dumbing down programming. Nothing is allowed to be shiny, shiny is evil. Nothing is allowed to be complex, complex is evil.

[exussum0]

As the Ruby creator (Matz) said, remember what it felt like to program when you were 14? Are you having as much fun now? Over-design is the mind-killer. This is, to some extent, the pitfall that "Software Engineering" has become. Maintaining code. Coding so that less-skilled developers can work equally well with your code. Dumbing down programming. Nothing is allowed to be shiny, shiny is evil. Nothing is allowed to be complex, complex is evil.

(snip)

For those who have read the SimCity 2000 strategy guide (ha!), the term is "Nucleated Expansion". It's a horrible way to design cities, but it allows needs to be treated as they arise, rather than pretending to have an omniscient view and building "architecture". Meanwhile, it is ALWAYS important to have modules and modularity, but the general concept is "meta-modularity" -- do not design modularity for specific things into the program -- design modularity for everything, knowing that if you need to change something -- anything -- you can change it, not just certain layers. And don't design modularity that you don't need immediately, just be able to add it later. Code is allowed to change.

Bingo. You hit it right on the head. This is what a true engineer would do. Engineering is a process where you make it best for the situation. When you under engineer, it's just over simplification. When you over, you are just complicating matters. The crux of the situation is there are no drawn lines anywhere to measure against. I agree and disagree. Making things overly complex is always bad. Using patterns is almost always good. Patterns promote consistency. There are anti patterns or ill placed patterns which prevent growth.

For instance, I had the impression that a system I would design, would use very simple queries to report data with little variance on how the report would look. (missing columns or reordering the columns). low and behold, they do matter and my design goes out the door. next version does they displaying a little more flexible, things are easier to develop for, everyone works a bit better.. a bit faster. This is the engineering that needs to take place. Is a toothpick the answer, or something with a lot of gears?

[KeighleHawk]

I wish somebody had commented on who the "sage" was you refer to. I would like more reading on this subject.

I have seen functional programming come up a few times in various forums and need to do some reading there myself.

I have spent some time myself thinking on coding style and seperation of concerns. From much I have seen so far I do not actually seem to agree with the MVC model, though I seem to agree with the intent. I say **seem** because I have a sneaking suspicion the code bases I have seen using Templates are just done badly.

However, I still have a feeling I would not like MVC because it seems to place the "emphasis" if you will in the wrong place. Too often, when working in so called MVC frameworks, I find myself "wandering" the templates to troubleshoot code. This just **feels** wrong. It seems to me there should be a more directed way to quickly trace or travel to the problem code.

I realize my commentary here is somewhat fragmented and distracted. Again, I think it is simply because my experience to date has involved only bad examples of MVC and template usage. But to waffle once again, if a design concept makes it so easy to do it wrong, is there not something about the concept itself that is flawed?

Basically, I think MVC somehow revolves too much around the user experience (screens) and thus the templates become the focus for all coding. Eventually this translates to having to "walk the screens" to troubleshoot every problem. I don't think this is inherent in MVC intent, just too easy a trap to fall into. When you use a templating system that can call other templates, the result can be downright dizzying.

To summarize, I agree with you, I feel this approach is bad, I just can't properly articulate why...

[dragonchild]

You haven't seen good MVC architectures, then. The ideas behind MVC are very simple:
• Every thing that happens has one and only one place in the code that does it.
• Every piece of code is broken up into functional areas. You generally have
  • Formatting and layout (aka, View)
  • Business-related code (aka, Model)
  • Datastorage code (DBI-related stuff, generally)
  • The glue code that actually runs the stuff (aka, Controller)

Your complaint about having to wander templates has to do more with how bugs are reported. Generally, the user says "Screen XYZ is doing foo when it should be doing bar". So, the debugging generally goes something like:

1. Look at the tepmlating for screen XYZ. Is it not displaying the right stuff?
2. Look at the engine routine that makes the data for screen XYZ. Is it not calling the right business objects?
3. Look at the various business objects called for screen XYZ. Are they doing the right thing(s)?
4. Look at the datastorage behind screen XYZ. Is it storing the right stuff in the right way?

So far, you're still doing the same amount of work. In fact, you might even be doing more than you would in a more naively-architected system, because you have to open more files and follow through more subroutines. And, I suspect, this is where you're stuck at.

That's because you don't see the point behind MVC. MVC adds some complexity to the architecture of a system. But, it does so in order to reduce complexity at another stage. Once you fix something in MVC, it stays fixed all across the entire application. Period.

And, there's another point behind MVC - people with different skills can collaborate on the same project. Most web applications require that everyone know language X. MVC-architected applications require that

• the layout guys know HTML, CSS, and the templating language
• the coding guys know language X
• the datastorage guys know SQL and a little bit of language X

So, now, the coding guys don't have to know CSS. Wow, is that a relief!