the disadvantages of mini-languages

[metaperl]

Dominus in the Text::Template docs

When people make a template module like this one, they almost always start by inventing a special syntax for substitutions. For example, they build it so that a string like %%VAR%% is replaced with the value of $VAR. Then they realize the need extra formatting, so they put in some special syntax for formatting. Then they need a loop, so they invent a loop syntax. Pretty soon they have a new little template language.

This approach has two problems: First, their little language is crippled. If you need to do something the author hasn't thought of, you lose. Second: Who wants to learn another language? You already know Perl, so why not use it?

Rob Nagler in the "Re: Doubts about nested %if in spec file" thread on the Redhat RPM mailing list

Me not Pythoneer. Me Perler. Greenspun used Tcl for ACS. The commonality is that we don't invent our own languages to solve our problems.

Rob Nagler in the "Re: separating C from V in MVC" thread on the modperl mailing list

 
Andy Wardley writes: 
> Because Perl is a general purpose programming language. 
> TT implements 
> a general purpose presentation language.  A different 
> kettle of fish altogether. 
 
These are the reserve words of TT: 
 
    GET CALL SET DEFAULT INSERT INCLUDE PROCESS WRAPPER  
    IF UNLESS ELSE ELSIF FOR FOREACH WHILE SWITCH CASE 
    USE PLUGIN FILTER MACRO PERL RAWPERL BLOCK META 
    TRY THROW CATCH FINAL NEXT LAST BREAK RETURN STOP  
    CLEAR TO STEP AND OR NOT MOD DIV END 
 
Looks an awful lot like the same keywords in any general-purpose 
programming language.

Considered by htoug: "Point out that the original node has been heavily updated" (keep/edit/delete vote = 7/31/2).
Unconsidered by davido: Consideration reason documented. No further action requested.

[Corion]

While your exposition is good and raises good points, the conclusion is wrong. Mini-Languages have their place, as domain-specific tools for expressing domain specific rules. Two examples:

• make - like its greater sibling, Prolog, make employs its own minilanguage for specifying rules and actions which will be verified respectively executed in a fashion to reach a goal. The make syntax has no nice counterpart in Perl.
• SQL - SQL is a very restricted language with one goal - to unify queries against database servers. While DBI does not have its own proprietary minilanguage, it employs SQL as a widerspread minilanguage.

Your ad-hominem attacks on the authors of CGI::Application and CGI::Prototype don't help either.

Update: Struck out the part of the reply that criticized the (now removed) attack.

[metaperl]

1. make as a language has shown itself to be inadequate. why do you think people invent things like cook, MakeMaker, Module::Build and makepp (a pure perl implementation of GNU make which extends it with Perl functionality?
2. ditto for SQL. Why was it extended with stored procedures? Why is it necessary or far more convenient to handle complex processing client side with Perl instead of server-side with SQL.

summary: neither language provided does anything other than show that mini-languages are limited and inflexible. They make easy things easy and hard things difficult or impossible.

[hardburn]

Each of those has replacements. Those replacements are in the form of other domain-specific languages. In either case, the domain-specific part was not fundamentally flawed; rather, the designers had insufficient understanding of the problem domain.

[Anonymous Monk]

make as a language has shown itself to be inadequate. why do you think people invent things like cook, MakeMaker, Module::Build and makepp (a pure perl implementation of GNU make which extends it with Perl functionality?

I don't know cook or makepp, so I won't comment on that. MakeMaker was created to create Makefiles for a specific purpose. MakeMaker is poorly equiped to generate a Makefile for, oh, say, a Linux distribution. Module::Build was created because some people don't like MakeMaker.

ditto for SQL. Why was it extended with stored procedures? Why is it necessary or far more convenient to handle complex processing client side with Perl instead of server-side with SQL.

Actually, it's not. It's never necessary to do data processing in Perl. As for it being "more convenient", it being more convenient usually stems from programmer blinds - creating tunnel vision. Most programmers can only program in one type of language category very well, and Perl (iterative programming - a language category that also contains C, Java, Python and PHP) and SQL (set algebra) are vastly different categories. It usually goes "Oh, I need to do something for every piece of data. Thus I need a loop. But loops are hard in SQL. Hence I need to do it in Perl."
Sometimes it's easier to do it outside the database. Sometimes, it's even more efficient to do it outside the database. But that happens less often than many people think.

[dragonchild]

Very often, mini-languages are meant to constrain the user for various reasons. Some reasons I can think of are:
• Performance - HTML::Template is a lot faster than Text::Template. The price is the lack of features. But, that was the conscious design decision when samtregar wrote it as a reaction to the Template Toolkit version 1.
• Maintainability - I have worked with Text::Template, and it's ugly. You have Perl code in your templates - this means that you can do anything. This includes the stuff you shouldn't be doing in templates per Separation of Concerns, like calling out to the mainframe to count something.
• Ease of use - not everyone knows Perl. In fact, most people who know Perl don't know Perl very well. As someone who has studied programming, in general, and Perl, in specific, for a long time, I have a very clear idea of how I want the Perl code to look and what I want it to do. If I have a web designer working on my templates (which I love to happen because it is part of MVC), I don't want to have to teach them Perl. In fact, I don't want them to even know that Perl will be calling their templates. I want them to work with HTML and CSS, plus a very simple mini-language like HTML::Template's or a subset of Template Toolkit's. That's all they need to know, which makes their and my jobs easier. They have to worry about less to get their job done and I have to worry about less to maintain what I've done.

So, I hope you see that Separation of Concerns can actually be used to argue for the use of mini-languages. It's not as black-and-white as you seem to hope for.

As an aside - I'd like to note that making an argument based on the thoughts of one person, however respected, is a dangerous proposition. I have no idea who Rob Nagler is, and he sounds pretty smart based on the quotes you've shown, but he's still one person with opinions, some of which are both right and some of which are wrong. A better argument would be based on quotes from a multitude of sources, at least some of which have been seen to argue with some of the others. That would provide your argument with a sounder footing and it would be more compelling.

[metaperl]

I'd like to note that making an argument based on the thoughts of one person, however respected, is a dangerous proposition.

Rob Nagler is simply echoing my thoughts and opinions in a far more articulate, powerful and informed manner than I am capable of.

Likewise, when it comes to CPAN modules, both CGI::Prototype and Class::DBI have done for CGI and DBI what I had been inching towards but not adept enough to actually implement.

[hardburn]

hardburn tosses metaperl down a Turing Tarpit

"General Purpose" does not mean it's good for everything; only that it's possible. BrainF*ck is technically a General Purpose language (in the sense of being Turing complete), but do you really want to write everything in BF?

What do you do when your main language doesn't make it easy to solve your problem? (This will happen in any language, even one as large as Perl). You have two options:

1. Write an API
2. Write a mini-language

As far as your maintance programmer is concerned, the two actually don't make a big difference. They either have to learn a new API, or a new language. If the problem is small, then either solution should be small. If the problem is large, then either solution should also be large. But I bet one of the above solutions will be significantly easier than the other, and a responsible programmer should learn to create both and apply them as necessary.

Forgot to add a feature that the end user needs? In either of the above cases, they still need to come back to you to add it.

No, we don't need another templating language (or API, for that matter). But what does HTML::Seamstress work on? HTML, which is a kind of mini-language. So you're not getting away from mini-languages by using that module.

What I can't stand is the attitude that Perl is the One True Way. TIMTOWTDI does not mean "there is more than one way, but that way is done in Perl". There are some really interesting ideas out there that Perl doesn't use (like Hindey-Milner type inferencing), and would have to be changed into an entirely new language to support.

[metaperl]

What do you do when your main language doesn't make it easy to solve your problem? (This will happen in any language, even one as large as Perl). You have two options: 1. Write an API 2. Write a mini-language

Or, in the case of Parse::RecDescent you write an API which in places requires a mini-language of sort.

[metaperl]

Forgot to add a feature that the end user needs? In either of the above cases, they still need to come back to you to add it.

And how if functionality extended in a mini-language versus a general purpose language? Read the quote by Dominus above carefully before answering.

[hardburn]

It seems to me that the hypothetical template author (and no doubt this story has happened many times in Perl's history) had no problem extending their mini-language. Their real issue was solving a problem that is already solved.

[jdporter]

I downvoted your post for the use of inappropriate language. I kind of like your point, but I think you've gotten way too excited about it. It's an interesting but controversial subject. I think you could have simply made your points and asked for discussion, rather than saying "mini-languages are evil" and insulting everyone who thinks they're in any way useful.

[Anonymous Monk]

mini-languages suck

While I agree that many mini-languages suck (just like many languages) I don't think the concept of mini-languages sucks. Mini-languages are good if they help us to express outselves less ambigiously, more clear, and/or more efficient. Perl consists of many mini-languages, some obvious, some less obvious. Take them away, and you're left with a language that isn't as useful as it's now. To name a few:

• Formats. Obvious mini-language, but hardly used nowadays.
• Format arguments of (s)sprintf, and POSIX::strftime. Wouldn't want to miss them - having to construct strings in an other way would be awkward.
• Format arguments for pack/unpack. See above.
• Prototype declaration. A simple mini-language, but it's still one.
• Numerical literals. You might not think of it as a mini-language, but it sure is one, and more complicated then at first sight.
• Second argument to open.
• Regular expressions. Can't live without them.
• Character classes inside a regular expression. A mini-language inside a mini-language.
• POSIX character classes. They appear inside character classes, giving a mini-language inside a mini-language inside a mini-language.
And I'm sure you can think of others.

As for other mini-languages, you can pry SQL, vi commands, *roff/pic from my dead, cold fingers, but not before that.

But feel free to burn all HTML templates to the ground. I've no use for them.

[BrentDax]

Before you pan mini-languages, take a look at the Parrot distribution. Parrot uses preprocessing and mini-languages extensively to make it easier to maintain. Opcodes, the JIT compiler, and PMC (variable) classes are largely written in mini-languages. Documentation is in a mini-language you may have heard of called POD. Configure uses a simple templating mini-language extensively--often to generate Makefiles, another mini-language. Code for Parrot is written in Parrot assembly, but a huge portion of the core is devoted to handling Parrot Intermediate Representation, a mini-language that makes the assembler more usable by offering friendlier syntax and automatically allocating registers.

Used judiciously, mini-languages can save a great deal of time. HTML templating isn't necessarily the right time, but that doesn't mean they're not useful in other situations.

[PodMaster]

No person in their right mind would have required HTML::Template for CGI::Application.

It's only optionally required (not a prerequisite). If you don't use it, it's not required. Get your facts straight before trying to make some point.

[itub]

I'm confused by the replies to this post. Was the original post edited heavily? That would be a Bad Thing.

[metaperl]

If you had seen the original post, you would not think that the current post is a Bad Thing At All :)

I just want to say that I am grateful. Grateful for editors at perlmonks who are not overly restrictive and let me go through a phase of venting. Grateful that we have world famous perl experts like merlyn here in our chatterbox participating with us just like another guy at the bar. Grateful that we have an outlet where we can meet and share ideas.

[itub]

Just to clarify, I don't think it's a bad thing if you change your mind or even decide to retract a post altogether. What is a bad thing is all the confusion caused by reading replies that are out of context when the original post is edited or removed. But don't worry, let's move along...

[autarch]

Why did you use the same quote as I used on alzabo.org? Is it even relevant to your page? And why did you use the same styles too?

[Aristotle]

This is rather a moot point. Mini-languages suck insofar as they are often restrictive and insufficiently expressive, but whether you use Perl inside your templates or a template-specific language, assuming the language is fairly expressive, is irrelevant.

Think about it: would you write a system kernel in Perl? Or rewrite a very simple Shell script in Perl instead? A complex Perl script in C? Or a Shell script in C? There are more language examples, some already mentioned, like SQL and make.

The fact is that languages are geared toward certain domains. Perl is a decent language for templates (not nearly as annoying as C would be for the purposes of complex Perl scripts, f.ex), but templating-specific languages can still make things easier for writing templates. They just need to be expressive enough to handle the job; something which homegrown templating systems generally fall short of. The Template Toolkit OTOH was designed with an abundance of functionality from the word go, and indeed it does a very good job at what it's made for.

[aufflick]

I have often been through similar frustrations. It's one of the reasons that I like HTML::Mason.

Mini-languages basically bring greater structure or different forms of expression. They definately have a place, but I think they can be over-used.

HTML templating, though, is one place where I think they have value. If you believe that there shouldn't be any significant logic in the template files, then an enforced mini-language is good. Also if your template writers are html types using dreamweaver, then a well designed mini-language designed to look like tags (cf. Apache::ASP, Aolserver).

Another benefit can be a different form of expression. Not that SQL is a mini-language, but as an embeded language it holds benefit because some things are better represented by declarations.

One pet peeve about the over-use of mini-languages is config files. There are SO many config file formats, and you often can't do what you want, so you write bogus scripts to read and edit them for you (cf. cobalt qube). I once blogged that Tcl is an ideal config file language: it is simple to embed; simple to learn; and if you don't need any evaluation in your config, then the file will look like a simple text format config file.

(Of humerous note: the thing that prompted my blog entry was discovering, to my painful amusement, that the sendmail.cf format is turing complete...)

I guess my point here is not that mini-languages are bad, but that there are too many poorly concieved re-inventions - if you think you need a mini-language, why not use an existing one like ASP or Tcl.

[hardburn]

The point of a mini domain-specific lanugage is to make a single task really, really easy. You almost certainly have a host language, but you're choosing not to use that because it doesn't make that single task really, really easy.

Imagine if, every time you wanted to parse text, you had to build your own Finite State Machine, node-by-node. You could hardly get any work done. Fortunately, Perl provides a regular expression domain-specific langauge that builds the FSM for you.

Would replacing regexen with Tcl help? No, not at all. Deal with it :)

[aufflick]

Regular expressions are a GOOD example of where domain-specific languages are beneficial.

A turing complete sendmail.cf is a domain-specific language gone BAD.

Apache rewrite rules, for example, use regular expressions - that is good. They could have invented their own language to match urls - that would have been bad.

Should I have added "or SQL or regular expressions" to the last sentance of my original comment?

[mstone]

Okay.. First of all, you need to spend some time meditating on the definition of the word 'language' as it applies to computer programming. It's a heavily overloaded term, and your post deals with a specific confusion of meaning between two very different definitions. On the one hand, we have 'languages' like Perl which are a collection of keywords and syntax. On the other hand, we have 'languages' like finite automata, pushdown automata, and Turing machines.

Languages like Perl are really just syntactic sugar coating for a set of basic concepts which we use to express automata or Turing machines. A 'general purpose' language (of which there are about ten bazillion) supports the concepts necessary to create a Universal Turing Machine, an idea more commonly expressed as: "a language that can't compile itself isn't worth using." Of course, that isn't terribly difficult.. you can build a UTM with the operations '++' and '--', two arbitrarily large named integer variables, a big lookup table, and a whole lot of time.

Things like finite automata and Turing machines are the theoretical languages which identify the various types of problems a computer programmer can solve. Broadly speaking, there are five such languages:

1. Simple expressions which support Atomic values
2. Parametric expressions which support Simply enumerable sets
3. Finite automata which support Regular sets
4. Pushdown automata which support Context free grammars
5. Turing machines which support Recursively enumerable sets

And at the end of the day, ain't a damnthing any programmer can do that doesn't fall into one of those categories.

Syntactic-sugar languages like Perl support several different ways to express any one of those theoretical languages. %Hashes and @lists are parametric expressions, but so is a function that takes a parameter and returns a value. Perl's regular expression syntax gives us a convenient way to define finite automata, but that doesn't stop any program from having its own state chart. You can build a pushdown automaton with a finite automaton and a stack, or you can just make a set of nested function calls.

When you get down to basics, programming consists of nothing but creating new syntactic-sugar languages appropriate to the task at hand. Every function you name is a new keyword with its own set of semantics. And unless you're really into frustration and wasted time, the language you create has to be powerful enough to solve the problem at hand. You can't balance parentheses with a finite automaton (1), and you can't do binary multiplication with a pushdown automaton.

(1) Yes, yes.. I know Perl's regex engine has a pragma for just-in-time compilation of targets which supports infinitely extensible, recursively defined patterns. A recursively-defined *anything* is a context free grammar at the very least. The pragma boosts Perl's regex engine into the realm of pushdown automata rather than proving you can solve PA-complete problems with a finite automaton. If you wish to argue otherwise, please include an implementation of the just-in-time pragma written only in Perl's regex syntax without the pragma.

Now, it's interesting that you chose to complain about the mini-languages associated with template systems, because arbitrary string replacement -- at least if you do it repeatedly -- happens to be a Turing-complete problem. In fact, 'arbitrary string replacement' is a fairly good description of lambda calculus.

So.. given that you need a Turing-complete language just to describe the problem any template system is trying to solve, you're pretty much stuck with two options:

1. Go the PHP route (like Greenspun), and turn every page into a first-class executable code object, or
2. Create a new language which supports string substitution, and write a compiler to process the pages.

Now, speaking as someone who's had to manage sites composed of first-class executable pages, I can say with conviction that the separation of concern issues others have raised are NOTHING compared to the synchronization and version-control issues inherent in trying to ride herd over a jillion different page/programs all loosely tied to the same code base. Sifting through all the pages in a site trying to find and update the cut-and-paste code for a given display item is time consuming. Trying to ferret out all the instances of cut-paste-and-modify code is exhausting. Factoring out a common code base for the entire site and then bringing all the pages in line is a dot-X upgrade at the very least. And it lasts right up to the point where someone wants to do something new.

So.. it seems to me that you're objecting to the creation of a specific kind of syntactic-sugar language, which just happens to be Turing complete because that's necessary to describe the problem at hand, without presenting any really detailed argument as to why this particular new lanugage might be bad. Your only two real points: "If you need to do something the author hasn't thought of, you lose," and "who wants to learn another language?" apply equally well to the API of any Perl module on CPAN. Those are mini-languages too, and some of them are fairly complex. Show us some evidence that you're familiar with the problem domain before dismissing one kind of solution out of hand.

[robnagler]

We don't program Turing Machines, just like we don't program in machine code any more. Languages are a means of expressing what we think in terms a computer can understand. If we can't express ourselves, we can't tell the computer what to do. Put another way:

Programming languages teach you not to want what they cannot provide. You have to think in a language to write programs in it, and it's hard to want something you can't describe.

Paul Graham, ANSI Common Lisp, Prentice Hall, 1996, p2.

Perl is an ugly language, because it has evolved over the past two decades driven by real customer requirements. That's a good thing. Languages like Lisp, Prolog, and Haskell are beautiful, but I would guess that more function has been delivered in Perl than in the other three combined. Perl is available, and availability matters more than beauty to most people.

print "Hello, world!\n";

This code is available to just about any programmer. The \n is a term from a well-known mini-language. print is a well-known verb. "Hello, world!" is a well-known example. We understand the above, because it is familiar to our context.

main:   mov     dx,str
        mov     ah,9
        int     021
        ret
str:    db      "Hello World!$"

You probably understand this, because you recognize it is a canonical example. And, I already established the context with the first example. However, unless you are an old DOS hacker, you probably don't know what 021 is. It's the name of an API, and ah and dx hold its parameters. Just like, [] is an API as in ("Hello, World!")[0], and ("Hello, World!") and 0 are its parameters. If you don't know Perl, it might be hard to see that, especially if it looked like: (0)[0]. What does that smiley mean?

You probably understand this sentence:

http://www.bivio.com/goto?x=http://yahoo.com

However, do you understand this one?

http://www.bivio.com/my-club-site/mail/post?fc=aMTc_!cZmlzY3Vz

There are many mini-languages in the above sentence, but all arose out of necessity. ASCII provides a common character set, and won out over EBCDIC, for random reasons. http:// was needed to allow the browser to switch between multiple protocols. www.bivio.com arose, because the Internet needed a distributed naming system when the hosts file got too large to be shipped around every night. my-club-site arose, because our support team needed to give people generic URLs to their private clubs. /mail/post is just a name for a message, that's readable and short. It allows our programmers to read the server logs easily. fc= evolved, because HTML is stateless. The stuff after the fc= tells the interpreter where the particular browser window should return to after you post the message or cancel without storing anything on the server.

The above mini-languages are simple, declarative, and satisfy a basic rule: keep it simple. They aren't beautiful.

Mini-languages which diverge from the KISS principle are an excellent distraction from what you are being paid to do. That's when the mini-language takes on a life of its own, and ceases to be a tool to solve a problem, but a problem in itself. This can be excellent for job security. However, you are programming ineffectively, and you want to be a good programmer. Someone has already generalized and abstracted most programming constructs for you. If you ignore that fact, you are working for yourself, and cheating your customer.

An excellent example of this is the syntax for formatting this reply. HTML was an excellent generalization. However, the special syntax for links is not HTML (although you can use HTML). While it is totally cool that you can make a link with square brackets, it's pretty odd all the same. It's not POD, it's not HTML, and it conflicts one of the most common syntactic elements in the subject matter of this site. You'd have to look at the source of this text to know that [ is the way to get a [ in your posts. And, I didn't have a ready example, and it took quite a bit of searching to figure out why my square brackets weren't showing up. This may be funny in this context. It isn't funny if it is Friday night, and you have to debug somebody else's undocumented, mini-language before you can check-in and go home.

[metaperl]

Languages like Lisp, Prolog, and Haskell are beautiful, but I would guess that more function has been delivered in Perl than in the other three combined.

Funny you should mention Haskell. I am teaching myself this language right now. It took 1st and 2nd place in the 2004 Functional Programming Contest (which any language can enter).

[dimar]

...

Update:Uggh original post totally re-edited, and toned-down obviating this response. Removed to avoid being taken out-of-context.

[Solo]

...original post totally re-edited, and toned-down ...

Yes, it was--and changed without including any Update notice as politeness and common community practice would dictate.

[borisz]

I love mini-languages.
When it comes to MVC, I have the advantage that someone else can work on the mini-language. The other person can learn the new language in some hours. But not my voodoo perl!.
The mini-language is the glue between the designer and the programmer. And both do only that part that they really can do.