Design Patterns Still Aren't

[simonm]

A Pattern of Misunderstanding: Fetishizing the Gang of Four

Over the last few weeks, after reading the first two articles in Phil Crow's "Perl Design Patterns" series on Perl.com, (1, 2) I've been troubled by a feeling that I couldn't quite put my finger on.

Finding myself with some unexpectedly-free time a few days ago (which is to say, while the blackout kept me offline) I paged through some of my patterns books (by candlelight) and thought about what had made me uncomfortable.

By the time the lights came on, I'd concluded that the Perl.com articles did not seem to have connected with the essential intent of the patterns literature. Moreover, the author's reading of the "Gang of Four" book was downright hostile and fetishistic: rather than responding to the content of the book, the focus was on dismissing or deflating the hype that has built up around it.

The reason the Gang of Four book is such a famous reference in the software world has a lot to do with the history of the patterns movement (and with the way fads develop and play out), but to focus on the patterns cataloged in that book, to the exclusion of the rest of the patterns literature, gives a lop-sided impression of what "software patterns" are all about.

What are Software Patterns?

In an attempt to refine a clearer definition of the software patterns field, I revisited a handful of books by different authors:

• Design Patterns: Elements of Reusable Object-Oriented Software (E. Gamma et al, aka "the Gang of Four")
• Analysis Patterns: Reusable Object Models (M. Fowler)
• AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis (W. Brown et al)
• A System of Patterns: Pattern-Oriented Software Architecture (F. Buschmann et al)
• Smalltalk Best Practice Patterns (K. Beck)

Although there's a fair degree of variation between the styles and subjects of these books, there did seem to be a handful of areas of consensus; I identified five shared characteristics, explained below along with some quotes from the original materials:

Patterns are A Type Of Documentation: As used in the software methodology field, patterns are a form of best-practices documentation. Their intent is to communicate the observations, analysis, and recommendations of experienced developers to others in an effective way.

"Patterns and pattern languages are ways to describe best practices, good designs, and capture experience in a way that it is possible for others to reuse this experience."
The Hillside Group, hillside.net/patterns

"Patterns are a recent software engineering problem-solving discipline that emerged from the object-oriented community... The goal of the pattern community is to build a body of literature to support design and development in general. There is less focus on technology than on a culture to document and support sound design."
J. Coplien, hillside.net/patterns/definition.html

"A pattern is a decision an expert makes over and over. For example, naming an instance variable after the role it plays is a pattern. Even though all the results (in this case, all the variable names) are different, they have a quality that is constant."
K. Beck, Smalltalk Best Practice Patterns, Preface

Patterns are Contextual Solutions: Each pattern is expressed as a written description of a design, solution, or technique that other developers have found repeatedly useful, along with a discussion of the context and challenges that the solution is attempting to address. (Anti-patterns are similar, but shift the focus to the problems they describe instead of the solutions.)

"Each pattern is a three-part rule, which expresses a relation between a certain context, a problem, and a solution."
C. Alexander, The Timeless Way of Building, p. 247

"A pattern for software architecture describes a particular recurring design problem that arises in specific design contexts, and presents a well-proven generic scheme for its solution."
F. Buschmann et al, A System of Patterns, p. 8

"This is a book of design patterns that describes simple and elegant solutions to specific problems in object oriented software design. "
E. Gamma et al, Design Patterns, Preface

Patterns are Both Observations and Recommendations: Patterns play a dual role as both descriptive and prescriptive documents. On the one hand, they recognize and name some technique that developers are already using. On the other hand, the pattern acts as a recommendation to others: if you're trying to solve the problem described in the pattern, consider using some variation of the solution it describes.

"The pattern is... at the same time a thing, which happens in the world, and the rule which tells us how to create that thing, and when we must create it. It is both a process and a thing: both a description of a thing which is alive, and a description of the process which will generate that thing."
C. Alexander, The Timeless Way of Building, p. 247

"Design patterns capture solutions that have developed and evolved over time... in a succinct and easily applied form."
E. Gamma et al, Design Patterns, Preface

Patterns are Interconnected: Patterns are collected in interconnected networks, called pattern systems. (A pattern system that aspires to fully cover some topic can also be called a pattern language, although various communities use these terms differently.) It's common for a pattern description to suggest other patterns that work with it (add an Iterator to your Composite!), or alternate patterns that might solve the problem differently. In concrete terms, this means that pattern-oriented books and sites are organized as a interconnected network of writeups, rather than a rigid sequential procedure or a hierarchical outline.

"Patterns do not exist in isolation -- there are many interdependencies between them... A pattern system ties its constituent patterns together. It describes how the patterns are connected and how they complement each other."
F. Buschmann et al, A System of Patterns, p. 360

"There is a structure on the patterns, which describes how each pattern is itself a pattern of other smaller patterns. And there also rules, embedded in the patterns, which describe the way that they... must be arranged with respect to other patterns."
C. Alexander, The Timeless Way of Building, p. 185

Patterns are Distinctly Named: Each pattern within a pattern system is given a distinct name. The name can serve as an abstraction that helps the developer think at a higher level: instead of seeing a soup of a dozen methods spread out over a handful of classes, you can think in terms of "a Publish-and-Subscribe interaction" or whatever. The name also facilitates communication between developers who share the same patterns: a developer familiar with the Observer pattern can guess what $target->add_observer( $self ) is trying to accomplish.

"The pattern name is a handle we can use to describe a design problem, its solutions, and consequences in a word or two. Naming a pattern immediately increases our design vocabulary. It lets us design at a higher level of abstraction. Having vocabulary for patterns lets us talk about them with our colleagues, in our documentation, and even to ourselves."
E. Gamma et al, Design Patterns, p. 3

What Software Patterns Aren't

Sadly, the idea of software patterns was popularized in a superficial way, such that many people have heard of them but lack a solid understanding of what they are. Thus, as originally pointed out by Mark Jason Dominus, what many people think are design patterns, aren't.

Inverting the consensus definition I came up with earlier, we can state that:

Patterns aren't Language Features: Patterns are a type of documentation that describe how experienced programmers use those languages, not optional features that the languages might or might not possess. At best, a programming language facilitates useful patterns, such that dozens of lines of Java can be replaced by a few lines of idiomatic Perl.

Patterns aren't Universally Applicable: Patterns are contextual solutions that describe how a particular problem fits with a particular solution. The Gang of Four patterns focus on challenges in object-oriented C++ code (with secondary consideration given to Smalltalk); that these might not be directly applicable to Perl programmers who prefer procedural code should come as no surprise.

Patterns aren't Platonic Ideals: Patterns are real-world observations from a particular development community. The patterns described by the Gang of Four aren't free-floating abstractions that somehow exist outside of time and space and were waiting for us to come and discover them; they're practical responses to various day-to-day challenges, including the limitations of C++. An attempt to document Perl Design Patterns should be looking at the best practices of the Perl development community, not just the parts that resemble a ten-year-old collection of patterns from a different language.

Patterns aren't Total Solutions: Patterns are interconnected with other patterns that support or compete with them; no pattern stands on its own as a silver-bullet for your software development challenges. The description of each pattern should review the context in which it is used, acknowledge the tradeoffs involved, and refer to other patterns that offer an alternative approach.

Patterns aren't Generic Concepts: Patterns are distinctly named so that developers can use that name as an abstraction and in communication with others. The Gang of Four knew that C++ had arrays and loop operators like for and while, but they weren't trying to write a general reference about all forms of looping. Trying to gather all of the looping-related techniques into one pattern would result in something too general to be of much use.

Patterns Mistaken

Re-reading the Perl.com articles with these ideas in mind, I was struck by the differences between the articles and the book they critique.

Here are three examples of confusing or contradictory references to patterns, taken just from the Iterators section of the first article. (Please understand that I don't mean to pick on the author of the Perl.com articles; such confusion is widespread and this is far from the worst offense.)

"If a pattern is really valuable, then it should be part of the core language." On the contrary, software patterns typically aren't "part of" a programming language, and it's often not clear what that would even mean. For example, Template Method is a pretty valuable pattern, but how would you make it "a part of Perl" in some concrete way?

"The inclusion of iteration as a core concept represents Perl design at its finest... Perl incorporates this pattern into the core of the language." This description confuses two related terms: "iteration" is a nearly-universal programming concept (cf. HQ9+'s 9 operator), whereas "the Iterator Pattern" is a specific object-oriented design solution (cf. the Gang of Four). The author goes on to describe a Perl pattern that is quite different from the Gang of Four's Iterator objects; one might instead call it the "Return Aggregates as a List" pattern. Blurring the distinction between these different concepts reduces the communication value of the names they've been given.

"To see that foreach fully implements the iterator pattern, even for user-defined modules, consider an example from CPAN: XML::DOM." To see an instance of what the Gang of Four meant by "the iterator pattern," consider another example from CPAN: DBI. Sure, there are times when you can simply call $sth->fetchall_arrayref() to put all of the matching rows into a list, but in some circumstances you want an iterator object (aka a "cursor") that lets you loop through the records one by one with while ( @row = $sth->fetchrow_array ) { ... }. That's clearly an instance of the "Use an Iterator Object" pattern rather than the "Return a List" pattern.

More broadly, the idea of describing best practices in a general, reusable way seems to have been lost somewhere along the way. For example, I would expect a presentation of the Template Method pattern to show how the master method defined in the superclass calls different internal methods depending on the subclass of an individual object. Having the subclass insert the methods into the superclass' namespace, such that you can not define different behaviors for different subclasses, is not a widely reusable approach, and is certainly not the type of functionality envisioned by the Gang of Four Template Method pattern.

Towards a Perl Iterator Pattern

So if software patterns, such as using iterator objects, do play a role in Perl, albeit a reduced one due to its dynamic nature and widespread use of the other patterns like "Return a List," what might a description of such a Perl pattern look like?

The Gang of Four spend fifteen pages discussing their Iterator pattern, and I'm not going to embed an equally long writeup, but here's a very rough draft of what I think such a pattern might look like, in pseudo-perldoc format. I've used closures in the below examples, in response to the Perl.com article's insistence that "objects are overkill," but a complete pattern writeup would also illustrate the use of classes of iterator objects. See the PerlDesignPatterns IteratorInterface page for a further effort in this direction.

NAME
Iterator Pattern - Use a separate entity to facilitate a loop.

PROBLEM
* A complex data structure may need to allow looping over its contents.
* There are several types of data structures that you need to loop over.
* You might need to loop over a data structure in different ways.
* You might be fetching data elements incrementally to avoid memory bloat.

SOLUTION
Create a new entity which is responsible for selecting the elements in order. Modify your loop code to create the proper iterator, then request the elements from it.

RELATED PATTERNS
If your collection is not particularly large or complex, you might prefer to "Return Aggregates as Lists".

DISCUSSION
There are many ways of implementing this pattern in Perl.

The iterator can be a closure or a full-blown class hierarchy.

The iterator's interface can be "active" or "passive":
* active or external iterator: allows the caller to step through the loop and perform its operation at each step;
* passive or internal iterator: controls the loop and allows the client to pass in an operation to be performed on the items.

EXAMPLES
For example, consider the following code, which uses a foreach loop to print out the keys and values in a hash:

    sub hash_pair_printer {
	my $hash = shift;
	foreach my $key (keys %$hash) {
	    print "$key\t$hash{$key}\n";
	}
    }
    
    # The normal use we intended to support
    my %hash = ( foo => 'Foozle', bar => 'Bozzle' );
    hash_pair_printer( \%hash );

Obviously, the foreach loop shown above is a perfectly fine piece of code, and if that's all there is to the task, nothing need be done to it. However, if we want it to be more flexible, or if our requirements change, it'll need to get more complicated.

For example, we might want to be able to print out either the contents of several different kinds of data structures or objects. In some cases, we might be able to just extract the relevant values and place them into a hash to be passed in, but sometimes other constraints prevent this. We could write specialized variations of the function that operated on different kinds of objects, but much of their logic would be duplicated, and we'd need to add another one every time someone asked us to support another data type.

The Iterator pattern suggests a way of decomposing this operation into separate "loop control" and "for each item" capabilities that can be varied independently, allowing users the flexibility of recombining software in new ways. The examples below show simple iterator implementations using code refs, but object-oriented solutions are also common.

External Iterator: It's easy to create an external iterator using a closure:

    sub pair_printer {
	my $iterator = shift;
	while ( my ($key, $value) = $iterator->() )
	    print "$key\t$value\n";
	}
    }

    sub hash_iterator {
	my $hash = shift;
	my @keys = keys %$hash;
	return sub { 
	    scalar @keys or return; 
	    my $key = shift @keys; 
	    return $key, $hash->{$key} 
	}
    };
    
    # The normal use we intended to support
    my %hash = ( foo => 'Foozle', bar => 'Bozzle' );
    pair_printer( hash_iterator( \%hash ) );

In return for the added complexity of having separated the iterator into a separate entity, we get the flexibility to reuse this code in new ways:

    my $dbh = DBI->connect( ... );
    my $sth = $dbh->prepare('select id, name from students');
    pair_printer( sub { $sth->fetchrow_array } );

Internal Iterator: It's also easy to create an internal iterator:

    sub pair_printer {
	my ($key, $value) = @_;
        print "$key\t$value\n";
    };

    sub hash_iterator {
	my $hash = shift;
	my $action = shift;
	foreach my $key (keys %$hash) {
	    $action->( $key, $hash->{$key} );
	}
    }
    
    # The normal use we intended to support
    my %hash = ( foo => 'Foozle', bar => 'Bozzle' );
    hash_iterator( \%hash, \&pair_printer );

In return for the added complexity of having separated the iterator into a separate entity, we get the flexibility to reuse this code in new ways:

    my $headers = HTTP::Headers->new( ... );
    $headers->scan( \&pair_printer );

COMMENTS
Every Perl hash is equipped with its own built-in iterator, accessible through each %hash, but its use is somewhat prone to failure; for example, the following attempt to add an iteration count accidentally causes an infinite loop instead because calling keys also resets the iterator used by each.

    sub hash_pair_printer {
	my $hash = shift;
	while ( my ($key, $value) = each %$hash) {
	    print "$key\t$value\n";
	    warn "Iteration ", ++ $counter, " of ", scalar keys %$hash;
	}
    }

What Next for Perl Patterns?

Although discussions of software patterns in the context of Perl have come up regularly over the last handful of years, they haven't seemd to play a significant role in the way most people approach Perl programming.

Indeed, on the surface, the Perl world's reaction to the patterns movement has been dominated by indifference, flecked with pockets of hostility. But on a closer look, it seems that the hostility is aimed less at the concept of design patterns, than at the hyperbole surrounding the Gang of Four's "Design Patterns" book.

In fact, the documentation practices of the Perl community seem to share some characteristics with patterns work, with web sites like this one, and books like "Effective Perl Programming" and the "Perl Cookbook", organized as collections of contextual solutions.

I believe that there's value in documentation efforts to collect good Perl practices across a range of scales and organized along a patterns model, and I'd like to imagine that such a Perl Patterns Repository could serve as a useful community resource, operating as a kind of FAQ for information about Perl developer techniques, designs, and idioms.

However, for one reason or another, current efforts in this direction haven't generated much momentum. Since a key aspect of writing descriptions software patterns is exposing them to discussion, feedback, and improvement by other developers, aspiring Perl patterns authors might be better off contributing to existing repositories like the Perl Monks Categorized Questions and Answers.

Regardless of whether patterns documentation becomes trendy or not in the Perl community, I have been seeing more comments indicating curious interest or quiet enthusiasm for the topic, and I think we would be best served by remaining mindful of the original intent of the software patterns movement.

Further Reading

Perl Pattern Collections
* www.perldesignpatterns.com -- A large but somewhat jumbled collection with some nice advanced tricks.
* www.patternsinperl.com -- An older site; seems to be down.
* panix.com/~ziggy/perl_patterns.html -- Lists some Gang of Four patterns implemented in Perl.

More About Patterns
* perl.plover.com/yak/design -- The "Design Patterns Aren't" presentation.
* www.samag.com/documents/s=1280/sam02010001 -- An article about applying patterns.
* www.norvig.com/design-patterns -- A presentation on patterns in dynamic languages like Perl.
* hillside.net/patterns -- The industry association of the patterns community.

Some Perl Monks Threads
* Design Patterns Considered Harmful (Dec 20, 2001)
* Are design patterns worth it? (Aug 28, 2002)
* Perl Design Patterns Book (Oct 03, 2002)
* Perl Design Patterns (Jun 14, 2003)

edited by ybiC: fixed "logout" PM links

[chunlou]

Besides prior training, I found the success or failure of introducing and using published patterns in the workplace have a lot to do the the culture and personality of the team.

I met a development team led by several Swiss programmers, they did everything by the book, Java and Java patterns all the way. Practically everything they code is according to some associated published pattern. They're like airplane engineers and their codes are very robust.

On the other hand, a lot of Web development houses seem to be a bit more on the artistic side than engineering side culture-wise. Many of them kind of abhor the idea of using someone else's ideas (i.e. patterns, reusable solutions). They prefer self-innovation.

But every programmer uses "patterns" unknowingly anyway, the minute they reuse their own solution they used before, whether or not they have a name for it. Nothing inherently good or bad about patterns. It's just a way to solve problems.


_______________
Incidentally, a Perl programmer happened to review an application written by those Java programmers. He called the application "overdesigned" since there're some 200 Java classes (the ones they wrote on their own) sitting on top of slightly less than 20 database tables. Quite a contrast on design philosophies they have.

[menolly]

Part of the problem, I think, is overly simple examples. It's necessary to simplify in order to manage length and clarity, but when it comes to patterns, what I mostly see are examples that make it look like a lot of extra work for little or no gain. Web development tends towards the rapid development end of the spectrum, so it's difficult to grasp why we'd want to jump through all these hoops. It appears to be the sort of increased complexity that can help long-term maintainability if everyone involved understands it, but can harm it for someone new coming in, unfamiliar with the patterns in use.

[chunlou]

Right. A common problem is not so much about what is taught but how it's taught.

It causes a lot of resentment when the teaching of, say, UML, CMM, patterns, or what have you, is done independent of what the programmers are currently working on, instead of complementing it. (A programmer is pressed for deadline; it makes no sense to him to be told and taken away to attend a lecture that didn't help him make the deadline.)

It happens frequently when the instructor (especially the outside one) teaches based on whatever preestablished syllabus. It annoyed the hell out of some database programmers when a consultant told them they could apply OOP even to SQL.

The teaching process seems to work better if a "lead" consciously introduces some new concept or pattern (just one or two at a time) during a project, say, the design phase. That is, integrate the teaching into a project, instead of making it an independent event. People learn and use a pattern right away without their precious time taken away.

[scrottie]

Whoa. Now this is a well written, well conceived, well organized article. I'd like to see this on http://perl.com. It gives a better taste of what patterns are about than any introduction I've seen.

I don't really have anything new to say that I haven't already said, or someone hasn't said better, but I should make my position known on the whole deal.

Go to the bookstore and look at the computer language section. The difference between the style of the books is amazing. PHP has nothing but intro books, and the programmers apparently lose interest in the language there. Java has leigons of books on design, correctness, and style: "Enterprise Java", Java certification manuals, "Bitter Java" and loads of other patterns-in-Java books. Perl "learning" books dominate, in their various permutations: "Learn Web Programming with Perl", "Learning Perl on Win32 Systems", "Perl for System Administration", "Perl for Web Site Management", "Beginning Perl for Bioinformatics". Non-intro books are around too, but they either give their subject a far more simple treatment than C - "Mastering Algorithms with Perl" has very few algorithms, very little theory, and my "Introduction to Algorithms" (which is anything but an introduction) has about a factor of 10 more girth than the Perl book and is packed full of dense technical explanations and ultra- concise, difficult to grok examples. Someone looking down their nose on PHP could easily lump Perl and Perl programmers into the same category: hacks who don't care a whit about what they do, just in it to make a quick paycheck at the expense of their clients, whom they leave with mountains of crud. No reguard for the art. Arrogent ignorant little kids.

I have to agree that the 3 perl.com design patterns articles kind of waved their hands and dismissed patterns, but at the same time, they dismissed design in general. Most perlers I know can handle a serious treatment of any subject (and would buy more books if publishers took Perl as seriously as they took Java, in fact). A lot of Perlers I don't know won't bother to crack the covers of the free online copy of "Beginning Perl" (hello #perlhelp newbies!). Because this "perl programmers are ignorant arrogant kids" perception is true for a portion of the population, I don't think we should all be subjected to it. To be fair to Perl book publishers, the books are usually far more pragmatic than the thick C tomes, but that doesn't mean that they can't give the subject a serious treatment.

I like Perl a lot. Probably for different reasons than you. I like B::Generate. I like self-modifying code, clever hacks, Acme::, and all of the idioms. But I also like being able to use Perl for work, and write good, clean, serious Perl that scales and people can work on. But I don't get that chance. I have to work in Java. Because of the half-true perception that Perl programmers are arrogant, ignorant kids.

-scott

[bronto]

I know I am OT from the original thread, but I can't keep me from saying that I don't agree with you in putting Perl for system administration in the "Learning" series.

I bought that book as soon as it was published and now it's about one year since I last read something in it, but anyway: IMHO it definitely isn't a book for beginners. Surely it explains things in a simple manner and from the ground up, but I believe that people ranging from total perl beginners to short-time newbies could run into difficulties in understanding what's going on.

But maybe I am wrong, after all it's a long time since I last open that book, and I don't have it at hand now...

Ciao!
--bronto

---

The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
--John M. Dlugosz

[simonm]

The third and final article in the series was published on Perl.com after I started writing the above homily.

To my eye, the third article represents an improvement on the earlier two, and the discussions of the Composite and Proxy patterns do manage to convey the essence of the patterns. Unfortunately, the article still suffers from a few flaws; for example, the section labeled "Abstract Factory" only discusses concrete factories, and ignores the specific issues that the "abstract" part of the Abstract Factory pattern addresses.

(I've also written the author of the series, Phil Crow, to give him an opportunity to respond here if he'd like.)

[adrianh]

++. Several times if I could.

The articles published on O'Reilly have annoyed the heck out of me and I've had writing a long response on my todo ever since the first one came out. Now I don't have to!

Somebody frontpage this now :-)

[simonm]

Thanks! I know it got a bit long-winded, but as you say, I'd kinda gotten my dander up.

[sharkey]

In defence of the Perl Design Patterns, I thought they were quite refreshing, and protrayed well how design patterns should be appropriately used in perl. There *is* a lot of hype around design patterns, and much of it *needs* to be deflated.

"Simple things should be simple" is the perl mindset, and the article does an excellent job of showing how to use them simply and effectively.

Design Patterns are a prime example of academic self-indulgence, where everything *must* be complex, because simple things are not publishable. Even the simplest pattern, Singleton, is an exercise in seeing how much complexity one can layer on top of a global variable, while providing zero benefits, and lingering problems. (Did anyone notice the note about timely and correct destruction being non-trivial--in the Knuth sense of the word?) In the end you have a Class in your global namespace instead of a variable, and a lot of useless, bloated code.

Patterns aren't Language Features. Maybe they aren't in java or C++, but if they *are* in perl or ruby, does that mean we still have to code it ourselves to make it a pattern? This is *exactly* the kind of artificial-complexity that turns design patterns into a tool of evil.

I'm not saying that patterns are always useless and bloated. (Well, maybe Singletons are.) They have their uses for solving complex problems, but not all problems are complex, and not all implementations need to be complex. Unfortunately, a culture of complexity has grown around design patterns, such that average programmers have a hard time applying or implementing them correctly. (I say that after having to fix applications by removing the design patterns from them.)

Patterns aren't Universally Applicable. and ... how a particular problem fits with a particular solution. Accepted patterns are too complex to be univerally applicable, but too many programmers try to apply them to everything anyway. You should be looking for solutions to your problems, not problems for your solution! It would help immensely if we could dispense with the complexity, and recognize iteration, repetition, and decision as valid patterns. Then patterns *would* be universally applicable, and it would be OK to apply simple solutions to one's problems.

Patterns aren't Platonic Ideals ... a ten-year-old collection of patterns from a different language. Um, well said. :) The problem with "distinctly named" patterns is that they become canonic: you should do it this way, even if perl has an easier and better way of doing it. Perl Design Patterns was trying to show the easier and better ways of doing it, instead of showing how to write 10-year-old C++ programs in perl. Is there something wrong with that?

Patterns aren't Total Solutions: ... The description of each pattern should review the context in which it is used, acknowledge the tradeoffs involved, and refer to other patterns that offer an alternative approach. First, I agree completely. Second, when was the last time you included comments like this in your code? (Oh wait, we only have to justify the pattern when we invent it, not when we use it, because any use of a pattern must be better than a non-pattern.) If people actually considered the tradeoffs involved (and documented them), complex patterns would only be used where the complexity is appropriate and necessary, or find a simpler way to do it.

Patterns aren't Generic Concepts. So looping is not a pattern because it is too generic? Why should I use a loop when an iterator will work just as well? It makes me feel elite, because I've written 10kLOC today.

Design Patterns provide solutions for problems, and standardize the names for those solutions. This is good.

Design Pattern Culture worships complexity to the point were any simple solution is rejected as a design pattern, and thus tacitly rejected as a solution. This is evil.

[scrottie]

The only point I want to nitpick here is complexity. A large program becomes complex for one reason. Fredrick Brooks (of the infamous Mythical Man Month) termed Natural Complexity and Accidental Complexity. A friend of mine worked for a month at a place that needed some new features added to their program. The program is 10,000 lines, all in one file, and completely incomprehensable. This is one kind of complexity that doesn't need to exist. Further, the code had no apparent structure. It was peicemeal. No relationships between parts could be discerned, and in fact, it was hard to identify anything as being a part, seperate from the rest. And fittingly, this is the reputation Perl has.

I don't advocate making programs more complex than they need to be, which is to say, they shouldn't be more complex than the problem they are meant to solve warrents. Patterns are meant to structure large programs - not small ones. Out of context, they look, well, dumb. This makes it hard to write examples and even harder to imagine what the example would look like in its native habitat. I have a copy of "Lions' Commentary on UNIX 6th Edition", ISBN 1-57398-013-7. This book rocks. It is half explanation, half source code listing (mostly C, very little assembly through in). The commentary on the code was done by a college professor for a class, and it points out idea behind the code, structure, how things were built, how the parts fit together - picking on things that students are likely to someday have benefited from learning. Patterns. Having structure to the code doesn't make it complex - quite the opposite. It makes it interesting and valuable and long lasting. Look - we're still using Unix, even though it had to fit into 128k on a PDP-11!

My advice to people writing small one-off scripts and quick hacks is to ignore the whole patterns thing just like you ignore objects. The tool doesn't fit the job. And trying to make it will just warp the job - as you say, adding complexity. However, at the first hint that the code is starting to outgrow what little structure it has, future programmers doomed to work on the project will thank you if you start early trying to fit the code to some higher level idioms. You use low level idioms, why not higher level ones?

Oh yeah. I forgot. Perl does that for you. Look. There are two kinds of programming tools in the world. 4GLs and things designed to make you not have to think about design or logic, and refactoring browsers and other things designed to augment the abilities of a human as they design. ASP fits the first category, along with dozens of failed graphical-builder tools. lint, yacc, c-tags, that postscript hack that makes a poster of the Linux kernel sources, Purify, Rational Rose and other UML or Use-Case modeling tools fit the second category. Things in the first category tend not to be any help at all to an expert - in fact, they are associated with amatures. Things in the latter category are associated with skilled professionals, who are highly effective at their job. These tools have stood the test of time and repeatedly proven themselves. Rather than trying to do a job for the programmer or make a job vanish, they augment what human can do. Think of humans augmented with computers rather than humans replaced with computers, if you want a mnenomic.

To say that Perl has high level design built in so you don't have to think about it is to cast Perl as a member of the former group. I don't like that very much. To say that Perl is a language more than capable of accomodating the technical needs of design in a large project when skilled professionals employ the right tools - that I like. Does Perl do the job of thinking for you, or does it expand what you're capable of? Do design issues not exist in Perl like they do in Java, or do they exist more readily as Perl increases programmer productivity, allowing them to build more quickly? When you go to the book store, do you read just the Perl books, or do you wander over and actually peek into these computer sciency rags?

Remember, at one time, no one would touch objects, and they were the fetish of a few colleges, never seen outside. It wasn't until the 1980's - 20 years after their invention - did they get foothold in global conciousness. Before that, no serious language would ever consider a lambda closure feature - people don't need that to do business reports in RPG or COBOL! Both of these things slowly seeped out. Lack of closure bit Java on the butt - inner classes are a horrible kludge, but absolutely needed the way Java makes you define objects of a certain class to be used as callbacks. Closures make code shorter, and to someone wise in their ways, more readable - they do great things to reduce the scope of variables, breaking the code down into obviously seperate, cleanly divided parts. Stupid CS stuff, trying to make our lives more complex! Tell yourself that patterns really aren't anything new, just a better way to learn objects - and I mean really learn objects - C++ shops have struggled something awful the last 15 years - objects didn't click into place for their programmers except for a very special few. None of the prophacies of objects making code reusable, modular, resistant to change came true for the ordinary man because they didn't grasp the single driving concept. They didn't have that "ahh" moment. Scheme and Lisp programmers never had any problem - they had "Structure and Interpretation of Computer Programs" to learn from. After an entire generation of doing it wrong, we have a generation of object programmers doing it right. So, to put it another way, the Perl global mindset is still stuck in last generations misused, misunderstood, ineffective object usage. If you need me, I'll be in the Java section.

Whoops, I got pretty far afield here. I always do that. I meant to offer perspective on your comment on complexity - the rest has little or nothing to do with your note. Otherwise, I agree. I've focued on patterns that apply to Perl and Perl programmers, not direct Perl translations of C++ patterns - on perldesignpatterns.com. Patterns shouldn't be mindless sprinkled about - normal code should stay normal code, to paraphrase Larry Wall's "perl should stay perl". In a large system, you need glue between the normal code or you have an endless procession of code. Like a patent, a pattern should be interesting and non-obvious - otherwise, people won't want to read them. Most people don't like to be told the obvious. It is insulting. And so on. There are very good reasons why an iterator needs to be something more than a foreach statement - you're exposing the internals of one object to another object through a predefined interface. Returning an array is okey sometimes, but the internal datastructure may not jive with that. perldesignpatterns.com's example of an iterator - or one of the examples - exposes the contents of a network of objects. Rather than building a massive array, the natural recurssion is exposed through the interface. Tieing and overloading are also very powerful - Perl shouldn't lose its expressiveness because of other idioms - but sometimes the idiom of using an interface and objects packs more of a punch than that of an array. Sometimes. In large programs.

-scott