Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Best way to 'add modules' to web app?

by BUU (Prior)
on Jul 04, 2003 at 20:07 UTC ( #271544=perlquestion: print w/replies, xml ) Need Help??
BUU has asked for the wisdom of the Perl Monks concerning the following question:

At the moment I'm in the planning stages of a project that will probably turn into something like a forum/bulletin board and while I was designing, it occurred to me it would very nifty if I could add some sort of modules to the output/input. When I say modules, I don't really mean the perl modules, or atleast the standard ones, but user defined modules that they could either add from another source or simply write themselves. These modules could preform a variety of actions, one example would be attaching a module to the output that would convert all urls surrounded in brackets into hyperlinks and things of that nature. So my question is, what do you think the best way to setup the program in the begginning so it would be easy to install these modules?

Replies are listed 'Best First'.
Re: Best way to 'add modules' to web app?
by chromatic (Archbishop) on Jul 04, 2003 at 20:44 UTC

    I wouldn't worry about it yet. If you do only what you really need to do to meet your current needs, you'll probably find it easier to add modules in the future when you know what you need. Since you don't have much code now, you'll have better results getting the basic forum and bulletin board code working solidly.

    When people start asking for two or three features that you think would make good modules, you'll have a better idea what people want for modules and how to add them to the system. That's when I'd start thinking about them.

    The best way I know of to make software easy to extend is not to extend it until I know exactly what I need. One rule of thumb is, "If I'm thinking about features I don't need yet, I'm probably going to design and implement them wrong and will have to rework them when I really do need them."

      >>One rule of thumb is, "If I'm thinking about features I don't need yet, I'm probably going to design and implement them wrong and will have to rework them when I really do need them."

      Really? It seems to me that the advice most often given here is to "always plan ahead". Why do you reccomend something different?

      I've had some personal experience with not planning ahead and having to rewrite large portions of a project and I'd like to try to avoid that this go-round.

      As to the higher idea of modules in general, I hadn't planned on implementing this modules myself, but rather providing an easy to use framework that people could easily develop their own or share their developed modules. Basically what I'm looking for is a way to provide a 'patching service' to change how one or more features work, but without resorting to some projects that have patches like "go to line 23 in foo.php and replace all the lines until the next semicolon and then go to line 56 and replace two variables" and so on and so forth.

        I've never planned more than a few features ahead without having to revise the plan heavily. Instead, I prefer code that I can change easily. This is exactly how I avoid rewriting.

        When I program, I continually ask myself two questions. The first question is does this work as I need it to work? and the second is is this as simple as I can make it right now?. My goal is provably working software that's as simple as possible.

        Simple is a loaded term, but I mean that my code should only be as complex as necessary to pass the tests. Part of that comes with experience. Part of that comes as I recognize more and more patterns in my code. Since I have oodles of tests, it's reasonably cheap to continue to simplify.

        Because I work in small steps, adding one feature at a time, always testing and always simplifying, I'm confident that I'll always be able to add new features relatively inexpensively. It's also nice to say, "I'm done already? I can't think of any more tests to write. The feature works. I wrote far less code than I thought I would need."

        I guess there's different between planning based on what you know for sure you need and planning for contingencies. It's too hard and impractical to try to tackle all potential contingencies--not that you shouldn't plan at all, just make sure you know correctly your priority.

      Uh.. what?!

      That's possibly the worst advice. NOT planning for expanding is a good way to have to rewrite your software later. Granted, it doesn't have to be overly-normalized, but my God... If you need to expand later, you better plan for it now.

      If you know your message board needs to interpolate with say, several clients, then doing somethign like SOAP or the ilkes will be required before hand. If you need it to do RSS feeds, better not making it too disorganized, otherwise, you'll have to litterally punch it in later on.

        chromatic is obviously advocating YAGNI approach to solve design problems. In short the idea is that you split your development process in short iterations: in each iteration you select functionality to implement, then you design the system in the simpliest possible way to pass functional tests. While doing it you religiosly refactor code to remove code smells (i.e. code dublication, too big methods and subs, etc). In my experience it is actually very efficient approach which at the end produces more high quality and more flexible code compared with water fall style development process when you try to design everything from the begining. Too often initial assumptions became wrong assumptions, functional requirements change and end result becomes big ball of mud.

        Word of warning though. If you try to develop using YAGNI way then TDD is a must, constant refactoring is a must. If you don't follow these practices you'd better stick with waterfall way or result will be a disaster. In a sence waterfall is simplier model though less effective.

        Ilya Martynov,
        CTO IPonWEB (UK) Ltd
        Quality Perl Programming and Unix Support UK managed @ offshore prices -
        Personal website -

        What if you guess wrong? You'll still have to rewrite.

        I'd rather add ten features to a system that's well-tested and regularly refactored than one system that has been completely planned from the beginning. I'd bet that it's also much cheaper to change the first system than the second.

        I much prefer software that's so simple you can change it than software that's so flexible you don't have to change it — because someone always has to change it and that's usually me.

Re: Best way to 'add modules' to web app?
by The Mad Hatter (Priest) on Jul 04, 2003 at 21:00 UTC

    Here's a few tips from my experience (re)writing my own forum software (in Perl of course):

    • Make sure from the start (one of my mistakes that I'm dealing with right now) that the code is clean, modular, and overall easy to modify. Overall, write the code in a way that makes sense and allows for changing of little details without breaking everything. This means that you shouldn't hardcode lots of stuff. If you use it a lot and it has the slightest chance of changing, put it in some sort of "options file" that can be accessed from any of your other code.
    • If you plan on doing something later, or even have an inkling that you might do it, write the code to easily allow for that. For example, in my code, I have a formatter class with a process method that is called on almost all user input/output. By calling the general method and not specific formatting methods, I can easily change how text is formatted by changing that one method.
    • And lastly, KISS: Keep It Simple, Stupid. There is a lot to be said for simplicity. While you should plan somewhat in the beginning so you don't end up having to rewrite/modify lots of code, overplanning can be and often is even worse.
Re: Best way to 'add modules' to web app?
by gjb (Vicar) on Jul 04, 2003 at 21:13 UTC

    Although I agree with the other contributers (chromatic and The Mad Hatter) that it is hard to design in advance to unknown requirements, it might be useful to keep a few design patterns in mind.

    Specifically, you might have a look at the Filter pattern (GoF pattern), as well as the Dynamic Linkage pattern (Grand98).

    References are Design Patterns and Patterns in Java.

    Just my 2 cents, -gjb-

      If you need a FREE patterns book, better: Bruce Eckell's Thinking in Patterns. He is a great teacher and an expert.

      But I guess that before thinking in patterns one should be aware of what is structuring well some code...

Re: Best way to 'add modules' to web app?
by chunlou (Curate) on Jul 05, 2003 at 00:32 UTC
    Sounded like a "make it flexible" requirement.

    The other monks already offered valid advices to heed on.

    Suppose you want or need to go ahead. It depending on whether your "modules" are more configuration-like or API-like, there're pretty of real life examples how other people did it.

    In either case, XML is increasingly commonly used as a meta-language, instead of self-invented pseudo (markup) language, say, used in some configuration files by some programs.

    In Why I like functional programming posted a long while back, it showed you how to use hashes (and functional programming, of course) to set up a series of tag and handler pairs to more conveniently manage the processing of any special tags, HTML or otherwise.

    In a "module" framework, the hashes above could be replaced by XML. The definition of a corresponding handler could be some RegEx or Javascript allowed and appropriate.

    Microsoft WSC (Windows Script Component) is a nice example, where you can use XML and scripting language to build COMs (which otherwise is not always necessarily an easy task).

    Apache itself is another more complicated example, where you could build a module without having to compile the whole thing all over again, considering especially "component model" is not naively part of Unix or Linux by default.
Re: Best way to 'add modules' to web app?
by DBX (Pilgrim) on Jul 05, 2003 at 15:07 UTC
    It seems to me that if you write your system with good object-oriented technique, you shouldn't have a problem allowing the extension of functionality, regardless of the planning.

    Keep in mind I know nothing about the specifics, but let's say, for example, you have a Forum::Format module. This module has the methods:

    $f->formatPost() $f->outputPost()
    You can also define the abstract or stub methods:
    $f->preFormatPost() $f->postFormatPost() $f->preOutputPost() $f->postOutputPost()
    If some other class is calling the  $f->formatPost() method, make sure it calls $f->preFormatPost() before and $f->postFormatPost() after. Even if the abstract versions of these classes do nothing, future module writers can have their classes inherit from Forum::Format and override these methods or even rewrite your Forum::Format module without changing a thing except for these two methods. I realize your system may be a bit more complex than this and having future writers rewrite one of your base classes is a bad idea, but the idea of leaving "hooks" for pre/post formatting (or other actions) is still sound.

    Additionally, anything you do inside the Format method could call other methods that can be overridden. So if, within Format, you call $f->modifyHTML($text) or something similar, that method can be overridden to modify HTML differently in a child of the Forum::Format class.

      The idea of providing easily over-rideable methods that would allow the 'client' to easily change functionality is basically what I was going for. However, I'm having a problem conceptualizing exactly how you tell the program that you over rode a sub method. From your example, if I write a Forum::Format module, then all my code dealing with it looks like: my $ff=new Forum::Format;. If someone doesn't like part of it and wants to inherit and over ride that sub, say his new module is Forum::Format::NoHtml how does he tell my application that it should be using this New and Improved class/object instead of the regular old Forum::Format
        The problem sounds more like a functional programming one (where you focus on evaluating "expression")than OOP one. I don't think inheritance (though it might not what you really meant) it's necessarily good for customization and even less for flexibility. It leads to too tight a coupling.

        OOP is a good way to deal with design time customization. At runtime, or so you could think, it's another story. Think about a calculator. It evaluates arbitrary expression at runtime. It all depends on how you model the problem and/or your spec.

        As to how one might conceptualize overriding a sub, I might think of a sub a value in a hash with rules determining which key (and therefore sub) to pick, either at runtime or design time. Think about this (trivial) example:
        sub make_binary { eval "sub { $_[0] }" } my %op ; $op{add} = make_binary '$_[0] + $_[1]' ; $op{sub} = make_binary '$_[0] - $_[1]' ; $op{mul} = make_binary '$_[0] * $_[1]' ; $op{div} = make_binary '$_[0] / $_[1]' ; $op{max} = make_binary '$_[0] > $_[1] ? $_[0] : $_[1]' ; for (sort keys %op) { print "2 $_ 3 = " . $op{$_}->(2,3) . "\n"; }
        Kind of customization via "templating."
        I'd explain in detail how you do this, but I'm walking out the door as I speak. Instead, take a look at the code for CGI::Kwiki and how they accomplish it. (Hint: Use a simple, global config file that contains what classes you should use.)
        From your example, if I write a Forum::Format module, then all my code dealing with it looks like: my $ff=new Forum::Format;. If someone doesn't like part of it and wants to inherit and over ride that sub, say his new module is Forum::Format::NoHtml how does he tell my application that it should be using this New and Improved class/object instead of the regular old Forum::Format

        Don't hard code the class name. Have a factory that gives you the appropriate formatter object based on a configuration file, environment variable, mod_perl setting or whatever.

        Take a look at Class::Factory for one way of doing this.

Re: Best way to 'add modules' to web app?
by logan (Curate) on Jul 05, 2003 at 05:54 UTC
    Not knowing any of the specifics of your project, I can only go by the description as a "forum/bulletin board". I have to ask: have you considered slashcode? It's free, well-documented, stable, robust, and you're using it right now. Why reinvent the wheel?

    "What do I want? I'm an American. I want more."

      Uh, PerlMonks uses a heavily modified version of the Everything engine, not slashcode.

      As for why he's doing this, maybe it is a learning experience, in which case he doesn't want to simply copy an existing system. Or maybe he just wanted to write his own so that there is little bloat and he knows exactly what was going on under the hood. There are lots of reasons. (And if all he wants to do is get a forum running, yeah, he should use something already out there, but I doubt he'd be thinking about writing his own if that is the case.)

Modules as Application "Plugins"
by simonm (Vicar) on Jul 07, 2003 at 19:29 UTC
    One useful extension technique for a server-side application like this is to support a "plugins" directory in to which users can place Perl modules that will be automatically loaded when your application is initializing itself. Then you can document some key functions within your code, and show people how to override them in their plugin.

    Here's an untested schematic example:

    # In myscript.cgi, or your mod_perl configuration, or wherever: # This could be relative to the script path or config file my $plugin_dir = '/opt/myscript/plugins'; sub load_plugins { # Read in the names of all .pm or .pl files in the plugins dir. # (Even better would be a recursive search of subdirectories.) unless ( opendir(PLUGINDIR, $plugin_dir) { warn "Plugin dir '$plugin_dir' is missing or unreadable."; return; } my @filenames = grep /\.p[ml]/, readdir(PLUGINDIR); closedir(PLUGINDIR); # Put the plugins directory in our library search path. # This allows one plugin to use or require another one. local @INC = ( $plugin_dir, @INC ); # Try to load all of the plugin files, but don't die trying. foreach my $plugin ( sort @filenames ) { eval "require $plugin"; if ( $@ ) { warn "Skipping plugin $plugin: $@" } } } sub print_output { print @_; } load_plugins(); print_output( "<html>Here's my CGI script output.</html>" );

    Then, as a user, if I install this script, and read your documentation, I can "add on" some functionality by writing my own code into a file in the plugin_dir that modifies your original functions. For example:

    # In $plugins_dir/ sub print_output { print map { uc } @_; }

    A key benefit of this approach is that when the original author releases an updated version of the main program, users can install it without worrying about overwriting their local customizations. Of course, if you make big changes to the main script, the plugins may not work properly anymore, so you need to either promise to maintain a backwards-compatible API, or check version numbers or such.

    Following standard techniques of good program design when writing your application should generally also make it easier for your application to be extended this way. For example, finding common sequences of operations and breaking them out into subroutines with clear, descriptive names helps construct a clear API that someone else can later come along and extend.

    If you have multiple plugins that all want to be involved in the same process, they'll need to wrap around each other or use some kind of event-handler logic to let them cooperate.

    Here's a brute-force wrapping technique:

    # In $plugins_dir/ my $inner_print_output; BEGIN { $inner_print_output = \&print_output } sub print_output { &$inner_print_output( map { uc } @_ ); }
    # In $plugins_dir/ my $inner_print_output; BEGIN { $inner_print_output = \&print_output } sub print_output { &$inner_print_output( map { s/\s//g } @_ ); }

    Because of the "sort @filenames" expression in load_plugins, these files will be loaded in alphabetical order; first wraps around the original print_output, then wraps around that. The result is that's print_output calls print_output which calls our original print_output. Feel free to use some other sort order, but make sure it's consistent so that users can control the order they want things loaded in.

    Note that the core concept described here is orthogonal to whether you're using functions or objects; if your web application is implemented using objects and factories, as described in other responses to this question, the "plugins" modules can define new classes and register them as needed.

Re: Best way to 'add modules' to web app?
by eric256 (Parson) on Jul 06, 2003 at 01:20 UTC
    Have you considered an event/template driven approach. (This would work mostly for Changes the look and actions of a button) Then each 'module' could register a handler for certain events, and template could call certain events.

    Like the header module might have DoEvent("DrawTitleBar") Then the engine could call all handlers associated with that event. Each 'module' could then add its own buttons and/or formating to the title bar by adding a handler to that event.

    The template would then obviously need to handle these event calls, and transform button clicks into Event calls etc, and you would need some standard data format that each module would have access too.

    I hope this helps or at least provides some spark of thought. Anyone have ideas/comments about my idea?
      Thats a very interesting idea/concept, and one I had not though of before. I'll definately have to give it some thought. I'm not sure of how the users would register handles for an event though..
        Frankly i'm not either. I've been working on concepts for a Community site with many of the features of Everything, Slashcode, Forums, Blogs, and link databases all combined into one.

        With perl it shouldn't be too hard to have each module have a standard init function that is called when its installed/loaded. That init function could then call a sub giving it the event to handle and a reference to the sub to call. I'd also think that instead of passing events arguments, either pass it a hashref with the arguments in it, or just have one common hash that is used to communicate between events, and the board/forum system.

        Some problems would be, how to handle multiple occurences of them all? or let them overpower the original in some way? Maybe that could be one of the options for the function to set handlers. So some handlers are used in addition to the current one, and some replace the original. It would me anyone making modules needs to play nice, but thats true anyway. You could also run the modules in a Safe compartment to control what access they do have for some added security.

        I look forward to seeing what you do come up with!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://271544]
Approved by gjb
Front-paged by valdez
and nobody stirs...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (4)
As of 2018-02-24 00:52 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (310 votes). Check out past polls.