Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Writing a web message board from scratch

by tomazos (Deacon)
on Apr 19, 2002 at 13:13 UTC ( #160520=perlquestion: print w/ replies, xml ) Need Help??
tomazos has asked for the wisdom of the Perl Monks concerning the following question:

The mission was to write a basic message board on the company web store where customers can login, leave public messages and answer other people's public messages. Kind of similar to PerlMonks although of course much simpler.

I have apache, mod_perl and mysql installed, tested and played with on my company server. All the basic building blocks of a great web application.

After studying all the right O'Reilly books and learning which modules I need to use and how to use them - it occurs to me that I have no idea where to start on the actual project itself, even though I can see the final picture I want to create and how it fits together. It just feels too big to offer a starting point, if that makes sense?

I could just start hacking away piece at a time, but it feels like I should be doing some kind of design first. I don't want to fall into the trap of creating a hard-to-maintain disaster that so many others have. I want to be able to add features later easily. I want it to be reliable and the code to be "clean".

What are the first steps to planning the implementation of a largish project like this? How do you plan the code? Backend or frontend first? How do you decide what to factor out and what not to? How do you write code in such a way that it is extensible? etc, etc.

I guess what I'm asking is, what are the steps between seeing the design of version 1.0 of a web application like this in your mind and starting to writing the first line of perl itself. Or are there no steps in between for you guys, and you can just start coding at this point?

Comment on Writing a web message board from scratch
Re: Writing a web message board from scratch
by Molt (Chaplain) on Apr 19, 2002 at 13:29 UTC

    Is there any reason you're intending to write this all from scratch? As you say, it's similar to what's here.. so why not use or adopt code from something that already exists?

    This has the advantage that the code has been bashed upon repeatedly and the bits that were liable to fall off have fallen off and been replaced by more resilient bits. Security holes have been patched, lock-ups removed, code made more friendly and customisable.

    If you want to really hit the O'Reilly then I'd recommend you have a look at 'Running Weblogs With Slash'. Slashcode may not be the friendliest thing to set up, but compared to coding your own replacement from scratch it's a walk in the proverbial park.

    Or, if you're after something more akin to the Perlmonks site, why not have a look at the code it's run on?

    Both of these are nice and free. Enjoy.

    Don't reinvent, spend your time doing things more productive.

      For better or worse the decision has already been made to roll-our-own. There are a few details that we need control of for our specific application and I'm sure it would be easier to do it from scratch rather than adapting existing code to it.

      Let's assume that for some reason you have to write it from scratch and re-using an existing "web message board" codebase is not an option - all you can use is well-tested CPAN modules like DBI and CGI.pm etc.

      How do you do it?

        Let's assume that for some reason you have to write it from scratch and re-using an existing "web message board" codebase is not an option - all you can use is well-tested CPAN modules like DBI and CGI.pm etc.
        Well, SlashCode and E2 are both in the same status as CPAN modules, and are both readily available for stealing parts or just seeing "how you do it". In particular, Slashcode runs http://www.slashdot.com (a high-traffic, high-visibility site), and is highly reconfigurable: enough that you wouldn't recognize the externals.

        It's perfectly OK to "reinvent wheels", presuming you study existing art first. Please don't ask us to spoonfeed to you what these guys have already discovered. Check out prior art first!

        -- Randal L. Schwartz, Perl hacker

        In this case it's rapidly becoming a matter of having difficulty answering this in any way it's going to be at all useful for you as how I probably doesn't resemble one bit of how you would do it.

        I'd first of all make absolutely certain there's no way on Earth that I can't adopt another solution. If I adopt I know what needs changing, I know my tasts. If I don't adopt I have to solve problems that I don't even know about- How do I stop cross-site scripting attacks? How do I authenticate users and handle sessions? How do I prevent someone from knocking up a two minute Perl script to hammer out slightly-different offensive messages to all posts on the board at a ridiculous rate?

        Never assume the worst won't happen, especially with any kind of communication forum where emotions can run high. Expect to be attacked from every point, expect your servers to fall over regularly.

        Now it'd come to programming. Personally I'd approach it the same as I approach any other large project, but as I said you will vary. To start I'd build a list of features, and user-stories, going through it all in my head and on paper. From this I'd make a list of components such as in this case session-management, templating, database access, authentication, possibly XML parsing, input validation, email handling, and I'd try and match them up with CPAN modules. Almost certainly I'd end up with modules I'd not used before so I'd try and learn a bit about them and write a few test programs to get to grips.

        Now I'd try and think about where I expect problems and try and make sure I don't hit dead-ends there. User-abuse, how's it dealt with? Bad data gets into the database, what happens?

        Now I'd try and build a small and simple program with the core functionality, essentially post a couple of messages unauthenticated and recall them. I'd now build round it, adding all the features I've listed in an order that makes sense to me, using unit testing to make sure it all hangs together. Eventually I'd hope to work towards a working solution.

        Now, for you. From the way you phrased your question I'd actually think you'd not done too many large projects, if this is true then expect to walk across hot coals in order to get something this complex working, and even then expect to spend a good long time fixing problems. Save time by doing your homework, read up on software engineering methodologies and actually try and use one, I personally tend to use the test-early-test-often, do-the-minimum approach from Extreme Programming, but I don't use the peer programming and few diagrams. You need to find what works for you, however.

        I do wish you luck in this, but I also have the horrible feeling you'll need it.

        You said
        For better or worse the decision has already been made to roll-our-own. There are a few details that we need control of for our specific application and I'm sure it would be easier to do it from scratch rather than adapting existing code to it.

        Which makes me ask, if you haven't done this kind of thing before, just why are you sure that it would be easier to do from scratch rather than adapt? Particularly code that was written with adaption in mind as the already done modules were (ok, maybe not originally, but they are by now given the number of times they have been changed...)

        What I'd suggest before anything else, is that you acquire the code from slash or from here on PM and research just what all would be required to implement the 'few details'. Along the way, take a look at what is required for such a project-- this would also help you begin your own design if it comes to that...

        –hsm

        "Never try to teach a pig to sing…it wastes your time and it annoys the pig."
Re: Writing a web message board from scratch
by Biker (Priest) on Apr 19, 2002 at 14:00 UTC

    Sit down with paper and pen (and posibly a colleague of yours). Talk yourself through the usage of the future system. Go through everything that should be possible and how.

    Whenever you pronounce a noun, put it down as a potential class. If you also hear yourself mentioning that the noun must be able to verb, then you write down this as a potential method of the class.

    In Perl you're not forced to use objects and very often you will use procedural code to handle entities. These entities will become objects implemented in you class modules.

    Implement the skeletons for the classes and repeat the verbal session. Ask yourself if your classes really represent the nouns you need in the system and if they are able to perform all the verbs that must be done. Redo. ;-)

    Another thing to think about is a way to handle the numerous constant values and common Perl subs that must be shared by your (presumably) large number of Perl scripts.

    Decide the strategy for a/the database backend.


    Everything went worng, just as foreseen.

      Wow. Cool. What do you call this brain-storming procedure? Is there an official name for this area of knowledge? Are there books and/or web sites about this sort of thing?

      I don't mean to sound like an idiot, but I am self-taught and don't know this sort of stuff.

        There are probably hundreds of publications on the subject. (Booch comes to mind.) Search the web for OOD (Object Oriented Design) and you will find lot's of it.

        The way I do it is a mix of reading many books about several methods mixed up with (too long a) time using it in real life.

        "I don't mean to sound like an idiot..."

        Not asking makes an idiot. ;-) The only stupid question is the question never asked. (OTOH, chosing the right fora for a question may be an art in it self.)


        Everything went worng, just as foreseen.

        Tomazos:
          What do you call this brain-storming procedure?
          Is there an official name for this area of knowledge?
          Are there books and/or web sites about this sort of thing?

        Just some keywords (& authors):

        • Try story telling to elicit typical user/system interactions, i.e. use cases from your users.
        • Find out about the important classes and their interdependencies:
          • Organize sessions with CRC cards: Class Responsibility Collaboration and
          • Do lexical analysis on business documents
        • Prioritize requirements / use cases in a group with MoSCoW-rules: Must / Should have / Could have / Won't have this time around
        • Schedule features / use cases in timeboxes

          Authors:

          • Ivar Jacobson
          • Martin Fowler (Very readable and small book: UML Distilled)
          • DSDM

          HTH

      ...this is great advice.

      Determining what "things" will be in your system will guide you as you decide what needs to be built next.

      Advice above that you carefully examine what there is in the world already is sound guidance. But don't believe that the task is difficult and insurmountable. Perl is designed to make things like this easier.

      If you begin with a clear picture (it doesn't matter if it's a tree, an idea diagram, a flowchart, or a "Bacchus-Pinkus-Zeus standard compressed paranormal data continuum diagram," as long as it helps you clarify your thoughts about how the big picture fits together,) you're on the right track.

      Plan to build something that you can see and test early on. Then each iteration of your coding operation will build upon a working foundation.

      Good luck. This really isn't hard to do.

      ---v

(crazyinsomniac) Re: Writing a web message board from scratch
by crazyinsomniac (Prior) on Apr 19, 2002 at 14:16 UTC
    The mission was to write a basic message board on the company web store where customers can login, leave public messages and answer other people's public messages.
    Great, now what kind of data about the customers will you be gathering? How are you going to store it? Security?
    After studying all the right O'Reilly books and learning which modules I need to use and how to use them - it occurs to me that I have no idea where to start on the actual project itself, even though I can see the final picture I want to create and how it fits together. It just feels too big to offer a starting point, if that makes sense?
    What modules? You say you have a picture in your head, well the first step is to try and write it down as best you can, and when you do that, it will become even more clearly defined. You build you application around a a toolset: what is your toolset?
    I could just start hacking away piece at a time, but it feels like I should be doing some kind of design first. I don't want to fall into the trap of creating a hard-to-maintain disaster that so many others have. I want to be able to add features later easily. I want it to be reliable and the code to be "clean".
    Now you're thinking. When you say HACK it almost immediately implies lack of discipline/reliability/cleanliness. You recognize your needs (design) and that is a kickass step in the right direction, but you're wasting time, you should be writing things down already ;).
    What are the first steps to planning the implementation of a largish project like this?
    Setting down minimal criteria -- minimal functionality, and you've got it:
    1) customers login, which means authentication and possibly state
    2) they leave/answer public messages, which means you have to store/format these messages
    3) there has to be a user interface, in which the users views/contributes to the forum

    Now you've got the beginnings of a tree, expand each of the items into subitems and so on and so forth, and by the time you draw that picture, you've got yourself one heck of a map for a great plan.

    How do you plan the code?
    You set you goals, evaluate what skills you have (toolkit available ...), and you begin to scheme (like above)
    Backend or frontend first?
    There is no backend or frontend. What exactly does that mean to you? First you decide what you want to do, then how it's going to work and what you need to make it work so (like what kind of database schema)
    How do you decide what to factor out and what not to?
    You should not have make this decision -- by the time you finish that tree, you should have a clear picture.
    How do you write code in such a way that it is extensible? etc, etc.
    Well modules, modules, similar functionality, modules ... modules
    I guess what I'm asking is, what are the steps between seeing the design of version 1.0 of a web application like this in your mind and starting to writing the first line of perl itself. Or are there no steps in between for you guys, and you can just start coding at this point?
    After the tree, which is a rough draft, you come up with a pretty good plan, which is draft 2, in which you begin to actually define your schema and your app flow, and then in draft 3 you refine both, and start making your app logic clearer. At this point you've not writtena single line of code. After draft 4, which is yet another revisement, you've got a clear picture of how to organize everything and you go at it piece by piece, generating tests as you complete each piece, so when it's time to glue it all together, you'll have no doubts ...

    So there you have it, asses, asses, asses, get a 2nd opinion, and asses, and tree, and plan, and revise, and revise and revise, and review, and get a 2nd opinion .... you get the picture. Keep your timetable in mind ... read web site design, or lack thereof for wisdom.

    That's all I could muster up, enjoy (and I wouldn't even think of suggesting you adapt the everything2 engine)

     
    ______crazyinsomniac_____________________________
    Of all the things I've lost, I miss my mind the most.
    perl -e "$q=$_;map({chr unpack qq;H*;,$_}split(q;;,q*H*));print;$q/$q;"

Re: Writing a web message board from scratch
by jerrygarciuh (Curate) on Apr 19, 2002 at 14:38 UTC
    Having worked on similar projects, one of the big questions to ponder before coding is how to store the data. As a neophyte coder who has been working with DBI and mySQL for about a month, I would heartly recommend using mySQL this would open up lots of doors ie easily searching messages for a word or phrase. And moreover, DBI used to intimidate me but I find it is really not that hard!

    I have written a lot of widgets that store delimited information and the problem always seems to be the myriad ways a user can break it when reading their input back in. Also, many messageboards try to save on processing time by writing and appending static HTML pages. I would definitely avoid this and no matter what store each message seperately. If you enmesh seperate entities I garauntee some honcho will ask for seperate delivery one day. One approach many boards take is producing 2 flatfiles like post12503.content and post12503.dat with the dat carrying any info relevant to processing but not needed in content. I realize this is somewhat rambling but these are all issues I encountered in writing messageboard type apps.
    HTH
    jg

    _____________________________________________________
    Think a race on a horse on a ball with a fish! TG
Re: Writing a web message board from scratch
by pdcawley (Hermit) on Apr 19, 2002 at 15:15 UTC
    This may sound like vaguely inane advice, but I've found it useful. And it applies whether you're writing from scratch or adapting existing code. The cycle goes like this
    1. Pick the most important part.
    2. Make it run.
    3. Make it right.
    4. Go to step 1.
    Don't overcomplicate things. Keep it as simple as possible, but no simpler. Whatever you do, don't ever skimp on making it right. Code that isn't easy to understand is hard to extend. And the extended code will be even harder to extend. And the cruft will kill you.

    There's another important step that I haven't put in there, 'Write tests'. Some say they should go before step 2, some say after. But not having them is a recipe for disaster. Every time you get a bug report, write a test that fails because of the bug; the process of doing this will often give you the information you need to make fixing it easy. And it's one more test.

Re: Writing a web message board from scratch
by snellm (Monk) on Apr 19, 2002 at 15:35 UTC
    Create HTML mockups, preferably linked together in the same way as the live system will be.

    Use these to discuss functionality with business users, and to give yourself an idea of the structure that will be required to back these.

    Use the mockups as templates in the live system.

    -- Michael Snell
    -- michael@snell.com

Re: Writing a web message board from scratch
by lolly (Novice) on Apr 19, 2002 at 16:08 UTC
    you want to build a perl cgi from scratch?? i dont know much about object orientated perl if thats what your after... anyway, personally i would get it looking however you want it first, then when you test your perl script you can see how it works. for the perl
    use CGI; $query = new CGI; # start the html output print STDOUT $query -> header(); print STDOUT $query -> start_html ( -title=> "Message board.cgi", -BGC +OLOR=>"yellow"); print STDOUT "\n<h1><align=center>PUT YOUR TITLE HERE</CENTRE></H1>\n" +; # retrive info from the html form using he param function # e.g $people = $query ->param ('people'); # do this for all your variables # design some failsafes using perls regular expressions so that # if nothing has been entered in the message box they get a warning # message if ($message !~/\w+/) { print STDOUT "<P> you have not entered a message\!\n"; print STDOUT $query -> end_html; exit (0); }
    on your html form you need to a comand that accesses the perl script. it should look something like this
    <form method ="post" action ="http://www.whereever/directory/myCGI.cgi" enctype="multipart/form-data">
    Hope this was what you were after. Lolly :)
      $people = $query ->param ('people');

      I would THINK about doing some taint checking here. Having validated data is the first step to writing a secure site....

Re: Writing a web message board from scratch
by Maclir (Curate) on Apr 19, 2002 at 17:07 UTC
    Now, after sorting out the responses "don't build something from scratch when there are perfectly good working examples you can use", you clarify your question with:
    I guess my question isn't as much to do with the specifics of writing a web message board as it has to do with the general process of beginning a project of about that size.

    What you are really asking is "How do I design an application?".

    Answer: Employ an experienced business analyst.

    Writing code is programming. Determining what that code should do and what the underlying business problems that the code will address is analysis. Too many people confuse the two, and think that any old programmer can just whip out a brand new application without any difficulties.

    In fact, in any (non-trivial) application, the time spent doing the analysis, design, going over user requirements and so on is anywhere up to ten time the time taken to actually write the code.

    There are many steps between "we need a foobar application" and the writing of the first lines of code. To summarise them:

    1. Requirements Analysis. What business problems and processes will this application support. What functions must the application perform to be considered a success?
    2. Logical Design. Process modeling (high level), logical data modeling, initial estimates of transaction volumes, data volumes. Noting any special requirements - security, auditability.
    3. Physical Design. Specification of individual modules / processing actions / physical data model. Development of testing procedures. User interface specification.
    4. Program Development. Now you can write your code.
    5. Testing. Often requires loops back to program development, even changes to the physical design.
    6. Implementation.

    Others may have different names for the steps, the "Extreme Programming" advocates will criticise my proposal of outdated development methodologies. But the functions I have listed still need to be done.

    Bottom line: If you don't have people with analysis skills and experience, you will most likely repeat many of the mistakes of other failed development projects. An anaylst can determine if an existing product (slash, zope, whatever) meets your company's business needs, and whether the stated requirement of "we have our unique requirements and have to write our own program" is really a furphy, and which of the build / buy (or adapt) alteratives is properly cost justified.

Re: Writing a web message board from scratch
by ignatz (Vicar) on Apr 19, 2002 at 17:55 UTC
    Here's how I go about it:
    1. What's my stored data?
      Design the structure of the data for the application and classes that will access the data, starting with the tests that verify that they are doing the right things.
    2. How will I display my data?
      Write classes that will call the data and filter it out into a display. At first write some dummy data to play with. (These days I write pseudo data tables with the classes so that I can call them easily for testing. DBD::CSV is a fav.)
    3. How will I write my data?
      Write forms that will send the data via its classes to the database.
    4. How will it look?
      Create templates that the data can be punched into.
    These steps can be very simple or very complex, depending.
    ()-()
     \"/
      `                                                    
    
Re: Writing a web message board from scratch
by rbc (Curate) on Apr 19, 2002 at 18:11 UTC
    Along with the great suggestions that have already been posted.
    I find it helpful to have a State Transition Diagram(STD)
    as part of the design.
    Here's the search results to check out.
Re: Writing a web message board from scratch
by coreolyn (Parson) on Apr 19, 2002 at 20:41 UTC

    What are the first steps to planning the implementation of a largish project like this?

    This following has been said several ways already, but I get the feeling you may not have recieved a concise answer to this question.

    Reasearch what has already been done!

    coreolyn
      It's a good question ... and I'm not sure I know a book that really helps you with this stage of your professional development. A junior programmer can write one module to specifications, a software architect can design a whole large application, but this is intermediate: the design stage of a medium-sized project. One way to look at this is, how do you know when you are ready to start actually coding. I think a complete project plan should 1) list milestones 2) Describe them in enough details that you can make a time estimate for each, to within a factor of two 3) identify the major risks for the project ... normally you perform firs the task that is most likely to fail. By coincidence, I've just written a message board. (Like you, I had good reasons for not using something off the shelf.) I'd be happy to send you a copy of my project plan - email me at chrishmorris at uk2 dot net.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://160520]
Approved by crazyinsomniac
Front-paged by crazyinsomniac
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (5)
As of 2014-08-29 11:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (280 votes), past polls