Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Speed, and my sanity.

by Dylan (Monk)
on Aug 26, 2001 at 09:51 UTC ( #107895=perlquestion: print w/replies, xml ) Need Help??

Dylan has asked for the wisdom of the Perl Monks concerning the following question:

I have, for a long time now, been writing a perl program (gasp!:).

It is called The Manifold Engine, and it does many things, and will someday do *everything* for my web site. Mainly, it parses and 'compiles' my pages. (Manifold: having many forms or features) It gives no error messages now, with -W and use strict, and is nice to work with. some of things it does are: SSI's, custom tags like <@TAG@> could equal "The chickens, they scare the children", and I have 'Commands', like <$randomquote$=1$> that returns a random quote. See, it is the entire site, like, I guess. page=pagename, or form=login, etc.

What I want to know is, is there anyway to make such a insanely large thing (400 lines, I think) run faster? I've not had to put any load on it, only me testing it, and it takes only 2 secs longer than a static html page of exact same size.

I *have* tried to do this myself, by the way. it has many 'modes' (colors), and instead of "if mode is this, do this, elsif this, do that" etc, I did this:

if (defined($params{modes}->{$mode})) { $params{modes}->{$mode}->(); }
Also, I am going to use MySQL instead of plain text files, for larger amounts of data (news boards, downloads, fun things), and my server will be using mod_perl, soon. I've used most of the ideas (all?) from the Camel partaining to speed. Also, I would like to know if this All In One Perl CGI, is a bad idea.. Is it the work of a mad man?

I can post a link, if someone wants to view it.
Sorry if this post was long.
If anything is unclear, sorry.
I thank you for taking the time to help crazy perl. :) P.S. Oh yeah, loading it from localhost, it takes 1/2 second to load... maybe it is fast, but the server it will be running on is not a 512mb system with a 1ghz processor, like my home system.

Replies are listed 'Best First'.
Re: Speed, and my sanity.
by chromatic (Archbishop) on Aug 26, 2001 at 10:09 UTC
    Template Toolkit and Everything, the latter powering the site, actually turn a template into real Perl code. Once you have that, you can stow it somewhere safe. That's a big win, because you can skip the parsing step every time.

    It's an even bigger win if you have components that rarely change. If you only have to compile a component once, it'll run at full Perl speed each time through.

    If you have a working parser right now, it's not too difficult to generate Perl code. You'll certainly learn a lot. Feel free to download either package and dig through to see how it works.

      Wow, a reply in a very short ammount of time.. :)

      Well. I don't quite understand
      what do you mean "turn a template into real Perl code"? I already have perl code. maybe I was unclear (I am sometimes..)

      Well, sort of everything changes...

      it parsers generates HTML, not Perl code. and by 'compile' I ment it puts the same head and foot file on every page.

      I use the <@TAGS@> for all the colors.. When called like: manifold.cgi?mode=b it has basic, black and white and blue colors. and so on. So, my site is thus skinned. Perhaps I should post the code..

        Turning a template into Perl code means turning a template inside out: typical templating systems use a mixture of HTML and Perl (or Perl and "something else", which isn't necessarily HTML), where the Perl is "inside" the HTML. Like this, sort of:
        <b>Name: </b><% $name %><br>
        The HTML is the outer layer, and the Perl is contained within special sections marked off by some delimiters. In this case, <% and %>.

        Most templating systems will turn something like the above into this:

        $_out->("<b>Name: </b>"); $_out->( $name ); $_out->("<br>");
        Imagine $_out is a subroutine reference that is basically just
        my $_out = sub { print @_ }
        So essentially, "turning a template into Perl code" has created a piece of Perl code with a bunch of print statements, which is what you might have written if you hadn't used templates at all. This is what I mean by turning the template inside out: you have taken a template that was HTML with embedded Perl, and have turned it into a Perl script with embedded HTML.

        The advantage to Perl code is that Perl code is fast. :) Ie. if you have a block of Perl code, you can execute it directly; you don't have to parse it as a template. So you can store it in this intermediate Perl script form, and then just run it again without reparsing the original template.

        This is a relatively standard way of caching templates for many Perl templating engines. And obviously, this is just a small example, but it shows the basics of how it works.

        Oh dear. In my question, I said a crazy perl and I ment to say a crazy person..
Re: Speed, and my sanity.
by mugwumpjism (Hermit) on Aug 26, 2001 at 14:04 UTC

    Which web server are you using? If you're using Apache or Zeus, then I must recommend using FastCGI. Basically, if your program looked something like this:

    my $cgi = new CGI; $my_code->{$cgi->param("action"))->($cgi);

    You'd change it to:

    my $req = FCGI::Request(...); while ($req->Accept()) { my $cgi = new CGI; $my_code->{$cgi->param("action"))->($cgi); }

    Then your perl program is just sitting in a little loop calling the code to generate a page when the web server connects to it. You should be able to get of the order of hundreds of dynamic page impressions per second. I'm getting 50 a second with a project I'm working on on my system, which includes some heavy database access.

Re: Speed, and my sanity.
by perrin (Chancellor) on Aug 26, 2001 at 23:01 UTC
    As far as speed is concerned, using mod_perl will mostly take care of it. If it's still slow after that, you should do some profiling on it and find the slow part.

    For your sanity, you might consider switching over to one of the standard templating tools like Apache::SSI or Template Toolkit. You can read my recent article on for some background on these. You should also think about breaking up this 400 lines monster into modules. It will make maintenance easier.

      I think using ePerl would be a good idea.

      Why? Well, look at this: Manifold.txt
      Ugly, and very long. I have a bet with myself that I can make it shorter, with ePerl. :)

      If it's still slow after that, you should do some profiling on it and find the slow part.

      And what exactly do you do when you realise the slow part is the system paging, from having to keep a seperate perl process for each and every web server child that most of the time is blocked on network I/O?

        You have at that point many options.
        1. Buy more RAM.
        2. Reconfigure Apache to have fewer processes.
        3. Identify parts of your system that take up the most memory. Then you can:
          1. Rewrite them to be less memory intensive.
          2. Turn them back to CGI.
          3. Use a dedicated serving process (eg FastCGI) for those components.
          4. Use shared memory segments.
        4. Reduce the number of requests that a given child will handle before restarting.
        5. Move to an OS with on demand paging (rather than per process) and give it lots of temp space. (This mainly applies if you are not already using some form of *nix.)
        6. Buy faster hard drives.
        There is no universal solution to poor performance. However the vast majority of the time mod_perl is a massive improvement over straight CGI. Time and money spent optimizing CGIs to run faster is typically penny wise and pound foolish. Hardware is cheap. With the spare hardware from the dotbombs, it is really cheap. Unless you are doing something ridiculous, the odds are very good that hardware is the best solution at hand.
        Maybe eliminate the network I/O for your heavy mod_perl procs using a proxy. Perlmonth has a great article on how to properly set this up.


    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Speed, and my sanity.
by Dylan (Monk) on Sep 06, 2001 at 10:10 UTC
    I've put a few things in modules, and it is better now.

    I have a few things longer than 400 lines: they are much easier to read...

    A few things are still hairy, but it is getting better.

    I figured if I put most of the neat functions in a module, than I could use them if I ever want to use some other templating toolkit.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://107895]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (5)
As of 2023-03-23 16:56 GMT
Find Nodes?
    Voting Booth?
    Which type of climate do you prefer to live in?

    Results (60 votes). Check out past polls.