http://www.perlmonks.org?node_id=186137

Sasquire has asked for the wisdom of the Perl Monks concerning the following question:

I've designed a shopping cart in Perl/CGI which I was planning to 'rent out' to numerous customers. The customers using my shopping cart to sell their own products would be using my server (third party webhosting).
Recently however, I've read about performance issues with using Perl over the web and was wondering if I haven't wasted my time (perhaps should have designed it all using ASP).

So I was wondering...
1) Is it going to slow down the server to excess if I have 100+ of the same cart running, each accessing a unique flat file database of 500k or so? (I wish to use flat file because the customers wish to export tab delimited databases from their accounting software). Assume that my server is just the run of the mill webhosting that you recieve at $15p/m sort of thing.
2) If yes, would MySQL sort the problem and how would I get flat file into SQL?
3) If such a feat is too much for the average webhosting service, what sort of machine will I need to accomplish this on my own? (eg I'll have to steal God's computer)
4) And ultimately, what does PERL/CGI have over ASP and similar scripting/languages when it comes to e-commerce websites?

I would also appreciate any help/links/material in dealing with mass use of a CGI program over the web.

Please excuse if these seem like basic questions but I'm certaintly no expert.

Thanking all ye Monks in advance...

A wee little novice

Replies are listed 'Best First'.
Re: Perl/CGI Performance for a Shopping Cart
by gav^ (Curate) on Jul 30, 2002 at 10:42 UTC
    This might sound like a harsh reply and it is meant to be.

    What the world does not need is yet another PERL/CGI (sic) programmer churning out yet another poorly designed shopping cart script. It is obvious to me that from Sasquire's post that he does not understand some of the important issues and will be doing his clients a disservice.

    A shopping cart is not a simple thing. It definatly isn't something that a beginning perl programmer should be attempting in any other context than a learning one. There are some big issues at stake here:

    Performance: How does the app scale? Can it handle peak loads? Talking about '$15p/m' hosting worries me. You won't nearly have the level of control you need.

    Security: Is it something I can trust? How are credit card details stored? Can something go wrong and they end up web accessible? What about other customer data? Does the script make any of the obvious mistakes like putting prices in hidden fields?

    Stability: Is it going to be up 24x7? Is there any degree of redundancy? Is there a plan for failure (ISP going under, hardware dying, etc)? Are backups done? How secure are these?

    Customization: How easy is it to respond to requirement changes? Can the client modify templates? What happens when they want to add features like cross selling? Is the tool powerful enough to handle changes?

    For these reasons I would never recommend that a small client bases their business on something written by a somebody who isn't an 'expert'. This is why a platform such as Yahoo Store is a win-win situation for somebody who doesn't want to spend a lot of money.

    For more advanced stores there are tools such as such as Interchange and Intershop.

    gav^

Re: Perl/CGI Performance for a Shopping Cart
by dws (Chancellor) on Jul 30, 2002 at 07:58 UTC
    So I was wondering...

    You ask a number of questions to which the proper answer is some variation on "it depends". For high volumes of traffic, you don't want to have to start up a new process external to the web server for each hit. That rules out CGI, though mod_perl is certainly a candidate. Or, you could arrange to run a separate shopping cart process that a thin web servlet (ASP, PHP, mod_perl, whatever) could delegate to via socket. I've seen setups like that handle high data rates, though they're more complicated to construct.

    If your database is 500K, you could benefit from either keeping it in RAM, or loading it into MySQL. (Loading data into MySQL is left as a (simple) exercise.)

    If you've already designed the shopping cart as a CGI, you have a good basis for experimentation. See how it scales against a 500K CSV on risk, then try moving it to RAM, then load it into MySQL. With a bit of stress testing, you should be able to make predictions on how the cart will scale.

    If you're going to be renting your cart out, you'll also need to consider availablity, which leads to load balancing, which leads to more hardware and a bit more complexity in the software.

    perrin has a good writeup on how eToys handled huge loads. See Building a Large-scale E-commerce Site with Apache and mod_perl.

Re: Perl/CGI Performance
by ajt (Prior) on Jul 30, 2002 at 08:30 UTC

    Tough question....

    You have several things mixed up, and I think it's best to separate them out.

    CGI
    Common Gateway Interface is a standard way that web servers can run programs, collecting input from web-forms, and returning results to the browser. It is slow but stable, and almost all servers support it. Most people use Perl when using CGI, but you do not have to, you can use virtually anything you want, e.g. BASIC, shell or c.
    In-Process / embedded architectures
    There are several embbedded technologies for web servers. Here the language interpretor is embedded into the web server, which makes it much faster, but it's more complex to use. Examples include ASP (where you can use one of several languages, e.g. Visual BASIC or Perl), and some with only one language, e.g. PHP/PHP, JSP/Java, mod_Perl/Perl.
    Perl
    Perl is a high-level language which is fast to develop in, and can run as a CGI script on a wide range of web servers and a wide range of operating systems. It can also be made to run much faster by using ASP or the mod_Perl interface. The CPAN library of pre-written code contains code to do most things you could think off - no other language has anything on this scale...

    Returning to your questions. CGI/Perl is not the fastest way of doing things, but it's pretty good for low to medium volumes. If you have written the code well, it can probably be converted from CGI to mod_Perl, in which case it will run much faster.

    There are many advantages of using a DB for storing data, and it should be easy to export data using Perl into any simple output fromat. Proper database systems have all the advantages of file-locking, multi-user capability, and potentially higher speeds. Examples include: MySQL, PostgreSQL or even DBD::SQLite all should be easy to use from Perl.

    A basic Linux/Apache/Perl box with a 1GHz+ processor and 1Gb RAM, will run very quickly, and be adequate for most needs, but cost more than USD15/month. You may find that at your suggested price range your options are more limited.

    One word of caution, you say you are a novice, in which case I would urge extreme caution if you are going to be moving money about. Web security is easy to get wrong, and you may be better off using code that someone else has written, and has been scrutinised by the public in general. I don't claim to know of such software.

    Additional links to read:

    Update: Extra Links added, minior typographical changes and corrections.
    ajt
Re: Perl/CGI Performance
by IlyaM (Parson) on Jul 30, 2002 at 08:22 UTC
    1. I guess by saing "I have 100+ of the same cart running" you mean it will be used by 100+ customers. Actualy the only important number is the rate of requests for this CGI in the peek time. In my experience unless you do very trivial stuff Perl CGI doesn't scale for rates much more than 1 request per secound. Of course it is very rough approximation. If you want more your need power of mod_perl. Read its guide for more info.
    2. Flat files doesn't scale for large amounts of data. They will be slow no matter you are using Perl or anything else. So yes, using MySQL (or other SQL database) helps a lot. Probably you may find perfomance of your scripts acceptable once they ported to MySQL so you will not even have to use something like mod_perl.
    3. This question is hard to answer. Really depends on expected load (i.e. the request rate).
    4. First of all I'd like to mention that ASP is not a scripting language. It is the way to embed scripting languages in web pages. And yes, you can use Perl with ASP too. As I understand on Win32 you need IIS and ActiveState Perl. On Unix you can use Apache::ASP with mod_perl. As for Perl vs other scripting languages. I find such discussions quite pointless. IMHO there is nothing you cannot do in Perl you can with other popular scripting languages. And this is true for reverse. Use tool you are most comfortable. If it is Perl then use Perl. If it is, say, PHP or VisualBasic than use them.

    --
    Ilya Martynov (http://martynov.org/)

Re: PERL/CGI Performance
by Ryszard (Priest) on Jul 30, 2002 at 08:25 UTC

    The performance of your site isnt *only* related to perl. We're talking network latency, machine speed, ram, IO, disk speed et al. Much of the overhead with perl CGI's is related to brining up the perl interpreter. (one instance per CGI). Check out mod_perl, it is apache with the perl binary build into it at compile time (Only have to load perl when you start apache).

    DONT USE FLAT FILES!, I'm thinking of several reasons:

  • Integrity - you may trash your "database" via a fatfingers (or late night) typo, very easy to do if you delete where you should have inserted.
  • Normalisation - much easier to do in an RDBMS than using flat files. If you're unsure as to what normalisation is, i reckon it could be a good research topic.. :-)
  • Scalability - It is much easier to preserve normalisation and extend your database using SQL than flatfiles.
  • Data integrity - Foreign key constraints. Enforcement of primary key, data typing, all native to an RDBMS you "should" use..

    Why use perl over ASP? Microsoft.. :-) completly irrational argument. TBH both may do the job, both platforms have quirks and the performance is highly dependant on application design. I'd choose perl as its easy, lots of support (you posted the question here didnt you?), and portable, (you can run it on solaris, linux, windows...). Also means your choice of host provider is not as limited.

    What makes MySQL better than others? Not much IMO. The latest stable release doesnt support transactions (Postgres, Oracle does). Both Postgres and MySQL are free, both have heaps of support... If youre leaning toward more of an "Enterprise" solution, Postgres may be the way to go as it is transaction based. If you're not particularly worried about this feature, MySQL might be the way to go.

    Of course if you have lots of cash, i'd say Oracle, which would probably be complete overkill for a small(ish) application.. :-)

    Anyways, good luck.

    Update: Whoops, i stand corrected on my ASP comments after reading IlyaM's post.

      Integrity - you may trash your "database" via a fatfingers (or late night) typo, very easy to do if you delete where you should have inserted.

      I've heard so many horror stories about late night mistakes like forgetting about WHERE part of UPDATE or DELETE queries that I don't think that SQL database is any safer than filesystem in this respect. In reallity no matter what database backend you use backups are must have for important data.

      The latest stable release doesnt support transactions

      Not true (for quite long time) with InnoDB table type which is part of source MySQL distro and part of MySQL-Max binary distro.

      --
      Ilya Martynov (http://martynov.org/)

        I've heard so many horror stories about late night mistakes like forgetting about WHERE part of UPDATE or DELETE queries that I don't think that SQL database is any safer than filesystem in this respect. In reallity no matter what database backend you use backups are must have for important data.

        Jepp! Two years ago I damaged all ongoing projects of my company from home by forgetting a where-clause; the backups were not @home :-)

        pure luck I wasn't killed in a car-crash that day...while we hurried to the companys rooms

        So while typos aren't an argument for DBMS :-), an rdbms is the way to go if scaleabilty matters, and it matters for every project not typed at the console-prompt (and somtimes even there...)


        regards,
        tomte
Re: Perl/CGI Performance for a Shopping Cart
by anolte (Novice) on Jul 30, 2002 at 16:03 UTC
    What really draws on performance, if you spawn a new perl interpreter for every request - so you should use s.th. like mod_perl under apache for this.

    If you want to recycle some code: look at a toolkit like openinteract ( sourceforge.net/projects/openinteract ), embperl or mason etc. Here you usually also find pointers to mod_perl etc.

    Andreas

Re: Perl/CGI Performance for a Shopping Cart
by Spenser (Friar) on Aug 04, 2002 at 17:58 UTC
    I used to use Perl with flat database files because I thought it was the simplest and best way to go.  Then I discovered mySQL--this reads like a laundry detergent commercial.  Anyway, mySQL has worked out much better for me.  It does take a bit of an effort to get started, but it's not too bad with the proper books and some patience.  As for its performance compared to other databases, it ranks as high as Oracle these days, according to e-week.

    Regarding your concerns about exporting to delimited text files, that's a breeze with mySQL and Perl and the Perl DBI module.  As with everything else with Perl, there are many ways of doing it, but below is a simple overview of an export program.  This export program is assuming that you've already set up a database and table in mySQL, of course.

    #!/usr/bin/perl -w use DBI; # Connect to the mySQL server and # get today's orders. my $dbh = DBI->connect("DBI:mysql:orders:localhost","user","passwd") || die "Could not connect:".DBI->errstr; my $sql_stmnt = "SELECT order_id, item_no, qty, price FROM orders WHERE order_date=CURRENT_DATE'"; my $sth = $dbh->prepare($sql_stmnt); $sth->execute(); while(@orders = $sth->fetchrow_array()) { $order_id = $orders[0]; $item_no = $orders[1]; $qty = $orders[2]; $price = $orders[3]; # Create record with bars as delimiters # and push record into array for exporting. $order = $order_id . "|" . $item_no . "|" . $qty . "|" . $price . " +\n"; push(@export, $order); } $sth->finish(); $dbh->disconnect(); # Now loop through array of orders # and save to text file. open(EXP, ">client-orders.txt") || die "Could not open export file: $!"; foreach(@export){ print EXP; } close(EXP);

    This bit of code is a lot wordier than it needs to be and can be consolidated into few lines.  However, when starting out I think its better to spread the code out to keep your sanity and to make learning and trouble-shooting easier.

    -Spenser

    That's Spenser, with an "s" like the detective.