http://www.perlmonks.org?node_id=565877

techcode has asked for the wisdom of the Perl Monks concerning the following question:

Hello fellow Monks.

I have checked Google and this website for some samples on BIG projects done in Perl - but without sucess. Closest thing was this node here.

Without much discussion of what is big and all that I would just like a list to be able to present to some people that say that big projects can't be developed in Perl.

Thanks.

Have you tried freelancing? Check out Scriptlance - I work there. For more info about Scriptlance and freelancing in general check out my home node.

Replies are listed 'Best First'.
Re: Samples of big projects done in Perl
by explorer (Chaplain) on Aug 06, 2006 at 23:06 UTC

    Please visit PDL... It have big projects at success page.

    My project only have 20.000 lines (18.500 working, 1.500 testing), and is working NOW. Many barbarians with fire in their hands... :-(

Re: Samples of big projects done in Perl
by GrandFather (Saint) on Aug 06, 2006 at 23:18 UTC

    Actually the thread you reference has a fair discussion of using Perl for BIG projects. At the end of the day you can use Perl for BIG projects involving a very large number of lines of code, but it most likely is not the most productive language for that sort of use.

    There is no single tool that does every job. That is just as true for languages as it is for DIY equipment. Sure, you can drive screws in with a hammer, but that's not the best way to do it. The closer the fit you get between the job at hand and the tool to do the job the easier the job is to do. Sad as it may seem, Perl is not always the best tool and one area where it is not the best tool is in writing very large applications.

    Actually, discussing what constitutes BIG may be fairly important. To me a big project is 50,000 lines and up, involves more than one person and writing likely spans more than a year. YMMV.


    DWIM is Perl's answer to Gödel

      ++ for a good article. From my (admittedly very narrow) experience in the field I must disagree with your general opinion, though.

      Right you are that Perl isn't always the right tool for every job, but in my own worklife experience I've seen many projects of the scope you mention, and miraculously, the ones done in Perl went quite remarkably well many more times than those done in other languages (Java, C++, Python, C, PHP, VB, Ruby in descending order of their use.)
      Just to say, there might be places where Perl might be the right tool for larger projects.

      What would you suggest would be the computer languages and technologies that are more likely suited for bigger projects?

      Cheers, Sören

        Ovid mentions a 1/4 million line Perl project he worked on which certinally shows that Perl can be used for BIG projects in the right hands.

        My argument is more to do with "stricter" languages picking up egregious mistakes earlier when they cost less (by whatever measure) to find and fix. As projects get larger the cost of finding and fixing such mistakes gets higher, almost exponentially.

        To put this all slightly in perspective: in many ways I'm a sloppy programmer who depends too much on my programming tools to find silly errors for me. For large projects such errors have a better chance of being detected early with C++ than with Perl.

        I don't have enough experience with a wide range of languages to say which might be "The Best". But from my experience, at one extreame, cost per line of assembly language is much higher (and does much less) than anything else I've used and it doesn't scale well at all. C++ and Perl are definately toward the other end of the scale. I've not written code for a BIG project in Perl and I suspect that for me and my style of programming, Perl wouldn't scale at all well. I am involved in a BIG C++ project and know that, despite my sloppy coding, the project is progressing in a saticfactory fashion with about 6 people working on different parts of it and that it is maintainable.


        DWIM is Perl's answer to Gödel
Re: Samples of big projects done in Perl
by Tanktalus (Canon) on Aug 07, 2006 at 03:30 UTC

    It really depends on what you mean by BIG. Based on GrandFather's definition, I'd have to say that the code we have at work is medium-large. 33,000 lines (using a simple "find . -name '*.p[ml]' -print0 | xargs -0 cat | wc -l" - so really much less as code), and taken about 8 person-years to get here (approximately 5 years, with 1 person full-time on it, and myself part-time on it). Two of three requirements met, and the third requirement nearly met. (If they hadn't done such a bang-up job at changing requirements on us, it would have been a bit less time to get here.)

    That said, by the time it is doing everything we want, it'll be at least 37,000-40,000 lines, and take another year or so.

    Is that "big"?

      Thinking about it a bit more - BIG is probably a function of language. 20,000 lines is getting BIG for Perl, but is smallish for C++. Our main code base is about 1,000,000 lines of C++, written by 3 - 6 people over about 10 years. I'd consider that to be BIG, but it is still manageable and maintainable. I suspect a Perl project of that size would be getting to be something of a handfull!

      Bigness could also be measured in terms of functionality provided. A Perl equivelent of the 1e6 line C++ program (presuming it were practical) may be half the number of lines of code, but I doubt it would be anywhere near as managable as the C++. Different rules apply as code gets larger so scaling a size/functionality ratio for Perl compared with other languages is unlikely to be meaningfull as projects get BIG.


      DWIM is Perl's answer to Gödel

        My rough estimate (informed by Peter Scott and more or less confirmed by Nicholas Clark) is that there's about an order of magnitude difference between C and Perl SLOC. C++ can be a little better, but I doubt it's as good as two to one. I expect a million line C++ project to take perhaps 200-250k lines of Perl. It depends on the domain though.

        I always wondered: Why would a 500,000 lines project in Perl be less easier to maintain than a project with 1,000,000 lines of C++?

        CountZero

        "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

      I don't think that's "big" big. You're probably not running a space shuttle or emacs or anything. It's beyond shell script size, though.

      I'm aware of the "order of magnitude" argument mentioned elsewhere in this thread, saying that 1 line of Perl is like 10 lines of C. It sounds nice, but I'm not sure I really buy it. Since 1 line of Perl is 10 lines of C, assuming that's true in the first place, then in the Perl code you're freed to think at a higher level of abstraction. It's not as if you're juggling 10 times as much in each line of Perl code.

      I think that argument is more for motivating people to write cleaner Perl code in the first place. "Should I put these 100 lines in a module?" "Would you put 1000 lines of C in a .c file?" "Sure." "Put your 100 Perl lines in a module, then." It seems to me to be more a way of determining at what size you want to start thinking about things at a higher level. But.. I mean if you're using a C library or a Perl module, it doesn't really matter how much code is inside them. They have (ideally) some external interface that you use, and that's it. Using 10 C libraries or 10 Perl modules, neither is substantially more complicated than the other, is it?

      How many lines of code is Firefox, for example? How many lines would the equivalent implemented in Perl be? Would the Perl version be 10 times simpler? What if you included also the Perl modules it uses? Perl itself?

      Using your shell command I found 190,000 lines in the latest release of Bricolage. But a lot of that is POD. Has anyone written a line counter with POD sections stripped out? (Maybe comments should also be removed; I'm not sure what exactly the definition of "line of code" is. On the other hand, Bricolage is a web application, so we should probably also include JavaScript, CSS, HTML, etc. as code.)

        Has anyone written a line counter with POD sections stripped out?

        Here's the somewhat rough-and-ready code I use to count lines in my work application:

        #!/usr/bin/perl -w use strict; use File::Find; my($d, $c, $h, $q, $s, $x) = (0, 0, 0, 0, 0, 0); for my $dir (qw/ cgi lib util /) { find(sub { return if /^\.#/ || /,v$/ || /\.(swp|gif|jpg|png|ps|tr|o|a)$/ || / +~$/ || /^core$/; $File::Find::prune = 1, return if $File::Find::dir =~ /\bCVS\b/; return unless -f && -T; local *F; my $file = $_; open F, $file or warn "$File::Find::dir/$file: $!\n"; my $cut = 0; my $here = undef; my $where; while (<F>) { if ($cut) { ++$$where; $cut = 0 if $cut > 0 && /^=cut/; } elsif ($here) { ++$$where; $here = undef if /$here/; } elsif (/^=/) { ++$d; $cut = 1; $where = \$d; } elsif (!/\S/) { ++$s; } elsif (/^\s*#/) { ++$c; } elsif (/<<(?:'(\w+)'|(\w+))/) { my $style = $1 || $2; $where = ($style eq 'SQL') ? \$q : \$h; $here = qr/^$style$/; ++$x; } elsif (/^__(DATA|END)__$/) { $cut = -1; $where = \$h; } else { ++$x; } } }, $dir); } print "doc $d, comment $c, SQL $q, text $h, space $s, code $x\n";

        This currently reports: doc 5699, comment 1685, SQL 675, text 4295, space 4382, code 37030.

        Hugo

Re: Samples of big projects done in Perl
by nite_man (Deacon) on Aug 07, 2006 at 12:38 UTC

    Maybe we should talk about success of using Perl to build enterprise applications. Because, the BIG project is nothing. You can write more than 1 million rows of code and it wouldn't good example.

    If we are talking about E-commerce the most famous example is eToys.com. Livejournal also can be a good example of BIG project based on Perl.

    RT is an excellent example of short (it's totally about 200,000 lines) but well developed application.

    I've developed for 4 years in the Billing System which based on Perl and Embperl (for user interface). We have more than 300,000 lines of code. It's not so much but our system's worked success about 3 years. It does all work: from receiving events (VoIP, VOD, Internet, Telephony, Web hosting etc) to issue invoices and processing payments. I cannot say that Perl is the best choice for this kind of applications. But I'm sure that it's much better then Java.

    So, the number of lines in the code cannot be a good criteria of aptitude of some language IMHO. It'd be better to use the cost of development which may include number of developers, the time of development, the number of already made parts which can be used in the development, the cost of hardware needed to run application and so on.

    ---
    Michael Stepanov aka nite_man

    It's only my opinion and it doesn't have pretensions of absoluteness!

Re: Samples of big projects done in Perl
by Anonymous Monk on Aug 07, 2006 at 12:15 UTC
      We've got around 200,000 lines of Perl in a system that manages and checks medical laboratory data coming off automated instruments. The usage is 24x7, and people's health depends on the accuracy of the numbers we maintain and verify. The data comes in many different forms, in terms of its inherent structure, as well as how a specific instrument manufacturer has chosen to package it. This makes the agility of Perl a very important feature.

      We have another million or so lines of code, mostly in C, that we will be porting to Perl over the next several years.

      Hope this helps you in your decisions.

Re: Samples of big projects done in Perl
by aquarium (Curate) on Aug 07, 2006 at 11:48 UTC
    horses for courses. how many lines of code do you have to write (from scratch btw) to achieve basic regular expression matching capability for a small project? i guess you'd call it a big project in C, and a one or two-liner in Perl. the open style of the perl language allows for flexibility in expressing a solution to a problem...with structured languages it's all about declaring the correct size variables and watching that you de-allocate what you allocate first. as perl was written for the domain of text processing, with thousands of modules written, you don't have to re-invent the wheel.
    the hardest line to type correctly is: stty erase ^H
Re: Samples of big projects done in Perl
by blahblahblah (Priest) on Aug 08, 2006 at 01:46 UTC
    The product which I work on consists of about 450,000 lines of Perl code, plus a handful of C, Java, and javascript files. We've grown to 8 full time programmers currently. I don't have any experience with large projects in other languages, so I can't offer any personal arguments about why Perl is better or worse than Java, C, etc.

    Joe

Re: Samples of big projects done in Perl
by jzb (Hermit) on Aug 07, 2006 at 13:41 UTC

    In a previous job my boss and I wrote a daily installation servers for nightly builds entirely in perl. He went as far as to build clients with perlcc (not an easy task at the time).
    I can't think of a better language to do such things...

    s/your reality/my reality/g
Re: Samples of big projects done in Perl
by nite_man (Deacon) on Aug 08, 2006 at 08:02 UTC

    Well, if we're talking about enterprise applications here is a couple of Perl-based frameworks which can help you build BIG project: Catalyst - well-known framework to develop web-based applications, Gantry - young but perspective framework, Maypole - framework for MVC-oriented web applications.

    It's difficult to develop a large project from scratch based on any programming language. The problem is not in Perl. The problem in lack of knowledge of appropriate tools IMHO.

    ---
    Michael Stepanov aka nite_man

    It's only my opinion and it doesn't have pretensions of absoluteness!

      Full Ack. Programming is not just stuffing lines of code together, it's a form of engineering. It involves planning, conception *and* execution. A C programmer that is unexperienced with Perl will probably not be able to manage a 500_000 lines project in Perl. On the other hand, I, as a Perl dev, wouldn't be able to handle a 500_000 lines project in C.

      Language does matter, as well as the knowledge of that language. So, in my eyes, all those "C is more manageable than Perl" comparisons/fights/arguments are (excuse the word) crap. If everyone could just admit that *he* couldn't, I'd feel much more comfortable talking about differences of programming languages. There's a big space between "I don't believe a 1 million Perl project is manageable" and "I don't believe I could manage a 1 million Perl project." Just like it is with the difference of "C is more manageable than Perl" and "I'd be able to manage larger projects in C than in Perl."

      Ordinary morality is for ordinary people. -- Aleister Crowley
      I agree with you. But if I understand correctly we're talking about possibility and effectiveness to use Perl in the enterprise (I don't like this word) projects. In my mind, to do that it's needed to know the Perl tools. Many developers start the project from the scratch and then tell how difficult develop using Perl.

      ---
      Michael Stepanov aka nite_man

      It's only my opinion and it doesn't have pretensions of absoluteness!