Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Large Scale Perl Projects

by Anonymous Monk
on Oct 06, 2009 at 16:24 UTC ( #799520=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

What types of Perl projects are considered large scale ? whicn ones you have done? I see CUFP, Snippets, and and other 'code's on but not a large scaled one to get appropriate ideas from.


Replies are listed 'Best First'.
Re: Large Scale Perl Projects
by Old_Gray_Bear (Bishop) on Oct 06, 2009 at 17:03 UTC
    Let's see -- Apache, Slash (the /. engine), BugZilla, TWiki, ....

    And these are only the ones in the public sector that come immediately to mind.

    There are two banks that I personally know of (I've worked contracts with them) with multi-thousand-line Perl back-end systems. I suspect the there are a number of Scientific projects that also fall under the 'large scale' rubric, if only for the volumes of data they are crunching.

    Also take a look at this thread from 2006.

    Update: Cleaned up the syntax a bit. I gotta stop responding before coffee.

    I Go Back to Sleep, Now.


Re: Large Scale Perl Projects
by ikegami (Pope) on Oct 06, 2009 at 16:29 UTC
      Thank you. It is a good site. However good they are, I find them not updated and at least 5 year old. Any recent one would be more useful. While I agree that Business needs may not much differ, but I am also looking for anything that is slightly more useful in current context.
Re: Large Scale Perl Projects
by ELISHEVA (Prior) on Oct 06, 2009 at 16:59 UTC

    See also What is the major Perl successful story ?

    The code on PerlMonks SOPW nodes has to be short - it exists to demonstrate bugs in a focused manner or illustrate the core ideas behind an algorithm. CUFP and Snippets are only meant for snippets. Meditations sometimes have longer code samples, usually marked RFC. But when there is a great deal of implementation code, many people only post the interface definitions or even a general statement of concept, not the actual code.

    Large projects are usually found on CPAN and sometimes on SourceForge. One of the most well known is Bugzilla, but there are many, many others. For example, AWStats is used for website traffic analysis. Live Journal and Blosxum are two blogging systems written in Perl.

    Many large projects probably fly under the public radar screen because their main role is behind the scenes supplying infrastructure. Moose defines an entire OOP framework for Perl. Catalyst is a web development framework. CPAN handles all of your installation and deployment. BioPerl is a rich library for bioinformatics. POE is a framework for event-handling. DBI is a framework for database integration.

    And then there are the in-house systems, outside of the view of both CPAN and SourceForge. My own company has somewhere between 75K and 100K LOC of Perl code and test modules supporting various parts of our system administration, deployment and build system. These are designed to work cooperatively (we are reuse mavens) and so they function essentially as a single application with a large number of entry points. They are certainly managed that way.

    Best, beth

Re: Large Scale Perl Projects
by JavaFan (Canon) on Oct 06, 2009 at 16:32 UTC
    I don't think $WORK (or anyone else) would appreciate me posting a couple of million lines of code.

    "Snippets" is all you'll be getting from me.

      A coupla million LOC eh ? That's one way to use up bandwidth (and not a little time, I'll be bound) if you were to try :-D

      I suspect that the OP is just trying to establish the break-over point is between large scale and other perl projects ... however ...get appropriate ideas gives me great cause for wonder .oO(should that be concern ??).

      A user level that continues to overstate my experience :-))
        however ...get appropriate ideas gives me great cause for wonder .oO(should that be concern ??)

        Better than "get inappropriate ideas", I suppose... ;-)


Re: Large Scale Perl Projects
by stvn (Monsignor) on Oct 07, 2009 at 21:01 UTC
    What types of Perl projects are considered large scale ?

    Well, Perl tends to be more concise then Java or C++, so what might be a several million LOC Java/C++ project might be a several hundred thousand LOC Perl project. And honestly LOC is not always a good measure since CPAN often times helps reduce the size of the local codebase by allowing a lot of functionality to be done by libraries.

    What you really want to be asking is "does perl have the features necessary to handle large scale development?". To which the answer is a resounding "YES!". In order to do large scale development (IMO) you need at least the following language/toolchain features:

    • Namespacing

      Any time you have hundreds of thousands or millions of lines of code, you need to keep it properly organized. This is not just an issue of files and folders, but also of names. Perl has a really powerful package system that makes it really easy to both organize your code physically (files and folders) but also in the abstract (namespacing). Perl also has a lot of powerful tools on CPAN to help manage these namespaces with (ex: Sub::Exporter, Class::Inspector, Module::Pluggable, and all sort of other tools like these and more).

    • Seperate Compilation Units

      You might initially think this is not really relevant to Perl as much as it would be to a more traditional "compiled" language, but not having to build *everything* at once and the ability to dynamically load Perl modules is really important in large scale dev. This means you can better control the amount of coupling in your system and sub-systems since you can hot-load code at runtime.

    • Strong Testing Support

      Perl has a very strong testing culture, I would even suspect it is possibly the strongest. This means we have a TON of great testing tools as well. When doing large scale dev you want to be able to test the smaller components in isolation as well as within the context of the larger system or sub-system and then the final product. Perl has testing tools to make all this possible and then some.

    whicn ones you have done?

    At $work we have done perl projects that have gotten up into the hundred thousand LOC range and in some cases if you combined the separate projects for a single client (they are distinct systems but work together towards the same ultimate goal) we have hit the couple hundred thousand LOC range. I would consider these to be medium to medium-large scale projects.

    For non-$work, I am the original author of the Moose framework, which is probably only a couple thousand LOC (much more if you include the test suite), but has a pretty large-scale scope.

    I see CUFP, Snippets, and and other 'code's on but not a large scaled one to get appropriate ideas from.

    Here is the list of companies that use Moose and therefore Perl (this was compiled for the upcoming new Moose website). Of course this is by no means definitive, nor do all these companies exclusively use Perl. What it does signify is that these companies have at least one non-trivial (you wouldn't use Moose for a trivial project) new (Moose is only 3 years old) and modern (Moose is one of the driving forces behind the modern perl movement) Perl project going on.

    Best Practical
    Infinity Interactive
    Hearst Corporation
    Omni Hotels
    Open Data
    Capitol Advantage
    Cloudtone Studios
    Shopzilla / BizRate
    Current TV
    The Genome Center at Washington University
    Bioinformatics Group at University College London (UCL)
    Hopefully this will give you the information you need.

Re: Large Scale Perl Projects
by merlyn (Sage) on Oct 06, 2009 at 19:04 UTC
    $ cd $WORKGIT $ wc -l **/*.pm ... 1502543 total $
    That's $client's code, in production, making money every day.

    -- Randal L. Schwartz, Perl hacker

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

      Do for the real juice:

      wc -c *.pm | tail -1

        Even better, if I wasn't curious about the individual items, I would have done:
        $ cat **/*.pm | wc -l

        Followup: see Re^5: Large Scale Perl Projects.. this syntax means something in bash and zsh.

        -- Randal L. Schwartz, Perl hacker

        The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Re: Large Scale Perl Projects
by biohisham (Priest) on Oct 07, 2009 at 09:07 UTC
    Please Check How Perl Saved the Human Genome Project to know how large a scale Perl reached that it is intertwining with Humanity, and also check BioPerl to see how a large library of many Perl modules are adopted as the primary front to facilitate Biological Sciences chores and errands using Object Oriented philosophy.
    Update: let me add some buzz, BioPerl maybe large, but seemingly small snippets of Perl can do large things that can make a Perl programmer a Robin Hood to some and a fugitive to others, check Threat Level, isn't it strange that his name rhymes with Schwartz?.

    About this first link, you may say that this is over a decade ago but knowing Perl stepped in means Perl is actively engaged -with dominance- there in the support and maintenance of many Biological Sciences IT interfaces and information management and is being as indispensable to a bioinformaticist as a lab coat to a doc is.

    Excellence is an Endeavor of Persistence. Chance Favors a Prepared Mind.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://799520]
Approved by ikegami
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (7)
As of 2020-02-26 17:22 GMT
Find Nodes?
    Voting Booth?
    What numbers are you going to focus on primarily in 2020?

    Results (117 votes). Check out past polls.