Your Mother has asked for the wisdom of the Perl Monks concerning the following question:

Three part longish question to explore the scope of Perl's power. Since there are so many heavy-hitters here, this, I hope, should get some very interesting info.


Addition: Excellent stuff so far! Keep it coming. I'll see about doing a précis of the replies when the thread has died down.

  • Comment on What's the biggest piece of work you've done alone with Perl?

Replies are listed 'Best First'.
Re: What's the biggest piece of work you've done alone with Perl?
by dragonchild (Archbishop) on Apr 21, 2003 at 22:26 UTC
    While at Motorola, I rebuilt (practically from scratch) a system called BOOST, which was used for 90% of all sub-system, system, and regression testing of the software used in Motorola's CDMA base stations. I inherited this system and it was a mess when I got it. I left it in a very beautiful OO state.

    Strategies? Your basic refactoring strategies, I guess.

    • Get everything to run under strict.
    • Identify common code that seems to be repeated and get it to live in one place.
    • Identify bits of business logic that should live together, then make them live together, preferably in some object.
    • Identify bits of application logic that yadda, yadda, yadda.
    • If there's a way to make Perl do your work for you, then be intelligently-lazy. For example, all the messaging was done by having each message be its own class. But, I didn't want to maintain 1000+ messages in three different strains by hand. So, I developed a common data structure that would be inflated by Perl into the classes. (That was over 75% of the code and about 45% of the success.)
    • Build a logging methodology and put it everywhere. Unless you know exactly what went wrong, you'll never make it right.
    • Get the buy-in from your group leader, but ignore anyone higher. All they want to know is that it works, not the cool things you did to get it there.
    • Write migration tools. Part of this effort involved changing the configuration files that the userbase used. Absolutely necessary. However, if I wanted acceptance, I had to make it as painless as possible to migrate.

    Mistakes? The biggest mistake anyone can make is to not have a design document, preferably coupled with a requirements document. This isn't an ivory-tower nicety. This is a real-world, in-the-trenches, life-and-death necessity. Spend the 100 hours to make one and you will recoup that 100 hours in the next four weeks. (In case you think I'm exagerrating, ask Elian what he thinks of design documents.)

    We are the carpenters and bricklayers of the Information Age.

    Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

    Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

Re: What's the biggest piece of work you've done alone with Perl?
by Abigail-II (Bishop) on Apr 21, 2003 at 23:49 UTC
    Probably the time I became a database administrator in the company I worked for. We had about 20 database servers (the amount varied), with 400 to 500 databases running on them (the amount of databases varied daily).

    The previous administrator had made a mess of things - he would just sit in his cubicle, and only move his ass when people started to complain they could no longer do their job. Nothing was being monitored. Backups were automated, but since everything was hardcoded, and not touched for a long time, most databases, including the more important ones were not updated - and noone was inspecting the output for errors. When I took over, I was working 10+ hours/day just to keep things running. At my first day as a dba, I hacked my first monitoring program together, and at least I would get paged if a database became unavailable, instead having to wait till user started to complain. Within months, everything was database and every server was being monitored for their vital statistics, all databases would be backed up daily (no recoding needed if databases were added or deleted), databases would be dbcc-ed regulary, and many problems would be solved automatically within a minute of them occurring, and I was down of spending less than 4 hours a day on my dba tasks, while getting an uptime of over 99.95% outside of the maintainance windows. With the vital statistics of the most important statistics being graphed - to please manglement.

    This turned out to be my biggest mistake as well. When the company had to lay off people, they decided a dba was no longer needed. :(

    As for strategies/techniques/modules, the code itself wasn't too much. All monitoring less than 1000 lines of code. The Sybase reference and sysadmin modules of the documentation where vital though. I used Sybase::DBlib to make the database connections, gnuplot for the graphing, and /bin/mailx for the paging.


Re: What's the biggest piece of work you've done alone with Perl?
by dws (Chancellor) on Apr 22, 2003 at 00:31 UTC
    Three part longish question to explore the scope of Perl's power.

    If you're exploring Perl's power, I suggest you rethink your first question, or consider adding a fourth. Perl is suitable for more than just solo programming. Teams can use it, too. I once managed a team that had written an 80KLOC application server in Perl. (The core of the team that wrote it was ~6 people, more were added later to extend it).

    Personally, the biggest single thing I have in Perl is about 1500 lines, plus another 1000 lines of unit tests, but not including CPAN stuff (HTML::Template, DBI, plus a couple of others).

    A strategy that helped me pull it off was to use Test-Driven Development. Figuring out how to write tests first helped me think through APIs with an actual client (the test) in mind. That helped avoid a lot of nonsense. Having a large body of unit tests gave me confidence that I wasn't breaking things as I went.

    The biggest mistake I made was not getting the code body under source-code control soon enough. (I've since set up a home CVS pserver.)

      I agree about the team vs. solo. If someone told me he had written a huge application in Perl all by himself and 90% of the code was his, I would not see that as a good thing. Much better to hear something like "I wrote a successful application with a zillion features that was able to turn on a dime when the business asked for changes, and took only a few hundred lines of code, and I did it by collaborating well and reusing tons of open source code from CPAN."
        Team or individual, it's prudent to see the quality of the code before judging, wouldn't you agree? 90% The Conway™ just might stand a chance of being better than 90% temp-SDEs. There have been fantastic large things done mostly by an individual with Perl (like Mason, right? or like the amazing sounding wireless stuff dragonchild wrote about, or for that matter—almost 1/2 of its 6,400 lines is code and I know that many have worked on it today but at one point it was the work of one and that single module drove a staggering portion of web development).

        The power of the communit(y|ies) is evident just by visiting this amazing site. I'm curious about the extreme edge of the leverage that Perl gives the individual. There are many, many users here who own scads of modules or big parts of Perl. What better group to put this question to?

        The second question does include "what modules?" I'm not seeking the Alceste formula... unless of course you have it. :)

Re: What's the biggest piece of work you've done alone with Perl?
by Juerd (Abbot) on Apr 22, 2003 at 00:24 UTC

    What is the largest system|code|site you have (>=90% of the work was your own) supported|created|inherited which was primarily in Perl?

    I don't know since most of my projects are under 2000 lines of code (Perl's too damn powerful).

    What are the main strategies that allowed you to pull it off?

    No feature overkill, no complex designs, no bureaucratic nonsense.

    Other than that, document every choice and every API for you WILL forget the specifics.

    What are the main techniques|modules that allowed you to pull it off?

    Stuff things in modules if it makes sense. Use OO where it makes sense. Release code you use in almost every project to CPAN (let others discover bugs for you so you can fix them before things go terribly wrong *grin*). That's why PLP, DBIx::Simple, again and Exporter::Tidy are on CPAN. They wouldn't be if I didn't use them myself.

    how about a bulleted list?

    Bulleted lists form a great strategy, a reasonable technique and perhaps a nice module. But they're a terrible brand of coffee!

    What was the single biggest mistake (most costly to your own time) you made while doing it?

    Not documenting was my biggest mistake. Lack of documentation saves time at first, but takes a LOT of time when you need to change the code a few years later.

    Another VERY big mistake was starting to code in Perl without actually knowing the language well enough. I do know Perl quite well now, but my old code is disastrous. Even though I did make enough money off my first projects to start my own business, I do regret that I did not wait another year or maybe two. And I wish I had a good book back then: I learnt most of what I know about Perl from perldocs and IRC.

    One very good thing you can do is letting people here and on IRC review your code. Disclosing source usually also means tidier code :)

    Another piece of advice: choose a readable and CONSISTENT programming style. Whitespace DOES matter!

    - (do not use).

Re: What's the biggest piece of work you've done alone with Perl?
by JPaul (Hermit) on Apr 22, 2003 at 15:10 UTC
    I was hired on to an Internet Provider to automate their billing system and ended up writing the dreaded eCommerce platform.
    And in this case 'Platform' is the operative word; From the get-go the specs changed from week to week depending on what idiot the manager had talked to and told him what his application had to have and how it had to function in just specifically a certain way.
    As the bank we had partnered with became more and more difficult to deal with the manager and accountant would think up more and more ridiculously obscure and abstract ways of accomplishing tasks that had become over-regulated.
    The system ended up being big. Like, way big. Like, you don't really want to know how big. The only part of the software not written in perl was the code used to talk to the bank via a frame relay VPN for credit card transactions, and that was only because it was based on existing code (That didn't work).

    • All shop owners would host their data on our site, but could link our site into theirs
    • We had an internal mail system between shop owners and customers, including an address book and all
    • Based on figures from various international courier/freighters the shop owner picked a freight company, and we worked out all the charges for them and contacted the freight company for item pick-up ourselves
    • The entire system was insured, and as such had an Insurance module allowing representatives of the insurance company to review any insurances claims (Which had to be made internally) on bad sellers
    • All sales history was naturally recorded, and there was a mountain of ways of displaying and graphing this data, thanks to the accounts two floors down
    • Our partner bank decided not to allow us to pay money into a credit card, as such we would collect all of our shop owners profits during the day and then wire amounts to each country appropriately. Then we would manually dial in via modem to whatever chosen issuing bank we had talked to, and spread the money out to bank accounts from there, thusly avoiding a telegraphic transfer charge for every bank account we sent money to.
    Those are the major features I can think of, off the top of my head. Most of those are still pretty frightening to me, particularly the insane method of paying into bank accounts... But there you go. Please laugh, it's all I can do now :P

    -- Alexander Widdlemouse undid his bellybutton and his bum dropped off --

Re: What's the biggest piece of work you've done alone with Perl?
by Anonymous Monk on Apr 22, 2003 at 03:51 UTC
    >=90% of the work was your own

    Has such a project ever been written, by anyone? Have you looked at the modules you're using? My Perl projects have all had more reused code written by others than original code written by me. Is this a bad thing? No, because that's not what I use Perl for. If I want to write an entire application from scratch, I'd choose a different language. Perl's power comes directly from third-party modules.

(z) Re: What's the biggest piece of work you've done alone with Perl?
by zigdon (Deacon) on Apr 22, 2003 at 17:16 UTC
Re: What's the biggest piece of work you've done alone with Perl?
by The Mad Hatter (Priest) on Apr 22, 2003 at 17:28 UTC
    The largest project in Perl I've done is the Public Mooseboard, a discussion board for my friends and myself.

    CGI, DBI, and HTML::Template (plus a few of my own modules) made the job a lot easier.

    My biggest mistake was not modularizing the code from the beginning. I copy and pasted instead of creating my own modules for general functions. This led to problems when I updated code in one area but not another and general bloat. It also made it harder to switch backends. I've since shoved most of the general stuff in modules, but there is still some things that I haven't gotten around to yet.

Re: What's the biggest piece of work you've done alone with Perl?
by cyberconte (Scribe) on Apr 22, 2003 at 18:11 UTC
    Not counting that failed startup i was a part of (which got big, but it doesn't count because a team worked on it, and you asked for 90% 'your own'), i'd say Phynd 4, including the home site. It also has a little bit of C (for an external helper program, not embedded), but id say its 95% in perl.
    • Vanilla Coke. By the bucket. Diet counts too.
    • Modularize where you can. (Althoug this was mostly appropriate due to the project.)
    • DBI is your friend.
    • Keep things neat and organized! I can't tell you how much of a difference that made (both good and bad when done/not done, respectivly). subs and "comment blocked" code, so that its easy to come back to a section after working on an entirly different part of the code for long periods of time. As your project gets large, this will occur more frequently and thus becomes more important.

    Probably the biggest mistake i made was trying to do everything in one shot. Working on a project alone, every time i would complete 1 task, i would htink of 3 others that would be great to have. The end result was a .0 release that, in all honesty, should've been a .9 release. As of right now, i'm working on a .1 release which is what i imagined the .0 release should've been.

    If i had a technical document, i would've been much better off and had much clearer goals. I would reccomend one.
Re: What's the biggest piece of work you've done alone with Perl?
by thom (Beadle) on Apr 22, 2003 at 18:36 UTC
    What is the largest system|code|site you have (>=90% of the work was your own) supported|created|inherited which was primarily in Perl?

    I wrote a 10'000 lines network test utility (database with hosts, pinging, host/IP resolving, tracerouting and proxy check) with a HTML interface. At the moment I'm writing a library of useful classes for system administration scripts which has about 5'000 lines.

    What are the main strategies|techniques|modules|brands-of-coffee that allowed you to pull it off (how about a bulleted list)?

    Planning: Flow charts for scripts and UML for bigger libraries. Bulleted lists and TODO/FEATURES lists also help a lot. Also look at CPAN if someone has already done something similar.

    Modules: CGI, HTML::Template, CGI::Application, DBI

    What was the single biggest mistake (most costly to your own time) you made while doing it?

    Improper design at the beginning -> rewrinting some parts from scratch and having done very little documentation (that was before I discovered POD :)).


      YABM (Yet Another Biggest Mistake): Roling my own webapplication framework with HTML::Template which did approximately the same as CGI::Application but with a procedural interface. I wrote only one application with it and abandoned it.

Re: What's the biggest piece of work you've done alone with Perl?
by hsmyers (Canon) on Apr 22, 2003 at 14:47 UTC
    Here is a list I've tried to live by...
    • Change Nothing
    • If you have to change it, make the smallest change possible
    • Make all changes in parallel---implement the new code and compare it's results with the previous
    • Change Nothing without a test to back it up with...
    And if given a choice, it is often better to be lucky than good.


    "Never try to teach a pig to wastes your time and it annoys the pig."
Re: What's the biggest piece of work you've done alone with Perl?
by gawatkins (Monsignor) on Apr 22, 2003 at 22:29 UTC

    The largest single piece of code that I have written takes the back end database files from our Avaya CMS server, renames them, inserts them to .xls files, and writes them to a database.

    The first part involves the renaming the files from their default name of chrXXX.XXX where the X's are numbers. One of our remote reporting teams likes to receive them as csv files, so I simply open the directory and copy the files to a directory with the extension csv.

    The second part was a bit more difficult, I have been using the Spreadsheet::WriteExcel::Big CPAN module to write the files to an excel spreadsheet (for one manager that only uses MS Excel) containing all of the call records each day. I had some difficulties, but solved them by chopping the files at about 14000 lines.

    The last piece involved writing the daily call records to a MS SQL database, then a MYSQL database for our local reporting team to generate reports from.

    My main strategies involved trying to go out of my way to get everyone and what format they wanted and to help out local reporting guy who was spending all his time doing this task manually. As for the modules, I found Spreadsheet::WriteExcel::Big to be very good.

    My biggest mistake was using MS SQl as a db the first time. The tables csv files have 55 columns, somewhere between 20,000 and 30,000 records each day. I had a terrible time with the DBI, each record was taking around 4 second to write. Finally I wised up and was able to write the all the records in just a couple of hours each day to the MYSQL db with the exact same piece of code, go figure.


    Greg W

Re: What's the biggest piece of work you've done alone with Perl?
by sfink (Deacon) on Apr 28, 2003 at 03:59 UTC
    A graphical debugger for LALR(1) parsers (about 6k lines). I wrote the LALR(1) parser generator, a debugger, a state machine visualizer with automatic graph layout, and an XML-fed hyperlink widget for Perl/Tk. I think of myself as having written 100%, despite using Perl/Tk and several other modules -- because I find that if I think of things that way, I'm both happier and more likely to reuse CPAN modules.

    Things that enabled me to pull it off:

    • Primarily, focusing on having something releasable without being perfect. If I had "finished" any single part of it to my satisfaction, I never would have released anything.
    • Heavy use of CPAN modules, of course!
    • Don't think too hard about making everything nicely modular initially. It's much easier to write a huge wad of stream-of-consciousness code and then go back and figure out what can be refactored into reusable pieces. (This might be a personal bias, because I suck so badly at up-front design.)
    • Come up with a small number of nontrivial and representative test cases early on. I personally don't care whether it's before you start implementing or during, and I tend to leave the exhaustive test cases until afterwards. During the main implementation push, I prefer to work from a small number of tests that I get to know intimately well.
    • Try very hard to plow through the things that are more likely to be showstoppers early on, when you're still enthusiastic and flying along.

    Biggest mistake: probably allowing myself to flail when I get stuck on things. Instead of stepping back and figuring out a simpler way of approaching the whole area I was having problems with, I would tend to accrete special cases, debugging code, large commented state machines, painstakingly detailed descriptions of what some insanely complicated function should be doing, etc. There's probably a better way to summarize that -- perhaps it would be "my biggest mistake is trying to solve problems by making them more complex instead of simpler"? Something like that.