why i may have to leave perl...

[eduardo]

I don't know to what degree this qualifies a meditation, however, it is something that I have been pondering for a considerable amount of time now, and it is something that I wanted to ask of the perlmonks community.

I find myself in a very interesting situation at this particular point and time in my life. I have spent a great many years in the trenches, coding, debugging, profiling, designing... and I have discovered what my favorite tools as a programmer are. I have discovered the methodologies by which I enjoy programming the most, as well as the languages and enviroments which caused me to be more productive, as well as happier. However, I realize that I have been blessed, and that the people I have had the good fortune of working with, were not representative of the programming community, and did not need the "crutches" that other enviroments present. Professionally, at my last job, I had a small team of programmers working under me... I had total faith in this team, and I knew that they were amazingly talented, so I had no problems giving them more than enough rope to hang themselves with... so of course, we wrote everything in perl.

The productivity my programmers had, thanks to perl, is of course amazing. The sense of happyness that they had, by working in such a great enviroment, was of course itself great. And they communicated, and spoke, and shared their designs, and interactions... and the objects talked to each other, and the modules communicate perfectly, and all of the tools worked exactly as I would have wanted them to.<br.

During this time, however, my attitude changed slightly. I began to see that the very power which perl gave you, the flexibility and expressability that it created, would also be it's biggest downfall. In an environment with only 5 programmers, the communicational heirarchy was small enough that it was trivial to ask about the subtelties of object interactions, aspects of instantiation and parameter passing, etc... In such a small team, the very large software engineering issues, which can be encountered, were manageble, despite of a language that realistically, gives very little restriction and form to the programmer. TMTOWTDI. that was the mantra... and i have reached a stage in my professional life, where I fear it.

I find myself the VP of Strategy and Technology at a firm, and I have been charged with developing and designing the software engineering plans, methodologies, and platforms that we will use, and build a software services firm around. And, for a series of reasons, I am having a very hard time finding a way, any way, to use perl as our main development platform.

If you chose other platforms, something like java for example, you have a great many choices for middleware application servers. From a blue martini to an enhydra, there are standardized platforms you can utilize. If you want a software design methodology, you can shoot for something as simple as the Java Servlet Specification, and try to be J2EE compliant. The language gives you compile type checking, strictness, and enforces a software engineering methodology (OO) that will alow for large firms to be able to communicate and guarantee their software processes. that is of course not to say that java is a panacea... but an example... let us not look at this meditation as a religious war...

In a perfect world, I would be able to reach down into the pools of developers that exist, and pluck out any developer intelligent enough to read the pod of an object, interact the subtleties of a hash or a hashref... intelligent enough to go to CPAN, and use POD, and know that you CAN put whitespace in regexps to make them readable by a human. However, this is not a real world, and I am not so sure that the "average" developer, can be trusted with this much flexibility, in an enviroment where he is just working as a small piece of a very large puzzle... where he may not even see the entire picture.

So my meditation becomes... are there any "trully large" perl shops? Shops where 100k line programs, with thousands of objects, and thousands of interactors are written in perl? Are there application servers that provide a complete framework for me to utilize in perl? Are there software engineering practices that would make an 80 man software team using perl a viable, fruitful, and productive team... that was able to produce fault tolerant, secure, and powerful applications?

This language, my favorite language, has shown me so much power and beauty, yet I fear that I will have to desert it for a stricter world... show me the way, tell me how you have dealt with it... tell me how 100 person perl teams function, where the caveats are, and how I convince myself that this is the right way to go.

Thank you, and here's to hoping that we find a way.

[Nitsuj]

The right choice of language in any given project is essential.
If you don't feel that this project is manageble in perl, do it in a different language.
You shouldn't feel obligated to write your projects in perl merely because you enjoy it.
You should feel obligated to write your projects in perl because it makes it simpler.
Remember that in large projects, the planning, the engineering, the treatment of portions on code as objects( well, it makes it easier ), and the proper segmentation of code are important.
Treating your code in a strict and uniform way is important.
Everybody being on the same page at the macroscopic level is important.
While PERL is great for MANY things, other languages have their strengths.
Not that perl isn't great, but I can certainly think of languages that are better for the situation that you describe.
Just remember, that doing a project, not in perl, isn't the end of the world. Also, remember, that you can always do PARTS OF IT in perl.

Just Another Perl Hacker

[coreolyn]

I feel your pain

I wish that my openning line was funnier than it is but I find myself in a very similar position. I have been trying for several months to write a whitepaper for bringing perl into a large, closed, cathedralesque financial organization.

Every methodology I try to establish just seems to take the fun out of perl by taking the creativity and freedom out of the code itself. CPAN, version control, support, style conventions, security, All seem like round holes and all I have are square pegs to deal with.

At the Perl 4.0 Conference I made a great effort to find out the size and scope of the businesses that 'deploy' perl. The term 'deploy' confused 99.9% of everyone I asked and seemed to catch even merlyn by surprise.

My conclusion is that Enterprise perl does not exist and like my perl predecessors it's up to me, and other's like me to make it exist. It may not be as romantic as cleaning up garbage collection in perl 6, but I believe that it is even more important to the immediate growth of perl as an industry accepted language.

Most perl people have no clue that roadblocks exits. The dumbfounded and shaking heads I ran into in Monterey could have shifted tetonic plates when I explained that Perl was not an approved development language in our organization.

As for solutions the three primary things I see that need to be created are:

1. An internal CPAN module that logs and tracks module installation for version control.
2. Separating the system perl (especially in the case of Solaris) from the individual developers perl enviornments.
3. Creation of an internal perl support site where the same type of interaction that occurred in your small team can be shared and participated in by the entire organization.
4. Strict enforcement of some style conventions such as:
   1. perl -Tw (at least through the system test/readiness phases
   2. Embedded POD documentation
   3. Utilization of perl's internal VERSION control

I'm onto about my 30th crack at this whitepaper so if you come up with anything please share!

coreolyn Duct tape devotee.

[knight]

Excellent points. I think the Perl community does have its head in the sand about some of the realities facing acceptance of Perl in other computing environments, especially traditional IT or MIS departments. I know just enough about those places to know that it's a different world from the cozy UNIX environments I grew up in.

Separating the system perl... from the individual developers perl environments.

A lot of Open Source projects assume that either you have root access to the system yourself, or have really friendly administrators that will install whatever you want just for the asking. Neither is true for a lot of people out there.

I think it would help the more widespread acceptance of Perl (and other tools) if they supported real relocatable installation, beyond the ability to specify your own absolute path name at compilation time. The idea is for the tool to support a library and directory structure with relative paths, not absolute paths, so you can check in a verified copy of the tool and know that it's not pulling in any library or module from outside the source tree.

I haven't tried to concoct a version of Perl that does this, although I did try to create a similar version of GCC once. (It can be made to work, but requires a lot of really fragile symlinking and oddball configuration.) Perl allows you to rely on a specific tool version better than a lot of other systems by supporting side-by-side /usr/lib/perl5/$VERSION/ directory structures, but this still relies on coordinating with the sysadmins to install new versions and modules. I have this gut belief that using Perl in a traditional IT environment would be a lot simpler for all involved if it could be used without relying on installation outside of the source tree, but maybe I'm making this up.

Does any of this apply to the environment you're targetting?

[coreolyn]

Knight,

Your comments are on target for the type of enviornment(s) I'm working to deploy. In the 6 months that I've been pinging on this topic on the various mailing lists. This current thread seems to have recieved more understanding readers than anywhere else ( Yay PerlMonks.org! :)

I'm going to put together a new thread that points to the various links (sometime today) with a Title of "Perl In The Enterprise" In the hopes of bringing more attention to this issue, and getting as much information on what has been done, and should be done in this area.




coreolyn Duct tape devotee.

[Anonymous Monk]

There exists within Perl the ability to support such a development and deployment framework. I know, I built one for a Fortune 1000 (620's). It requires dedication to the process and framework, and most important of all, a toolmaster. The easy part is the coding standards surrounding use of the framework to implement override libraries. Then, your applications deploy with whatever versions of libraries you want to use. You can even tier the libraries for enterprise, platform (e.g. dev/stg/prd), and project. Regardless of the complexity of the framework, it must be trivially easy to embed within your applications. Otherwise, your programmers will find the path of least resistance, and following your framework won't be it. I'd be glad to post some code if desired.

[coreolyn]

I would be VERY interested to see any code, and hear any more comments you might have on this subject.

coreolyn Duct tape devotee.

[JanneVee]

This is the reason I have advocated COBOL as a Strict language(also why I'm considerd a heretic!). Not that I would want program in it. But I have pointed out in the wars COBOL vs. Perl that COBOL is designed for something specific! Large financial systems!

You could apply strong dictatorship on the perl files and say that they must pass a parser that checks the code for the proper things. A standards parser then it might work!

[pos]

You could apply strong dictatorship on the perl files and say that they must pass a parser that checks the code for the proper things. A standards parser then it might work!

This is exactly one of the things I was trying to get at with my last node. I would like to check a huge pile of code, spread across multiple dirs, users, etc... and format them all corectly, document them in pod, convert the pod into an html api listing, check for inconsistancies in the api ( check code calls to functions against the included module's qw ( api_function1 ) statements and finally, check the code into cvs.

It seems like that's quite a tall order.

We are also trying to scale perl up to a size that it probably doesn't naturally fit into. TMTOWTDI probably means that the way you decide to do it may be completely foreign to your fellow developers, any kind of standards checker/parser, or even yourself in a few months ;-) However, a small team of people fluent in conversational Perl will be able to write the code that a company needs faster than a fleet of "baby talkers". Trouble is that in this job market, all most companies can seem to hire/retain are the less fluent, translation dictionary toting, "just visiting for the money" computer science tourists. It's reassuring to hear that other people are in the same situation, but I don't hear that many success stories. That's quite discouraging.

I hope this post doesn't come off as a rant because I don't mean for it to be so... I would like to see problems like these affect a change in Perl rather than result in a lot of complaining and hot air. We are going to try to make it happen in our company... If we can't we'll be forced to move some code to other, stricter, languages

-pos

The truth is more important than the facts.
-Frank Lloyd Wright

[JanneVee]

I applied for a company that worked with a huge perl based website. They had developed a good standard to handle the scripts by running them through a "wizard" that put in the right constructs for the "shop standard". They had about 5 person working on the code

Now you're might be asking for what does this apply to your problem:

Trouble is that in this job market, all most companies can seem to hire/retain are the less fluent, translation dictionary toting, "just visiting for the money" computer science tourists. It's reassuring to hear that other people are in the same situation, but I don't hear that many success stories. That's quite discouraging.

They wouldn't hire me. The reason they gave me was that I was not quite right on their specification.

So they actually looked carefully on the testcode that I did on the interview and said to themselves: "This guy isn't going to follow our specification"!

So their are companies that actually try to avoid the "tourists" and people that aren't as adaptable to the standard that they apply! /JanneVee

[Nitsuj]

I have seen a lot in this thread, and indeed in general talk, about how perl should be extended, or managed, in order to make it a panacea for all that needs to be programmed.

Don't get me wrong... but all languages have their unique niche.

I love perl, perl is a GREAT language for the things that I do with it... but it's not the right programming language for every situation.

Perl stands for Practical Extraction and Report Language... not Perfectly Everything's Replacement Language, for every other language and system on the planet.

Yes, improve, work, make perl WONDERFUL! Just don't make it impossible and useless. You can only change it so much before losing sight of the fact that it is GREAT for certain things.

I still think that in certain situations, it is better to embrace the fact that a different language should be used, rather than forcing a nice square peg like perl, that fits wonderfully into nice square holes, into a round one.

This will just make the people who had THAT experience with perl as their first real experience not want to embrace it like we have.

Whatever you do with perl, whatever solution you come up with... make sure that it is the BEST solution to the problem that you can thing of. If you think that lisp is the solution to the problem, and you can code it in 20 lines in lisp, why make an issue of it and vow to use perl in this project. Instead, take what you have learned from the experience, and if it seems practical to extend perl to make such endevours practical under it, then come back, and give this back to perl.

This is the way that we make the language that we love the language that everybody loves, rather than the language that is cryptic and painful to learn.

Just Another Perl Backpacker

[tilly]

Perl was originally intended for parsing and printing formatted reports.

When was the last time you personally used formats? (merlyn can skip that question. :-)

Perl has changed niches before and undoubtably will in the future. Please don't lose sight of the fact that Perl was meant to evolve over time. As long as that happens within reason, it can do that and keep its flavour. Just as it has in the past.

[Nitsuj]

Yes, but change and evolution takes time.
Perl as we know it has evolved over many years.
And I do love the language.
Heck, if you want to change the package management and such of it, go right ahead.
What I am saying though, is don't use it for a purpose that it is not CURRENTLY designed for, that will only cause it's audience to shrink, rather than grow.

Just Another Perl Backpacker

[knight]

I have seen a lot in this thread...about how perl should be extended, or managed, in order to make it a panacea for all that needs to be programmed.

I don't see this thread as suggesting that perl is the answer to every programming problem. I do see people wanting to use perl to solve some programming problems in an enterprise computing environment, and finding that some aspects of how it's packaged and its culture has evolved throw up some barriers to acceptance in that environment.

Implicitly, you're suggesting that these barriers are intrinsic to perl and that, essentially, it isn't an appropriate language in the enterprise--that's simply not perl's niche. I think the point is that there are enterprise programming tasks for which perl would be the right language, if only people could satisfy some of the traditional MIS concerns about introducing any new tool to the environment: packaging, administration, engineering process...

The question isn't, "Can we extend perl's syntax and modules so that every enterprise RPG/Cobol/Lisp/whatever program can be rewritten in perl?" It's, "Can perl's surrounding infrastructure be cleaned up a bit so that an enterprise programmer can choose to use perl when it's appropriate without scaring management and the IT department?"

[Nitsuj]

Well, no, I'm saying that there are SOME tasks, but this situation doesn't sound like one of them. I generally don't look at perl as a language whose properties loan themselves well to such large groups. There are ways to do it. If you can separate EVERY task into a PM, and test those properly, then you have a fighting chance.

Just Another Perl Backpacker

[coreolyn]

I disagree with the attitude of not trying to expand perl's utilization beyond what it is currently being used for. It seems to me that if perl folk embrace that philosophy it will fade as a once popular 'fad' language. Finding new innovative ways to use perl is part of what the language is about.

Currently 'speed to market' is a huge key variable in web application development. Nothing beats perl for rapid prototyping. Additionaly I don't know what Larry put in perl that gave it it's 'people power', but this too is a quality that large MIS Departments can only strive for. I don't know any other language that is as FUN as perl, and the 'fun' factor is everything to anyone that works with code for a living.

What is frustrating is that the powers that be in the perl community could solve the problems of Perl in the enterprise simply by providing easier methods for perl's installation by decentralizing it's core distribution modules. In other words making the default installation according to the user's enviornment. While the mechanisms exitst for such installations, documentation is scarce, (References would be appreciated), and they are far from the default. MAKING PERL'S DEFAULT INSTALLATION CONTINGENT UPON THE INSTALLING USER'S ENVIORNMENT RESOLVES THE MAJORITY OF ISSUES SURROUNDING PERL IN LARGE ENTERPRISE ENVIORNMENTS. Furthermore, it makes ISP system administration, and standard system administration easier as well. It also eliminates the mess in /usr/local/lib that comes with utilizing the different perl rev's/distros

This appears to be a mostly trivial change for perl that could open huge doors for the language. The failure of perl people to see the benifits of coming through that door cause me to worry that perl is having a hard time getting past it's adolesence and taking it's well deserved place in large scale commercial application development. What if the original system administrators that exploited perl didn't extend it's use into the web because it was only good as a sysadmin tool?

coreolyn Duct tape devotee.

[Nitsuj]

I am not saying don't expand perl's utilization. What I am saying, is that if it ain't the right tool for the job, use the right tool, instead of banging on a screw with a hammer, get a damned screwdriver, don't try to use a nail, when a screw is what you were asked for. If you see something that can be improved in perl, improve perl. I can CERTAINLY see improving perl to fit a situational problem. What I can't see is using it in a project that you don't think can be accomplished with it, it just doesn't make sense.

Just Another Perl Backpacker

[Poetic Justice]

Aw man, I'm dealing with the same thing. I'm working on an application that will be deployed organization wide. There are only 2 Perl programmers here. One of which is myself, the other is a recent convert. I could do the app in Delphi, Visual Basic, Visual C++, Java, but I want to use my Perl and it's my call what gets used but I have to take other issues into consideration, code maintenance being first and foremost. I love my Perl and I'll be loath to use anything else at this point. The other perl programmer has just picked up the language about 3 weeks ago and he's becoming a fanatic. I guess I'll just have to set some guidelines for code maintenance and start the process of evangelizing and get some converts into the fold. Thanks for the post eduardo. Poetic Justice "Licenses.....We don't neeed no steeeekin' licenses....

[grackle]

It's hard to imagine any role that Perl couldn't fill, with the proper adjustment. But how many different people with how many different needs can "clean Perl up" before it becomes just another programming language? If you consult Programming Perl, you'll see that Perl was consciously designed as a language that would allow you to program in ways inappropriate "for complex problems demanding complex data structures." The idea was that Perl would intentionally violate rules known to be prudent, if not necessary, for large projects, thus allowing small jobs to be handled with unprecedented ease.

Currently, talented and careful programmers can do almost anything in Perl. That's not good enough for all situations; some projects require a language like Java, so that managers can coax usable code out of any idiot who can use a keyboard. (By the way, I love Java and use it every day :-)

Now managers are salivating over all the great code in CPAN and all the talented programmers who want to use Perl.

The Problem

<cite>I love my Perl and I'll be loath to use anything else at this point.</cite>

The Antidote

<cite>The right choice of language in any given project is essential. You shouldn't feel obligated to write your projects in perl merely because you enjoy it.</cite>

Let's leave it up to the people who understand Perl the best -- the developers -- to decide whose demands can be met without turning Perl into one of those "industrial-strength languages" that "make it equally difficult to do almost everything."

Will we ever throw another script away?
Flirt gently for an evening;
Recall it on a summer's day?
Will we ever throw another script away?

[jcwren]

It's hard to imagine any role that Perl couldn't fill, with the proper adjustment.

(Okay, I'm aware I'm taking this one statement a little out of context. But this is a pet peeve of mine for people who see Perl as a hammer, and everything else as nails.)

Actually, it's easy to imagine dozens of roles that Perl is completely unsuitable for. It will never be found in automotive embedded systems (engine/transmission computers), life support applications, microwave ovens, cellular phones, or any other application where the hardware cost is driven by volume, or certain types of compliances must be met.

There are a couple of factors here: One is that the lowest hardware cost is almost always achieved by using a minimal hardware configuration. Unless you're willing to use a 386 or better processor, and a corresponding environment that will support a real O/S (read 4+ MB memory, 2+ MB ROM), this is rarely co-incident with lowest cost. Perl is an application that requires an underlying operating system to be effective. People may port it to the Palm Pilot, but one has to ask oneself "Is this *really* useful?". Most of the time, such exercises are for the demonstration of "coolness", not practicality. This is evidenced by the fact that Linux and it's cousins still aren't practical on the hardware used by most PDAs. And this is also why MS has a market for WinCE (or whatever they call it now), and Palms are running Palm O/S.

Considering that minimum hardware cost for a 386, 4MB of RAM, 2MB ROM, supporting circuitry, yada yada yada is still about $50 in large volumes, are you going to pay that surcharge so your microwave oven can have a "Linux/Perl Inside" sticker? I sure won't. And neither will the normal consumer.

Perl will never be suitable for writing kernel mode device drivers. I'm sure someone will do it, just to show that it can be done, but it's not the correct model for driver development. Remember that Perl is still written in 'C'. There's a good reason for this. It's because 'C' is still the right tool for writing as close to assembly as possible, and maintaining portability. Possibly this might change when Perl-to-EXE actually works, but I seriously doubt it.

Another factor is that Perl will (in all probability) never be certified for military and life support programming, such as Ada is. There are a number of reasons for this, ranging from the requirements for strong type checking of variables and arguments, (current) lack of ANSI certification to compile time evaluation (How many times have you not realized you misspelled a subroutine name until it coughs a hairball at runtime?) and practical automated CASE tools.

I've said this before, I'll say it again: Perl is a hell of a language. It's very full featured, it has features that few others languages have, it's well accepted, etc. But, remember this: One of the things that makes Perl so useful is CPAN. Without the people that have written these modules and made them publically available (GPL, Copyleft, whatever), Perl would be seriously lacking. How'd you like to lay out $100 for CGI.pm, under the typical licensing schemes for most "DOS/Windows" libraries? I'm pretty sure it wouldn't be anywhere as popular as it is now. One of the major reasons (I believe) that Perl is so widely accepted is because of the underlying distribution model.

On a slightly different note, one little thing that bothers me about Perl is the lack of structures such as 'C' supports. I know that structures can be emulated, but the ability that 'C' has of mapping a block of memory to a structure, accessing the individual elements with ease, then stepping a pointer to the next block is missing. This makes it much more difficult to implement access to memory mapped devices, and other objects that can't be conveinently remapped into Perl idioms. I'm sure someone will immediately correct me on this (probably tilly...), but that's my take on it as a long time 'C'/assembly/Forth programmer.

Good programmers know when to use the right tools for the right job. These decisions are, of course, mediated by management (always runaway when management endorses a methodology. Nothing like an edict "all future code will be object-oriented" from people who don't even know how to reboot their own computers), development time & time to market, maintainability, and half a dozen other factors. Know when to hack Perl, when to sling 'C', and when to hand-craft assembly. Just because you know a way to make it *work* in Perl doesn't mean that Perl is the answer.

--Chris

e-mail jcwren

[tilly]

My home node says my beliefs about criticism, and this rant just went into the list of links I keep there. Perl is not and never should be all things to all people. Indeed anything that sincerely tries will fail. And some of your points I have said before and will again, for instance in Re: FS OS sysprog I pointed out that Perl is bad for kernel programming.

I do disagree on some things though. Linux is not actually a bad fit for a lot of the embedded market. Sure, at the very bottom end it doesn't make sense. But betting against it is IMO betting against Moore, and Moore has a few years yet to run. Plus as soon as you want an embedded device that is able to network using standard protocols, you won't wind up with requirements much below Linux' and guess which is less work?

As for memory mapped devices, it is possible in theory but the same hiding of details that makes Perl so easy to develop in makes it hard. OTOH take a look at IPC::Shareable and you see that Perl can be taught to be careful with what it does after all.

There is just one point I would like to wind up with. And that is that there is a very good reason that Perl doesn't work like a lot of other languages. Most people who think about maintainability start with the idea that they are going to keep a large project under control. In saying that they have already lost sight of the fact that the act of keeping something short and sweet makes it easier to maintain. Perl is a master of that school of design.

In order to be that, Perl goes out of its way to be expressive, emulates a way of thinking that people find natural, concentrates on simplicity of interfaces to a relatively small number of important concepts, and in general finds reason to break most of the classical CS rules. But in the end it works! The classical CS people are right that you wouldn't want to maintain many Perl programs with 30K lines. OTOH what takes 30K in C will take an order of magnitude less in Perl - with better debugging information yet!

Not to mention fewer buffer overflows...