Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Perl vs Java in Heavyweight Filesystem Processing

by higle (Chaplain)
on May 22, 2003 at 21:59 UTC ( #260245=perlmeditation: print w/ replies, xml ) Need Help??

I have the privilege of working with a development department that is married, with children, to Java. Somehow I've managed to squeak a couple of applications written in Perl through, but now I've come to a point that I've expectantly dreaded: having to justify, in writing, the use of a Perl CGI in lieu of a JSP or Servlet for a particular project.

Here's quick and dirty background on the project, our environment, and why I feel I should use Perl instead of Java --

Project:
The requirements for my part of the project are asking for an Intranet-based "GUI" that will essentially list the contents of a particularly large subset of filesystem directories, parse the file names and parts of the files into organized chunks of information, show this to the user, and allow them to manually flag files for use in another application (short description: lots of filesystem interaction and text parsing with a little bit of database work, tied to an HTML front end. Not so much a GUI, but more like a text crunching, file slurping monster with a pretty face).

Environment:
We're using the iPlanet HTTP server, which of course doesn't support FastCGI (or anything similar, that I'm aware). This is the main argument that our architecture governance committee hangs on whenever CGI programming is brought up, saying that the resource overhead of starting CGI processes is too inefficient (on two load-balanced, 4 processor machines which run 98% idle most of the time). To top it off, only 1 person at a time will be using this particular application at any given time, making that argument irrelevant anyway.

Perl instead of Java:
Whereas I seem to have at least some ability to weigh practicality over marketing spiel, my colleagues regularly regurgitate the Java RhetoricTM that encourages developers to write bloated and ultimately inefficient software. I agree that, in our environment, JSP is a fine technology for writing database-driven applications used by thousands of employees, compared to CGI. But in this case, Java is simply the wrong tool for the job.

For those unfamiliar with Java's java.io.File package, consider yourselves lucky. In comparison to Perl's File modules, Java's API is clumsy, slow, and incomplete. The text processing piece is a no brainer; clearly Perl has the upper hand.

So why am I posting this Meditation?
Because, other than the generalities that I've listed above, I don't have what I believe is a case that can sway the argument in my favor. I have no extensive hard and fast data dealing with this very specific comparison/contrast of Perl & Java. I'm starting from a disadvantaged position; I fear that if I push the issue any further without proving my stance, I'll be digging my own grave, or at best I'll look like a complete jerk. This is my conundrum.

I don't have time to do benchmark testing with both languages, and I haven't been successful in finding benchmarks on the Web that deal with these specific issues. So I pretty much have to wing it, if I choose to follow through this time. On the project level, it really doesn't matter a whole lot if we end up using Java and have a slow application as a result. But in the big picture, I will have effectively been assimilated, Perl will lose more face in our organization, and I will be trifectively pissed off (not to mention that I'll be known as the guy who wrote the slow program).

Should I just give in? Is my hubris leading me down the path to Disaster? I don't know. I just don't know.

  higle

Comment on Perl vs Java in Heavyweight Filesystem Processing
Select or Download Code
Re: Perl vs Java in Heavyweight Filesystem Processing
by perrin (Chancellor) on May 22, 2003 at 22:15 UTC
    There are some benchmarks of Perl and Java for reading files on this site. The main things they will tell you is that Perl is plenty fast, and it takes tons more code to do a decent job of it in Java. You already said that the performance of the application is irrelevant in this case, so it comes down to the cost of development and maintenance. If you can build and maintain this app twice as fast in Perl as in Java, you will save your company money. It's that simple.

      You already said that the performance of the application is irrelevant in this case, so it comes down to the cost of development and maintenance.

      From the sound of it, it seems that while the OP realizes this, his PHBs may care deeply about performance. Perl is at a natural disadvantage in this case, since it sounds like their running a persistant environment for Java, but not for Perl.

      ----
      I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
      -- Schemer

      Note: All code is untested, unless otherwise stated

        Actually, I suspect Perl will mop the floor with Java in this case, unless someone goes to great lengths to use the new direct io classes in the latest Java release. This is not a case where a small script is being launched over and over (which would favor a persistent environment over CGI), but rather a very heavy script that is run infrequently. In that situation, the actual file-processing speed is much more important than the startup time.
Re: Perl vs Java in Heavyweight Filesystem Processing
by Abigail-II (Bishop) on May 22, 2003 at 22:15 UTC
    Of what you are posting, I only find one thing that might be relevant in your quest. You say that Java is the wrong tool for the job. Whether that is true or not, I cannot judge from your posting, but that, IMO, is the only thing you have you can use as a pivot point.

    What you have against you is that your platform (iPlanet) isn't geared towards Perl, but to Java, that you don't have benchmarks (which would only be useful if the Java solution was slow anyway), and that you seem to be the only person in your team wanting (or perhaps able) to program in Perl.

    If I were a manager, I'd weight the latter very heavily.

    Abigail

Re: Perl vs Java in Heavyweight Filesystem Processing
by chromatic (Archbishop) on May 22, 2003 at 22:32 UTC

    What would happen if you were to write a mockup in Perl and let people play with it for a few days? It'd likely be faster to develop than a mockup in Java. Having something concrete and working would give everyone a feel for how the technology works.

      Yes, and if the mockup is reasonably complete, the bosses may ask how long it will take to finish. This has happened to me more than once and my 'proof of concept' CGI scripts are still running in production :)

Re: Perl vs Java in Heavyweight Filesystem Processing
by submersible_toaster (Chaplain) on May 23, 2003 at 03:25 UTC

    While I don't have quite the same case to relate, let me tell you about a daemon I wrote once. I worked with a company that used a java , web-faced system to track big lumps of data around the building (with versioning and various bells). This system become interactive enough that users could create a work order to copy chunk-of-data-BLAH from someplace to someOTHERplace.

    I was tasked to write the poor little slave process to do the work orders. My initial try was with perl , it was quick, smart and IMO very cool. But it was deemed that rather than a CGI perl service, this was to be done using Tomcat instead. Not knowing java , I ported the worker to python, then jythonized it to run with Tomcat.

    The developer descends on me, howling about the slowness of this new service. Files were taking 10 times longer to transfer , no network or disk IO bottlenecks could be found. I went straight for the .py file , changed some shutil.copyfile()'s with actual system calls to cp and magically everything went a LOT faster , but still not as quick as perl.

    I doubt this gives you much ammunition, but it's a Java vs perl story, where java won (but sucked) and perl lost (but got the job done). I am suspicious of other things pushing the bias toward java , certainly it was seen as more acceptable to use a python/java hack than a clever perl CGI. Why? I just dont know.


    I can't believe it's not psellchecked
      When you were hired, did they require Java skills? I'd have fired you for jythonization

        • I was never hired as a programmer
        • My employ pre-dated the use of java for internal services
        • I programmed in perl to avoid sysadmin fear and loathing
        • Tomcat was not my idea.
        • The perl solution solved the problem.
        • The perl solution did not create a problem
        • You'd fire me for jythonising? Like a care

        I can't believe it's not psellchecked
      This is completely irrelevant as you didn't program anything in Java, you ported a perl script to a python script, and run the python script via a Python interpreter implemented in Java. That harldy qualifies a Java "program".

        Correct - I programmed nothing in java, but ultimately the python ran in a python interpreter, implemented in java. So what java classes does the j-python interpreter call for file IO?

        For those unfamiliar with Java's java.io.File package, consider yourselves lucky. In comparison to Perl's File modules, Java's API is clumsy, slow, and incomplete. The text processing piece is a no brainer; clearly Perl has the upper hand.

        Irrelevant ? I can't really say, I don't program in java.


        hunting /. grammar nazis since when?
Re: Perl vs Java in Heavyweight Filesystem Processing
by hardburn (Abbot) on May 23, 2003 at 14:05 UTC

    Present a challenge to your superiors. Two teams, one writing the Java version, the other writing the Perl version. If the people on the Perl team are decent with the langague, they'll have the Perl version done while the Java people are still hagaling over the design.

    The Perl team could probably be as small as one person. The success of Perl in this regard will depend heavily on that person's familiartiy with Perl and CPAN (which is true for any Perl project).

    Your description of the actual program was a bit vague, but it sounds like something that I could whip up in an afternoon, including debugging. There might be details I'm not aware of, though.

    Your superiors will no doubt think you're setting yourself up for disaster by having a team of only yourself against the company's Best and Brightest Java programmers. Again, the details of the application matter a lot here, but if you have more than passing familiarity with Perl and CPAN, you'll likely have your project done in at least 1/10th the time of the Java team.

    ----
    I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
    -- Schemer

    Note: All code is untested, unless otherwise stated

Re: Perl vs Java in Heavyweight Filesystem Processing
by adamk (Chaplain) on May 23, 2003 at 15:33 UTC
    I've worked in a company where we had to do time budgets for jobs. Our general rule of thumb was writing typical code, from scratch, in perl was about 2.5 times more efficient. That was normal code, and basic HTML fluff.

    For every major API you have to interact with, CPAN vs Java stuff, add another factor of 2.

    So for the typical example of working with 1. A database API, and 2. Filesystem search/modify/replace, you get 2.5 x 2 x 2 == about 10 times faster. A Java guru I work with agrees with these figures. I've written automation systems in about a month, and then waited 18 months for an ( admittedly better ) Java version to replace it.

    With respect to CGI, remember the process creation/teardown is only relavent above about 1-2 hits per second per CPU. Below that rate, it hardly matters.

    Also, CGI applications have zero resource usage when not in use. I'm not a Java person, but from memory most of the speed from Java comes from having a permanently running VM, kinda like mod_perl. I used to work for Cisco, and on their big main worldwide E10000 intranet, they had thousands of perl scripts. They could do this because each script wasn't using up permanent resources.

    Cisco later switched to Java corporation wide, with "ALL NEW APPLICATIONS IN JAVA!". They filled up a dedicated E10000 in less than 2 months just from NEW apps, because they could no longer add all the VMs required by all these apps.

    Hope some of this helps

Re: Perl vs Java in Heavyweight Filesystem Processing
by defyance (Curate) on May 23, 2003 at 16:14 UTC
    hig, As a fellow employee, I feel compelled to say something here.

    In recent years our company have always been open for ideas and suggestions. We were a smaller company, and trying to find our niche. Hell, they even let me write a few scripts and things for them, remember? Now that we have gone "metro", things have really changed, I'm sure you've noticed. I'd have to say that due to the recent acquisitions and our growth spurt, uniformity is key for organization. Basically, it looks like they found a niche in Java. I'd say that right now it would be in your best interest to lay low until things get settled. This is just a suggestion, but I'm sure you've noticed tensions have been high in our building recently. I'm not sure if your going to be doing this for the PHB in Corporate, but I'm sure that things are changing up there as well..

    So, in short, I'd just lay low.

    I could be really, really wrong here, but in nearly 5 years with our company, I've tried to pick up on how things work around here...

    Peace, Love, and Chicken Grease.

    --
    A conclusion is simply the place where someone got tired of thinking.

Re: Perl vs Java in Heavyweight Filesystem Processing
by petesmiley (Friar) on May 23, 2003 at 16:49 UTC
    The thing that truly revolts me about this question (++) is that you could probably write the damn thing in C (how's that for speed) almost as fast as you could write it in Jyva. The benefits of using Jyva probably wouldn't come in to play much with a project like this, especially if you are on this project by yourself. The whole world seems to think that you have to kill knats with a sledge hammer that's painful to lift.

    I vote for the "write a test version in perl". It seems like you could write something in no time flat. If you want perl to work for you, you have to work for perl. Work late one day if you can't get it done during your regular work schedule. Then tell them how long it took. If you're worried about there being repercussions, just say it was a working model for the java version ;) Remember, it's better to ask for forgiveness than to ask for permission.

    Or if you think you're going down the toilet anyway, just write it in perl, hand it to them and say nothing. See how long it takes them to notice the difference ;) (just kidding)

    smiles
    (thankfully I no longer program in C for most things)

    PS I have never worried about there being other perl programmers around to maintain my work. Currently I'm the only perl programmer where I work. Like most work places, mine likes to make me feel like I'm easily replaceable. So I act like I am ;) Oddly enough, most of my programs have never had problems, and usually they are retired before they ever do (the oldest running program I have is 3 years old). All because I have the time to test my programs because I'm working in perl.

      Currently I'm the only perl programmer where I work.
      /me tries not to take offence to that comment.

      :)

      -silent11
        Hehe, let me rephrase. I'm the only perl programmer that's been hired to program. There are other slackers where I work that are perl programmers.

        smiles

Re: Perl vs Java in Heavyweight Filesystem Processing
by linux454 (Pilgrim) on May 23, 2003 at 18:07 UTC
    I've been programming in perl for quite some time, and I've picked Java up in the past several years. And I've found that I love both languages equally, but for different things. Perl can do any job that Java can do, somethings it does better and somethings it doesn't and vice versa. The way I see your situation is that perl may be the more efficient tool, but that does not necessarily mean that it is the best tool for the job. Meaning that if you are the only developer who knows perl, in an all java environment, then what happens to that little sliver of perl in the sea of java if you kick the bucket, are fired, or when you move on to bigger and better things? Well it will either have to be replaced or someone will have to learn perl. Both are in my opinion unacceptable from a business perspective. On the other side of things, it's sounds like you aren't going to have to process file contents just file meta-data (filename, size, etc.). If that is indeed the case then there should really be no argument java is not going to be that much slower (if any) when performing those types of operations. If however, you will have to operate on a large amount of file contents (i.e. parse text from inside the file etc.) then I would hate to use java, perl is the king of file slurping. But if speed is not of paramount concern, I say just suck it up, take one for the team, and right the thing in Java. Because java really is a great language if you give it a chance. ((This whole post is written by a converted perl bigot))

      If Perl allows the employees to finish their tasks sooner and better, then it's good in the business perspective if a few more are forced to learn it.

      Jenda
      Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
         -- Rick Osborne

      Edit by castaway: Closed small tag in signature

Re: Perl vs Java in Heavyweight Filesystem Processing
by higle (Chaplain) on May 23, 2003 at 18:12 UTC
    Thank you, everyone, for your comments, suggestions, and/or support.

    I've been mulling this over for quite some time; this isn't the first time that I've gotten some flak for using or wanting to use a "scripting" language in an enterprise application. I suppose that my problem really consists of two things:
    • My coworkers are more concerned with the "latest and greatest industry standard" rather than leveraging the effectiveness of an existing tool or platform. To them Perl is old news, only good for writing throwaway scripts. Java is Big Business, and we've invested (wasted?) Big Money in our existing platform, which changes every year, continues to bloat, and eats our budget like candy corn.
    • I've never stood my ground, pointed out the wasteful spending, or generally argued this point to anyone in our company who's in a place to do something about it until now.

    What I'm actually fighting is a mindset. It's not that they don't know how or can't learn to code in Perl; they're just the kind of people who'll buy the first thing the vendor drops on the table and insist we use it, for better or worse.

    I've decided to press the issue further, perhaps using subversion ("Oh, uh, I've already written a prototype in Perl... is that OK?"), or perhaps going the business route and making a dollars & cents case, or maybe both.

    It matters to me on a somewhat personal level, not because I'm a proud Perl zealot or Java basher, but because I've recently agreed to move for the company and I've signed a contract obligating me for the next year. I personally don't want to live with and maintain crappy, troublesome software, de facto, for the next year.

    Is this pursuit dangerous? Maybe. Is it worth it? Maybe.

    Only those who will risk going too far can possibly find out how far one can go. -- T.S. Elliot

      higle

    Can I crash at your place if this backfires? -- Higle

      Just arrived at this node via "Selected Best Nodes" and I wonder ... how did this end? If you are still around I think it would be interesting to hear your further adventures in Javaungle.

      Jenda
      Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
         -- Rick Osborne

      Edit by castaway: Closed small tag in signature

Re: Perl vs Java in Heavyweight Filesystem Processing
by Anonymous Monk on May 14, 2014 at 10:29 UTC
    It is nice to see how the collective brilliance of human minds combined with the test of time ultimately reveals answers to some of the questions posed in this post and its comments. Generally, there exists an unfortunate trait of programmers to get stuck in their ways. If someone has learnt and specialized in older languages such as Perl ('87) and newer languages such as Java ('95) arrive on the scene, no matter what the benefits, there will generally be a distorted view from the older programmer in favor of what they already know so well. As we have all seen, Java has exploded and with good reason. Scripting of old is dying. Of course there are scenarios in which the scripting is considerably more appropriate. I had to make a comment here as it was so interesting to read this article and its comments, the last of which was made in 2004, with the advantage of time passed. I realize this might cause flame responses, particularly as this comment is posted on a website for perl. But if we put emotions and pride aside, and reflect with rational reasoning and logic there is much insight to be gained from this article.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://260245]
Approved by ChemBoy
Front-paged by Louis_Wu
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (9)
As of 2014-09-15 09:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (145 votes), past polls