How we use Perl.

[osunderdog]

Intro

Here's a description of at least one of the ways we are using perl at the company I work for. Hope it's an interesting read.

Description

A few years ago I lead a group of developers that wrote an event dispatch mechanism in C++ using ACE. It works on Windows, Solaris, and Linux. This server acts as central point for message received on various different transports: HTTP, TCP, UDP, Serial, 3rd Party IPC, etc. For each transport, messages can be received in various different grammars: HTML, XML, Binary, Label-Data pairs, in-house, etc. The goal was to produce a server framework that would be capable of understanding the combinations of transports and syntaxes. We also wanted the customer (server developers) be able to plug and play transports and grammars as they saw fit.

What do we do with all these messages? As each message is received it is sent through a dispatching mechanism that maps the message to a function that is written to handle that message. Handlers have a simple signature, they receive a single parameter that is the message that triggered them. Handlers can be written in a C/C++ library that is linked in with the server. Alternatively, the server can create an instance of the Perl interpreter and Perl handlers can be loaded into the instance at startup. The server will invoke handlers that are registered, Once handlers are registered with the dispatcher, be they C/C++ or Perl handlers, the server will invoke the handler that is registered for a message based on some key data in the message. For example, a bridge server that receives a binary message on a serial port tweaks the data and retransmits it over UDP in an XML syntax.

For the Perl case, the state of the server is held in a Perl instance that can be loaded at runtime. A global variable can be declared to keep the state of the server. A handler can assign into that variable and another handler can access that variable. Sounds simple enough.. For example, a connected database (DBI of course!) handle can be stored in the global hash and handlers can access that db handle to perform database operations. Having a connection all logged in and ready to go can save tons of time between handler calls. I know this is a problem with typical CGI servers where the connection has to be re-established each time a particular URL is accessed. (Although I haven't looked, I believe this is one of the advantages of mod_perl.)

Handlers executed using a 'run to completion' model. This means that only one handler can run at a time. Any messages that are received while a handler is running are queued and handled in the order that they were received. 'Run To Completion' (RTC) has been a much debated features of the server. The primary advantage of a RTC model is that each handler can assume that no other conflicting handlers will fire during its execution. Since only one handler is fired at a time, there is no confusion about the potential sequence of events that will occur. The second significant advantage of the RTC model is that debugging can be performed on any handler and messages received during a debugging session will be queued by the server until the debugger releases control. On the other hand, A handler that has a considerable amount of code or calls synchronous APIs will result in a slow running handler. Slow handlers block the execution of other, possibly higher priority, handlers. It is possible for a single server instance to become bogged-down and unresponsive if the queue of pending messages is considerable and each handler takes a while to complete.

On the topic of debugging, we are able to pass parameters to the Perl instance on server start up and enable the perl debugging mode (Equivalent to setting -d on the command line.) Combine this with a $DB::single=1 in the handler you're trying to debug and Voila! The Perl debugger prompt shows up and you can step through a handler inspecting all the nitty gritties. When you have finished debugging a handler, just continue and the server will run through the queued up messages.

We write quite a few servers and this tool has enabled us to write them much faster than before. We have experienced a few Socio/Political problems along the way. 1) Quality in-house Perl Packages, 2) Packages used in long running servers, & 3) Spooked C++ developers.

1. Quality in-house Perl packages

It's very difficult to get reliable, high quality Perl Packages from developers who just think that Perl is a just a good way to search through log files. The documentation for XS is pretty good, however you really have to immerse yourself in it for a while to become proficient. I also have the distinct impression that they would rather be working on other 'Shiny' new languages and frameworks (C# & .Net).

2. Packages used in long running servers

Not to name any names, but a we have had memory allocation issues with a particular DBD package. The memory leak was slow and it only appeared when disconnecting from data sources. It seems as though most people using Perl and testing Perl packages don't have them running continuously 24x7. We reported the error and supplied information to get it fixed, but there wasn't much interest in fixing it because the suggested remedy was to restart the app. (Which we could only do at specific times.)

3. Spooked C++ developers

It's amazing how annoyed C++ developers get when they realize that their turf has been invaded by Perl upstarts. Sure they might be able to write a faster server, but it'll take 'em twice as long to do it. (Perhaps 3 times as long if you count memory allocation issues.)

Conclusions

Overall we have had great success with Perl for developing 'soft realtime' servers. The development has been considerably faster than I have seen in the past developing C++ servers. We have gained a considerable bit of functionalty from using the Packages that are available in CPAN. All Hail Larry Wall!

[pg]

"It's very difficult to get reliable, high quality Perl Packages from developers who just think that Perl is a just a good way to search through log files."

Although Perl is a good tool to search through log files, mainly because its strong regexp support, Perl can obviously do much much more than that. For whoever thinks Perl is only good for that, there is not much need to reason with them.

However, on the other hand, it is definitely a valid thought, for some people not to use Perl as a language for full-scale applications, but only use it to develop various tools to ease their life. Perl is handy and support rapid development very well. This is just one school of thought, however it is a valid one, and lots of people who belong to this school are capable of creating quantity Perl code and packages ;-)

[osunderdog]

The Quality in-house Perl packages item is related to groups in our organization that are tasked with providing Perl packages for existing in-house libraries. Most of them are very familiar with C/C++ and have used Perl for simple things.

They get wrapped around the axle when converting C/C++ apis that follow the memory allocation axiom:

He who allocates memory shall deallocate memory.

For those that have had this axiom burned into their brain, it is very difficult to pass back a reference to something in any language that they have created in the function.

We fear change. --Garth

[mpeppler]

Not to name any names, but a we have had memory allocation issues with a particular DBD package. The memory leak was slow and it only appeared when disconnecting from data sources.

I'm probably not the target of that remark, but I know that there is a leak in DBD::Sybase that is related to closing and re-opening connections. Finding and fixing these sorts of leaks can be tricky (in my case for example valgrind doesn't report the leak), and I've again spent some time trying to find it. I've discovered one possible cause that is buried in the Sybase libraries, though a Sybase engineer that I discussed this with didn't seem to think that there was a leak there - so maybe this "leak" is due to memory fragmentation.

Anyway - this may not be related to your situation, and maybe the fix for the driver you are using is comparatively simple - but I still wanted to point out that fixing memory allocation issues in things like perl isn't always simple...

Michael

[osunderdog]

Not at all! DBI is a great package and DBD::Sybase works well. It's safe to say we wouldn't have been able to get this far without DBD::Sybase.

Also, I know that the connect/disconnect leak is a bugger. I tried to find it myself. I'm pretty sure that it is in on the sybase api side. But Sybase isn't interested in fixing it -- probably because they are fighting to stay in business.

I'm just trying to point out some of the pitfalls of running Perl for a long time. Test packages you intend to use over a long time and benchmark Memory consumption.

[mpeppler]

Actually from my testing the leak is in allocating/deallocating the CS_LOCALE structure. If you run a connect/disconnect loop written in C that doesn't allocate a CS_LOCALE struct then the memory usage is dead even for thousands of iterations. If you add the CS_LOCALE alloc/dealloc the memory grows.

I may decide to open a tech support case on this issue to see if I can get this escalated.

Michael

[sfink]

Well, there are at least three different languages called "Perl" out there:
• A fairly robust application development language, with some clunky OO syntax, excessively verbose and unsafe custom data structure access ($o->{games}{$GAME}{$player}{health}), and a few too many special cases.
• A very robust scripting language for utility programs, with somewhat more tendency than shell scripting to backslide into monstrous hairballs of unmaintainable code (probably as a result of the vastly greater functionality in the same amount of code)
• A line item on a resume, used by many contractors to qualify for a job. This is the language where you can poke repeatedly at a large script that you know practically nothing about. Just keep rerunning it until it more or less gains whatever functionality was requested, and doesn't seem to gain any serious problems -- at least, none that will be noticeable before you leave.
If you use all three of those languages, then you'll come to realize that Perl fully deserves just about everything people say about it. Even the mutually contradictory things.

Oh, and if you feel that you're above using #3, try setting up a tinderbox. (Or insert some other hairball. There seem to be a number of them -- they provide amazingly useful functionality, but are a severe test of your gag reflex. You may be sorely tempted to rewrite them, but it won't be worth it.)

Personally, I use all three constantly.

[sweetblood]

Not to name any names, but a we have had memory allocation issues with a particular DBD package. The memory leak was slow and it only appeared when disconnecting from data sources.

I for one wish you would name names. One of the advantages of The Monastery is that we all may benifit from each other experiences with available modules, good or bad. It certainly may benifit others if they knew of a possible memory leak.

You could save someone a lot of debugging effort.

[rcaputo]

You may want to look at POE if you find you're writing servers with message handlers written mostly in Perl. Of course this may spook your C++ programmers even more. :)

-- Rocco Caputo - http://poe.perl.org/