[RESOLVED*] Logging run-time warnings from an embedded perl interpreter

[Hercynium]

I've got a doozy of an application I've been working with, and it's heart and soul revolves around it's ability to evaluate Perl code which is written my the user (me) and then executed within an embedded interpreter.

I don't want to plug their product, and I'll try not to unnecessarily berate it either, but it may help my fellow monks to know that it is OpenService NerveCenter, formerly sold by VERITAS. (I'll take no offense at all if the mods wish me to delete this paragraph)

I believe the code it evaluates is (usually) done so as if they were pieces of a long-running program - data in package scoped variables is available to all code after declaration, while lexically-scoped (my) variables *seem* to disappear after a chunk of code is evaluated.

I dare not use strict or warnings, because there appears to be a lot going on behind the scenes and I have no clue how it may affect things.

The problem I am facing is in trying to debug this beast. I'll spare the saga of what brings me here instead of the vendor's tech support, but suffice to say something makes this thing unstable as all hell and I want to know if there's anything in my Perl code that may be doing it, especially something I can work around. Being able to see run-time errors in a log file may be the key.

I can has halp plz? :)

Update:

At the recommendation of sundialsvc4, I'm trying to explain the 'saga' in another node below, posted anonymously... it's long winded and mostly irrelevant, but some monks may be interested...

Update 2:

Thanks for all your help. Because of the advice in your responses, I was able to get some info out of this beast at the time of a malfunction. There's still no evidence it's my code, but that may be a good thing! The asterisk on RESOLVED is simply because I'm still open to learning new techniques for debugging this application :)

[gamache]

I dare not use strict or warnings, because there appears to be a lot going on behind the scenes and I have no clue how it may affect things.

warnings and strict are very useful for precisely this situation; they will let you know if weird shit is happening behind the scenes. There also exists use diagnostics; for even more verbosity. I strongly recommend you enable warnings and strict checking, and see if something pops out at you.

In general, if warnings and/or strict break things, things were already broken to begin with.

[Hercynium]

I just put strict and warnings into my code and now I remember why I don't do that with this software :)

When a user-specified block of code is evaluated, some variables are already created in it's scope. If you use those variables in your code and then enable strict and warnings the application won't save your code to run later.

I side-stepped the warnings with the ever-so-icky my $pre_set_var if 0; construct (which I learned about here just a few weeks ago!)

Anyhow, I've enabled strict and warnings and even diagnostics, but I've still no clue how to get the stuff those spit out on STDERR into a log file for an embedded interpreter!

[Corion]

I haven't had to do that, but maybe the following works for you?

BEGIN {
    close STDERR;
    my $logfile = '/some/hardcoded/path/where/to_put_your.log';
    open STDERR, ">", $logfile
        or die "Couldn't create logfile: $!";
};

Put that code somewhere where it gets run as early as possible (for user code) - maybe you can even specify modules to be loaded when you start the program?

[Anonymous Monk]

As far as I know "use strict" and "use warnings" use warn/die to "spit" out messages. I have use "$SIG{__WARN__}" and "$SIG{__DIE__}" to catch/log them.

[jbert]

That sounds as though it might be the problem?

If you've got variable name clashes with the app itself, then there's a reasonable chance you're hurting its internals.

Unless such names are documented ways of interacting with the app, it seems like poor design. I would have thought they could have scoped the names into packages and had a limited/no lexical scope over your code.

The my $x if 0; construct is a bad (and unreliable in that's its behaviour may change in different versions of perl) way of generating what C programmers know as a static variable, i.e. one which maintains it's value across invocations of the function. This may not be what you're expecting, I don't know.

[Hercynium]

I hope this node becomes useful for others who are struggling with the vagaries of NerveCenter.

Something else I found that looks highly interesting...

Look inside this file: /opt/OSInc/bin/ncperl.pl

I can't paste the code here 'cause I don't own it, but as I read through it some things are beginning to make more sense now. :)

Update 2007.12.04: All the problems I've been having have one thing in common - RedHat EL 4. It's not RedHat's fault, but some experimenting has revealed that NerveCenter runs like a champ on RHEL 3 and Windows 2003. The version changes in glibc, kernel, and (I think) compiler between RHEL 3 and 4 are substantial, and the app probably uses some cpde specific to one of those.

To be fair, the folks over at OpenService are probably just used to how apps tend to simply keep working from one version of Solaris to another. The same often goes for Win32. They're new to supporting Linux and, while it's no excuse, helps me to make sense out of it all.

BTW, I'd like to make one BIG apology. Their engineers are anything *but* clueless. That was my frustration talking. I've been in their shoes before, and though it could have been handled better, they were truly working to find a *solution*. I even got some face-time with some of the senior tech folk... maybe I can get them to support to Debian or Ubuntu... *devilish grin*

[Hercynium]

One more thing, unrelated to Perl, but pertinent to those of us using this application - I've been having these issues with the app running on x86 RedHat Enterprise Linux 4 AS (RHEL4 AS). Before this upgrade we were running a much older version on Solaris (SPARC) with relatively few problems, at least not *nearly* as bad as we're experiencing now.

Last week I teased out of their engineers that their Linux development is being done on RHEL 3. G-d knows why, but it is what it is. Linux support was only added a few years ago.

However, this app has been running on Win32 since it's inception so I tested everything on a Windows 2003 server and have experienced NO problems whatsoever. No more memory leaks, no more crashes, no more inexplicable refusals to communicate with nodes.

Now I just have to port a pile of in-house management and integration scripts from *nix to Win32 *grumble*

[sundialsvc4]

Maybe "the saga of what brings you" (here...) would be just the thing we need to know, to help you. After all, we want to help you solve your problem, not the brain-dead-vendor issue that caused it...

[Anonymous Monk]

I would begin, perhaps gratuitously, with the interesting tidbit that NerveCenter was first created by a company called Netlabs - while a fellow by the name of St. Larry Wall worked there. It's got his holy fingerprints all over it's conceptual design :)

Since the mid-90's NerveCenter has been a well-regarded application that, while having a few warts, did it's job reasonably well. OpenService bought it from VERITAS in 2002 and supposedly has spent the time fixing bugs and improving the architecture.

I've been working with an older version for several years, and for the most part it worked as advertised. We recently upgraded to the newest to take advantage of better scalability and some new features.

Problem is, they've taken a useful (though esoteric) tool and made it so brittle as to be almost unusable.

To use an analogy: It's like a car jack that can lift 2000lbs, and it works fine as long as you don't shake the car or forget to lock the handle down. You get underneath and do your work with a reasonable assurance you won't get flattened.

Now you need to lift 5000lbs so you buy the newest jack which is made for heavier lifting. It can do the job, so long as you don't even breathe on the vehicle you need to work on. And by the way, nobody knows for sure if the handle lock is working or not.

OK, so that's my rant.

The situation I face is that I'm stuck with this product, the vendor support is less-than-stellar, and my boss believes I'm the one causing the issues. If I can show him that the issues we're having do not stem from my code then I may have something I can use to either force OpenService to fix what's broken or get the PHB to push for a breach-of-contract or something. On the flip side, if the problem *is* my code, I want to know, because most of my job is to know the idiosyncrasies of our applications inside and out and keep things running smooth.