Looking For The Obvious - Debugging Perl

[greenFox]

I started my working life in electronics as an apprentice working on communications equipment. I spent a lot of time doing component level diagnosis and repair of communication equipment, everything from two way radio sets to analog and digital microwave bearer equipment. There was one card which we had so many spares for that nobody bothered to repair them until the spares started to run out. At which point I setup a test rack and made a start on a box of 200 or more faulty cards.

I soon knew all the common faults this card had and I had a series of notes on how to pick them- waveform diagrams and voltage measurements. One card had me stumped though, I worked on it on and off for 2 or 3 days. Eventually one of the other technicians stopped by to see how I was doing. I told him about the progress and how I was stuck with this particular card. I showed him what I had checked and the weird waveforms at various points on the circuit board. His response was "Have you checked the power supply?". I told him that the power supply was fine and to demonstrate I connected the oscilloscope and showed him various waveforms on the board. He asked again if I had checked the boards power supply. I showed him some more wave forms. When he persisted I gave in and checked the power supply only to find that the only fault this board had was a missing fuse! I could easily have spotted the fault with a visual inspection! (The waveforms I could see on the card were coming from the other connected devices in the test rack.) It was a powerful lesson and one I thought I had not forgotten.

Last week I was asked by a colleague to help with a Perl problem. The error message was for a missing module IniConf. I jumped over to CPAN and found the module and becuase it hadn't been updated since August 2000 I checked the Readme and found that Config::IniFiles is a drop in replacement. I downloaded and installed Config::IniFiles edited the script to call the new module and told my colleague it was all good.

He soon complained again though. The error messages indicated a problem with the script other than the edit I had done. I asked him if it had worked before and he said it had. I jumped in and found 3 missing quoation marks from a section of code. He then recalled having edited that code so I fixed it and gave it back to him and went back to what I was doing. Now it wasn't producing any parse or compile errors but it was still failing at run time.

The original program was written in 1998 and did not use -w or stict and didn't check return codes in many places. But it had worked prior to my colleagues edits and I had fixed them. The IniConf initialisation was in a 20 line commented subroutine which did nothing other than

 $cfg = Config::IniFiles->new( -file => "/path/configfile.ini" );

so I removed the subroutine and just in case checked again. It was failing on reads from the inifile "Can't call method "val" on an undefined value". The section of code which was calling method val was a 16 line commented subroutine which did nothing other than

 $val = $cfg->val( 'Section', 'Parameter' );

I replaced that code as well. It felt good but it didn't fix the problem. I checked the return value from Config::IniFiles new method. $cfg was not getting set. I then wasted some time insisting that it was the script that was at fault (even though the evidence indicated otherwise) after all I hadn't found any reason to trust it so far. Eventually I gave in and wrote a 5 line test case which demonstrated the problem. I didn't believe that Config::IniFiles would be at fault (it had passed its tests when I installed it) but I wasted some more time looking at it any-way. I read the documentation again, the new method "Returns a new configuration object (or "undef" if the configuration file has an error)."

I had had a (brief) look at the ini file and dismissed it because it was pretty simple and it had "worked before". Time to look again. I deleted all the config file apart from one section and one name value pair. This time it worked. I asked my colleague if he had edited the ini file "Nope I just added a couple of comments". Great. Delete all the comments. Worked again. Taking a closer look at the comments showed one line starting with a ":" instead of a ";" (which proves that a "#" is a far superior comment character to ";"!).

Going back to the real program with the fixed ini file and it worked. Just to be complete I replaced

 $cfg = Config::IniFiles->new( -file => "/path/configfile.ini" );

with

 $cfg_file = Config::IniFiles->new( -file => $ini_file ) 
     || die "Config::IniFile method new failed\nProbably error in $ini
+_file\n";

so that the next poor schmuck is pointed in the right direction. (It would be much better if Config::IniFiles set $! on failure, a bug report/feature request is still outstanding.)

Lessons:

• use strict and -w (although fixing this 3000 line monster was out of my scope)
• Always check return codes.
• If someone edited something find out exactly what and where and check that first.
• If you are a module author set $! for failures.
see here, thanks Jenda

What "check the bleeding obvious" lessons have you learnt?

[bradcathey]

As the quote in my signature states, and to paraphrase, don't delete code that looks like it does nothing until you are sure it doesn't. I have opened legacy scripts, spotted what appeared to be useless code, deleted it, and then tried to run it‐setting off all the alarms.

As to your comment about using strict, and while it was impossible in your situation, I find that if it is the second thing I type in my new script, after the shebang, use strict acts as a mini-debugger right from the get go. Of course, for other debugging chores use CGI::Carp; use warnings, use Data::Dumper are all indispensible. They will all help help you catch the "bleeding obvious" from the start.

Update: This popular node has some helpful tips as well.

[greenFox]

"Don't ever take a fence down until you know the reason it was put up. " G. K. Chesterton

I agree and I am normally very hesitant about deleting or replacing someone elses code. I will sit with it and try and work out why it has been done a certain way and what I am missing, sometimes I will ask for a second opinion. I always have a backup so that reversion is no problem. In this instance though both subs were doing no more than I said and both were more comments than code.

[Old_Gray_Bear]

Maintain a style consistent with the legacy code I am modifying. If they used CamelCasedCaps, so should I, even though my first thought is "ugh, %s/CamelCasedCaps/camel_case_caps/g". As I learn the code base, I get expect certains patterns to appear in certain modules, because I recognize the original authors (now long gone). Failing to adhere to the historical coding precident is a certain way to Warnings and Madness. (Now, when I re-write this sucker.....)

[zentara]

What "check the bleeding obvious" lessons have you learnt?

I think the first thing I check, is does the script have Windows line endings. It can give errors like "module not found", even though it's there; or "no such file or directory". It can be very exasperating to a new troubleshooter.

[guha]

On a couple of occasions I have been debugging some script which uses an external datafile for input. I had the script and the datafile up in the editor, laboriously following the execution path, adding print here and there and nothing makes sense. You get the picture.

Hey, even basic stuff don't come out as expected. Only Hubris refrained me seeking help as I sensed something was "rotten in Denmark".

Only after too much time was spent I realized the the datafile I was viewing and the script was reading was not the same. In fact there was never anything wrong with the script in the first place!!

So I guess the morale of the story is that a "bug" can be caused by the script, data or yourself, more seldom in my experience by hardware or the OS.

---

Error, mistake, slip, blunder, lapse? I prefer to think of it as an 'unfortunate juxtaposition of circumstances'.

[husker]

"Is it plugged in?" or some form of that.

[eric256]

Quick Story: My father worked as computer tech support for a local power planet. He got called in the middle of the night once and spent 40-50 minutes on the phone trouble shooting why the terminals weren't booting up correctly. Finaly he gives up and says "Well before I come down, reset the server. Its the large grey cabinet on the far wall." The reply he got "where is it? I can't see it because the lights went out." "The lights are out?", "Yea the power died an hour ago, is that a problem?"?.

[Jenda]

As a module author you can't use $! to signal your module's errors. The list of possible values of $! is very restricted and is OS dependent. Try this for yourself, you can assign a small integer to $! and get the OS's error message, but you can't assign an adhoc message to the variable.

Your module has to use a different variable or die() with the error message.

[Abigail-II]

{Story about faulty cards}
It was a powerful lesson

Except for the "look at the obvious" lesson, I think there's an even more useful lesson in this story: listen to the advice of others.

Abigail

[greenFox]

"Except for the "look at the obvious" lesson, I think there's an even more useful lesson in this story: listen to the advice of others."

Laughing. I think I was only 19 or so at the time, I like to think I have learned that one as well *grin*

[andyf]

I like the story of reductionist debug, how you distil the problem at each stage, partitioning into what the problem cannot be and what the problem might be. So finally you converged on the ini file, and after the fact it seems so easy to say 'well of course the problem was in the ini file', but was it? As a hardware guy I'm sure you will like the analogy of repairing a cable with break. Best bet is cut it in the middle and one part will surely work. But usually the break is near one end. But which end? Your story proves your logic and method worked, but when you nailed the sucker it was at the wrong end, and you had a whole pile of chopped up bits :)
Andy

[slife]

Learned behaviour can sometimes be a hindrance, even when the behaviour in question is good practice.

One of the habits I acquired in my earliest days as a Perl programmer was to protect myself from the common 'gotcha' of regex capture variables being left at the value of the previous match if the current match fails, e.g. instead of this dangerous code:

# Bad code. Don't copy
while(<>) {
    /^(\w+?)/;
    my($val)=$1;
    # do something with $val.
}

I'd instead do this:

while(<>) {
    my($val);
    if(/^(\w+?)/) {
        ($val) = $1;
    } else {
        # Handle failed regex.
    }

    # Do something with $val.
}

and this served me well over the years.

However, on one occasion, I was helping a colleague debug a legacy program which was producing bizarre behaviour: it was supposed to read one line at a time from a data file, extracted some values and inserted them into a database. Recently, it had started to intermittently insert apparently random nonsense.

Well, my colleague and I spent hours debugging this thing, trying increasingly obscure and desparate 'fixes', until, finally, it dawned on me that the program was doing example one above, and the format in the data file had changed so that that regex sometimes failed. Problem solved.

Why did this take so long to spot? My colleague and I were so used to programming defensively against this behaviour that we simply didn't recognise it when we saw it. We assumed that because we did the right thing, everyone else did, too.

The lesson here is to always look at what the program is actually doing, not what you think it is, or even should be, doing. Sometimes, this means 'unlearning' good habits.

NB - I would probably have spotted this in minutes had I used the Perl debugger: lesson 2 here is not to neglect your toolset because "it's only an easy bug".

[demerphq]

(which proves that a "#" is a far superior comment character to ";"!).

Just in case you didnt know Config::Inifiles tolerates # as a comment character. Also its worth knowing that Config::Inifiles is not a good solution for huge inifiles. For smaller ones its fine, for large ones its awful.

[Anonymous Monk]

Simple request, "Do me a favor, let's run the original code so I can see the original behavior."

If the reply is "sure" and they do, you can run a diff on the files, as well as possibly trust the information this person gives you.

If the answer is "no <insert reason or personality defect here>", don full bomb squad protective gear, activate all sensing devices, prepare for battle on all fronts. This person is either hiding something (won't run original code) or is too <place insulting reference to sub normal intelegence here> to have made a back up of the original code. Either case requires extreme caution as you are going to be held responsible until the code works.

No Fear, No Brains. If we didn't pay attention to some of our fears, we would do stupid or possibly deadly things like smoke, talk on cellphones in traffic, drive vehicles while drunk and so on.

dageek