Debugging Perl over the phone

[thelenm]

I just had a funny but mentally draining experience I wanted to share. I received a call from one of my company's clients, who was having a problem with a Perl program that someone at my company (not me) had originally written. I had the original program available, but the client had modified it heavily, and for security purposes our employees are not allowed to see the changes they've made. Also, the guy who is maintaining their program is not very familiar with Perl. (What a messed up situation, I know.)

So I was tasked with helping our client, who is not familiar with Perl, debug his Perl program, which I had not written and could not see, over the phone. The conversation went a lot like this:

{

Him: "Well, there's a while loop that says while left parenthesis left diamond capital I capital N right diamond bracket right parenthesis left curly brace and some if elsifs and inside the elsif I have C-H-O-M-P semicolon followed by on the next line, dollar type equals tilde lower-case S forward slash backslash N forward slash space character forward slash lower-case G semicolon, does that sound right?"

Me: "Whoa."

  redo;
}

It was a little more complicated than transliterating characters, though, since I had to deduce the structure of the code, and had to ask questions I thought would lead in promising directions to find the source of the error he needed to fix. Eventually I discovered a bad assumption that was being broken somewhere other than where he thought the problem was. The fix I had him make was also not pretty, but the best I could do over the phone.

Mostly I just wanted to get that experience off my chest, but I'm also curious: have any of you Monks had similar experiences? How did you handle them?

[dws]

I'm also curious: have any of you Monks had similar experiences? How did you handle them?

I've done over-the-phone debugging, but only with code that I was familiar with. It really gets fun when you're talking with someone is has an exaggerated sense of their own competence.

Me: First, type <string of chars> return

(Pause, listen to the sound of too many keyclicks)

Him: Nothing happened.

Me: That's odd. What, exactly, did you type?

Him: <string of chars> return

Me: Uh, how many keystrokes in "return"?

Him: six

(Bang head on table)



PDP-10/TOPS-10 folks might recall that it was possible to patch the running operating system. Try that over the phone.

[converter]

This was why I developed the habit of telling customers: "now press the enter key" or "...press the spacebar".

[Anonymous Monk]

It really gets fun when you're talking with someone is has an exaggerated sense of their own competence.

If you read what you wrote you sure can get the impression that it is you whose got the exagerated sense of your own competence. C'mon why did ya say type return when you shoulda said press enter?

[ferrency]

This reminds me of a quote I saw in Real Programmers Don't Use Pascal:

One of my favorite Real Programmers was a systems programmer for Texas Instruments. One day, he got a long distance call from a user whose system had crashed in the middle of saving some important work. Jim was able to repair the damage over the phone, getting the user to toggle in disk I/O instructions at the front panel, repairing system tables in hex, reading register contents back over the phone. The moral of this story: while a Real Programmer usually includes a keypunch and lineprinter in his toolkit, he can get along with just a front panel and a telephone in emergencies.

:)

On a more serious note, on occasion (mostly when my wrists hurt from typing too much) I have thought about what it would take to optimize a programming language for voice input/output systems. This could be very useful for visually impaired people who use reading software as their output device, and to those who can't use a keyboard input device, but who want to continue to program efficiently.

You mentioned that your primary problem was not the character to voice conversion; however, in a usable system, pronunciation of the special characters in Perl would create quite a bottleneck for efficient input/output via voice interfaces. Has anyone come up with alternate pronunciations, or even standard pronunciations of commonly used special characters? Does anyone know of any research already done on this subject? Is anyone out there programming Perl with a voice input or output device on a regular basis?

Thanks!

Alan

[Juerd]

I try to let people describe what goes wrong, and if possible, I ask question that the other one has to answer. Having error messages helps a lot, but even without, it can be doable. When debugging CGI applications over the phone, don't forget to ask about the error log. When you ask about error logs, don't ask for the last $few lines, but ask them to tail -f it, and read out loud what it says as the error occurs, and don't let them skip important things (with error messages, only the first one is important, as the other ones are probably results of something else being wrong).

When someone spells out c-h-o-m-p, I let them say every word as it is, and I complain about every word needlessly spelled out.

A problem I run into often is that people tend to forget dollars as they occur much. This is even more so if the person in question has no Perl experience. print $foo and print foo are not things you want pronounced equally.

I like it most when both ends have the same color-highlighting editor. That way I can ask for certain colors (namely those for comments and strings) to be ignored. (vim++ btw)

If you have e-mail or a fax, let them send code to you. It's a lot more efficient. If they can't, because the code is copyrighted, tell them that telling you the code is a violation too and that you have already written down everything he/she said.

- Yes, I reinvent wheels.
- Spam: Visit eurotraQ.

[VSarkiss]

The fix I had him make was also not pretty, but the best I could do over the phone.

thelenm: OK, get to a shell prompt.
customer: All right, I'm there.
thelenm: Type "R M space dash F space" and the name of the file.
customer: Uhm...
thelenm: No, no, just listen. Now type "C A T greater-than space" and the name of the file.
thelenm: Now, listen very carefully....

Sorry, couldn't resist. ;-)

The closest I came to this was when I worked for a scientific computer manufacturer, and we delivered a computer to a client. I realized when it booted that the interface driver that had been shipped with the machine was way old. I had written most of it, so I pulled up the file in vi, called back to the shop, and asked them to read me the text of particular subroutines. It was not a pretty sight.

And yes, there were choice words when I got back.

[dthacker]

My worst phone troubleshooting experience was trying to walk a person through editing kermit configuration files on a PC running MS-DOS 4.something. We had to use edlin. He was in Cancun and I was in Omaha. I knew no Spanish and he knew very little English. It took a long time....

Dave

---

Code On!

[mojotoad]

I prefer to avoid phone-coding if at all possible. Typically this means jumping through whatever hoops are neccessary to either access the script in question via a temporary web page or email -- then we can step through it line by line, in tandem, over the phone and use phrases such as "line 86, see where $current_value is assigned to $booger ? Good. Go down two more lines, etc..."

Verbalizing the code is too inefficient and akin to the blind leading the blind.

Matt

[seattlejohn]

Seems like a great reason to use the instant-messenger app of your choice (while simultaneously talking on the phone). Instead of reading aloud a couple of lines of code, the customer could paste in the relevant line or two so you could see it for yourself and meditate for a moment. Then you could respond with commands, code patches, or whatever without concern they'd get mangled in the translation. You could converse as this is happening: OK, can you paste in the error message? OK, what's on line 45? Can you try this input sequence? Blah blah blah...

Of course, from the way you describe the folks controlling things on the other end of the line, it sounds like they might not be amenable to something as <ahem> sophisticated as IMing... it sounds like a pretty bad situation all around, and I sympathize ;-)

[Mr. Muskrat]

Okay, so I've never done it over the phone...
The closest experience I had to that was trying to get a CGI script that I had written to work on NT 4 Server w/ IIS. I was in the midwest and my client was on the west coast. We started with email and quickly switched to ICQ when the initial "fix" didn't work.
It worked fine in Apache, Xitami & PWS but would crash with NT/IIS. After an hour or two of going back and forth, we finally got it working. My script used relative paths and urls. NT/IIS was expecting full paths and urls.

---

Matthew Musgrove
Who says that programmers can't work in the Marketing Department?
Or is that who says that Marketing people can't program?

[mattr]

Yes, I've had such an experience though not as bad (worse?) that is, the person calling me was a dangerous beginner perl programmer, not a complete newbie. As it got painful and starting taking a long time I tried to figure out what the basic problem was as quickly as possible and get off the phone, then offered to come in and fix it which they declined. In the end I did end up going in, and got kudos which lasts.

I recommend trying to figure out what is going on, then once you have some hypotheses, cut the guy off and say you've gone as far as is possible over the phone. Then go in and fix it after you talk to his boss, who will love you. He may even love you if you charge some money.

[chicks]

I had to debug an RPG program once from about 6 states away without any physical access, etc. Not surprisingly I had no interest in learning the AS/400 or RPG. The RPG program was communicating over the network with a perl program I had written for the client and the connections were randomly breaking. Sometimes they'd work and sometimes they wouldn't. No amount of debugging logs from the perl script showed any pattern whatsoever. So we got the client to fax me 100 pages of packet dumps from the LAN between the UNIX box with the perl script and the AS/400. All of these were spread out all over my house. From piecing them together we were able to find the mis-logic in the RPG program, explain it to the RPG programmer and everyone was happy. :-)

[SparkeyG]

While my programming via phone experience is limited to one regex problem a developer had (A node for a different time), the most painfull phone debug experience was debugging a SNA line problem while walking back to the office, leaving my wife to finish lunch by herself. SNA was old before I got into elemetry school, and I'm the resident 'expert.'

--SparkeyG

[mstone]

Been there, done that, burned the t-shirt.

My personal favorite was getting a call from a client on my cell phone while I was driving on the interstate. I didn't have a computer, I didn't have code, and I wouldn't have been able to use either one if I did. Fortunately, it was code that I'd just written, so I was able to step through it by memory.

My second-favorite phone-coding experience involved writing a WAV/AIFF codec in Fortran, which I haven't spoken in many, many years. The client was trying to port a program that relied heavily on pipes to subprocesses from Linux to Windows NT. The codec was one of the things that broke, and since the bulk of the code was in Fortran (heavy numerical processing), the client wanted me to replace the subprocess with built-in code.

The best advice I have for such situations is: use lots of small steps.

The biggest problem of working with code is getting lost.. even when you have the code right there in front of you. If you try to do too many things at once, you'll eventually suffer a mental stack overflow and segfault your brain. And when you're debugging by proxy, you also face the risk of segfaulting the other person's brain. Keep the tasks small, and make sure to clean up after yourself every step of the way.

Beyond that, debugging over the phone follows most of the same rules as regular debugging. The first (and usually biggest) problem is diagnosis. Start at the point where things obviously go wrong, and start inserting print statements. Dump out the variables to see what the values are, and locate the ones that don't look like they should. Trace those back until you find the statement or function where good input values produce bad output values, they try to figure out what's going wrong.

If you're lucky, the problem will be obvious once you know the exact location of the misbehaving code and have a sample of input that makes it misbehave. If you're not lucky, the problem will involve subtle interactions with other parts of the code, or possibly the OS. If it's a well-behaved bug, you should be able to chant off a code patch right there on the phone, and the client will go away feeling vaguly privileged to have worked with such a guru.

OTOH, it's not worth the pain of trying to debug a subtle bug over the phone. Identifying it as a subtle bug is enough for an ad-hoc debugging session, but the solution will probably involve deep design changes, and should be handled accordingly. If you discover that you're dealing with a subtle bug over the phone, just say, "Okay, this one is too tough to resolve here and now. You talk to your superiors and I'll talk to mine, and we'll work out a deal where we can solve this thing properly."

[Putzfrau]

TRUE STORY:
The following strory doesnt involve perl, but its a good tale... My neighbor,Bob an old assembler programmer, told me this story one stormy night of the teh call from HELL .It begins so... one stormy day( just like this one) he recieved a call from a client with a particular problem. Unfortunately for Bob his client's phone was defective so the only sound that can through was the touch tones. Using his superior intellect, Bob, decyphered the message(each series of 3 beeps represented 1 decimal ASCII character)which was a call for help followed by a program listing of some JCL( thats right....) luckily the error was quickly remedied so that Bob and Client lived happily ever after.

THE END
It's true i swear!

"Putzfrau no like heavy english based COBOL, Putzfrau SMASH!!"

[ehdonhon]

I think all OS's should come with screen reader software as part of the standard install. At a previous job, I had to do a lot of phone support, my favorite customer was one that was visually impaired. Mainly because he didn't have to tell me what was on his screen, he could just let the computer read it to me. It even echoed the characters as he typed them and said what menus were being selected. Made my life much easier when I could actually believe what I was being told.