Sub-initiate needs help getting started

[Lori713]

Hello all,

I've been given the task of learning Perl, Javascript, HTML and SQL on my own in order to take over a program written by a consultant (currently about 8000 lines of code with little/no documentation) that generate financial reports on the Web (I also have to develop some new reports from scratch). The current program works and is in production. To read more about my experience:

I currently don't know any of these languages, but am reading various books and nodes here to get ideas. My previous programming experience involves a little Visual Basic (i.e., "Programming Lite"), and something called nVision from PeopleSoft (i.e., "Spoonfeeding Heavy"). I'm fairly comfortable with all the new languages except Perl. The next Perl class available that I can go to is in late October.

Did I mention a draft of these new reports are due October 31st?

I'm not sure how to ask my question because I know so little at this point, but I'll try...

How does one debug a Perl program containing multiple subroutines that generate HTML pages that use Javascript on the Web (and uses SQL to pull data from the database)? To read more about what I've done so far:

I've read numerous nodes and articles about the Perl debugger (i.e., nate's article on "using the Perl debugger," the idiot's guide to debugging node). I've also explored ActiveState's site and watched a couple of recorded webcasts about VisualPerl. I also have the Camel book, the Llama book, an HTML and Javascript book, a Perl/CGI book, all of which I've scanned for various tips and tricks. I've also tried digging through perldoc perldoc, perlfaq1 through -9 and other various manpages.

Confession: I don't know how to run my program from a command line when my access to it is through an URL I type in. I make changes to a text document, FTP it to the Unix server, and it automagically appears when I go into Windows Explorer and type in my URL (I'm on Windows 2000 on a client PC). I have found someone to show me how telnet works... but I'm not sure how to debug in that environment since my program/subroutines generate web pages...

Any thoughts on how to set up an effective debugging system given that I'm playing with multiple languages on the Web? I'm having trouble pulling together all the information I've read over the last couple of weeks.

Just FYI: The program I'm taking over doesn't use the "-w" or "use strict" etc. (when I put these helpful things into the program, well, it's ugly - lots of "uninitialized variables"). Any dangers to putting "my" on every variable? The consultant said a lot of the variables are global, but I'm not sure what that means in the context of my one subroutine library (everything is in one file). He named the global variables "g_xxx," so I do know how to find them. I'd like to be able to use the "-w" and "use strict" for future help in debugging.

Thanks for any help you can provide. If you need more info, please let me know. Lori

P.S. Sorry this is so freaking long but I wanted to provide all the info I'd seen people ask for in previous posts.

[bobn]

Holy cow. Where to start?

1. The consultant was a flaming asshole. No -w, no strict, no comments, no doc and an 8000 line file? What a dipshit.


2. Get the Ram book aka The Perl Cookbook. Invaluable how-to recipes.


3. Get the leopard|cheetah|whatever book, Programming the Perl DBI.


4. Get command-line access to the server using ssh. Edit on the server and test from the command line that way.


5. Stock up on caffeine or stronger stimulants. Tell friends|lovers|family you won't be seeing much of them in the next 10 weeks.

You have a hell of a job ahead of you. I'm not convinced it's doable. Had the consultant left a useable code base, it would be much easier - but with variables named g_xxx in an 8000 line file with no comments, it sounds like he intentionally obfuscated his code. His code will probably be almost worthless except as a negative example.

Update: Use CPAN http://www.cpan.org - especiually the modules. You'll already be using CGI and DBI, but there may be many more there than can be uused to your advantage.

As for the "from scratch reports", that's how I'd say you should do it - from scratch. Use the consuiltant's code only for understanding table layouts. Write yours from scratch, using strict and warnings, which can save you much aggravation.

When in doubt, simplify.
I'd stay away from javascript unless you know for certain that only one type and version of browser will ever be used. Even then, I'd stay away from it - it's best use is for validating user input w/o server involvement, but it's a whole other language to learn that won't necessarily get you closer to getting the job done. Validate in perl instead, because security requres that the input be validated at the server in any case.)

Keep the HTML simple, too. Do not fall prey to the temptation to endlessly tweek the appearance of your web-pages. You don't have time, and it's futile anyhow - HTML is not meant for that, and it will look different in different browsers.

Use the standards for HTML. Do not write browser specific code. That way lies madness.

--Bob Niederman, http://bob-n.com

All code given here is UNTESTED unless otherwise stated.

[agentv]

...do not despair Lori713 in the face of this discouraging (and somewhat misguided) advice.

It is not useful to conclude, "He drives without a seatbelt, therefore he must be a terrible and careless driver." And it is similarly unhelpful to conclude (without seeing any code) that the code is worthless and unmaintainable on the basis of a few superficial points of dogma.

Don't get me wrong. I think that everyone learning or writing their own code from scratch should make a point of using strict and -w for everything they create. But my experience with production projects is that you should not feel stopped from proceeding if you cannot retrofit those civilities onto an existing code base. There are more important tests that you can conduct.

But go ahead guys. Keep on declaring any code that does not meet your personal standards for cleanliness or elegance to be "worthless" and watch the number of IT managers who refuse to let Perl anywhere near their shop grow. (It's already a depressingly large number of them already.) The cliche' of a consultant who leaves behind a legacy of ugly and maintainable code is a pervasive one. And part of that legend is created by weak and dogmatic programmers who know how to write things from scratch, but quiver in the face of complex code written by someone else whose thought process works differently.

And there's good reason for this. It's easier to pronounce code to be "worthless" than it is to admit, "I don't understand how this works."

So don't be discouraged Lori. And as other advice here indicates, you should simply take small parts of the problem and examine them one at a time. As you work with the system, you'll begin to see what it does, and what you can do with it in the future.

You've already identified several things that are important about your Perl parts of the problem. Use the seatbelt where you are creating new things by using -w and strict in your own code. Learn how to test your code from the command-line.

In addition, begin now to learn how your web server operates. Become familiar with the configuration files (or other configuration systems) and with the log files that are created. Sometimes those log files are going to contain the clues about what went wrong with your system.

Most of all relax. What you have to do can be done (in fact, you will do it) and you have already taken an important step by joining this community and becoming part of the conversation.

...All the world looks like -well- all the world, when your hammer is Perl.
---v

[bobn]

But go ahead guys. Keep on declaring any code that does not meet your personal standards for cleanliness or elegance to be "worthless" and watch the number of IT managers who refuse to let Perl anywhere near their shop grow.

1. The desire to rewrite rather than modify is not limited to perl programmers.


2. Shit code is shit code. Good code is good code. I know. I've written both. And one of the main differences is that when it comes time to make changes and add features, the shit code is sheer agony and the good code is surprisingly easy.

If the new reports are only trivially different from the existing reports, the OP might be able to tweak the program to meet the needs. Some parts of the program may be salvagable.

But a system that is contained within one 8000 line file with few or no comments, doc, no warnings and no strict, and lots of global variables named in the form g_xxx is almost certain to be shit code. It is hard to imagine anyone who cared writing anything decent that matched this description.

But it probably is a mistake to condemn it without seeing it. Lori, you might want to try and get some help with this. This is one place. Another might be at a meeting of your local Perl Mongers group. http://pm.org.

The advice given above about learning how your webserver works is excellent, especially the log files, especially the error logs because anything you output to STDERR (eg warn 'something') ends up there instead of in the webpage.

--Bob Niederman, http://bob-n.com

All code given here is UNTESTED unless otherwise stated.

[dash2]

Nonsense.

First of all, the reason that managers are often turned off Perl is nothing to do with "dogmatic programmers". It is much more to do with incompetent programmers who don't use strict or -w, and who, partly for these reasons, do write unreadable code.

It is possible that this consultant was a genius who doesn't need a seatbelt (see Paradigm Shift - Don't use strict for a famous example). But the 99% likelihood is that he was an idiot who didn't know how to use a seatbelt - especially if his code has a lot of globals, which is normally a sign of bad design.

Finally, hard-to-understand code is almost always bad code. (There are exceptions, like tightly optimized loops, or JAPHs.) And for a consultant, this is doubly true: if you've left a lot of code that other people find difficult to use, then you're doing a crummy job - no matter how brilliant your code may be, your first priority was to help the people you worked for. Beginning programmers often think that if they don't understand something, it must be very clever, deep code. Usually the best code is very easy to read, almost like pseudocode.

I think this is a tough job, but kudos to Lori713 for taking it on.

A massive flamewar beneath your chosen depth has not been shown here

[Popcorn Dave]

First off Welcome to the Monastary! :)

Get yourself a copy of Perl in 21 days. That is a good starting point for where you're at. The books you do have are invaulable, but I personally think they can be a bit overwhelming if you've never done a lot of programming.

Next, get yourself a copy of Perl - obviously. :) ActiveState's is good for windows. So it's sort of up to you whether you go with 5.6 or 5.8. Other monks here who have more experience with both versions would be better qualified to advise you on that. I'm using 5.6 and it works for me.

Now as far as your debugger, there is a graphical debugger that is available on the web <a href="http://world.std.com/~aep/ptkdb/>here. Personally, I prefer this debugger, but again your mileage may vary.

Also, as a beginner I would say yes, put in use strict, -w and perhaps even use diagnostics to help you catch errors. And remember there's a LOT of good people here that are willing to help.

Good luck!

There is no emoticon for what I'm feeling now.

[jdporter]

Get yourself a copy of Perl in 21 days.

Ugh. Terrible advice. Of all the Perl books out there, this is one to avoid.
For better advice on how to choose a Perl book:
• Choosing a Perl Book, by Simon Cozens (perl.com)
• Perl Books, by Uri Guttman (perl.com)
• Camel Critiques, by Tom Christiansen (perl.com)
Of course, one should check out our own Book Reviews, but I tend to take these with a grain of salt.

I personally would recommend the following excellent on-line Perl books:

• Picking Up Perl by Bradley Kuhn
• Beginning Perl by Simon Cozens
As Simon said,

You can't learn Perl in 24 hours, 21 days, 12 weeks, 9 months, or a year. I've been programming Perl for nearly five years and I'm still learning.

Same here.

jdporter
The 6th Rule of Perl Club is -- There is no Rule #6.

[bart]

The book that seems most appropriate in this particular case, is Perl Debugged. It's aimed at beginners, discusses ways to avoid and eliminate bugs, and gets quite favourable reviews. (Amazon, Perl Books)

There's also a title Debugging Perl.

[Popcorn Dave]

I don't think you can ever learn Perl, but you can get more proficient the more you do. That said, the reason I did suggest the 21 day book is I know a couple of people it did well for. Personally I learned via the camel, but it was the textbook for the class that I took.

And yes there are a bunch of lousy Perl books out there, the Dummies Guide being one of the worst IMHO. However, I still think that for the person with little or no programming experience, O'Reilly tends to be a bit heavy.

Just my opinion.

There is no emoticon for what I'm feeling now.

[bobn]

Next, get yourself a copy of Perl - obviously. :) ActiveState's is good for windows.

The OP stated that the app is already up and running on a Unix box. AS might be useful to take home on a laptop for practice, though.

--Bob Niederman, http://bob-n.com

All code given here is UNTESTED unless otherwise stated.

[Popcorn Dave]

You're right. After re-reading the post I realized I missed that point.

There is no emoticon for what I'm feeling now.

[Lori713]

Thanks for the hints and the welcome. I've downloaded Perl from ActiveState to my PC (and the fun begins...) and I'll be sure to use strict, -w in the future.

[dbwiz]

On learning Perl, woolfy has recently written a node on the available sources (Where and how to start learning Perl). Also, have a look at our Tutorials.

About SQL, if you want a start, here is an online tutorial: SQL for web nerds.

For HTML, there is the barebones.

About javascript, I don't know, sorry, but I think you have enough to read so far ... :).

[CombatSquirrel]

Lori713, first off: Good luck with your project. Secondly, in case you have never worked on an extensive project before, some tips which have helped me in the past (you may or may not consider them useful):

• Split the problem into sub-problems, and those into smaller problems, for example (very rough) data input, data processing, data output or (finer for data input) get data from database, parse data into appropriate formats for processing.
• For every larger problem group, generate an own module. This will split your code into readable portions. A larger problem group would for example be the output to an HTML table, possibly as submodule in a module that outputs the whole report. Test every module thoroughly after you have finished it.
• Every module can be split into functions as sketched above. For each function, you should note in a comment -- before you write the function body -- what input the function expects, what it does, and how the output is formatted. After you finished a single function, always test it with sample data. You should have small functions that do part of the dirty work of the larger functions to make your code more readable.
• Nesting is good (see above)
• Good luck again
This method provides you with a LEGO way of solving your problem: First, buld up parttial solutions from the bricks, group them into larger partial solutions and finally into the real solution.
The partial documenting "on the go" is a bit more work, but it'll pay offonce you wonder what arguments should be passed to that what-was-it-called function.

And the last thing: Whenever you have problems with a piece of code that you think should work, but doesn't, post it and you'll see (if it is a sensible small problem - splitting ;-) that we try to help with code as much as with theory.

Hope this helped.
CombatSquirrel.

Update: About the HTML part: Although you probably don't, if you should know German, have a look at Stefan Münz's Selfhtml tutorial; it is simply the best HTML tutorial I know.

---

Entropy is the tendency of everything going to hell.

[liz]

Additional to Re: Sub-initiate needs help getting started, you might find Difficult code (Resolutions) relevant for your situation.

And best of luck. But with the help of the Monastery, it should work out ok.

I wonder how much more expensive the not-documenting, using global variables consultants were.

Liz

Update: Removed "making" in the last line.

[williamp]

Hi, Good luck with this project. Once you have a test environment either on your unix box or PC, I would take a copy of the code and start going through it line by line, making LOTS of comments/note as you go. You do not have to become a Perl Geru by October 31st just a geru of your program. To find out how lines or sections work try pulling them out to small test scripts and pass them variables which you feel will test the code. The -w and use strict should be used here as well. Their messages are very useful. Hope this helps,

[shenme]

That's probably the simplest and most useful advice - try out the bits of code in the small if you have any questions what they do.   Whether in small test jig programs, in the debugger or typed-in on the command line, just try it out.  

What does .*? mean? Try it with and without the '?'   (pardon the Windows quoting):

perl -e " printf \"'%s'\n\", join \"', '\",  '...tangled web was woven
+...'=~m/(.*?w)/  ;"
'...tangled w'

perl -e " printf \"'%s'\n\", join \"', '\",  '...tangled web was woven
+...'=~m/(.*w)/  ;"
'...tangled web was w'

And even when you don't understand it, at least you'll know what it did ...

[skyknight]

The short answer is, and I'm not trying to be an asshole, that you are thoroughly screwed. There is nothing worse than being handed code that was shoddily written by an untalented hack of a consultant. It is a huge mistake for companies to hire random people without any regard for their talent level and then dump the maintenance onto some unwitting internal developer. The code that the consultant hands over may mostly work for a very specific task, but it will probably be extremely difficult to extend, and god forbid you should ever have to debug 8000 lines worth of spaghetti.

Many of the skills that are essential to being a good software engineer, and that go largely uncovered in school, involve social interactions. More than anything else, you have to be honest, and you have to rein in the unreasonable, illogical expectations of managers. Do not let managers make unreasonable promises, and do not make grandiose claims about how you'll learn four new languages, debug 8000 lines of code, and cure both AIDS and World Hunger in two months. When you fail, you'll look foolish, your manager will look foolish, and the clients will be hung out to dry.

Sometimes when you are handed a true monstrosity, it is better to glean what info you can from it, and then burn it to the ground and start over again. Given your gruesome description of the task at hand, I think that you should seriously consider this course of action.

[chantstophacking]

The epitome of effectiveness is to accomplish a goal without trying.

[Cody Pendant]

I'm going to join the chorus of "good luck" and also "the consultant should be shot".

I'm also going to say you're in a very sticky situation.

How's this for a question -- why are you attacking this problem from behind?

You've got 8,000 lines of bad code -- forget that for a second, what is it that you want to achieve?

If you've got a database and you've got some expected output, we might well be able to bridge that gap with a new script. Is the JavaScript necessary, or does your boss just need to see reports on x number of widgets sold on the east coast versus x number of widgets sold on the west coast?

Considering what we know about your consultant, they might very well not be very good reports in the first place.

So I'm saying, what if you go back to your boss and say "forget the exact details, layout etc of the reports you have now, reproducing them is a very complex job -- what do you really need to know? I'll start on that." and take it from there.

Getting data out of a database into a web page is not the hardest task you will ever face in Perl.

---




($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss')
 =~y~b-v~a-z~s;                             print

[blackstarr]

As someone who develops reporting structures for a living, I'll second that, forget what your inept predeccessor has done and redo it from scratch. Approach it as if it were a brand new request.

Forget the look & feel until you are answering the correct questions ... then the look and feel will be easy.
If you handle the data output (answers) as totally seperate from your presentation (format), then you will find that if either requirement changes, it will be a simple matter to update the code.

Good Luck & ...
So Long
blackstarr

[markguy]

I'd agree with you if it weren't for the fact that the "from scratch" part of this problem includes the fact that the (hi! welcome to the club!) new developer (here's your new hat!) is woefully unprepared to know what the issues/problems are and where to find solutions that fit them. Of course, I have that problem too and I've been doing this for years.

In any case, I don't think it's an answerable question short of seeing the code itself and ideally the functional specs for what the code is apparently solving. Figuring out legacy code can do wonders for jumpstarting your knowledge base (it's how I got started) and limits the feeling of helplessness that can hammer you if you have no experience in handling this sort of situation.

This may not fit the goals of the original poster, but you might consider hiring another consultant. And checking if they have a handle here for good measure ;). Hell, even if the company won't go for it, it might be worth paying some amount of your own personal money to get a little hand-holding as you work through it. Or just abuse the hell out of SoPW for the cheap, yet amazingly effective method!

[bobn]

Any dangers to putting "my" on every variable?

It's not always going to work. Here's a (contrived) example:

for ( 1..3 )
{
    $x = $_;
}
print "\$x is $x\n";

Results are:

$x is 3

Now look at:

for ( 1..3 )
{
   my $x = $_;
}
print "\$x is $x\n";

Results are:

$x is 

Reason: the my makes $x a "lexical" varible which is "in scope" only within the innermost { } pair. Ouside of there, it has no meaning, or other, unrelated meaning.

So you'll at least have to dig out all the globals and do the my at the filde scope level (eg outside of any { } blocks). They'll still be globals, but it lets you start using strict.

--Bob Niederman, http://bob-n.com

All code given here is UNTESTED unless otherwise stated.

[CountZero]

A good way to start (other than regularly visiting the Monastery) is to dig out the original/amended/updated specs of this project, so you can find out what it was meant to do and (perhaps) how it should look.

This may help you to understand the program (if it was written to conform to the specs, that is) and more easily find your way into it.

It may perhaps even show you that it is better to start over from scratch. This may seem a daunty task, but with the help of some templating systems (Template-Toolkit or Mason or another templating system (super-search the monastery to find more) it may even be less demanding than finding your way into the undocumented sub-standard code of someone else and at least it let you adapt future versions of this script with less effort.

Oh yes, and if you found the original specs, see if you can find how much the consultant charged for his "work". Perhaps you can negotiate a bonus along the same lines once the job is finished ;-) and don't forget to drop a small bit of your bonus in our Offering Plate.

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[PodMaster]

Start with Tutorials -> Web Programming Using Perl -> CGI Help Guide -> Concerned about security? Start with the FAQ's ...

How much time can/are you going to devote to this? I say give it a week, and then think about getting some help (a week ought to be enough time to help you decide if you can meet the deadline on your own).

MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"

I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).

** The third rule of perl club is a statement of fact: pod is sexy.

[Lori713]

Thanks to all who responded. There was a lot of excellent advice in all the replies, and I'm feeling MUCH better about this project. Just FYI, I work for an excellent manager who does listen to me when I say I can't get something done by a certain deadline. I had to say that very thing when this project was first given to me (and after banging my head against the wall for a week or so after trying to decipher the code). He immediately adjusted the deadlines on my other concurrent projects to give me some breathing room. Still a tough project, but I feel confident I can get into the Perl groove... (okay, so I'll probably post a lot of questions over the next couple of months ;-) ).


THANKS AGAIN!

[bm]

I highly recommend that you version control your work. Not being able to roll-back and control your changes may make life even more difficult than it already is.
Best of Luck!
--
bm

[wirrwarr]

I think you might want to try to setup a web server on your local client. So you can edit the code using your favorite editor on windows and test it at once.
I assume the unix box uses Apache as a web server. So go to apache.org, download the relevant version (either 1.x or 2.x, depending on what's installed on the unix box), and install it on your windows box.
If you have questions about this, a search on the web ("install apache and perl on windows") should give you plenty of help.

Don't panic!

[jonadab]

Any dangers to putting "my" on every variable?

Yes, don't do that. When you write your own code from scratch, you can write it to use all lexical variables, but if you try to retrofit that onto code that wasn't written that way you can get strange results. The "always use strict" advice only applies to new work; you shouldn't try to retrofit it to someone else's code. At least, not until you're confident you know what you're doing.

Warnings you should use. Now, for cgi you don't want warnings on for production use, because they'll mess up the interaction between the script, the web server, and the browser. So you can either test from the command line, or use one of the modules that sends errors and warnings to the browser. Or you can turn on warnings while you're debugging and turn them off again after, which is probably what I'd do. You do know how to tail web server logs, right?

My long-term advice to you is to plan on gradually rewriting the whole thing (or most of it) yourself, in bits and pieces. You'll have an easier time maintaining it that way. But you don't have time to do that all at once right now, so only write the bits you need to write in order to meet your deadline. Later you'll want to add or change more things, and you can (re)write more parts of it at that time.

---

$;=sub{$/};@;=map{my($a,$b)=($_,$;);$;=sub{$a.$b->()}}
split//,".rekcah lreP rehtona tsuJ";$\=$ ;->();print$/

[Wally Hartshorn]

How does one debug a Perl program containing multiple subroutines that generate HTML pages that use Javascript on the Web (and uses SQL to pull data from the database)?

If you're very lucky, he used some sort of HTML templating module. (HTML::Template is probably the easiest to get started with.) That can help immensely because it avoids having oodles of print statements spewing out HTML code at numerous places in your code.

Unfortunately, I suspect you aren't that lucky.

If you're lucky, the 8,000 line file is done as a separate module (something.pm), with just a short perl program calling the initial subroutine in that module. If that's the case, it's easy to write short perl programs of your own to call subroutines in that module to test things out. For example, if there is a subroutine in the module that accepts an employee name as a parameter and returns that employee's salary, you can easily write a short perl program to call that subroutine, pass it a hard-coded name for testing purposes, and then print out the salary that is returned on the command line. That allows you to debug just that tiny segment of code from the command line, rather than dealing with the entire CGI environment in a web browser.

Unfortunately, I suspect you aren't that lucky in this bit, either.

If the changes you will be making are quite minor, I would recommend making the changes first and verifying that they work before doing any significant structural work.

If the changes are going to be extensive, then I would suggest restructuring the existing code first to make it easier to make the future changes.

Suggested modules: CGI.pm, DBI.pm, HTML::Template, Config::IniFiles, Test::More, HTML::FillInForm, CGI::Application, CGI::Application::ValidateRM. These let you easily read data from a form (via CGI.pm), read from the database (via DBI.pm), display values in an HTML page (via HTML::Template), read from a configuration file (Config::IniFiles), run automated tests to ensure that a routine that you had working earlier hasn't been broken by later changes (Test::More), pre-populate a web form (HTML::FillInForm), handle CGI interactions via run-modes (CGI::Application), and validate data entered in a form (CGI::Application::ValidateRM).

An 8,000 line program without "use warnings" and "use strict" that incorporates HTML, JavaScript, and SQL without modules and automated tests is pretty much a textbook example of what NOT to drop on someone who is new to Perl.

Look on the bright side -- every Perl project you have after this one will be easier! :-)

Wally Hartshorn

(Plug: Visit JavaJunkies, PerlMonks for Java)

[dragonchild]

Having done a number of similar jobs in the past (albeit knowing Perl, CGI, SQL, JScript, and Unix quite well), I'd like to share a few pieces of experience I've gleaned.

Note: All of these suggestions are based on the assumption that this code will continue to be maintained and extended for a long time. If it's going to be retired soon, ignore everything I've said, do what you need to do (no matter how dirty), and be glad the monstrosity is dying a well-deserved death.

1. Get the specs. I don't care what you have to do - beg, borrow, steal, whatever. You can't (re)write or maintain anything unless you know what it's supposed to do. If there was a design, even if it's a bunch of scribbled-on cocktail napkins, get that too. Get all of the consultant's notes. If there was a tester, get their notes, too. Ask your boss for a few emails describing what s/he thinks this thing should do.
2. Take a week and build a bunch of test data. Make it resettable. Use this as a good first project in Perl and SQL. The idea is that you can do a test run, then reset the datasource(s) and run the testsuite again.
3. That test-suite thingy - make one. DO NOT DO ANY WORK ON THE CODE WITHOUT ONE. I have only recently come to see the light re: test suites and I cannot imagine working without one. That difference between 90% certain and 99.999% certain is a huge one.
4. Build your test-suite from the specs. Do not build it from the current code! For all you know, the code is full of bugs. Your first priority is detailing what this thing is supposed to do.
5. Once you have a test suite, run it against the current code. I can almost guarantee that the current code will fail at least 10-20% of the tests. (You can use this information later.)
6. Factor the code into smaller pieces. Do this one baby step at a time. Every single time you make a change, run your test suite. You're looking to see if your change introduced a bug.
7. Now, if your change introduced a bug, do not assume that you messed up. It could be that your change exposed a bug. Often, production code has two bugs that offset each other. When you fix one, you expose the other. This is a good thing!
8. Keep repeating the refactoring process until you understand what's going on. Now, at this point, the code most likely will not be where most of us would consider to be good. That's ok. You're in a remediation process.
9. Now, throughout this entire process, you're keeping notes. The notes include
   1. Places that need improvement
   2. Refactoring ideas
   3. Business flows
   4. Data flows
   5. Execution flows (all three are different)
   6. Ideas you have for improvements
   7. Modules you can leverage to help out
   Do not make any of these changes ... yet. (Except, of course, where the change is obvious and will make an immediate positive impact.)
10. Release a version of the code that's yours. Call it v2.0.0 and make your boss happy.
11. Add each report. When you design a report, design the tests first. Build the report. Test it. (You might want to consider writing a Report object that will handle most of the common tasks for you. At a previous position, I did that and reduced the development time for a new report from 40 hours to 25 hours, while improving maintainability and promoting a common look and feel.)

Good luck. If you ever need to vent/ask/whatever, please don't hesitate to come here. :-)

------
We are the carpenters and bricklayers of the Information Age.

The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

[MidLifeXis]

3. That test-suite thingy - make one. DO NOT DO ANY WORK ON THE CODE WITHOUT ONE.

Any recommendations or literature on doing this? I have a sysadmin / jack of all trades type background, and as such, am starting to reach the limit of my "how to test" knowlege. I am looking for some insomnia meds in the form of some heavy (or light) reading on this topic.

Thanks

[dragonchild]

Googling will give you a lot of good info, as will browsing Borders and/or Barnes & Nobles. However, there are a few good rules of thumb I've seen after supporting a test team for a year:
• Test the design first - make sure the design is sane. This includes the application, the database, the physical setup, the user interface, etc.
• Nail down the specs - if you don't know what you're testing to, you don't know if you passed or not.
• A corollary to the last point is that you shouldn't accept a spec if you can't test it. Remember, it's your responsibility to make sure that your code meets the spec. If you can't test a feature, you can't discharge your responsibility.
• Don't test application functionality, test user functionality. Remember, the users don't care if you have a neat-o-keen sorting procedure. They just want to know that their data returns in the order they asked for / were expecting.
• Test both coding units and logical units. You have to make sure that a given subroutine works, but you also have to make sure that a given feature works, too.
• Don't just test functionality - you want to test under different scenarios. Light load, heavy load, different types of users, etc.
• Automate your testing as much as possible. Ideally, you'll be running thousands of tests every time you change. It's a lot easier to have the computer do the grunt work for you.

A lot of these suggestions assume that you have a rational codebase. For example, if you cannot extract a given subroutine from its context and test it, you very well could have a problem.

Also, your testing design

------
We are the carpenters and bricklayers of the Information Age.

The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

[talexb]

Having written some truly ugly Perl back when I was starting as a consultant, I understand how a little 250 line script can grow to 8000 lines when the person who's paying the bills says things like "Re-write? Are you crazy? We don't have time! Stick in this new feature, and have it done by Tuesday!!".

You've already received some great advice -- just pop back here when you run out of brain cells and we'll help you as best we can. Probably the best advice is to version control as much of this as you can -- being able to 'roll back' to the previously working version is priceless. If you're running Linux, check out rcs -- very simple, but very solid. Failing that (like, if you're on Windows), make new sub-directories for each version. Not quite as convenient, but still very useful.

Good luck!

--t. alex
Life is short: get busy!

[Rudif]

version control as much of this as you can

I second that.

Perforce is an excellent VCS too. You can have it for free for 1 or 2 users, and it is available on all usual platforms.

Rudif

[johndageek]

Welcome to the sea of programming. Based on your statement, priorities need to be set.

Base unsderstanding - the evil scary program runs and does it's thing in an acceptable manner. You have little or no programming experience, let alone how these systems fit together. There is a systems person or group who understand and keep the database and web servers al running and in tip top shape:)

If the above is not true and you are the sole IT person, or most knowlageable it person the company has, let us know and we will try a different tack.

You do not have time to flail around learning perljavascriptsqlhtmlandotherbuzzwords.

First priority - get the following information:
1) What kind of database is in use (oracle, mysql etc) and version.
2) What webserver are we running? (apache, Windows etc) and version
3) Are we on the intranet or internet? (security is the issue here).
4) What information is on the reports from the evil program.

Now small step 1) The database, this is your friend in whom all good things are stored. You will want a client connection to the database (sqlplus, toad or torra for oracle). You will use this tool to access the database, get information about how tables are set up, and test your sql statements. Have them set you up with an account for the live database and the test database - you do have a test database I hope??? Ask the database people how to list the tables and columns in the database you will be using. Once you can see the tables and column names, try a select clause to pull some of the data out. You will need to find how the tables are connected. Run the evil report then try to pull the same information with sql.

Having pulled the data successfully (yes I would suggest several tests) Write some perl code using DBI to connect to your database using the same sql statements from above. Dont worry about html or formatting, just get the perl program to pull the data and print it out from a command line.

Success!

Now all you need to do is format with html, our esteemed local monks are much more familiar with packages to help you in this area than I am.

Good Luck!
thegeek

[qq]

Good luck, first of all.

There is a lot of good advice given so far about how to write better code than you've been given, but I'd be wary of making large structural changes given your deadline. It certainly needs it, but I doubt its feasible in the time frame.

So here is a quick and dirty approach:

Do the subs all look roughly the same - if you squint? It may be that there is just one main sub per report and some helper functions. Given that the code is so long, its probably not well structured and may be very repetitous. If the new reports you need to create are similar to the existing ones, you can just copy and paste one of the existings subs and make modifications until it works.

If the subs don't all look the same, and aren't 'one per report', this approach won't work at all.

Another possible route is to not change the existing code at all, but write any new report code from scratch. In the .pl file have a simple exit clause near the top so that if one of the new reports need to be processed, you use the new code, or else fall through to the original mess.

If the existing code is considered broken, this won't work.

Two more random points:

• If confidentiality clauses permit, feel free to post bits of code here for explanation.
• If you are not already use an editor with syntax highliting (TextPad, UltraEdit, anything). It makes a lot of difference, even if you don't use a full blown IDE.

And again, good luck.

qq

[Lori713]

Just some random clarifications:

I wasn't very clear in describing the current "program" I'm taking over. There are two main files, one is the .pl that starts the processing, and the other is a .lib that contains all the subroutines. The .lib file is the monster with about 8000 lines of code.

There are not a lot of comments in the actual .lib file (I will be adding these as I figure out what each subroutine does). Without getting into the politics, my manager had no say over what this programmer did, or left behind. We just ended up being the ones to maintain it.

The global variables do begin with "g_" and the "xxx" part was me being lazy... There are some decent variable names in there (but I'm often confusing them with the "m_" variables with the same "xxx" part).

Another confession: I don't know how to build a test suite, but I'll dig around this website for ideas before asking for input right now on this node.

The subroutines are mostly Print statements to get the HTML to generate on the Web page; I don't see where any HTML template module was used. I have figured out how to do a HERE on my own (I think that's what it's called) so I won't have to type PRINT all the time. I have also looked a little at some of the HTML modules, and I think I'll try to use them.

A lot of the new reports are very similar in look and feel to the existing ones. At this point (and the looming deadline), I'll probably copy one of the reports, clean it up, and use it for a template for the new reports. I'm hoping that by putting my new reports in a separate file I can use -w and use strict.

Thanks for even more great pointers. I'm actually getting a little jazzed about starting in on this wooly mammoth!

[dragonchild]

Ok. A few notes on what you just said:
1. Templates were not used. If they were, you would have been handed a bunch of template files and would not see many print statements. You will want to set up one template for every different page that exists. Templates allow you to separate the form (how it's displayed) from function (what is displayed). There's a lot you can do with this.
2. Don't use heredocs for CGI display. They're just glorified print statements, essentially. You will want to use them for your SQL statements. Something like

   my $sql = <<__END_SQL__;
   SELECT a.foo, b.bar
     FROM some_table a,
          some_other_table b
    WHERE a.id = b.id
      AND a.name = ?
   __END_SQL__
   

   If you're wondering what the question mark is in that sql statement, look up placeholders in the DBI documentation. (Follow the link ...)

3. Test suites. There are oodles of books on the topic of testsuites, and testing in general. There are also dozens of threads at the monastery regarding them. The basics are this:
   • You have a function / script / webpage / whatever that has stuff it has to be able to do.
   • You want to verify that your thingy does what it's supposed to do. More importantly, you want to verify that it will return the appropriate error if it's given bad input. (Testing error cases is more important than testing success cases. Why this is true is left as an exercise for the reader.)
   • Look up Test, Test::Harness, Test::Cmd, Test::Cmd::Common, and the hundreds of other test suites that exist. For testing the CGI functionality, you're probably going to have to go to something like RationalRose or do it by hand. See if the testing group in your company has access to something like RationalRose or the like. Hand-testing forms is really annoying work.
   • Before you actually write your test-suite in Perl (or whatever), write it out on paper. List all the different types of input your thingy can possibly receive. Then (and this is the hard part) think about all the different types of bad input your thingy can possibly receive. For example, let's say you have a subroutine that updates the total of a foo. Your subroutine is given the name of the foo and the value. Are you making sure that the value is actually a number? (Look at Regexp::Common for help with that.) Are you making sure that the name is a valid name for a foo? Stuff can work and still have tons of bugs. (Look at Windows for an example of this.)

As always, feel free to post more questions, ideas, concerns, etc. Many of us (myself included) are more than willing to help you 1-on-1. Feel free to ask! :-)

------
We are the carpenters and bricklayers of the Information Age.

The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

[duffbeer703]

Once you start getting a handle on things, you might find it helpful to use the Eclipse IDE (www.eclipse.org) with perl extensions (e-p-i-c.sourceforge.net). It's great for spotting syntax errors and figuring out what subroutines and regexes do.

[CountZero]

On the issue of IDE's and if your company can spare a few hundreds of dollars, the professional version of Komodo certainly warrants a closer look. I'm quite happy with it as it really speeded-up my programming. Of course, YMMV and many other Monks will surely tell you that the IDE they are using is far superior (and less expensive), but I'm happy with it and that's all I can say.

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[CountZero]

You may be not totally lost yet!

The "small" .pl program probably works as some kind of scheduler, calling the appropriate sub-routines from the .lib file as and when needed.

Mind you, using a .lib file instead of a properly packaged module (.pm) is certainly not considered a standard in the Perl-world.

Anyhow, the fact that subroutines are used in this .lib file is something "good".

Indeed, the way to go forward is trying to figure out what each sub-routine does. Even more important is to find out how the flow of execution runs through all these subroutines.

Knowing that, you can establish some hierarchy, see what data is passed around (in and out of the subroutines; do not forget that "global" variables may muddle/mess up this picture!) and what gets send to the web by which subroutine.

Having discovered all this you can then proceed, to carefully take apart the things which need to be changed and apply the necessary changes. It may seem like a chore but all this preparative work will help you later.

'Time spent in reconaissance is seldom wasted' they told us in the Army (to which one replied "A good scout is a dead scout" (Gen. Sherman, if I'm not mistaken))

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[bobn]

After re-reading the original post, I think I may have inferred more than I should have.

Specifcally,

• I don't know that the code is all in one file, the OP just said "a program", which might or might not mean that


• I don't know that the code is uncommented. The OP said there was little or no doc, which may or may not mean few or no comments.


• I don't know that the globals, stated as beinng form "g_xxx", are really 3 letter names preceded by "g_" - the names my well be longer and more descriptive than I thought.


• I don't know what's in this code - 8000 lines is a LOT of perl code.


So if I take the optimistic view of each of these questions, for all I know, this may not be such bad code. The lack of doc, lack of -w and lack of strict still make me suspicious, but maybe this code can be used and adapted.

I still think you have a very steep learning curve to climb, in that you have to learn a tremendous amount in a very short timeframe to be able to either modify the existing code with good understanding, or to write original code, however simplified, to make the new reports.

In addtion to perl, SQL, HTML and (possibly) javascript, you'll also need to understand webserver function, and a litle Unix experience at least in order to debug, so I am still not convinced that either of these is possible, on your own, by October 31st, even devoting full time to it.

--Bob Niederman, http://bob-n.com

All code given here is UNTESTED unless otherwise stated.

[barbie]

There are several good suggestions here, but none that really highlight the testing on your development platform.

WAMP (Windows-Apache-MySQL-Perl) is just as capable as LAMP (Linux-Apache-MySQL-Perl) for development. Okay there are some Linux dependant features that you can't use, but in the main most of what you are likely to want is portable between Windows and Linux. Apache (1), MySQL (2) and Perl (3) are all available for the Windows platform and with a little effort can all work together.

(1) http://www.apache.org/
(2) http://www.mysql.com/
(3) http://www.activestate.com/

You didn't mention which database you were using, so if not MySQL you might not be as successful at finding a Windows version. Although if your existing database is accessible from your desktop, then you won't need to have a database on your desktop, unless you want it for the learning experience. DBI can connect to a database anywhere, as long as it has permission. If you don't currently have access to your database, it may only require a quick update to the permissions table in the database to let you in. Ask the company's Database Administrator if in doubt.

In the short term there is likely to be a steep learning curve in putting all this together, but in the long run you will have a good experience of figuring out how Apache config files work and getting Perl talking to a database.

Another poster mentioned having a Version Control System, which is a must just in case anything goes wrong. CVS (4) is available on Windows, which is good for basic projects.

(4) http://www.wincvs.org/


Once you've set everything up, the turn around for developing and testing can be much quicker. At my last company, I managed to get the designers, graphics bods and developers all using the same CVS repository and database, but with their own copies of the websites and web servers. They were then able to run the scripts and see the results for themselves before commiting to CVS and then testing on the test server. The subsequent projects were quite often finished upto 50% quicker.

Debugging a CGI script can be painful. However, if you are using Apache then the error logs can be a godsend. However, a quicker error report can be obtained via the use of:

    use CGI::Carp qw(fatalsToBrowser);

Place the above in your CGI script, and any fatal errors will get sent straight to the browser. Though ensure it's remove or commented out for production code.

If you want to use your own trace logs, without interfering with the web server log files, have a look at Log::LogLite. A straightforward mechanism for recording processing information, with the added ability to decide the priority of a message, thus enabling you to filter what message to trace without having to rewrite huge chunks of your code.

Alternatively, is it possible run portions of the script stand alone? If so then test scripts, like the ones found in many CPAN distributions may be a guide to help you. Test::MockObject is a great help if you want to test code without having a connected CGI or DBI interface.

Several posts have mentioned templating. If all the HTML appears in your code, whether via straight prints or using 'Here Documents', then this can create all sorts of problems. It maybe too overwhelming to transpose these into a templating system now, but it is definitely something to think about. There are several good templating modules/systems recommended here and it would be a good idea to have a look through them to see what suits you best. By having a templating system you may have found that much of the code to create each report is actually duplication of effort. Thus a new report can be as simple as writing a new subroutine that extracts the data and stores it in a format ready for the presentation process, then creating a new template to present the data. Quite possibly a half a day job or less for each report. And what happens when your boss decides it has to be in XML format or any other format in the future? Templating allows you to change the presentation layout without any changes to the data extraction and preparation.

Learning HTML and JavaScript isn't too demanding and there are many books and online resources which you may have found already.

However, I would concur with others that JavaScript is not something you should be too eager to rely on. There are many aspects of JavaScript that are useful, such as the form validation aspects, but anything that comes from the outside world should still be validated server side too. 'Taint' is the keyword here. Node 80037 might be of interest.

From your subsequent post I suspect you're on the right track already and may have found your job of producing the new report a little easier than you first thought. There is much food for thought here, so I hope it doesn't overwhelm you. Good luck and welcome to the community.

--
Barbie | Birmingham Perl Mongers | http://birmingham.pm.org/

[Lori713]

Based on the advice above, here are some updates:

(1) I've loaded Eclipse and the Perl editor plug-in, and I've gotten it to work. It's given me a lovely list of modules and subroutines.

(2) We are on Sybase 12, Apache for the Unix box web server (I think - see (6) below), the reports are generated on the internet, and the reports present financial info (i.e., balance sheet, income statement).

(3) I've ordered some books that should be here this week (Debugging Pearl, the cheetah book) using my own funds. I work for the State, and we have NO money right now (they've actually starting laying off folks recently).

(4) The DBI module is not currently being used but I think there's a connection established using the Sybase::CTLib module.

(5) I am not the IT/DBA; I'm a reporting specialist who develops financial reports for our campus community.

(6) If I understood correctly, I can set up my desktop to run scripts. I currently FTP the edited text file to the Unix server, and then open a browser and type in my URL. This is not a difficult process, but it can be annoying at times. Confession: I'm not sure how all this stuff works together (i.e., the web server versus the Unix server versus the database server (BUT I will look this up on my own later today). I also plan to download Apache to my PC once I confirm that our Unix box uses that for its web server.

(7) There is minimal javascript being used, and it's pretty straightforward and understandable. I still have concerns about security (about which I know NOTHING at this point), but I'll talk to our DBAs about security issues later this week to make sure I'm covered.

(8) I have learned (painfully through experience) how vital version control is, so... 'nuff said on that.

Questions:

(a) What is the difference among the following: module, function, subroutine, script? I'd like to make sure I'm using the right terms (and understanding some of the advice given). CombatSquirrel mentioned creating some modules to help organize things. Is it best practice to have multiple "files", or have this similar-report generator all in one file? I'd appreciate advice on this. The main menu will have 7 reports, with a spot for input and a drop-down to select an accounting period. The reports will pull similar data and have a similar look, but there are differences among them.

(b) Is there any way to pull a list of variables used within a file/script/program? The Eclipse editor does a nice job of finding the subroutines and modules, but I haven't found a way to pull variables (I'd like to understand where these variables are initialized, how used, etc.).

Whew! Enough for now. I'm going to have to pitch a tent to block the evil fluorescent lights...

Thanks for all for the input and advice! Lori

[dragonchild]

• Function / Subroutine / Procedure - In Perl, these terms are interchangeable. (In other languages, like Pascal, function means something different than procedure, but don't worry about that.) This represents a grouping of statements, usually with a defined set of input and a defined set of output. You want these because they will simplify your life. (Instead of writing the same 10 lines over and over, just with different variable names, you create a function and pass the variables to it.) "sub" or "func" are common abbreviations.
• Script - this is a Perl executable. Your pages / reports will have a script associated with them. A script has a bunch of lines of code in it, many of which are usually in a bunch of subs. It will almost always farm out some of the work to a bunch of modules. (For example, CGI is a good example.)
• Module - a separate file that contains a bunch of functions which perform similar or related work. For example, the CGI module has over 100 different functions, all of which are related to input from and output to the Web. There are literally thousands of modules on CPAN and you are encouraged to write your own.
• Package - while often interchangeable with Module, this word has special meaning in Perl. It describes a namespace. Usually, one has one Package in one file, but not always.

As for your design, I would reccomend the following:

• Have your basic reporting work be in one module. This would include all the queries to the database and any basic transformations on the data that comes back that might be needed. A good rule of thumb is that if you have the same (or very similar) code in more than one place, you should put it in a module.
• Have one script for each report. This is where you keep what is unique about each report. It could be a formatting thing, or maybe one report is a detail report vs. another which is a summary report. That is the stuff that goes into the script that is specific to a given report.
• Have one script for each screen. So, you probably have a login screen, a choose-report screen, etc.
• When you get to using templates, you will want to have a template for every different screen you want to present to your users. So, you'd have a template for each report and a template for each screen.

------
We are the carpenters and bricklayers of the Information Age.

The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

[CombatSquirrel]

Okay, to question (a):
• module: A file (*.pm) that contains one or more packages
• package: The Perl equivalent of a C namespace; encapsulates subroutines or objects
• script: A program, so to speak. Just that it is not executed, but interpreted by the Perl interpreter
• subroutine: You could call it a procedure or a function. It is declared with sub blabla { ... and does the stuff you tell it to do
• function: Does not really exist in Perl, but would be (if you think Pascal), a sub that returns a value. Just understand it as sub

To (b): I think there is a way, but I could not get it working :/. Hope the others have ideas...
Hope this helped anyways.
CombatSquirrel.

---

Entropy is the tendency of everything going to hell.

[pboin]

I'll just skip the technical side of things. You seem to be getting your ducks in a row, and there's *tons* of good advice in other replies.

One side that IMO hasn't been addressed much, is your responsibility to yourself and to your management to get their expectations exactly where they need to be. If I were in your shoes, I would have expressed *huge* doubts about the October deadline. Maybe you'll make it, and maybe you won't, but you should be telling them when it can be done, not have them tell you when it *will* be done.

You may piss some people off. (I've literally had people red-faced and jumping up and down.) However, once you do it a few times, and people know your word is good, life becomes much easier on you, and your organization will be better for it.

Do not bow to pressure -- the truth will set you free.

[Lori713]

It's funny you should mention that... I've recently expressed my concern over this massive a project (given my limited programming experience and knowledge of the languages involved). He's taking it to the committee that decides on deadlines, and I should hear something back soon (a little more complex that that simple statement makes it sound - we're in the middle of a massive software upgrade). I'm pretty sure the deadline for these new reports will be pushed out. Cross your fingers.

On the other hand, the best you can do is the best you can do, and I try to duck under the political radar.

[PERLscienceman]

Greetings Lori713 and Welcome! I see (in the beginning of your post) that you are looking for books related to Perl, Java, HTML, etc. Somewhere in my digging around I found these links to a veritable compilation of books related to what you were looking for:

www.perl-books.com
www.javascript-books.com


I thought that I would bring these to the forefront so that you could have easy access to them.

I am also partial to the O'Reilly "Animal Books" I have found everyone of their books that I have encountered to be quite useful so here is their main link:

www.oreilly.com

Also... Take a look at Book Reviews and see what Perl Monks have to say about Perl related books.

Anyway good luck... I am certain that you will find your fellow Perl Monks to be a great resource! I know I have!