Refactoring

[stephen]

I've been using refactoring techniques for Perl programming, recently, and I was wondering about others' experiences with it. I'll start by defining refactoring and how I do it... there is an excellent book on the subject, Refactoring: Improving the Design of Existing Code, by Martin Fowler. The examples are in Java, but many of the refactoring techniques described therein can be applied to Perl.

Anyway, like the book title says, refactoring is the process of taking existing programs and improving their designs, step-by-step. I find it helpful when in the course of my job I need to add features to legacy code that someone else (or a previous, less learned version of myself) wrote. You can boil down refactoring to these steps applied repeatedly:

• Pick a patch of code that's desperately in need of improvement. (A subroutine, for example.)
• Write a small test program that tests the current functionality of that piece of code. (NOTE: Not the expected functionality... this is not bug fixing or feature addition. The current functionality.)
• Check your tests and code into some kind of version control system, so that you can always retreat to a previous version if you cause a bug.
• Make a small change that makes the code easier to comprehend and enhance. (For example, pull a small function out of a larger one. Turn a global variable into a function call.)
• Run the test you've just defined, plus any other tests you've defined for the program.
• Repeat.

Using the Test and Test::Harness modules that come along with Perl saves a lot of time while writing the tests.

In his book, Fowler defines a list of "refactorings"-- small procedures for improving code. You can get a list of them at his website, http://www.refactoring.com. I'm working on a similar list focused on Perl, and hope to put up a website collecting them. An example refactoring for Perl would be:

Transform Global Variable To Function Call
Indication: You've got a global or package variable that is being used by a number of different functions to communicate some value.
Solution: Replace the global variable with an exportable function which returns an equivalent value.
Steps:

1. Extract the part of code that initializes the global variable into an exportable function.
2. Tie the global variable to that function.
3. Test.
4. Replace instances of the global variable with calls to the new function.
5. Test.
6. Untie the global variable from the function.
7. Test.

Other Perl-oriented refactorings would include:

• Unreinvent Wheel: replace needlessly rewritten code (like a CGI parser or templating system) with a call to a CPAN module.
• Strictify Module: Gradually remove errors under 'use strict', testing as you go, until you can finally run the module under 'strict'.
• Localize Variable: A variable is used only in a single subroutine. Good! Localize it with 'my'.

These would be better with some examples; I'll post some as followup if people are interested. Many of these things are obvious, common-sense things to do, but it's frequently useful to have them written down someplace so that one isn't tempted to skip a step.

Refactoring is extremely useful to me because it gives me a middle-ground alternative between living with incomprehensible code and tossing and rewriting a large Perl project. For small projects it's not such a problem to rewrite from scratch, but I've inherited more than a couple of huge projects consisting of poorly-written RUNNING code. It's my experience that all specs and documentation for such projects are completely out of date. No one knows everything that the software does, but people are out there using it, and management isn't interested in funding a long rewriting project that may lead nowhere... they just want this one feature added. Oh, and this other one. Oh, and this other one, too...

Sorry to have run on for so long, but it's a big topic and I wanted to at least scratch the surface of it. What experiences do people have with this sort of thing? Any refactorings that you find helpful? Any questions?

stephen

Update: I am now maintaining a list of Perl refactorings on my homenode.

[chromatic]

The most important part of this is testing. Test, test, test.

If you don't know what your code is supposed to do, if you can't verify that it does what it's supposed to do, you can't ever refactor it. That's a problem.

A normal programming language gives you different ways to solve the same problem. Perl gives you more. If you've done a reasonably good job of encapsulation, you can often tweak things to improve them.

You won't be free to do this until you have external verification that the code you modified has the same (good) effect that it did before.

Refactoring is good. Refactoring without testing is nearly impossible.

Update: Okay, okay. Confident refactoring without testing is difficult. Avoid it if you can.

[dws]

Refactoring is good. Refactoring without testing is nearly impossible.

That's a bit strong. While cleaning up or migrating legacy code, I've done a lot of refactoring without having a solid base of unit tests. It can be done, but it's tough on the psyche.

I think a more accurate statement is:

It is difficult to refactor with confidence without having a complete set of regressions tests that cover the code you're refactoring.

In fact, having adopted the "write your tests first" dictum from eXtreme Programming, I'm finding it uncomfortable to do any development without having a set of regression tests at hand. An infectious notion, that.

[arhuman]

That seems to be a great Idea to me.

I like this concept, for it seems practical,
I've seen so many way of programming with a proven/therorical/formal design that were just unusable...

I REALLY like this method (paradigm?) beccause :
• The idea is simple (iterative 'enhance then check' step).
  
• The experience can be saved and transmitted (by describing refactoring recipe.)
• It seems that the process can be made partially automatic.
• It can be applyed to EXISITING project AND new ones.

However I'd definitly like to have more info on this (other links, books) and especially opinion of people who have been using it.
A step by step practical example of refactoring using Test, test::Harness on an existing code would be welcome too...

Anyway, thanks for this info, I've never been so excited about a programming method for long... (stephen++ ;-)

"Only Bad Coders Badly Code In Perl" (OBC2IP)

[Rudif]

You can get a list of them at his website, http://www.refactoring.org.
Nope, the correct URL seems to be http://www.refactoring.com.

I like the XP ideas and I practice them, though not formally and eXtremely. I tend to write and run scripts for functional testing of my modules as I develop them.

stephen, do you practice unit testing and did you use PerlUnit modules from http://www.xprogramming.com/software.htm?
If yes, could you comment on your experience? How does it relate to the standard Test and test::Harness modules?

Rudif

[stephen]

My testing practice is that I refuse to modify code unless at least one test has been defined on it. That lets me apply tests where they're needed, but means I don't need to interrupt development until I can develop a battery of tests for all known functionality. I use (as I suspect you do) the standard Test and Test::Harness modules, and am more-or-less satisfied with them.

Initially, I took a look at the PerlUnit module on xprogramming. In my humble opinion, it's not too well documented and lacks a standard install. Plus, PerlUnit seems to be by default packaged in a ZIP file-- slightly more Windows-specific than I'd prefer.

The CPAN TestUnit module seems more versatile and mature, and I'm a bit surprised that the xprogramming page doesn't link to it instead. (No offense to the author of PerlUnit, whoever she may be.) TestUnit generates both xUnit-friendly and Test::Harness-friendly output. (This is based primarily on reading the docs, not from actual use.) It's based on JUnit, which I've used and I like. Like JUnit, it has good support for setup and teardown of tests. I intend to try using it on future projects.

In terms of actual testing functionality, I haven't seen too much out there that the standard Test module doesn't offer. Test::Simple gives a way of naming tests. Test::Cmd is best for testing the output of external scripts and programs.

For CGI scripts, CGI::Test allows you to run your scripts from the command line as though they were running in CGI context. I haven't tried using it yet, but I will as part of this discussion, and will post the results here.

In actual practice, I use the regular Test module (like I said) and supplement it with some modules of my own, which I'm working on packaging up for CPAN (hopefully sometime this week). The primary things that Test is missing, in my experience, are output tests. Things like:

• Comparing the printed-to-STDOUT output of code against a pre-stored file
• Comparing e-mail sent to a test user against a pre-stored file
• Checking that records inserted into a given database match a hash or array of hashes

Anyway, I'm working up some examples, both of applying refactorings and of tests, and will be posting further followups. There's a great deal to explore here, and PerlMonks is the place to do it!

Update: Corrected a misstatement about Test::Cmd.

stephen

[Rudif]

Thank you, stephen, for sharing your experience and for pointing out the CPAN module TestUnit - it looks promising.

Rudif

[tye]

I was wondering if it was just me... These types of changes aren't at all what I think of when I hear the term "refactoring". At the least, these would be at the very bottom of the scale.

To me, refactoring a program is like refactoring in math. In math, refactoring could be used to change (x^2-2x-3)(x^2+2x-3)to (x^2-1)(x^2-9) So I tend to think of refactoring as redesigning the boundary between the pieces that make up your program, often allowing code that is repeated to be removed.

I have a hard time coming up with good specific examples as refactoring is usually rather complex in my book. Some rather lame examples might be taking a handful of global variables and putting them into an object or changing a subroutine that takes a ton of arguments into one that takes a hash of options where most of them are optional with good defaults so most calls no longer need to specify them.

I guess the examples given so far just seem too trivial. (:

        - tye (but my friends call me "Tye")

[gumpu]

A very nice site, with the same wealth of information as perlmonks is wiki. You can find tons of info about refactoring, unittesting, and extremeprogramming there (and lots more information about programming and design principles). It's a wonderful site. Each page contains links to many other pages, and there is well balanced discussion. You can spend hours there just following all the links and reading.

Have Fun.