Best practices and financial applications

[cleverett]

I am going through a phase where I really need to grow and develop professionally. In a recent post I was talking about what aspects of Perl (00, first class functions, etc) my group is prepared to use in our code.

I am about to stake out a position based on the responses to that post, which boils down to:

1. In the short term:
   • Run everything through perltidy
   • When we work on a .pl file, run it through perlcritic and fix the most pressing issues it finds
   • Write test code, where we easily can
2. Build in house code libraries for the things we do over and over which are specific to us, and plug them in as part of any fixes we make to existing code.
3. For new Perl OO work, use a well developed and well exercised Perl OO framework, rather than repeatedly coding OO bits and bobs from scratch. I'm leaning towards recommending MOOSE and DBIx::Class (if that's even possible given what we have).
4. Work towards being able to implement test driven development.

The reality is that the business side of the company will only release funds for patching code, not for doing wholesale rewrites from scratch. But we are hoping that over time we can adopt a strategy to improve the code in fundamental ways over the next few years.

If anyone has any ideas or advise, please, please, please chime in. Especially those of you who are doing financial (playing with actual dollars and cents, or Euros, Yen, Yuan, Pounds, what have you) stuff.

[BrowserUk]

The reality is that the business side of the company will only release funds for patching code, not for doing wholesale rewrites from scratch. But we are hoping that over time we can adopt a strategy to improve the code in fundamental ways over the next few years.

It sounds, (on the basis of very scant information, I may well be jumping to the wrong conclusion), like you are trying to expand what is primarily a maintenance role, into something more. Whilst I fully understand that motivation, I've been there myself, I strongly urge you to proceed with extreme caution.

If you break something during the process of refactoring for the sake of 'improving the codebase', or 'fixing' something that hasn't been explicitly documented as broken--by the owners or users of the codebase, not you or your team--then your good intentions will not save you from their wrath.

• Your very first priority should be to make sure that you get the code--every last line of it--into a version control system (VCS) if it isn't already.

  And if it is, make sure that a very clear checkpoint is made before you touch anything. Even supposedly benign things like beautifying with PerlTidy.

• Next, you need to set up regression tests.

  I'm not talking about cutesy ok/not ok one-liners. It is almost impossible to safely apply unit and functional verification tests to an existing codebase. Even if the code has been meticulously commented and/or documented, what they say the code does, and what it actually does are nearly always two quite different things.

  All code contains bugs. And in an established system, both up-stream and down-stream processes will have knowingly or inadvertently been constructed so as to work with or around those bugs. The simple act of correcting something obviously wrong in your part of the process can bring the system down around your ears.

• You need to obtain a complete set of real (or if privacy/security reasons prevent that, at least very realistic), inputs (and outputs as produced by the current codebase), exercising as much of the functionality as possible.

  And set up a test environment that allows you to verify that when you supply those inputs, you get back those outputs. This becomes your "spec" for what the system currently does. Which may be markedly different from what people and/or any written specs say it does.

  Only once you've successfully exercised the current code in your test environment and verified that you can produce the same outputs given realistic inputs, should you even consider making 'proactive' changes to the system. And then...

• Fork the codebase in the VCS, and confine your proactive changes to the fork whilst performing documented, assigned bug fixes to the original.

  Don't be tempted to merge the two back until you've a) proven it thoroughly through the test setup. b) documented the business benefits of the changes.

Move cautiously. Get sign-off for rolling unassigned refactoring back into production systems.

[djp]

+++++ to this reply, in my opinion this is some of the best advice I've ever seen on Perlmonks.

[cleverett]

Mostly, we're happy with doing maintenance. And our number of outstanding defects is way down. But the code we have, presents another set of risks (mostly long term) we'd like to mitigate.

We have a big commitment to release management here. We develop on one box, use another box to create and test release builds and do integration tests. After that, code migration and the DBAs run the release builds out to yet another platform for another group to do user acceptance testing, before anything goes to production. And, we try to keep a consistent environment all the way from development to production.

I couldn't be irresponsible and keep my job, even if I wanted to ...

And yeah, we have *everything* in a version control system. Actually we have 2: svn for day to day development and creating release builds, and a corporate repository run on another version control system that we commit releases to.

Regression testing is dependent on test data. We have a test data library with 30 days of anonymized production data. I'm working on a tool to use files from that library as templates for generating a stream of test files for out daily batch processes, by changing some dates and numbers in them.

The same tool should be usable for modifying another set of templates to create expected outputs. Regression testing should be straightforward from there.

[BrowserUk]

You are obviously well set up and ready to go.

Then my next step--after running everything through PerlTidy to get a consistant layout, and making sure it doesn't break anything--would run it all through perlcritic.

Or rather, I'd run one or two of the larger pieces of code through, and then manually cross-reference the output with the code, and decide which of its protestations are bogus, and work out how to turn them off. Once you've turned it down to the point where it is only bellyaching about things that you agree are worthy of note, then it would be a fairly painless process to then pass the entire codebase through.

That would give you two things:

1. An indication of where to start concentrating your refactoring efforts.
2. A set of documented (PBP) reasons that you can take to management as justification for the refactoring efforts.

Maybe that suggestion will go some way for my grandmother-eggs-vacuum thing. It's often difficult to access where someone is at from the wording of their questions.

[ww]

Don't cut the dog's tail off by inches:

Write test code, where we easily can

and

Work towards being able to i Implement test driven development.

This Rx has problems of course; problems beginning with "business side of...." but that's a copout too! The finance folk aren't going to shift their approach unless and until you demonstrate that swabbing a little money on the problem (that you recognize and you need to make them understand) will save a grundle of $,yen, euros or whatever in the (short-term) future.

[stvn]

For new Perl OO work, use a well developed and well exercised Perl OO framework, rather than repeatedly coding OO bits and bobs from scratch. I'm leaning towards recommending MOOSE and DBIx::Class (if that's even possible given what we have).

FWIW, Moose is also built to work well with existing CPAN modules and non-Moose OOP, the MooseX::NonMoose extension makes it pretty much painless. It works best with HASH based objects of course, but other people have had success subclassing GLOB based objects (see MooseX::GlobRef::Object and it's SYNOPSIS which shows it subclassing an IO::File object). Beyond that you can use the "handles" option in Moose to do delegation to any other type of Perl object. One caveat is that handles => REGEXPR will not work with objects that use AUTOLOAD because it cannot determine the method list to filter with the regexpr, but this can be worked around by simply specifying the full method list yourself.