How to define a package using a tweaked version of LWP::UserAgent?

[pwolfenden]

Folks,

I've written a small suite of methods using LWP::UserAgent to connect to a commercial API and do some useful things - like log in and out, and log out automatically if the script exits prematurely, etc.

I decided to put this logic into a Perl package (let's call it "My::Client" for the sake of this discussion), so I used h2xs to build myself a nice Makefile and everything was going nicely until I realized that in order to allow users to specify HTTP proxy settings (URL and credentials) on a "per client" basis (instead of setting $ENV{'HTTPS_PROXY'}, etc.) I would need to tweak the behavior of the get_basic_credentials() method. This is something that the LWP::Useragent documention suggests can be achieved by subclassing the package and overriding the get_basic_credentials() method (and for my purposes I also want to add a set_proxy_credentials() method). Let's give the name "RequestAgent" to this subclassed version of LWP::Useragent (in honor of the RequestAgent package defined in lwp-request).

I first naively attempted to "nest" the RequestAgent package declaration inside the declaration of My::Client, but of course that didn't work. After reading a bit more about Perl packages in general and AutoSplit in particular, I decided to define the RequestAgent and My::Client packages in the same file. But that didn't work, either. It seems that MakeMaker insists on having a nice match between the name of the package file and the name of the package defined inside.

So then I tried defining RequestAgent in a separate package file (with its own h2xs-generated suite of associated files), and I was able to build that just fine. But the My::Client package needs to "use" the RequestAgent package, and this made things complicated when I installed the packages in a local directory. Although I believe that all the use/include issues that I encountered can be overcome via the judicious use of @INC-modifying BEGIN blocks, it seems silly to have to pollute the Perl package namespace this way when all I really want to do is make local use of a trivial modification to the LWP::UserAgent package.

So I'm wondering - perhaps it would be better to modify the package LWP::UserAgent itself so that callback methods can be passed to setter methods at runtime (as function references) rather than being overridden by derived classes at compile time? That seems more in line with my notion of what a "callback method" ought to be in any case.

What do y'all think?

Thanks in advance for your time.

PS - I'm using a Perl 5.6.1 distribution.

[moritz]

If you want to write a second module, you don't have to start a whole new distribution. When you created your My::Client distribution, it will have a module in lib/My/Client.pm. Simply add a file lib/RequestAgent.pm.

Although I believe that all the use/include issues that I encountered can be overcome via the judicious use of @INC-modifying BEGIN blocks

You mean something as complicated as use lib 'lib/'; ?

PS - I'm using a Perl 5.6.1 distribution.

That's lagging two major versions behind, currently 5.10.0 is the last stable distribution, and 5.6.1 is not maintained any more. Do yourself a favour and upgrade, there are many good reasons why 5.10 is loads better than 5.6.1

[Corion]

(except of course an existing code base that works with 5.6.1 and lack of resources for doing the testing necessary before an upgrade, plus the hassle of recompiling and reinstalling the modules with external dependencies)

[moritz]

You've got a point there. But on the other hand, what would you do? Wait another few years, and upgrade then? Or hope that whatever you do will become superfluous soon, or will be re-done from scratch?

Or spend years maintaining an abandoned perl version, working around myriads of Unicode bugs, memory leaks in closures and similar hassles?

The original questions shows that he is actively developing stuff, and slowly new versions of CPAN modules require higher minimal perl versions. Which means that you also have to start maintaining old versions of CPAN modules on your own.

Working around a broken environment is a daunting and time killing task. (I didn't program perl when 5.6.1 was state of the art, but from what I heard on various IRC channels it seems to be broken, compared by todays standards and compared to what's available today)

I don't know if there are viable solutions, but continuing to use old stuff certainly isn't very future proof, and the longer you wait the larger your system grows, and the hard it will become to upgrade in the end.

[MidLifeXis]

And don't forget my favorite -- having a vendor of a third party library that you have to use starting development using the newest library only when pushed by their clients.

An application I have to support is about 1-2 years behind supporting newest OS / DB / Perl (for example, they are still validating and compiling against 5.8.x, just released Oracle 10g support in January, and have now clue what the current revision of HP-UX is). We don't do "dot-oh" releases, and we have a boatload of local apps to validate against their release, putting us even further behind.

It is a never ending battle to try to have them find out where we are or would like to go a couple of years out rather than asking the question where are you now (yes, they would ask surveys based on current platforms, and then come to the conclusion that moving forward was not a priority; thankfully the surveys have started to change).

[pwolfenden]

Indeed, as you suggest I can put the RequestAgent.pm file into directory 'lib' and add "use lib 'lib';" to Client.pm prior to "use RequestAgent;".

This works as desired when I use the -Mblib command line argument to tell Perl where to find my libraries, if I the "use lib" directive in Client.pm contains a path for 'lib' which is relative to the directory specified by -Mblib. This is good.

But when I try to run "make test" from the lib/My/Client directory, I find that I get the following error:

Can't locate RequestAgent.pm in @INC (@INC contains: lib blib/arch bli
+b/lib /usr/lib/perl5/5.6.1/i386-linux ...(snip snip)... .) at blib/li
+b/My/Client.pm line 18.

*unless* I either:
1. tweak the "use lib" line in Client.pm gives the *full path* to 'lib' (horrors), or
2. move RequestAgent.pm into lib/My/Client/ and tweak Client.pm to "use lib 'lib/my/Client'"
Since I hate hardcoded file paths I find myself forced to use the second option.

But this seems lame, since in order to properly maintain both packages in the same directory I'll need to manually hack the local Makefile - or become enough of a MakeMaker guru that I can intelligently hack the upstream Makefile.PL instead.

Am I missing another option (perhaps something more obvious) here? Again, my goal here is simply to cleanly override the LWP credentials handler.

[moritz]

But when I try to run "make test" from the lib/My/Client directory

Why would you want to run 'make test' in that directory? Usually the directory structure for a module looks like this:

Makefile.PL
Makefile
lib/My/Client.pm
lib/OtherModules.pm
t/test-one.t
t/test-two.t
# other stuff
META.yml
important-script.pl
README
examples/example1.pl

If you adhere to that standard, your Makefile lives in the root directory, and the tests in a directory sub directory of the root dir.

So running make test on the lib/Foo/ level would actually complain about a Makefile not being found. So tests are always run from the root level directory, in which case make tests will invoke your test scripts as perl t/test-file.t.

In this case a use lib 'lib'; in all scripts that are invoked with the root dir as the current working directory.

[pjotrik]

It shouldn't be a problem to 'use' your package:

use lib '/absolute_path_to_RequestAgent';

or

use lib $FindBin::RealBin.'/relative_path_to_RequestAgent';

[pwolfenden]

Re the $FindBin::RealBin idea: since the My::Client package is for use by clients on client machines we can't assume anything about the location of the running script in the filesystem. So unfortunately we cannot express the path to my package file relative to $FindBin::RealBin. And hardcoded paths are out of the question, since the My::Client package may have been installed anywhere.

I think we can safely assume that it is the responsibility of the author of any script which uses the My::Client package to tweak the @INC array (whether via "use lib", -Mblib on the command line, etc.) so that the Client.pm file can be found. But once this has been done IMO it should be the responsibility of the My::Client package to include the My::RequestAgent package.

[spacebat]

This might sound inelegant, but what I've done in the past is inject a subroutine into the LWP::UserAgent package that occludes the original get_basic_credentials(). I define this routine either to supply the credentials directly, or make it a closure so I can use it to install a callback.

I know its probably pretty far from best practice. I also think having to subclass the useragent to supply credentials is a bit naff.

[pwolfenden]

How exactly did you "inject" the subroutine? I'd love to see an example where you make it a closure and then install a callback method.

Thanks!