Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

How to unit test code that used LWP

by chrestomanci (Priest)
on Dec 17, 2012 at 10:56 UTC ( #1009146=perlquestion: print w/ replies, xml ) Need Help??
chrestomanci has asked for the wisdom of the Perl Monks concerning the following question:

Greetings wise brothers.

In the project I am working on, I have some code that attempts to fetch files over http via LWP. I am confused at how I can write unit tests for that functionality.

The overall design is that I have a global database that lists all the files available, along with the size and checksum, and a list of http servers that may or may not have a cached copy of any particular file, so when a client needs a file it might have to try several servers before it finds one that has the one it is looking for. In simplified form my code looks like this:

my $fileObj = $schema->resultset('Files')->some_query() # A DBIx::Cla +ss result object. my $ua: # An LWP::UserAgent instance HOST: foreach my $host ( @hosts2try ) { my $leafname = $fileObj->leafname(); my $file_url = 'http://'.$host.'/'.$leafname; $logger->debug("Will try fetching $leafname from $host"); # First check if the remote host has the file & the size is correc +t. my $response = $ua->head($file_url); if( ! $response->is_success() ) { # Not a problem as not every host has every file. $logger->debug("Looks like $host does not have $leafname. Will + try another host"); next HOST; } elsif( $imageObj->size() != $response->headers->header('content-le +ngth') ) { # An error. If a host has a file it should be correct. my $remote_size = $response->headers->header('content-length') +; $logger->error("Host $host has a copy of $leafname but the siz +e is wrong. Is $remote_size bytes but should be ".$imageObj->size() ) +; next HOST; } # Looks like the remote server has the file. Download it. $logger->info('Fetching '.$file_url); $ua->get($file_url, ':content_file' => $tmp_path); if( -f $tmp_path ) { if( ( $imageObj->size() == -s $tmp_path ) && ( $imageObj->checksum() eq get_sha1_checksum($tmp_path) ) ) { $logger->info("Successfully fetched $filename from $file_u +rl"); move($tmp_path, $final_path); return 1; } else { $logger->error("Serious: $filename fetched from $file_url +was truncated or corrupt") unlink $tmp_path; next HOST; } } else { $logger->error('Error fetching: '.$file_url); next HOST; } }

For the unit testing, I would like to cover the case where a remote host does not respond, does not have a file, and where the fetched file arrives truncated or corrupt. The unit tests need to fit nicely into the codebase and not require much in the way of external dependencies. The other developers on the project won't mind installing another Ubuntu package, but they will complain more if I ask them to install something from CPAN. There would be no point asking them to run a local web server on their machines, or suchlike just to run unit tests.

So far I looked into Test::LWP::UserAgent which looks promising, but is not available as an Ubuntu package, and Test::Mock::LWP which is packaged for Ubuntu, but does not have much helpful documentation. With both those mocking modules there appears to be a problem in that they don't pay attention to the hostname portion of the URL, so I can't easily simulate the case where one host has a file and another does not.

I am also considering just using Test::MockObject::Extends to mock the LWP::UserAgent and HTTP::Response objects. I also asked in Chatterbox on Friday, but the suggestions where not that helpful.

Can you offer any suggestions? I can't believe that this has not been extensively covered before, but a search here did not turn up anything recent or useful.

Comment on How to unit test code that used LWP
Download Code
Re: How to unit test code that used LWP
by Anonymous Monk on Dec 17, 2012 at 11:49 UTC
    use inc::Module::Install ; auto_bundle; and ship any testing modules your distribution requires
Re: How to unit test code that used LWP
by Corion (Pope) on Dec 17, 2012 at 12:38 UTC
    I use two approaches. The harder but more versarile approach is to distribute and spawn a custom webserver that exhibits the intended (error) behaviours under well-known URLs. See WWW::Mechanize and WWW::Mechanize::Firefox for examples of that. The other approach is to use fike:// URIs in the strategic places to return (broken) content with status 200 (or 404 for "not found").
Re: How to unit test code that used LWP
by tobyink (Abbot) on Dec 17, 2012 at 20:15 UTC

    Test::LWP::UserAgent is pure Perl, right? So if it's not available as an Ubuntu package, you could just bundle a copy in your project's t/lib/ dir.

    The other possibility is to do things the other way around, and bundle a test HTTP server; then you can use a real client. I often do this; e.g. the test suite for IO::Callback::HTTP.

    perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'

      Thanks. Test::HTTP::Server certainly looks useful, though there is the issue of which server is contacted. And you are right, Test::LWP::UserAgent is pure perl, so there is nothing stopping me checking it into the source tree, other than an irrational worry that doing so could be the start of a slippery slope that will end with half of CPAN checked in, and the tree growing to many gigabytes in size.

      In the end I decided that the simplest way to solve the problem was to directly mock the get and head methods in LWP::UserAgent. My mocked verson of head looks like this:

      sub mocked_http_head { my($self, $url) = @_; ( $url =~ m!http://(.*)/(.*)! ) or die "malformed URL"; my $hostname = $1; my $file = $2; my $test_path = '/tmp/fake_server/'.$hostname.'/'.$file; my $response = Test::MockObject::Extends->new('HTTP::Response'); if( -f $test_path ) { my $fakeHeaders = HTTP::Headers->new('content-length' => -s $t +est_path ); $response->set_true('is_success'); $response->mock('headers', sub{ return $fakeHeaders } ); } else { $response->set_false('is_success'); } return $response; }

      Using this approach I can easily simulate as many servers as I like, each with different content. I can simulate wrong content or corrupt downloads by putting a file in the /tmp/fake_server directory that differs from the database. The only thing I can't easily simulate is a server that is not listening (so LWP times out) or an aborted download so that the size from the content-length header differs from the file actualy delevered.

      The other thing this does not simulate is any server side scripting, but that is not a requirement for what I am doing.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1009146]
Approved by McDarren
Front-paged by Tommy
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2014-09-16 22:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (51 votes), past polls