How to unit test code that used LWP

[chrestomanci]

Greetings wise brothers.

In the project I am working on, I have some code that attempts to fetch files over http via LWP. I am confused at how I can write unit tests for that functionality.

The overall design is that I have a global database that lists all the files available, along with the size and checksum, and a list of http servers that may or may not have a cached copy of any particular file, so when a client needs a file it might have to try several servers before it finds one that has the one it is looking for. In simplified form my code looks like this:

my $fileObj = $schema->resultset('Files')->some_query()  # A DBIx::Cla
+ss result object.
my $ua:      # An LWP::UserAgent instance

HOST: foreach my $host ( @hosts2try )
{
    my $leafname = $fileObj->leafname();
    my $file_url = 'http://'.$host.'/'.$leafname;
    $logger->debug("Will try fetching $leafname from $host");

    # First check if the remote host has the file & the size is correc
+t.
    my $response = $ua->head($file_url);

    if( ! $response->is_success() )
    {
        # Not a problem as not every host has every file.
        $logger->debug("Looks like $host does not have $leafname. Will
+ try another host");
        next HOST;      
    }
    elsif( $imageObj->size() != $response->headers->header('content-le
+ngth') )
    {
        # An error. If a host has a file it should be correct.
        my $remote_size = $response->headers->header('content-length')
+;
        $logger->error("Host $host has a copy of $leafname but the siz
+e is wrong. Is $remote_size bytes but should be ".$imageObj->size() )
+;
        next HOST;
    }

    # Looks like the remote server has the file. Download it.
    $logger->info('Fetching '.$file_url);
    $ua->get($file_url, ':content_file' => $tmp_path);

    if( -f $tmp_path )
    {
        if(
            ( $imageObj->size() == -s $tmp_path )
            &&
            ( $imageObj->checksum() eq get_sha1_checksum($tmp_path) )
        )
        {
            $logger->info("Successfully fetched $filename from $file_u
+rl");
            move($tmp_path, $final_path);
            return 1;
        }
        else
        {
            $logger->error("Serious: $filename fetched from $file_url 
+was truncated or corrupt")
            unlink $tmp_path;
            next HOST;
        }
    }
    else
    {
        $logger->error('Error fetching: '.$file_url);
        next HOST;
    }
}

For the unit testing, I would like to cover the case where a remote host does not respond, does not have a file, and where the fetched file arrives truncated or corrupt. The unit tests need to fit nicely into the codebase and not require much in the way of external dependencies. The other developers on the project won't mind installing another Ubuntu package, but they will complain more if I ask them to install something from CPAN. There would be no point asking them to run a local web server on their machines, or suchlike just to run unit tests.

So far I looked into Test::LWP::UserAgent which looks promising, but is not available as an Ubuntu package, and Test::Mock::LWP which is packaged for Ubuntu, but does not have much helpful documentation. With both those mocking modules there appears to be a problem in that they don't pay attention to the hostname portion of the URL, so I can't easily simulate the case where one host has a file and another does not.

I am also considering just using Test::MockObject::Extends to mock the LWP::UserAgent and HTTP::Response objects. I also asked in Chatterbox on Friday, but the suggestions where not that helpful.

Can you offer any suggestions? I can't believe that this has not been extensively covered before, but a search here did not turn up anything recent or useful.