Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Greetings wise brothers.

In the project I am working on, I have some code that attempts to fetch files over http via LWP. I am confused at how I can write unit tests for that functionality.

The overall design is that I have a global database that lists all the files available, along with the size and checksum, and a list of http servers that may or may not have a cached copy of any particular file, so when a client needs a file it might have to try several servers before it finds one that has the one it is looking for. In simplified form my code looks like this:

my $fileObj = $schema->resultset('Files')->some_query() # A DBIx::Cla +ss result object. my $ua: # An LWP::UserAgent instance HOST: foreach my $host ( @hosts2try ) { my $leafname = $fileObj->leafname(); my $file_url = 'http://'.$host.'/'.$leafname; $logger->debug("Will try fetching $leafname from $host"); # First check if the remote host has the file & the size is correc +t. my $response = $ua->head($file_url); if( ! $response->is_success() ) { # Not a problem as not every host has every file. $logger->debug("Looks like $host does not have $leafname. Will + try another host"); next HOST; } elsif( $imageObj->size() != $response->headers->header('content-le +ngth') ) { # An error. If a host has a file it should be correct. my $remote_size = $response->headers->header('content-length') +; $logger->error("Host $host has a copy of $leafname but the siz +e is wrong. Is $remote_size bytes but should be ".$imageObj->size() ) +; next HOST; } # Looks like the remote server has the file. Download it. $logger->info('Fetching '.$file_url); $ua->get($file_url, ':content_file' => $tmp_path); if( -f $tmp_path ) { if( ( $imageObj->size() == -s $tmp_path ) && ( $imageObj->checksum() eq get_sha1_checksum($tmp_path) ) ) { $logger->info("Successfully fetched $filename from $file_u +rl"); move($tmp_path, $final_path); return 1; } else { $logger->error("Serious: $filename fetched from $file_url +was truncated or corrupt") unlink $tmp_path; next HOST; } } else { $logger->error('Error fetching: '.$file_url); next HOST; } }

For the unit testing, I would like to cover the case where a remote host does not respond, does not have a file, and where the fetched file arrives truncated or corrupt. The unit tests need to fit nicely into the codebase and not require much in the way of external dependencies. The other developers on the project won't mind installing another Ubuntu package, but they will complain more if I ask them to install something from CPAN. There would be no point asking them to run a local web server on their machines, or suchlike just to run unit tests.

So far I looked into Test::LWP::UserAgent which looks promising, but is not available as an Ubuntu package, and Test::Mock::LWP which is packaged for Ubuntu, but does not have much helpful documentation. With both those mocking modules there appears to be a problem in that they don't pay attention to the hostname portion of the URL, so I can't easily simulate the case where one host has a file and another does not.

I am also considering just using Test::MockObject::Extends to mock the LWP::UserAgent and HTTP::Response objects. I also asked in Chatterbox on Friday, but the suggestions where not that helpful.

Can you offer any suggestions? I can't believe that this has not been extensively covered before, but a search here did not turn up anything recent or useful.

In reply to How to unit test code that used LWP by chrestomanci

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others studying the Monastery: (3)
    As of 2020-01-23 22:55 GMT
    Find Nodes?
      Voting Booth?