Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Testing objects that cache

by dreadpiratepeter (Priest)
on May 15, 2009 at 13:18 UTC ( #764268=perlquestion: print w/ replies, xml ) Need Help??
dreadpiratepeter has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to write tests for an object that uses a cache.
The issue that I am running into is this. Normally, I write all my tests from outside the object, ie. my tests treat the object as a black-box and I only use the api for tests.
This shields me from changes in my object implementation causing changes in my tests, and validates that changes don't break the api, blah blah blah - standard testing stuff.
But, this object implements a cache internally, but from the outside there is no way of knowing if the cache was used or not.
What is the best way of testing the caching behavior?
  • I could add extra methods to the api to check for cache state, but that opens up a closed interface.
  • I could instantiate methods on the fly the show the cache state and have them only instantiate while testing, but then I'm not testing the same code I'm running when not testing.
  • Or I could look at the internals of the cache in my tests, which violates my test-from-the-outside principle.

    How do the wise test caching behavior?


    -pete
    "Worry is like a rocking chair. It gives you something to do, but it doesn't get you anywhere."
  • Comment on Testing objects that cache
    Re: Testing objects that cache
    by ELISHEVA (Prior) on May 15, 2009 at 13:47 UTC

      Are they really the same?. The first thing you probably want to test is that the cached object and the non-cached object in fact do have the same "black box" behavior. Hand crafted optimizations have a nasty habit of breaking the very thing they are meant to help.

      Does caching really improve things?The second thing to think about is what benefit is caching supposed to provide? Consider how you might put together some tests (or maybe benchmarks) that verify that the cached object really is an improvement over the non-caching object.

      The boundaries of a black box are a matter of perspective. For the consumer of a business object API, caching is just another way of implementing the same API. However, if your caching is implemented as a service to be used by many objects, then those who use it will see the caching behavior itself as the boundary of the black box. They may not care whether you use arrays or hashes to implement it, but they will care that the promised algorithm does what you claim it does. For example, if you have an algorithm that says that cache members should be polled for last use timestamp each time an object is fetched, then you can test for compliance with that specific contract.

      As for specific testing tools and practices, great question! I'm looking forward to the answers others have. It seems tricky to get right. Many of the things one might think first of testing for are transient effects. Furthermore, if you access objects to test the content of the cache you may end up disturbing the actual caching behavior. Yet another case of observation changing reality.

      Best, beth

        The first thing you probably want to test is that the cached object and the non-cached object in fact do have the same "black box" behavior.
        No, not really. Once the OP is that far, his problem is solved.

        His problem is "how do I test whether caching was used"? If you don't know an object you have is a cached object or not, any test that compares it against an object that isn't cached is moot (as both object may not be cached).

        Does caching really improve things?
        I don't think that "reconsider using X" is a very useful answer to "how do I test X happened".
        For the consumer of a business object API, caching is just another way of implementing the same API. However, if your caching is implemented as a service to be used by many objects, then those who use it will see the caching behavior itself as the boundary of the black box. They may not care whether you use arrays or hashes to implement it, but they will care that the promised algorithm does what you claim it does. For example, if you have an algorithm that says that cache members should be polled for last use timestamp each time an object is fetched, then you can test for compliance with that specific contract.
        I've read the above paragraph a couple of times. I still have no clue whether you're trying to make a point, or whether you're just stringing words together.
          No, not really. Once the OP is that far, his problem is solved.

          The claim that two objects behave the same (performance asside) is a testable claim. Therefore it should be tested.

          I don't think that "reconsider using X" is a very useful answer to "how do I test X happened".

          Presumably if we are implementing caching then caching does make a difference. Therefore a non-cached and cached implementation of the same object ought to have a statistically different performance profile. That is an assertion that can be tested. If no performance difference is observable, then we might wonder if indeed caching is happening.

          I've read the above paragraph a couple of times. I still have no clue whether you're trying to make a point, or whether you're just stringing words together.

          Unfortunately such a sarcastic remark does little to help me be more clear. What didn't you understand?

          Perhaps a more concrete example would help? Suppose I have a Person List API. I am responsible for programming a customer analysis report using that API. As the report writer I don't really care whether caching does or does not take place. I only care that the API provides whatever information I need to select the top 10 customers of the month in what my client subjectively feels is a reasonable amount of time. Testing performance time is OK, but testing specifically for caching is not. It violates the black box principle. How I get the performance doesn't matter - neither to my customers nor to me.

          Now suppose my customers don't like the performance. After doing some profiling I decide that the slow performance is due to too much paging. I decide to reduce the memory consumption of the Person List by adding caching. Not wanting to reinvent the wheel I look at several different caching libraries - some optimize for retrieval time, some optimize for memory usage. If the API lets me set the memory consumption of the cache , I care very much that the cache API does not exceed that memory threshold. I would want that claim tested. Caching is still an implementation detail. However, testing the caching library API to make sure it conforms to its promises does not violate the black box principle, because the API being tested is the caching behavior itself.

          Once I am done installing the code that uses the caching library, I go back to testing the customer analysis report. Once again I don't care about caching. I only care that my changed implementation seems fast enough to my customers. It really doesn't matter why it is faster, only that it is faster.

          Best, beth

        I liked this post(update: to be clear, the one by ELISHEVA). I think the original question is so general that there is no "one size fits all" answer. It think this has a lot to with what you are caching and why?

        Some examples.
        1. I have one function that always caches its results. It does a very expensive approximate pattern match on a DB and saves the result. How much memory it uses for this is documented (and directly related to DB size) and user can't turn it off. In a typical app, performance difference is like 5-10,000x between first and second call. Testing is easy as I just have to verify that consistent results are coming back and the performance difference is so huge that it is easy to see. The database is read only for each run, so this is easy to implement (no need to fiddle with "dirty pages", etc.

        2. I have another function that is not strictly "caching" as it precomputes a whole bunch of things that I think that you are going to ask, before you ask them (actually makes it more efficient to answer a whole bunch of questions at the expense of a lot of horsepower at the beginning). This is an example of a "cooperative" thing. You have to tell me that this dataset is special via a "study" call. If you do that intelligently then overall performance speeds up dramatically. The normal api calls aren't changed. So testing is working with and without "studying" this dataset first.

        3. There are lots of examples where caching although transparent to the user API, the user nevertheless knows that caching is going on. One example would be file I/O buffering, say to stdout. $|=1 turns this off.

        4. If you are caching results for something that is dynamic (data changes during execution of your program), that is usually something more appropriate for the module to mess with, not "something between you and it".

        5. Almost always it is not wise to cache something that is already being cached by the O/S. An example of this would be file I/O. I have one app with a large number of files >500 and after they've all been looked at once, they become memory resident due to O/S caching. I don't have to do anything. If I tried to cache these files, performance could go down as this could reduce amount of space that the O/S has to do this function more efficiently than I can.

        6. I personally have no problem adding api calls that would tell the user some stats on how well the caching is working. These would be extra optional things. For example the Perl hash allows you to query numBuckets and Buckets used. So what? You don't have to know this to use it, but if you are curious, you can find this out.

        7. I also have no problem with "private" calls that look at the "guts" and report how things are going. I don't export these methods or functions. Of course any user can call these with fully qualified name, at their own risk.

        8. I would have some sort of test() method that is not exported to be used for module verification.

        So, I see as separate things:
        A) How to test that this module is actually working? This is for developer and build process.
        B) How well is this caching gizmo working for me? This is mainly based upon: consistent results and measured performance differences.

        sorry that I was so long winded....

    Re: Testing objects that cache
    by almut (Canon) on May 15, 2009 at 14:43 UTC

      You could maybe compare some performance measure (response time, or whatever you'd expect to change as an effect of the caching) for repeated tests with the same input, under otherwise identical conditions.

      This could in theory allow you to test (1) whether caching is being used at all, and (2) if you're getting the same results in case the caching is being used.  (But it would assume that you can start with an empty cache...)

      Just some thoughts...  What exactly are the tests supposed to prove?

    Re: Testing objects that cache
    by JavaFan (Canon) on May 15, 2009 at 15:00 UTC
      I don't think there's a general solution for it, if you have the cache properly encapsulated. I've struggled with this problem before, and I haven't found an answer that has satisfied me.

      But you may be able to write some tests tailored to how you've implemented the cache for your specific problem. If you're caching database or network results, you might be able to trace the traffic over the database or network handles and conclude a query not repeated. Or you may be able to gain some insight using tied variables, or overloaded objects. But they are not general solutions, and will only work in some cases.

    Re: Testing objects that cache
    by NiJo (Friar) on May 15, 2009 at 15:31 UTC
      If your application is so elaborate to use caching, a debug output in some form is likely to already exist. This allows a practical approach:

      1) Reset Cache (or use a fresh instance)

      2) The first function call is known to be not cached

      3) The second call should be cached. In non-debug mode you can not test this, but results should be equal to 2)

      4) Start another instance in debug mode after a cache reset.

      5) Call the function two times

      6) Compare the debug output, which should contain 'I am (not) cached' type statements.

      7) Compare results of all 4 calls.

    Re: Testing objects that cache
    by CountZero (Bishop) on May 15, 2009 at 16:37 UTC
      As others have already said much better than I can: "It is a matter of perspective".

      If you want to test the published API and the cache is "hidden away", i.e. you cannot check through the API whether or not the cache "worked", there is no good way to test it.

      But nothing of course stops you from checking the internals of your modules in a different set of tests.

      I have had a similar problem, where I had written a framework into which a specific parser was to be plugged-in. The framework contains its own set of basic "utility" methods, some of which are used by a parer-plugins. One set of tests exercise the "utility" methods (which is the low-level internal API) and the tests linked to the parser plugins test whether the frame-work including the parser "works as advertised" by using the external API.

      Applying this to your case, I would say test both the API (for which the cache is not an issue) and also test the working of the cache itself by stepping inside of your API.

      CountZero

      A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    Re: Testing objects that cache
    by rir (Vicar) on May 15, 2009 at 17:41 UTC
      Scaling up the load of your normal tests will show if your code "works" in a pragmatic sense. That is difficult in your case or you have undisclosed motives.

      Using Devel::Cover could verify that your caching code is being exercised.

      If your cache is implemented with a call tree, i.e.

      sub fetch { return _cached_fetch($_[1]) if _cached( $_[1] ); return _via_normal_retrieval( $_[1] ); }
      there are packages that will insert code at the entry or exit of a set of routines. This would let you insert a logging function into fetch and _cached_fetch to generate statistics on cache performance. I cannot think of the name for this type of function modification.

      The insert code could be something like

      sub log_cache { local $, = " "; print SOMEWHERE "MyObj cache called", caller(1), "w/",$_[1],$/; }
      cleaned up for log readability.

      Be well,
      rir

    Re: Testing objects that cache
    by scorpio17 (Monsignor) on May 15, 2009 at 18:32 UTC

      I like to have a config file available, in which I can put something like, "CACHE = OFF". You could also use an environment variable to control something like this.

    Re: Testing objects that cache
    by tilly (Archbishop) on May 15, 2009 at 18:51 UTC
      My attitude is that unit tests should know about the internals of the object they are testing, and should test all of the boundary cases of the implementation. Because if you treat it as a black box, how will you ever find the nasty little off by one errors? Therefore you provide hooks for the unit tests which allow them to mess with the internals in a controlled way.

      For instance suppose that you have a caching layer implemented inside the main object. So it contains some sort of object which is responsible for accessing the cache. Then I'd have a set of unit tests for the cache object (just to be sure that it works properly). And then for the main object I'd have a way for the unit test to poke inside and replace the cache object. (Test::MockObject::Extends is useful for this.) This will let you write unit tests that test exactly how the caching works, and let you verify that the result without caching is the same as the result with caching.

      Now to address the "I'm not testing the same code" complaint, the code for the main object is exactly the same in the unit test and out. The code for its cache admittedly gets tampered with, but the API of the new code should be the same as the old. (After all you're mostly just delegating from the old code with some logging added.) Plus you have some of your tests running before the cache gets replaced, and you know that those don't get affected, it is only the piece where you are testing the exact caching behavior that has been touched.

    Re: Testing objects that cache
    by sgifford (Prior) on May 18, 2009 at 04:20 UTC
      I have solved similar problems by exposing some useful but very limited bits of the cache internals through an API. For example, a method to keep a count of the number of cache hits and misses. With that, you can tell whether your test hit or missed the cache by comparing the counters before and after. I often find these sorts of methods useful for logging after the fact.

    Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Node Status?
    node history
    Node Type: perlquestion [id://764268]
    Approved by marto
    Front-paged by ELISHEVA
    help
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others perusing the Monastery: (10)
    As of 2014-08-22 20:17 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      The best computer themed movie is:











      Results (164 votes), past polls