Why I avoid using verify in tests

When I first learned unit testing and Test Driven Development, test frameworks were very light and mocking frameworks were just coming into the picture. I was writing a lot of C++ back then, and if I wanted a mock I would implement my class interface and link my test to the mock version instead of the production version. Fast forward 15 years and we have extraordinarily advanced mocking frameworks (and i mostly write Java these days). While new libraries have come and gone, Mockito remains my favorite. It has a good balance of a straightforward syntax and a very complete feature set. In fact, a bit too complete if you ask me.

While there are a few ways to use Mockito (or other similar frameworks), there are two patterns that I see most frequently. The first uses Mockito to generate a dynamic mock object with a call like:

when(mock.method()).thenReturn(expectedValue).

The test interacts with the mock object and uses an assertion library to validate the result of some logic under test. The other pattern uses

 verify(mock, times(1)).method() 

as an assertion to determine if a specific method was (or was not) called using the provided parameters or parameter surrogates. This second form, which I call “verify”, is what I would like to discuss. In general, verify based tests do not meet the criteria of good tests and should be avoided.

What are we really testing when we use verify?

Verify asserts that the test author and developer are in agreement about what code should be written. In many cases, a single developer writes both the test and code, and it is at best a check against a typo in either the test or the code. In paring situations, it can serve as a stand-in for the conversation that should be taking place in the pair. I find more value in testing “business things” than “code things”. To break it down, let’s take a look at the typical reasons that we write tests and whether or not verify satisfies them:

  1. Validate that the product meets the business requirements
  2. Give cover for future change
  3. Document intent

Validate Business Requirements

The basic purpose of most tests is to validate correctness, and correctness is defined by meeting the business requirement. The only reliable way to accomplish this is by asserting that the result of a piece of code is the right business outcome. Verify does not meet this requirement in most cases. If the call being verified does the right thing for the business requirement, then the test will be correct, but there is nothing in the test that guarantees it. Assumptions could be missed, the behavior could be different, or the cumulative effects of calls produce a very different effect than intended. Verify calls validate that the code was written as expected, but not that any business logic is actually correct.

Cover for Future Change

Heavy use of verify results in one of the most pathological cases of unit testing, and one often cited as an argument against it. Detractors say that every time they refactor their code, they also have to rewrite all of their tests, causing them to work twice as hard rewriting twice as much code. Since refactoring should only change the structure of code but not the function, this should not be the case. Verify calls are tightly coupled to implementation, so they do not provide functional coverage which is essential for refactoring. Having to make significant change to both the code and the test is more of a rewrite than a refactor, and comes with increased risk.

Documentation of intent

Similar to “Validate Business Requirements”, documentation of intent is only effective when the tests demonstrate the expected business outcomes of the code. Tests that rely on verify show what other code we intended to call, but not why or how it impacts the business outcome. A developer new to the codebase cannot determine why those choices were made or whether the code would be appropriate for reuse in other circumstances; only what was expected to be called directly.

Up next

In my next post I will break down some examples of common verify tests and how I transform them to more durable, complete and business focused tests.

 

 

Author: Kevin Hickey

I have been a professional software engineer for over fifteen years. I currently work at Intuit as an architect. I have written bootloaders, ported the Linux kernel and Android to new platforms, written CPU diagnostics, developed control software for CPU manufacturing and worked on enterprise web sites. As both a developer and program manager I have been helping software organizations become more agile for over a decade. I am passionate about helping teams deliver world-class software solutions to interesting problems. My current focus is on pragmatic agility for the enterprise. When I’m not behind a keyboard I enjoy spending time with my wonderful wife Amanda, rock climbing and hanging out with my dog Tex. In the summer you can find me in my pool or climbing something. In the winter I count the days until summer returns to Texas (I never have to count too long).