Ignorance is bliss.

— Proverb

When you get used to refactoring with tests, one common pitfall people sometimes fall into is stopping writing tests and starting refactoring after they attain 100% code coverage. Code coverage can be a helpful metric under specific contexts and scales; refactoring is one. If you are refactoring code, looking only at code coverage as a metric is one-dimensional and can inspire false confidence and a false sense of security.

Let's take a look at the following trivial example,

fun increment(number: Int): Int {
  var maybeIncremented = number
  if (number % 2 == 0) {
    maybeIncremented = number + 1
  }
  return maybeIncremented
}

If we were to write the following test,

class SlipperySlopeTest {
  @Test
  fun `it should increment a number`() {
    // given & when
    val number = increment(4)

    // then
    assertThat(number)
      .isEqualTo(5)
  }
}

Running the test with coverage will yield the following result.

We have hit all statements and branches in the function and achieved 100% code coverage with one test.

You may have already realized that something's not right. In this case, the variable name maybeIncremented could have already been a dead giveaway that we don't have enough tests to refactor the function safely. But hey, this is a trivial example.

If you've relied on code coverage as an indication to stop writing tests and begin refactoring in the past, I wouldn't be surprised if this example had set off your alarm bells 🔔

Understanding code coverage

There are a couple of questions we must ask ourselves,

1. "What are we trying to find out using code coverage?"
2. "Are we using the right metric?"

For question 1, we are trying to see if we have enough tests to refactor a function safely. Right now, we are looking at a metric that says 100%. What does it mean? Does it mean we have all the tests in place? Often, we are accustomed to using code coverage as a proxy metric to see if we have the required tests before we begin refactoring.

What about question 2? What could be a better metric if code coverage is a proxy that can't give us the number of tests we should write? Enter cyclomatic complexity.

Cyclomatic complexity

Cyclomatic complexity measures the number of independent paths in a function. So, before we refactor, we should have tests covering each path. So, if a function has a cyclomatic complexity of 5, you need at least five tests that cover five different paths. The number of tests is irrelevant if they don't cover different paths in the program.

You can manually compute a function's cyclomatic complexity if you don't have a tool or plugin to do it for you.

You start at one, and then you count how many times if and for occurs. For each of these keywords you find, you increment the number (which started at 1) … The idea is to count branching and looping instructions. In C#, for example, you'd also have to include foreach, while, do, and each case in a switch block. In other languages, the keywords to count will differ.

— Mark Seemann

If we apply Mark's technique to our trivial example,

fun increment(number: Int): Int {
  var maybeIncremented = i
  if (number % 2 == 0) {
    maybeIncremented = number + 1
  }
  return maybeIncremented
}

We can see the cyclomatic complexity of our function is 2. This means we need at least two tests executing two different code paths. However, we only have one. Let's write the next test.

@Test
fun `it should not increment an odd number`() {
  // given & when
  val number = increment(3)

  // then
  assertThat(number)
    .isEqualTo(3)
}

The new test should cover the alternative path. You can verify if both paths are covered by running each test individually with coverage and visually verifying their execution paths.

Running all the tests must still yield 100% code coverage now.

Recap

We examined at how code coverage alone may not be a good indicator of safety for refactoring. We also found how to pair coverage information with cyclomatic complexity to identify if we have enough tests to get to safety before we begin refactoring. Please pay attention to the nuance that cyclomatic complexity indicates the number of unique code paths you must cover through tests, not just the number of tests. You can write more tests if you feel your situation needs more.

What's next?

In the upcoming article, we'll explore additional tools and techniques that can help increase confidence while writing tests for untested code. Stay tuned or subscribe to the website and get the next article delivered to your inbox!