PIT Mutation Testing

Mutation testing in Java: a practical example with PIT

Test coverage can tell you whether your tests execute your code. But does it tell you whether those tests are actually good enough to detect a fault?

That is where mutation testing comes in.

In previous projects, I was used to working with test coverage of roughly 70–90%. At one of our clients, however, the requirements went further. Before code could be merged into the develop branch, we worked with:

• JUnit tests
• Regression tests
• PIT mutation tests

The aim was to reach 100% across these testing requirements before merging.

JUnit, regression testing and mutation testing

Java and JUnit are familiar territory for us at Yuma, so I will not go into detail about unit testing here.

At this client, regression tests were implemented using Cucumber. Cucumber makes it possible to automate functional validation in a format that can be understood by business analysts, developers, testers and other stakeholders.

Cucumber executes tests written in Gherkin, a structured plain-text language that can be written in natural language.

How Gherkin works

In a Gherkin file, lines can start with specific keywords followed by a description in natural language. Two of the most important keywords are:

• Feature: a high-level description of the software functionality being tested. It can be viewed as a description of a use case.
• Scenario: a concrete example that illustrates a business rule or expected behavior.

Example of a Gherkin scenario

Scenario Outline: Successful login to Searchlight as a legit Searchlight user
    Given I am on the "Login" page
    When I fill in "username" field with "foo@mydomain.com"
    And I fill in "password" field with "CucumberIsMagic"
    And I click "submit" button
    Then I should be logged in successfully as "foo@mydomain.com"

What is mutation testing?

This was the part that was new to me.

Mutation testing is a form of software testing in which small changes, or mutations, are deliberately introduced into the source code. The existing test suite is then run against the mutated code to determine whether the tests detect the change.

The goal is to evaluate the quality and robustness of your tests, rather than simply measuring how much of the source code they execute.

Why code coverage alone is not enough

Traditional test coverage metrics, such as line, statement and branch coverage, measure which parts of your code are executed by your tests.

They do not necessarily tell you whether those tests can detect faults in the code they execute.

A piece of code can therefore have high test coverage while still being only partially tested. An extreme example would be a test without assertions, although more commonly the problem is that tests cover the normal path but fail to test important boundaries or alternative behavior.

PIT mutation testing in practice

PIT, also known as Pitest, introduces mutations into your Java code and checks whether your existing unit tests detect them.

Consider this simple method:

public boolean isPositive(int number) {
    boolean result = false;

    if (number >= 0) {
        result = true;
    }

    return result;
}

The method considers zero and all positive numbers to be positive. PIT can make small changes to this code and then run our tests against the mutated versions.

Mutation 1: change the conditional boundary

One possible mutation is changing >= to >:

public boolean isPositive(int number) {
    boolean result = false;

    // Mutator: changed conditional boundary
    if (number > 0) {
        result = true;
    }

    return result;
}

The method now behaves differently when the input is zero.

Mutation 2: negate the conditional

Another mutation can change the behavior of the condition entirely:

public boolean isPositive(int number) {
    boolean result = false;

    // Mutator: negated conditional
    if (false) {
        result = true;
    }

    return result;
}

Killing mutations with better unit tests

In mutation testing, a mutation is considered killed when one of your tests fails because of the change.

If all your tests still pass, the mutation has survived. That can indicate that your tests are not checking the affected behavior closely enough.

Consider this JUnit test:

@Test
public void testPositive() {
    CalculatorService obj = new CalculatorService();

    assertEquals(true, obj.isPositive(10));
}

This test detects the second mutation because changing the condition prevents the method from returning true for 10.

However, the first mutation survives.

Why? Because the test only uses the value 10. Both number >= 0 and number > 0 return the same result for that input.

Testing the boundary condition

To kill the first mutation, the test needs to cover the boundary value: zero.

The mutated version looks like this:

public boolean isPositive(int number) {
    boolean result = false;

    // Mutator: changed conditional boundary
    if (number > 0) {
        result = true;
    }

    return result;
}

We can improve the unit test by adding an assertion for zero:

@Test
public void testPositive() {
    CalculatorService obj = new CalculatorService();

    assertEquals(true, obj.isPositive(10));

    // Kill mutation 1
    assertEquals(true, obj.isPositive(0));
}

Now the test distinguishes between > and >=, meaning the changed conditional boundary is detected.

The result: 100% mutation coverage for this example.

What mutation testing tells you that code coverage cannot

This small example demonstrates the real value of mutation testing.

The original unit test executed the conditional and therefore contributed to traditional code coverage. But it did not fully test the behavior of that conditional.

Mutation testing exposed the missing boundary case.

In other words:

• Code coverage asks: did your tests execute this code?
• Mutation testing asks: would your tests notice if this code behaved differently?

Why use PIT for Java mutation testing?

There are other mutation testing systems for Java, but PIT is designed with real development teams in mind.

Some of its practical advantages include:

• It can analyse a codebase relatively quickly.
• It produces reports that combine line coverage and mutation coverage.
• It can be integrated with build tools such as Ant, Maven and Gradle.
• Plugins are available for development environments including Eclipse and IntelliJ IDEA.

What I learned from using mutation testing

Before working on this project, I had not encountered mutation testing in practice.

The biggest takeaway for me was that high code coverage does not automatically mean you have strong tests. Mutation testing adds another layer by checking whether those tests are actually capable of detecting meaningful changes in your application's behavior.

In this example, a seemingly reasonable unit test missed an important boundary condition. PIT made that gap visible and helped turn the test into a more robust one.

And that is one of the things I enjoy about working on different projects: every now and then, you come across a technique that changes the way you think about something you thought you already knew.