Mutation testing vs code coverage: what 88% coverage misses

Code coverage counts a line as tested the moment a test executes it, whether or not any assertion depends on what that line did. We ran StrykerJS against Gaffer’s own dashboard code to see how far mutation testing and code coverage disagree in practice. Gaffer’s report parsers have 88.4% line coverage and a 50.6% mutation score: 676 of the 2,018 scorable mutants Stryker made in them ran under at least one test and survived. This post shows the numbers per directory, one surviving mutant next to the tests that executed it, and how to run mutation testing in CI without adding minutes to every pull request.

What does code coverage actually measure?

Code coverage measures execution: a line, branch or function counts as covered when at least one test caused it to run. Nothing in the metric checks whether an assertion would fail if that line computed the wrong value.

That makes coverage a good detector of untested code and a weak detector of under-tested code. A test that calls a function and asserts only that it did not throw covers every line it passes through. If you already track coverage (the Vitest coverage reports guide covers the setup), you have the first half of the picture.

What is a mutation score?

A mutation score is the share of deliberately injected bugs that your tests catch. A mutation tool such as Stryker makes small changes to your source (flip a > to >=, replace a condition with false, empty a string), reruns the tests against each change, and records whether any test failed.

Each mutant ends in one of a few states:

Stryker’s formula:

mutation score = (killed + timeout) / (killed + timeout + survived + no coverage)

Compile and runtime errors are left out. The interesting bucket is survived: those are lines that line coverage reports as tested and that no test actually checks.

What does a survived mutant look like?

A survived mutant is a change Stryker made to a line that at least one test executed, after which every test still passed. Here is one from our run. getPrimaryReportFile in shared/utils/manifest.ts picks the report file out of an upload’s file manifest. It prefers XML or TRX and falls back to JSON:

// shared/utils/manifest.ts, lines 34-37
  ) || manifest.find(entry =>
    entry.filename.endsWith('.json')
    || entry.contentType === 'application/json',
  )

Stryker replaced the content-type check on line 36 with a constant:

|| entry.contentType === 'application/json',
|| false,

Three tests in tests/unit/shared/utils/manifest.spec.ts (under manifest utilities > getPrimaryReportFile) executed that line against the mutant and all passed:

• returns null when no report files exist
• finds .json files
• falls back to JSON when no XML/TRX exists

A second mutant on the same line, which replaced 'application/json' with an empty string, also survived.

The reason is mundane. Every fixture in that spec names its JSON file with a .json extension, so the filename check always answers first and the content-type check is never the deciding condition. If someone uploaded a JSON report named without the extension but sent with application/json, the function would stop finding it and no test would fail. Line 36 counts as covered the whole time.

Mutation testing vs code coverage, measured on our own code

Across three directories of Gaffer’s dashboard, mutation score trailed line coverage in every one, by 37.8 points in the report parsers. Both numbers come from the same tests: the dashboard’s unit Vitest project, 153 spec files and 2,664 tests, with line coverage from Vitest’s v8 provider and mutation score from StrykerJS 10.1.1 with the Vitest runner and per-test coverage analysis.

The parsers took 2m34s under Stryker; the two helper directories took 10s and 7s. Six runtime-error mutants in the parsers are excluded from the score. Some app/utils helpers are also exercised by a separate Nuxt-runtime test project that this run did not include, so its absolute numbers understate reality. The gap between the two columns is still a fair comparison, since both used the same tests.

shared/utils looks the healthiest because its coverage is honest: most of what its tests run, they check. The parsers are the opposite. Coverage near 90% suggests a well-tested area, and half the injected bugs went unnoticed.

Which files had the widest gap?

Per-file results for selected parser files, lowest mutation score first:

The adapters are small (12 mutants each), so the percentages swing on a handful of survivors: 4 killed and 8 survived in each of the first three. rspec.ts is the more telling case. It is a real parser with 376 mutants, and 181 of them survived under tests that covered 93.2% of its lines. 97 of those 181 survivors sit in the code that rebuilds nested describe and context names from each group’s margin-left nesting depth, which the tests run but rarely assert on in detail. playwright.ts, the parser with the most tests, is the only file in the directory near Stryker’s 80% “high” threshold.

At the other end, might-handle.ts has 17.4% line coverage and a 0% mutation score: all 116 of its mutants were on lines no test ran. That file is a coverage problem, and coverage already reports it.

Why AI-generated tests make this worse

AI coding assistants are good at producing tests that execute code and weak at producing assertions that pin down behavior, which is exactly the shape that inflates coverage without raising mutation score. Asked to “add tests for this module”, a model will call every function with plausible inputs. Whether each test then checks the value that matters is a separate question that coverage cannot answer.

One practitioner measured it directly. Markus Albrecht at Nimble Approach had a GenAI assistant write unit tests for a Python module and reported roughly 85% code coverage alongside a mutation kill rate of 57.3% under mutmut. A human-refined suite of 33 tests reached an 80.4% kill rate. It is one experiment with 12 generated tests, so treat it as an illustration, but the shape is close to our own parsers.

Meta reached the same conclusion at scale and built a generator around it. Their ACH system, described in Mutation-Guided LLM-based Test Generation at Meta (FSE 2025 industry track), first generates mutants that simulate a specific class of fault (privacy regressions) and then asks an LLM to write tests that kill them. Applied to 10,795 Android Kotlin classes across seven platforms, it produced 9,095 mutants and 571 privacy-hardening tests. In test-a-thons on Messenger and WhatsApp, engineers accepted 73% of the 191 tests they reviewed. The paper reports that 277 of those 571 tests (49%) would have been discarded if line coverage had been the only adequacy criterion. Coverage as a target throws away the tests that check behavior; mutants as a target keep them.

If assistants are writing a growing share of your tests, code coverage is not enough on its own: it is the metric they will satisfy, so you need a second one they cannot satisfy by execution alone.

How to add mutation testing to CI without slowing every PR

Run the full Stryker mutation testing suite on a schedule, and run only changed files on pull requests. A full run re-executes your tests once per mutant, so even with per-test coverage analysis (the Vitest runner’s default) it is far slower than a normal test run. Our parsers alone took over two and a half minutes for 19 files.

A StrykerJS config with the JSON reporter and incremental mode:

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "testRunner": "vitest",
  "mutate": ["src/**/*.ts", "!src/**/*.spec.ts"],
  "reporters": ["json", "clear-text", "progress"],
  "jsonReporter": { "fileName": "reports/mutation/mutation.json" },
  "incremental": true,
  "thresholds": { "high": 80, "low": 60, "break": null }
}

Three ways to keep the cost down:

1. Nightly or weekly full runs. Put stryker run on a cron schedule and keep it out of the PR path entirely.
2. Incremental mode. With incremental: true (or --incremental), StrykerJS stores results in an incremental file and reuses them for code and tests that have not changed since the last run. Cache that file between CI runs or it starts from scratch each time.
3. Diff-scoped runs. --mutate accepts a comma-separated file list, so a PR job can mutate only what the branch touched:

CHANGED=$(git diff --name-only --diff-filter=d origin/main...HEAD -- 'src/*.ts' \
  | grep -v '\.spec\.ts$' | paste -sd, -)
if [ -n "$CHANGED" ]; then
  npx stryker run --mutate "$CHANGED"
fi

The catch with the second and third options is that each report covers only part of the codebase. A diff-scoped run over 3 files produces a mutation score for those 3 files, and if you chart the score per report, the line jumps around depending on which files happened to change that day. To get a number that means something, merge results per file over time: keep the latest known status of each mutant, and let a new report update only the files it contains.

Seeing mutation score next to coverage per file

Gaffer parses Stryker’s JSON report format (StrykerJS, Stryker.NET, Stryker4s, and PIT through its mutation-testing-elements plugin) and computes the score with the same formula. It keeps the latest known status of every mutant per file and per branch, so a partial or incremental run merges into the existing picture instead of replacing it. The project’s mutation view lists each file with its line coverage beside its mutation score, and expanding a file shows its survived mutants: line, mutator, replacement, and the tests that covered it.

Mutation reports are in beta. Setup is in the mutation testing docs; the coverage side comes from the same uploads described in code coverage tracking, and both sit alongside the rest of your hosted test reports.

FAQ

What is the difference between mutation testing and code coverage?

Code coverage records which lines your tests executed. Mutation testing changes those lines in small ways and reruns the tests to see whether any of them fail. Coverage tells you a line ran; mutation score tells you a test would notice if that line were wrong.

What is a good mutation score?

A good mutation score is 80% or higher by Stryker’s default thresholds, which mark 80% and above as high and anything under 60% as low. The more useful number is the gap between a file’s mutation score and its line coverage: a file at 93% coverage and 46% mutation score has tests that run the code without checking it.

Does mutation testing replace code coverage?

No. Mutation testing depends on coverage: a mutant on a line no test runs is reported as no coverage and counts against the score. Coverage is cheap enough to run on every commit, while mutation runs take minutes to hours, so most teams keep coverage on every PR and run mutation testing nightly or on changed files.