Mutation Testing Reports (Beta)

A mutation testing tool changes your code in small ways and reruns your tests. If the tests still pass, the mutant survived. Line coverage only says a line ran. A surviving mutant says no test would notice if that line were wrong.

Supported Tools

Gaffer parses the Stryker mutation-testing-report-schema JSON, schema versions 1 and 2.

Tool	Language	How it produces the format
StrykerJS	JavaScript, TypeScript	`json` reporter
Stryker.NET	C#	`json` reporter
Stryker4s	Scala	`json` reporter
PIT	Java	pitest-mutation-testing-elements-plugin, `JSON` output format

Gaffer detects the format from the file content. The filename does not matter.

Enable the JSON Reporter

In StrykerJS, add json to reporters and set the output path:

{
  "reporters": ["json", "clear-text"],
  "jsonReporter": { "fileName": "reports/mutation/mutation.json" }
}

reports/mutation/mutation.json is the StrykerJS default. Run it with npx stryker run.

In Stryker.NET, pass -r "json" or list json under reporters in the config file. Pass --output <dir> on the command line to change the output directory, and set report-file-name in the config file to rename the report (default mutation-report). In Stryker4s, add json to reporters. The file is written next to the HTML report.

Upload

Uploading uses the existing paths. No new flag.

CLI

gaffer upload reports/mutation/mutation.json

GitHub Actions

name: Mutation tests

on:
  schedule:
    - cron: '0 3 * * *'

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v3
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: pnpm

      - run: pnpm install --frozen-lockfile

      - name: Run Stryker
        run: pnpm stryker run

      - name: Upload to Gaffer
        uses: gaffer-sh/gaffer-uploader@v2
        if: always()
        with:
          gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }}
          report_path: ./reports/mutation
          commit_sha: ${{ github.sha }}
          branch: ${{ github.ref_name }}

Upload API

curl -X POST https://app.gaffer.sh/api/upload \
  -H "X-API-Key: YOUR_PROJECT_TOKEN" \
  -F "files=@reports/mutation/mutation.json" \
  -F 'tags={"commitSha":"abc123","branch":"main"}'

See the Upload API for the full field reference.

Mutation Score

Gaffer computes the score the way Stryker does:

(killed + timeout) / (killed + timeout + survived + no coverage)

Ignored mutants, compile errors and runtime errors are left out of the score.

Partial and Nightly Runs

Mutation runs are slow, so many teams run them nightly, incrementally or only on changed files. A report that covers 12 files should not erase what you know about the other 400.

Gaffer keeps the latest known status of every mutant, per file and per branch. A new report updates only the files it contains. Every other file keeps its last known state. The weekly trend is drawn from this merged state, not from individual uploads. A diff-scoped run on Tuesday therefore does not make Tuesday’s score look like a collapse.

What the Dashboard Shows

Open the mutation view from the project page. For each file it lists:

Mutation score, beside line coverage from the latest coverage report
Under each file, the survived mutants: line, mutator, replacement and the tests that cover it

Start with files that have high coverage and a low mutation score. Their tests run the code but assert little. For a worked example on Gaffer’s own code, see what 88% coverage misses.