Gaffer

Gaffer

CTRF - Common Test Report Format

Use CTRF to upload test results from any framework to Gaffer in a standardized JSON format.

CTRF (Common Test Report Format) is a universal JSON schema for test results. It provides a standardized way to report test outcomes across any testing tool, framework, or language.

Why Use CTRF?

  • Universal: Works with any test framework that has a CTRF reporter
  • Consistent: Same format across Jest, Playwright, pytest, RSpec, and more
  • Rich data: Includes timing, retries, flaky test detection, and metadata
  • Open standard: Community-driven, MIT-licensed specification at ctrf.io

Supported Frameworks

CTRF has reporters for most popular test frameworks:

FrameworkPackageLanguage
Playwrightplaywright-ctrf-json-reporterJavaScript/TypeScript
Jestjest-ctrf-json-reporterJavaScript/TypeScript
Vitestvitest-ctrf-json-reporterJavaScript/TypeScript
Cypresscypress-ctrf-json-reporterJavaScript/TypeScript
Mochamocha-ctrf-json-reporterJavaScript/TypeScript
pytestpytest-ctrfPython
Goctrf-go-json-reporterGo
JUnitjunit-json-reporter-ctrfJava

See the full list at ctrf.io.

Installation

Playwright

npm install playwright-ctrf-json-reporter --save-dev
// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['playwright-ctrf-json-reporter', { outputFile: 'ctrf-report.json' }],
    ['list'], // Also show in console
  ],
});

Jest

npm install jest-ctrf-json-reporter --save-dev
// jest.config.js
module.exports = {
  reporters: [
    'default',
    ['jest-ctrf-json-reporter', { outputFile: 'ctrf-report.json' }],
  ],
};

Vitest

npm install vitest-ctrf-json-reporter --save-dev
// vitest.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    reporters: ['default', 'vitest-ctrf-json-reporter'],
  },
});

pytest

pip install pytest-ctrf
pytest --ctrf ctrf-report.json

Uploading CTRF Reports

Once you have a CTRF JSON file, upload it to Gaffer:

curl -X POST https://app.gaffer.sh/api/upload \
  -H "X-API-Key: YOUR_API_KEY" \
  -F "[email protected]" \
  -F 'tags={"commitSha":"abc123","branch":"main"}'

GitHub Actions

- name: Run tests
  run: npm test

- name: Upload CTRF report to Gaffer
  if: always()
  uses: gaffer-sh/gaffer-uploader@v1
  with:
    gaffer_api_key: ${{ secrets.GAFFER_API_KEY }}
    report_path: ./ctrf-report.json
    commit_sha: ${{ github.sha }}
    branch: ${{ github.ref_name }}

GitLab CI

test:
  script:
    - npm ci
    - npm test
  after_script:
    - |
      curl -X POST https://app.gaffer.sh/api/upload \
        -H "X-API-Key: $GAFFER_API_KEY" \
        -F "[email protected]" \
        -F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'"}'

CTRF Report Structure

A CTRF report contains standardized test result data:

{
  "results": {
    "tool": {
      "name": "playwright"
    },
    "summary": {
      "tests": 42,
      "passed": 40,
      "failed": 1,
      "pending": 0,
      "skipped": 1,
      "other": 0,
      "start": 1703520000000,
      "stop": 1703520060000
    },
    "tests": [
      {
        "name": "should login successfully",
        "status": "passed",
        "duration": 1234,
        "retries": 0,
        "flaky": false
      },
      {
        "name": "should display dashboard",
        "status": "failed",
        "duration": 5678,
        "message": "Element not found: #dashboard"
      }
    ]
  }
}

Benefits with Gaffer

When you upload CTRF reports to Gaffer, you get:

  • Structured analytics: Pass rates, duration trends, flaky test detection
  • Cross-framework comparison: Compare results from different test suites
  • AI insights: Automatic failure pattern detection
  • Historical tracking: See how tests perform over time

Tip: CTRF is particularly useful when you have multiple test frameworks in your project. Instead of learning different report formats, use CTRF everywhere for consistent analytics.

Next Steps

CI Provider Guides:

Reference: