```
## Badge Styles
Shields.io supports multiple badge styles. Add the `style` parameter to customize:
| Style | Example | Parameter |
|-------|---------|-----------|
| Flat (default) |  | `style=flat` |
| Flat Square |  | `style=flat-square` |
| Plastic |  | `style=plastic` |
| For the Badge |  | `style=for-the-badge` |
## Badge Colors
Badge colors automatically reflect your project's health using a 5-level scale:
### Tests Badge
| Pass Rate | Color |
|-----------|-------|
| ≥90% |  |
| ≥75% |  |
| ≥50% |  |
| ≥25% |  |
| {'<'}25% |  |
### Coverage Badge
| Coverage | Color |
|----------|-------|
| ≥80% |  |
| ≥60% |  |
| ≥40% |  |
| ≥20% |  |
| {'<'}20% |  |
### Flaky Badge
| Flaky Tests | Color |
|-------------|-------|
| 0 |  |
| 1-3 |  |
| 4-10 |  |
| >10 |  |
When no data is available (e.g., no test runs yet), badges display "N/A" with a neutral gray color.
## Caching
Badge data is cached for 5 minutes to ensure fast loading times and reduce API load.
After uploading new test results, badges will update within 5 minutes.
## Example
Here's how badges might look in a README:
**my-awesome-project**
  
A well-tested project with great coverage and no flaky tests!
## Next Steps
- [Getting Started](/docs/getting-started/) - Set up your first project
- [Analytics & Metrics](/docs/analytics/) - Understand your test health data
- [Slack Integration](/docs/integrations/slack/) - Get test results in Slack
---
## CLI Reference
Source: https://gaffer.sh/docs/cli/
import { Aside } from '@astrojs/starlight/components'
## Installation
Install via Homebrew (macOS, Linux):
```bash
brew install gaffer-sh/tap/gaffer
```
Or via the install script (macOS, Linux):
```bash
curl -fsSL https://app.gaffer.sh/install.sh | sh
```
The install script places the `gaffer` binary in `~/.local/bin`. Both methods support Linux (x86_64, aarch64) and macOS (Apple Silicon, Intel). For a guided walkthrough, see the [Getting Started guide](/docs/getting-started/).
## `gaffer test`
Run a test command and analyze results.
```bash
gaffer test -- npm test
gaffer test -- pytest -x
gaffer test -- go test ./...
gaffer test -- cargo test
```
### Flags
| Flag | Env var | Description |
|------|---------|-------------|
| `--token
CircleCI's own test insights only cover a single project's build history inside CircleCI. Gaffer adds cross-build history, flaky detection, and a report link that works for anyone, CircleCI account or not.
## Prerequisites
- A [Gaffer account](https://app.gaffer.sh/register?utm_source=gaffer&utm_medium=website&utm_campaign=docs&utm_content=inline) with a project
- Your [project token](/docs/upload-api/)
- A CircleCI project with a `config.yml` file
## How do I add my Gaffer project token to CircleCI?
**Add it as a project environment variable, or a context if you have more than one CircleCI project.**
For a single project:
1. Go to your CircleCI project settings
2. Navigate to **Environment Variables**
3. Click **Add Environment Variable**
4. Name: `GAFFER_PROJECT_TOKEN`
5. Value: your Gaffer project token
## How do I share one project token across multiple CircleCI projects?
**Use a CircleCI context instead of a project-level environment variable.** Contexts are defined once at the organization level and attached to jobs in any project, so you set `GAFFER_PROJECT_TOKEN` once instead of re-entering it per project.
1. Go to **Organization Settings** → **Contexts**
2. Create a context, e.g. `gaffer`
3. Add `GAFFER_PROJECT_TOKEN` as an environment variable on the context
4. Reference it in each project's `config.yml` with `context: gaffer`
## How do I upload a report after a CircleCI job runs?
**Add a `run` step with `when: always`.** Without it, the step is skipped whenever an earlier step in the job fails, which is exactly when you want the report.
```yaml
version: 2.1
jobs:
test:
docker:
- image: cimg/node:20.0
steps:
- checkout
- run:
name: Install dependencies
command: npm ci
- run:
name: Run tests
command: npm test
- run:
name: Upload to Gaffer
when: always
command: |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"$CIRCLE_BRANCH"'"}'
- store_test_results:
path: test-results
workflows:
test:
jobs:
- test
```
Keeping `store_test_results` alongside the Gaffer upload is intentional: it feeds CircleCI's own Insights timing charts for this one project, while Gaffer keeps the report and history independent of CircleCI.
## Which CircleCI variables carry commit and branch info?
| Variable | Description | Example |
|----------|-------------|---------|
| `$CIRCLE_SHA1` | Full commit SHA | `abc123def456...` |
| `$CIRCLE_BRANCH` | Branch name (empty on tag-triggered builds) | `main`, `feature/login` |
| `$CIRCLE_TAG` | Git tag name (only set on tag-triggered builds) | `v1.2.0` |
| `$CIRCLE_PR_NUMBER` | Pull request number (only available on forked PRs) | `42` |
| `$CIRCLE_NODE_INDEX` | Index of the current container under `parallelism` | `0`, `1`, `2` |
| `$CIRCLE_BUILD_NUM` | Build number | `123` |
## Framework examples
### Playwright
```yaml
version: 2.1
jobs:
playwright:
docker:
- image: mcr.microsoft.com/playwright:v1.40.0-jammy
steps:
- checkout
- run:
name: Install dependencies
command: npm ci
- run:
name: Run Playwright tests
command: npx playwright test
- run:
name: Upload to Gaffer
when: always
command: |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@playwright-report/index.html" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"$CIRCLE_BRANCH"'","test_framework":"playwright","test_suite":"e2e"}'
- store_artifacts:
path: playwright-report
```
### Jest with JUnit reporter
```yaml
version: 2.1
jobs:
test:
docker:
- image: cimg/node:20.0
steps:
- checkout
- run:
name: Install dependencies
command: npm ci
- run:
name: Run Jest tests
command: npm test -- --reporters=default --reporters=jest-junit
environment:
JEST_JUNIT_OUTPUT_DIR: ./test-results
- run:
name: Upload to Gaffer
when: always
command: |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"$CIRCLE_BRANCH"'","test_framework":"jest"}'
- store_test_results:
path: test-results
```
### pytest
```yaml
version: 2.1
jobs:
pytest:
docker:
- image: cimg/python:3.11
steps:
- checkout
- run:
name: Install dependencies
command: pip install pytest pytest-html
- run:
name: Run pytest
command: pytest --html=report.html --self-contained-html
- run:
name: Upload to Gaffer
when: always
command: |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@report.html" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"$CIRCLE_BRANCH"'","test_framework":"pytest"}'
- store_artifacts:
path: report.html
```
### Using CTRF format
For a standardized format across all your test frameworks, use [CTRF](/docs/guides/ctrf/):
```yaml
- run:
name: Install CTRF reporter
# Choose the reporter for your framework:
# npm install --save-dev jest-ctrf-json-reporter
# npm install --save-dev playwright-ctrf-json-reporter
# npm install --save-dev vitest-ctrf-json-reporter
command: npm install --save-dev jest-ctrf-json-reporter
- run:
name: Run tests with CTRF
command: npm test -- --reporter=jest-ctrf-json-reporter
- run:
name: Upload CTRF to Gaffer
when: always
command: |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@ctrf-report.json" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"$CIRCLE_BRANCH"'"}'
```
## Does CircleCI's own test insights replace Gaffer?
**No, they cover different scope.** `store_test_results` feeds CircleCI Insights, which summarizes timing and flake rate for a single project's own build history, visible only inside that CircleCI org. Gaffer keeps a report you can link to someone without CircleCI access, and correlates runs against the commit and branch tags you send rather than CircleCI's internal job graph. Keep `store_test_results` for CircleCI's native charts and add Gaffer for anything that needs to leave CircleCI, like a report link in a customer ticket or a Slack alert.
## Is there an official Gaffer CircleCI orb?
**No.** Gaffer publishes a GitHub Action and a CLI, but not a CircleCI orb. On CircleCI, the two supported paths are calling the upload API directly with `curl`, as every example on this page does, or running the `gaffer` CLI as a normal `run` step. If you want to avoid repeating the same `run` block across jobs, define it once as a reusable command in CircleCI config 2.1 rather than reaching for a third-party orb:
```yaml
version: 2.1
commands:
upload-to-gaffer:
parameters:
report_path:
type: string
test_framework:
type: string
default: ""
steps:
- run:
name: Upload to Gaffer
when: always
command: |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@<< parameters.report_path >>" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"${CIRCLE_BRANCH:-$CIRCLE_TAG}"'","test_framework":"<< parameters.test_framework >>"}'
jobs:
test:
docker:
- image: cimg/node:20.0
steps:
- checkout
- run: npm ci
- run: npm test
- upload-to-gaffer:
report_path: test-results/junit.xml
test_framework: jest
```
## Why is $CIRCLE_BRANCH empty on some builds?
**CircleCI only populates `$CIRCLE_BRANCH` on branch-triggered builds.** A build triggered by pushing a git tag sets `$CIRCLE_TAG` instead and leaves `$CIRCLE_BRANCH` unset. If your release pipeline runs on tags, an upload that only reads `$CIRCLE_BRANCH` sends an empty branch tag to Gaffer. Fall back explicitly:
```bash
BRANCH="${CIRCLE_BRANCH:-$CIRCLE_TAG}"
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"$BRANCH"'"}'
```
## How do I upload from CircleCI's parallel test splitting without runs colliding?
**Tag each container's upload with `$CIRCLE_NODE_INDEX`.** Gaffer records each `curl` call as an independent test run; `parallelism` fans the same job out across N containers, and without a per-container tag those N uploads land as N indistinguishable runs against the same commit.
```yaml
jobs:
test:
parallelism: 4
steps:
- run:
name: Run tests
command: |
circleci tests glob "**/*.spec.js" | circleci tests split | xargs npm test --
- run:
name: Upload to Gaffer
when: always
command: |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CIRCLE_SHA1"'","branch":"'"$CIRCLE_BRANCH"'","container":"'"$CIRCLE_NODE_INDEX"'"}'
```
## What goes wrong and how do I fix it?
| Symptom | Cause | Fix |
|---------|-------|-----|
| Upload step never runs | Missing `when: always` on the upload `run` step | Add `when: always`, CircleCI skips steps by default once an earlier step fails |
| Branch shows as empty in Gaffer | Build triggered by a tag, not a branch push, and `$CIRCLE_BRANCH` is unset in that case | Fall back with `${CIRCLE_BRANCH:-$CIRCLE_TAG}` |
| 401 Unauthorized on forked-PR builds | CircleCI withholds project environment variables from fork-triggered builds by design | Skip the upload on fork builds (see below). Do not turn on "Pass secrets to builds from forked pull requests" to work around this |
| Same commit shows N nearly-identical runs | `parallelism` fans a job across containers, and each container uploads independently | Add a `container` or similar tag using `$CIRCLE_NODE_INDEX` |
| 401 Unauthorized, variable looks correct | Token missing the `gfr_` prefix, or the variable is defined on the wrong project/context | Re-copy the token, confirm the context is attached to the job if you're using one |
## Next Steps
- [CTRF Guide](/docs/guides/ctrf/) - Use the universal test format
- [Upload API Reference](/docs/upload-api/) - Full API documentation
- [Slack Integration](/docs/integrations/slack/) - Get test results in Slack
**Other CI Providers:** [GitHub Actions](/docs/guides/github-actions/) · [GitLab CI](/docs/guides/gitlab-ci/) · [Jenkins](/docs/guides/jenkins/) · [Bitbucket](/docs/guides/bitbucket-pipelines/) · [Azure DevOps](/docs/guides/azure-devops/)
---
## Fix jest-ctrf-json-reporter ENOENT: CTRF Guide
Source: https://gaffer.sh/docs/guides/ctrf/
import { Aside } from '@astrojs/starlight/components';
CTRF (Common Test Report Format) is a universal JSON schema for test results.
## Why does jest-ctrf-json-reporter throw ENOENT?
Because a directory path was put inside `outputFile`. The reporter writes to `path.join(outputDir, outputFile)` and only creates `outputDir` (with `mkdirSync(..., { recursive: true })`), so any directory segments embedded in `outputFile` are never created and the write fails with `ENOENT: no such file or directory`. The fix is to move the directory into `outputDir` and keep `outputFile` a bare filename.
```javascript
// jest.config.js
['jest-ctrf-json-reporter', { outputFile: 'ctrf/coverage/report.json' }] // ENOENT
['jest-ctrf-json-reporter', { outputDir: 'ctrf/coverage', outputFile: 'report.json' }] // works
```
Both paths are resolved relative to the process working directory, which is your repo root under `npm test` but not necessarily inside a container or a monorepo package. There is no `CTRF_OUTPUT_PATH` environment variable in any CTRF reporter: `outputDir` in the config is the only supported way to move the file. `playwright-ctrf-json-reporter` and `cypress-ctrf-json-reporter` behave the same way. [Full troubleshooting below](/docs/guides/ctrf/#enoent-no-such-file-or-directory-writing-the-ctrf-report).
## Why Use CTRF?
CTRF gives you one report shape across every testing tool, framework, and language you run.
- **Universal**: Works with any test framework that has a CTRF reporter
- **Consistent**: Same format across Jest, Playwright, pytest, RSpec, and more
- **Rich data**: Includes timing, retries, flaky test detection, and metadata
- **Open standard**: Community-driven, MIT-licensed specification at [ctrf.io](https://ctrf.io)
## Supported Frameworks
CTRF has reporters for most popular test frameworks:
| Framework | Package | Language |
|-----------|---------|----------|
| Playwright | [`playwright-ctrf-json-reporter`](https://www.npmjs.com/package/playwright-ctrf-json-reporter) | JavaScript/TypeScript |
| Jest | [`jest-ctrf-json-reporter`](https://www.npmjs.com/package/jest-ctrf-json-reporter) | JavaScript/TypeScript |
| Vitest | [`vitest-ctrf-json-reporter`](https://www.npmjs.com/package/vitest-ctrf-json-reporter) | JavaScript/TypeScript |
| Cypress | [`cypress-ctrf-json-reporter`](https://www.npmjs.com/package/cypress-ctrf-json-reporter) | JavaScript/TypeScript |
| Mocha | [`mocha-ctrf-json-reporter`](https://www.npmjs.com/package/mocha-ctrf-json-reporter) | JavaScript/TypeScript |
| pytest | [`pytest-ctrf`](https://pypi.org/project/pytest-ctrf/) | Python |
| Go | [`ctrf-go-json-reporter`](https://github.com/ctrf-io/go-ctrf-json-reporter) | Go |
| JUnit XML | [`junit-to-ctrf`](https://github.com/ctrf-io/junit-to-ctrf) | Any (CLI converter, not a reporter) |
See the full list at [ctrf.io](https://ctrf.io).
## Installation
### Playwright
```bash
npm install playwright-ctrf-json-reporter --save-dev
```
```typescript
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
reporter: [
['playwright-ctrf-json-reporter', { outputDir: '.', outputFile: 'ctrf-report.json' }],
['list'], // Also show in console
],
});
```
The reporter writes to `
This guide is the plain-HTTP path: no CLI to install, no GitHub Action, no orb. Use it when your CI system doesn't have a dedicated Gaffer guide, when you're debugging what the CLI or Action does under the hood, or when you just want to see a report on your project without adding a dependency.
## Prerequisites
- A [Gaffer account](https://app.gaffer.sh/register?utm_source=gaffer&utm_medium=website&utm_campaign=docs&utm_content=inline) with a project
- Your [project token](/docs/upload-api/), a string starting with `gfr_` followed by 64 hex characters
- cURL installed (available by default on macOS and most Linux distributions)
## What's the simplest way to upload a report?
**POST the file to `/api/upload` as multipart form data with your project token in the `X-API-Key` header.**
```bash title="curl"
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: YOUR_PROJECT_TOKEN" \
-F "files=@path/to/test-report.html"
```
Replace `YOUR_PROJECT_TOKEN` with your actual project token, and update the file path to point to your test report. Don't set `Content-Type` yourself: curl generates the `multipart/form-data` boundary header for you when you use `-F`, and overriding it breaks the request.
## How do I attach commit and branch metadata?
**Pass a `tags` field alongside `files`, as a JSON string.** `commitSha` and `branch` are the two tags Gaffer uses to correlate a run with your code.
```bash title="curl"
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: YOUR_PROJECT_TOKEN" \
-F "files=@playwright-report/index.html" \
-F 'tags={"commitSha":"abc123def456","branch":"main","test_framework":"playwright","test_suite":"e2e"}'
```
## How do I upload more than one file at once?
**Repeat the `-F "files=..."` flag once per file.** Gaffer accepts multiple files in the same request and parses each one it recognizes.
```bash title="curl"
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: YOUR_PROJECT_TOKEN" \
-F "files=@playwright-report/index.html" \
-F "files=@playwright-report/data/report.json" \
-F "files=@playwright-report/screenshots/failed-test.png" \
-F 'tags={"commitSha":"abc123","branch":"feature/login"}'
```
## How do I pull git metadata in automatically?
**Use shell command substitution inside the `tags` field.** This works in any CI system with a git checkout, which is why it's the fallback used later in this guide for CI systems without a dedicated integration.
```bash title="curl"
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: YOUR_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$(git rev-parse HEAD)"'","branch":"'"$(git branch --show-current)"'"}'
```
## What's the full request shape?
The request is `multipart/form-data` over HTTPS. This is the complete contract, not just the fields the examples above use:
| Part | Required | Details |
|------|----------|---------|
| `X-API-Key` header | Yes | Your project token (`gfr_` + 64 hex chars). Missing header returns 401 with "API key required. Provide X-API-Key header." |
| `files` field | Yes | One or more files, repeat the field per file. At least one file with the exact field name `files` is required. |
| `tags` field | No | A JSON object of string keys to string values. Max 20 tags, keys max 64 characters, values max 256 characters, and the whole JSON blob capped at 64 KB. |
Filenames are sanitized server-side; a filename with path-traversal or control characters is rejected with a 400 before anything touches storage.
## What does a successful response look like?
**A 201 with a `testRun` object and a `files` array**, both derived from what you sent:
```json title="Response"
{
"testRun": {
"id": "abc123xyz",
"uniqueId": "1701234567890",
"projectId": "proj_abc123",
"commitSha": "abc123def456",
"branch": "main",
"tags": { "commitSha": "abc123def456", "branch": "main" },
"createdAt": "2024-01-15T10:30:00.000Z"
},
"files": [
{ "filename": "index.html", "size": 245678, "path": "org_123/proj_456/1701234567890/index.html", "contentType": "text/html" }
]
}
```
If you're scripting against this, pull fields out with `jq` instead of parsing JSON by hand:
```bash title="curl"
RESPONSE=$(curl -s -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: YOUR_PROJECT_TOKEN" \
-F "files=@report.html")
TEST_RUN_ID=$(echo "$RESPONSE" | jq -r '.testRun.id')
```
Your project token also authenticates read requests, so you can poll a run's processing status the same way without opening the dashboard:
```bash title="curl"
curl -s "https://app.gaffer.sh/api/v1/user/projects/$PROJECT_ID/upload-sessions/$SESSION_ID" \
-H "X-API-Key: YOUR_PROJECT_TOKEN" | jq -r '.session.processingStatus'
```
`processingStatus` moves through `pending` → `processing` → `completed` (or `error`).
## When do I need multipart upload instead of a single POST?
**Above 75 MB total request size.** `POST /api/upload` enforces a 75 MB ceiling and returns a 413 (`file_too_large_for_v1`) past it, well under Cloudflare's own edge limit, so the error is a clean, actionable one instead of a raw connection failure. For files up to 5 GB (R2's per-object ceiling), use the multipart upload (MPU) endpoints directly.
The MPU flow is three calls: create the upload, PUT each part, then complete it. The server tells you the part size, don't invent your own:
```bash title="mpu-upload.sh"
#!/bin/bash
set -euo pipefail
FILE="huge-playwright-trace.zip"
FILE_SIZE=$(stat -f%z "$FILE" 2>/dev/null || stat -c%s "$FILE")
# 1. Start the multipart upload
CREATE=$(curl -s -X POST https://app.gaffer.sh/api/upload/mpu/create \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"fileName\":\"$FILE\",\"fileSize\":$FILE_SIZE,\"tags\":{\"commitSha\":\"$(git rev-parse HEAD)\",\"branch\":\"$(git branch --show-current)\"}}")
UPLOAD_ID=$(echo "$CREATE" | jq -r '.uploadId')
SESSION_ID=$(echo "$CREATE" | jq -r '.uploadSessionId')
PART_SIZE=$(echo "$CREATE" | jq -r '.partSize')
# 2. Split the file and PUT each part, collecting the etags it returns
split -b "$PART_SIZE" -d -a 4 "$FILE" /tmp/gaffer-part-
PARTS="[]"
i=1
for part in /tmp/gaffer-part-*; do
ETAG=$(curl -s -X PUT "https://app.gaffer.sh/api/upload/mpu/part?uploadSessionId=$SESSION_ID&uploadId=$UPLOAD_ID&partNumber=$i" \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
--data-binary "@$part" | jq -r '.etag')
PARTS=$(echo "$PARTS" | jq --argjson pn "$i" --arg et "$ETAG" '. + [{"partNumber": $pn, "etag": $et}]')
i=$((i + 1))
done
rm -f /tmp/gaffer-part-*
# 3. Complete the upload once every part has landed
curl -s -X POST https://app.gaffer.sh/api/upload/mpu/complete \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"uploadSessionId\":\"$SESSION_ID\",\"uploadId\":\"$UPLOAD_ID\",\"parts\":$PARTS}"
```
The server picks the part size from your declared `fileSize`: 10 MB parts under 500 MB, 25 MB parts up to 2 GB, and 50 MB parts up to the 5 GB ceiling. If a part upload fails partway through, call `DELETE /api/upload/mpu/abort?uploadSessionId=...&uploadId=...` to clean up the orphaned R2 upload rather than leaving it dangling.
## What HTTP errors can the upload endpoint return?
Every error response is JSON with a `reason` field, useful if you're branching on it in a script rather than eyeballing logs.
| Status | `reason` | Cause | Fix |
|--------|----------|-------|-----|
| 400 | `no_files` | No `files` field, or a field named something else | Use `-F "files=@path"` with the exact field name |
| 400 | `invalid_tags` | `tags` isn't valid JSON, or exceeds 20 keys / 64-char keys / 256-char values | Fix the JSON, trim the tag count or length |
| 400 | `invalid_content_type` | Body wasn't sent as `multipart/form-data` | Let curl set `Content-Type`, don't pass your own |
| 400 | `invalid_filename` | Filename contains unsafe or malformed characters | Rename the file before uploading |
| 401 | (none, plain 401) | Missing, empty, or malformed `X-API-Key` | Confirm the header is set and the token starts with `gfr_` |
| 402 | `quota_exceeded` | Free tier only: the upload would push the organization over its storage quota. Pro and Team bill metered overage instead, so they never hit this | Upgrade the plan or delete old reports |
| 413 | `file_too_large_for_v1` | Request over 75 MB | Use the multipart flow above, the CLI, or the GitHub Action |
| 503 | `r2_error`, `db_error` | Transient storage or database failure | Retry, see the next section first |
The multipart endpoints add their own reasons: `invalid_body` and `invalid_query` for malformed create/complete requests, `invalid_part_number` for out-of-order or out-of-range parts, `size_mismatch` if the assembled object's size drifts more than 1% from what you declared in `fileSize`, and `mpu_error` (503) when a part upload or the final assembly fails against storage.
## Is it safe to retry a failed upload?
**Not blindly.** `POST /api/upload` has no client-supplied idempotency key: every call generates a fresh run ID server-side. If curl times out waiting for a response but the upload actually completed, retrying creates a second test run tagged with the same commit and branch.
Retry safely by distinguishing the failure mode:
- **curl never got a response** (connection refused, DNS failure, timeout before headers): safe to retry, nothing was created server-side in most of these cases.
- **curl got a 5xx**: the request was received but failed server-side; retrying is generally fine since these paths clean up on failure, but if you're not sure, check first.
- **curl got any 2xx, or you're unsure whether it landed**: query the upload-sessions API for an existing session with the same `commitSha` before uploading again.
```bash title="curl"
curl -s "https://app.gaffer.sh/api/v1/user/projects/$PROJECT_ID/upload-sessions?commitSha=$(git rev-parse HEAD)" \
-H "X-API-Key: YOUR_PROJECT_TOKEN" | jq '.sessions | length'
```
## How do I upload from a CI system without a Gaffer guide?
**Read git metadata directly instead of relying on CI-specific environment variables**, since those are the one thing that won't be documented for a CI system Gaffer hasn't written a guide for. This pattern works in Drone, Buildkite, TeamCity, Woodpecker, or a bare Docker container, anywhere a git checkout and curl are available:
```bash title="upload.sh"
#!/bin/bash
set -euo pipefail
REPORT_PATH="${1:?Usage: upload.sh }"
COMMIT_SHA=$(git rev-parse HEAD)
BRANCH=$(git rev-parse --abbrev-ref HEAD)
STATUS=$(curl -s -o /tmp/gaffer-response.json -w "%{http_code}" \
-X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@${REPORT_PATH}" \
-F "tags={\"commitSha\":\"${COMMIT_SHA}\",\"branch\":\"${BRANCH}\"}")
if [ "$STATUS" -ge 300 ]; then
echo "Gaffer upload failed with HTTP $STATUS:" >&2
cat /tmp/gaffer-response.json >&2
exit 1
fi
echo "Uploaded to Gaffer: $(jq -r '.testRun.id' /tmp/gaffer-response.json)"
```
Run it as a post-test step, whatever your CI calls that concept: `after_script`, `post`, a second pipeline step gated on the test step's exit code. The important part is running it even when tests fail, since a failing run is usually the one you most want a shareable report for.
## FAQ
### Does uploading to Gaffer require a specific curl version?
No. Every example on this page uses -F for multipart fields, which curl has supported since version 7.1 (1998). Any curl your CI image ships with will work. The only thing to check is that -F sends files as multipart/form-data, not as a raw request body.
### Can I upload with wget or another HTTP client instead of curl?
Yes. The endpoint is plain HTTP: a POST with an X-API-Key header and a multipart/form-data body. Any client that can build a multipart request works, including `wget --post-file`, Python's requests library, or a raw socket. curl is used here because every CI image already has it.
### Why do I get "No files provided" even though I passed -F?
The field name has to be exactly files, and the value has to start with @ so curl reads the file from disk instead of sending the path as a literal string. Using file instead of files, or leaving off the @, both trigger this error.
## Next Steps
- [Upload API Reference](/docs/upload-api/) - Field-by-field reference for the single-POST endpoint
- [GitHub Action](/docs/guides/github-actions/) - Automate uploads without hand-writing the multipart flow
---
## GitHub Actions
Source: https://gaffer.sh/docs/guides/github-actions/
import { Aside } from '@astrojs/starlight/components';
The official [Gaffer Uploader](https://github.com/gaffer-sh/gaffer-uploader) GitHub Action makes it easy to automatically upload test reports from your CI workflows.
## Prerequisites
- A [Gaffer account](https://app.gaffer.sh/register?utm_source=gaffer&utm_medium=website&utm_campaign=docs&utm_content=inline) with a project
- Your [project token](/docs/upload-api/)
- A GitHub repository with a test workflow
## Setup
### 1. Add your project token as a secret
1. Go to your GitHub repository
2. Navigate to **Settings** → **Secrets and variables** → **Actions**
3. Click **New repository secret**
4. Name: `GAFFER_PROJECT_TOKEN`
5. Value: Your Gaffer project token
### 2. Add the upload step to your workflow
Add the Gaffer uploader step after your test step. Use `if: always()` to ensure reports are uploaded even when tests fail.
### Without a stored token (OIDC)
Skip the `GAFFER_PROJECT_TOKEN` secret entirely by granting the job `id-token: write`. The `gaffer` CLI inside the uploader exchanges the runner's own GitHub Actions OIDC identity token for a project-scoped upload token, so there's nothing to create or rotate:
```yaml
name: Tests
on: push
jobs:
test:
runs-on: ubuntu-latest
permissions:
id-token: write # required: lets gaffer request a runner identity token
contents: read
steps:
- uses: actions/checkout@v4
- name: Run tests
run: npm test
- name: Upload to Gaffer
uses: gaffer-sh/gaffer-uploader@v2
if: always()
with:
report_path: ./test-results
commit_sha: ${{ github.sha }}
branch: ${{ github.ref_name }}
# no gaffer_upload_token
```
On the first upload from a repository, Gaffer automatically provisions an unclaimed organization and project for that repository, no signup step required. The job log prints a claim URL; a repo admin visits it once to claim the org into their own account. If `gaffer_upload_token` (or the deprecated `gaffer_api_key`) is set, it still works and takes precedence: OIDC is only the fallback when no token is configured.
## Inputs
| Input | Required | Description |
|-------|----------|-------------|
| `gaffer_upload_token` | No | Your Gaffer project token (the deprecated `gaffer_api_key` alias still works but emits a deprecation warning). Omit it under `permissions: id-token: write` to authenticate via OIDC instead; see [Without a stored token](#without-a-stored-token-oidc) |
| `report_path` | Yes | Path to the report file or directory to upload |
| `api_endpoint` | No | Custom API endpoint URL (for staging/preview environments) |
| `commit_sha` | No | Git commit SHA to associate with the test run |
| `branch` | No | Git branch name to associate with the test run |
| `test_framework` | No | Test framework used (e.g., `playwright`, `jest`, `pytest`) |
| `test_suite` | No | Name of the test suite (e.g., `unit`, `e2e`) |
## Examples
### Basic Usage
```yaml
name: Tests
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
- name: Upload to Gaffer
uses: gaffer-sh/gaffer-uploader@v2
if: always()
with:
gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }}
report_path: ./test-results
commit_sha: ${{ github.sha }}
branch: ${{ github.ref_name }}
```
### Playwright Example
A complete example for Playwright tests:
```yaml
name: Playwright Tests
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v3
with:
version: 9
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'pnpm'
- name: Install dependencies
run: pnpm install
- name: Install Playwright browsers
run: pnpm exec playwright install --with-deps
- name: Run Playwright tests
run: pnpm exec playwright test --reporter=html
- name: Upload Playwright Report to Gaffer
uses: gaffer-sh/gaffer-uploader@v2
if: always()
with:
gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }}
report_path: ./playwright-report
commit_sha: ${{ github.sha }}
branch: ${{ github.ref_name }}
test_framework: playwright
test_suite: e2e
```
`--reporter=html` is what creates the `playwright-report/` directory. Playwright's default reporter writes to the terminal only (`list` locally, `dot` in CI), so without it the upload step has nothing to find. If your `playwright.config.ts` already sets `reporter: 'html'`, you can drop the flag.
### Pull Request Workflow
For pull request workflows, use `github.head_ref` to get the correct branch name:
```yaml
name: PR Tests
on:
pull_request:
branches: [main]
env:
branch_name: ${{ github.head_ref || github.ref_name }}
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
run: npm test
- name: Upload to Gaffer
uses: gaffer-sh/gaffer-uploader@v2
if: always()
with:
gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }}
report_path: ./test-results
commit_sha: ${{ github.event.pull_request.head.sha }}
branch: ${{ env.branch_name }}
test_framework: jest
test_suite: unit
```
## Using CTRF Format
For a standardized format across all your test frameworks, consider using [CTRF](/docs/guides/ctrf/). Select the reporter in your test config, not on the command line:
```javascript
// jest.config.js
module.exports = {
reporters: [
'default',
['jest-ctrf-json-reporter', { outputDir: '.', outputFile: 'ctrf-report.json' }],
],
};
```
```yaml
- name: Install CTRF reporter
run: npm install --save-dev jest-ctrf-json-reporter # or playwright-ctrf-json-reporter, vitest-ctrf-json-reporter
- name: Run tests with CTRF reporter
run: npm test
- name: Upload CTRF report to Gaffer
uses: gaffer-sh/gaffer-uploader@v2
if: always()
with:
gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }}
report_path: ./ctrf-report.json
commit_sha: ${{ github.sha }}
branch: ${{ github.ref_name }}
```
## Environment Variables
GitHub Actions provides these variables for your workflow:
| Variable | Description | Example |
|----------|-------------|---------|
| `${{ github.sha }}` | Full commit SHA | `abc123def456...` |
| `${{ github.ref_name }}` | Branch or tag name | `main`, `feature/login` |
| `${{ github.head_ref }}` | PR source branch (PR workflows only) | `feature/login` |
| `${{ github.event.pull_request.head.sha }}` | PR head commit SHA | `def456abc789...` |
## Troubleshooting
### Report not uploading
- Verify the `report_path` points to the correct file or directory
- Confirm your test command actually writes a report there. Several runners print results to the terminal by default and produce no file at all until you select a file-based reporter (Playwright needs `--reporter=html` or `reporter: 'html'` in its config)
- Check that your project token secret is named exactly `GAFFER_PROJECT_TOKEN`
- Ensure `if: always()` is set so the step runs even when tests fail
### Wrong branch showing
For pull request workflows, use `github.head_ref || github.ref_name` pattern to get the correct source branch name.
### 401 Unauthorized
- Check your project token starts with `gfr_`
- Verify the token is correctly copied (no extra spaces)
- Make sure the secret is accessible to the workflow
## Next Steps
- [CTRF Guide](/docs/guides/ctrf/) - Use the universal test format
- [Upload API Reference](/docs/upload-api/) - Full API documentation
- [cURL Guide](/docs/guides/curl/) - Manual uploads for debugging
- [Slack Integration](/docs/integrations/slack/) - Get test results in Slack
**Other CI Providers:** [GitLab CI](/docs/guides/gitlab-ci/) · [CircleCI](/docs/guides/circleci/) · [Jenkins](/docs/guides/jenkins/) · [Bitbucket](/docs/guides/bitbucket-pipelines/) · [Azure DevOps](/docs/guides/azure-devops/)
---
## GitLab CI
Source: https://gaffer.sh/docs/guides/gitlab-ci/
import { Aside } from '@astrojs/starlight/components'
import JsonLd from '../../../../components/JsonLd.astro'
GitLab CI/CD runs tests on every push, but its own reporting resets at the merge request boundary. Gaffer adds run history, flaky detection, and a link you can hand to anyone, whether or not they have GitLab access.
## Prerequisites
- A [Gaffer account](https://app.gaffer.sh/register?utm_source=gaffer&utm_medium=website&utm_campaign=docs&utm_content=inline) with a project
- Your [project token](/docs/upload-api/)
- A GitLab repository with a `.gitlab-ci.yml` file
## How do I add my Gaffer project token to a GitLab pipeline?
**Store it as a masked CI/CD variable**, not inline in the YAML.
1. Go to your GitLab project
2. Navigate to **Settings** → **CI/CD** → **Variables**
3. Click **Add variable**
4. Key: `GAFFER_PROJECT_TOKEN`
5. Value: your Gaffer project token
6. Check **Mask variable** to hide it in job logs
## How do I upload a report after the test job runs?
**Add the curl call to `after_script`.** Unlike `script`, `after_script` runs even when the job's `script` section fails, which is when you most need the report.
```yaml
test:
stage: test
script:
- npm ci
- npm test
after_script:
- |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'"}'
artifacts:
when: always
reports:
junit: test-results/junit.xml
```
Keeping `artifacts:reports:junit` alongside the Gaffer upload is intentional: GitLab's widget gives reviewers an inline pass/fail count on the merge request itself, and Gaffer gives you the history that widget doesn't keep once the artifact retention window passes.
## Which GitLab CI variables carry commit and branch info?
| Variable | Description | Example |
|----------|-------------|---------|
| `$CI_COMMIT_SHA` | Full commit SHA | `abc123def456...` |
| `$CI_COMMIT_REF_NAME` | Branch or tag name | `main`, `feature/login` |
| `$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | Source branch in MR pipelines | `feature/login` |
| `$CI_NODE_INDEX` | 1-based index of the current job in a `parallel:` job. Not set by `parallel:matrix:` | `1`, `2`, `3` |
| `$CI_PROJECT_NAME` | Project name | `my-app` |
## Framework examples
### Playwright
```yaml
playwright:
stage: test
image: mcr.microsoft.com/playwright:v1.40.0-jammy
script:
- npm ci
- npx playwright install --with-deps
- npx playwright test
after_script:
- |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@playwright-report/index.html" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'","test_framework":"playwright","test_suite":"e2e"}'
artifacts:
when: always
paths:
- playwright-report/
```
### Jest with JUnit reporter
```yaml
jest:
stage: test
script:
- npm ci
- npm test -- --reporters=default --reporters=jest-junit
after_script:
- |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@junit.xml" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'","test_framework":"jest"}'
artifacts:
when: always
reports:
junit: junit.xml
```
### pytest
```yaml
pytest:
stage: test
image: python:3.11
script:
- pip install pytest pytest-html
- pytest --html=report.html --self-contained-html
after_script:
- |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@report.html" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'","test_framework":"pytest"}'
artifacts:
when: always
paths:
- report.html
```
### Using CTRF format
For a standardized format across all your test frameworks, use [CTRF](/docs/guides/ctrf/):
```yaml
test:
script:
- npm ci
# Install the CTRF reporter for your framework:
# npm install --save-dev jest-ctrf-json-reporter
# npm install --save-dev playwright-ctrf-json-reporter
# npm install --save-dev vitest-ctrf-json-reporter
- npm test -- --reporter=jest-ctrf-json-reporter
after_script:
- |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@ctrf-report.json" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'"}'
```
## Does Gaffer replace GitLab's JUnit test report widget?
**No, run both.** GitLab's `artifacts:reports:junit` widget shows pass/fail counts inline on a single merge request and disappears once that job's artifact retention expires. Gaffer keeps every run's full history, flags tests that flip pass/fail across branches, and gives you a link that works for people without GitLab access, like a customer asking why a release slipped. Use the widget for the at-a-glance MR check and Gaffer for anything that needs to survive past that one pipeline.
## How do I upload from merge request pipelines specifically?
**Use `rules:` to detect an MR pipeline, and tag the branch with `$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME`.** That variable is set only in merge request pipelines and names the source branch explicitly. `$CI_COMMIT_REF_NAME` resolves to the same branch there, so keep it as the fallback for ordinary branch pipelines. The synthetic ref `refs/merge-requests/42/head` lives in `$CI_MERGE_REQUEST_REF_PATH`, which is not what you want in a branch tag:
```yaml
test:
rules:
- if: $CI_MERGE_REQUEST_IID
script:
- npm test
after_script:
- |
BRANCH="${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME:-$CI_COMMIT_REF_NAME}"
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$BRANCH"'"}'
```
## How do I structure a test-then-upload pipeline with `needs:`?
**Split the upload into its own job that `needs:` the test job's artifacts**, so it runs as soon as the test job finishes instead of waiting for the whole stage to complete:
```yaml
stages:
- test
- report
test:
stage: test
script:
- npm ci
- npm test || true # let the pipeline continue to the report job
artifacts:
when: always
paths:
- test-results/
upload-to-gaffer:
stage: report
needs:
- job: test
artifacts: true
script:
- |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'"}'
```
`needs:` creates a direct dependency edge rather than a stage-order one, so `upload-to-gaffer` starts the moment `test` finishes instead of waiting on everything else in the `test` stage. This only works because `artifacts: true` passes the report file across the job boundary. A GitLab job's workspace doesn't persist between jobs on its own: if you split test and upload into separate jobs and skip `artifacts:`, the upload job won't find the report file at all, since it's running on a clean checkout with none of the previous job's output.
## Can I upload to Gaffer from a job that only runs on failure?
**Yes, and `after_script` already does it.** `after_script` runs even when the preceding `script` section fails, which is the GitLab equivalent of running a step regardless of exit code. If you would rather have the upload as its own job than as a script hook, gate that job with `rules: - when: on_failure`, which GitLab evaluates only once a job in an earlier stage has failed.
## How do I upload from a GitLab parallel matrix job?
**Tag each upload with one of your matrix variables**, such as the `$SHARD` value below, so you can tell the runs apart later. `$CI_NODE_INDEX` is not set in `parallel:matrix:` jobs, only in `parallel:` jobs that take an integer. Gaffer records each `curl` call as a separate test run; without a distinguishing tag, four parallel shards produce four runs on the same commit and branch with no way to tell which shard is which.
```yaml
test:
parallel:
matrix:
- SHARD: [1, 2, 3, 4]
script:
- npm test -- --shard=$SHARD/4
after_script:
- |
curl -X POST https://app.gaffer.sh/api/upload \
-H "X-API-Key: $GAFFER_PROJECT_TOKEN" \
-F "files=@test-results/junit.xml" \
-F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'","shard":"'"$SHARD"'"}'
```
## What goes wrong and how do I fix it?
| Symptom | Cause | Fix |
|---------|-------|-----|
| 401 API key required, and the variable looks correctly set | The variable is marked **Protected** but the pipeline is running on an unprotected branch or a fork MR | Unprotect the variable, or restrict the upload job to protected refs with `rules: - if: $CI_COMMIT_REF_PROTECTED` |
| Upload never runs | The curl call is in `script` instead of `after_script`, and the test step failed first | Move the upload to `after_script`, which runs regardless of the preceding step's exit code |
| curl can't find the report file in a later job | Job workspaces don't persist between jobs without `artifacts:` | Add `artifacts: paths:` to the test job and `needs: - job: test artifacts: true` to the upload job |
| Branch shows as a merge-request ref like `refs/merge-requests/42/head` | Tagging with `$CI_MERGE_REQUEST_REF_PATH`, which holds the synthetic MR ref rather than a branch name | Use `$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` instead |
| 401 Unauthorized, variable is set correctly | Token missing the `gfr_` prefix, or extra whitespace from copy-paste | Re-copy the token from project settings, check for trailing newlines in the variable value |
## Next Steps
- [CTRF Guide](/docs/guides/ctrf/) - Use the universal test format
- [Upload API Reference](/docs/upload-api/) - Full API documentation
- [Slack Integration](/docs/integrations/slack/) - Get test results in Slack
**Other CI Providers:** [GitHub Actions](/docs/guides/github-actions/) · [CircleCI](/docs/guides/circleci/) · [Jenkins](/docs/guides/jenkins/) · [Bitbucket](/docs/guides/bitbucket-pipelines/) · [Azure DevOps](/docs/guides/azure-devops/)
---
## CI Provider Guides
Source: https://gaffer.sh/docs/guides/index/
import { LinkCard, CardGrid } from '@astrojs/starlight/components'
import { Aside } from '@astrojs/starlight/components'
Choose your CI provider below to get started with automatic test report uploads.
## Universal Format: CTRF
CTRF (Common Test Report Format) provides a standardized JSON format that works with any CI provider and test framework. Use it for consistent analytics across your entire test suite.
[Learn about CTRF →](/docs/guides/ctrf/)
## CI Providers
Field Reference
| Field | Type | Notes | |-------|------|-------| | `upload_id` | string | Parent upload ID (`upl_*`). Stable across schema changes. Useful for correlating the webhook to dashboard views and to the `hosted_report_url`. | | `dashboard_url` | string | Deep link to the Gaffer dashboard for this run or coverage report. Opening this link in a browser authenticates via session or prompts sign-in. | | `hosted_report_url` | string or null | Link to the hosted HTML report (e.g., Playwright `index.html`) when the upload includes one. `null` when the upload is a raw report file (JUnit XML, lcov, etc.) with no browsable HTML. Opening requires sign-in or a valid share — the link is designed to drop recipients straight into Gaffer. Coverage events may return `null` for every upload today. | | `url` | string | **Deprecated.** Alias of `dashboard_url`, retained for backwards compatibility. Switch to `dashboard_url`. Will be removed on **Sat, 01 Aug 2026**. Outbound requests include `Deprecation` and `Link: rel="successor-version"` headers pointing here. | ### Test Event When you click "Test" in the dashboard, Gaffer sends a test payload: ```json title="test event" { "event": "test", "timestamp": "2026-01-25T14:30:00.000Z", "delivery_id": "770e8400-e29b-41d4-a716-446655440000", "test": true, "data": { "message": "This is a test webhook from Gaffer", "webhook_id": "whk_abc123", "webhook_name": "My Webhook", "organization_id": "org_ghi789" } } ``` ## HTTP Headers Every webhook request includes these headers: | Header | Description | |--------|-------------| | `Content-Type` | `application/json` | | `X-Gaffer-Event` | Event type: `test_run`, `coverage`, or `test` | | `X-Gaffer-Signature` | HMAC-SHA256 signature of the request body | | `X-Gaffer-Delivery` | Unique delivery ID (UUID) for idempotency | | `User-Agent` | `Gaffer-Webhooks/1.0` | | `Deprecation` | Present on `test_run` and `coverage` events. HTTP-date of the sunset for the deprecated `url` field. See [Field Reference](#v2-payload). | | `Link` | Present on `test_run` and `coverage` events. Points at this documentation section with `rel="successor-version"`. | ## Verifying Signatures Every webhook is signed with your webhook's secret using HMAC-SHA256. Always verify the signature before processing the payload to confirm it came from Gaffer. ### Node.js ```javascript title="verify-signature.js" const crypto = require('crypto'); function verifySignature(payload, signature, secret) { const expected = 'sha256=' + crypto .createHmac('sha256', secret) .update(payload) .digest('hex'); return crypto.timingSafeEqual( Buffer.from(signature), Buffer.from(expected) ); } // In your request handler: const payload = JSON.stringify(req.body); const signature = req.headers['x-gaffer-signature']; if (!verifySignature(payload, signature, process.env.GAFFER_WEBHOOK_SECRET)) { return res.status(401).send('Invalid signature'); } ``` ### Python ```python title="verify_signature.py" import hmac import hashlib def verify_signature(payload: bytes, signature: str, secret: str) -> bool: expected = 'sha256=' + hmac.new( secret.encode(), payload, hashlib.sha256 ).hexdigest() return hmac.compare_digest(signature, expected) # In your request handler: payload = request.get_data() signature = request.headers.get('X-Gaffer-Signature') if not verify_signature(payload, signature, os.environ['GAFFER_WEBHOOK_SECRET']): abort(401) ``` ## Retry Behavior Gaffer retries webhook deliveries when your endpoint returns a 5xx status code or a network error occurs. Deliveries that receive a 4xx response are not retried — these indicate a permanent issue with the request (e.g., your endpoint rejected it). Each delivery has a unique `X-Gaffer-Delivery` ID. Use this for idempotency — if you receive the same delivery ID twice, you can safely skip the duplicate. ## Testing Webhooks 1. Create your webhook in **Settings > Webhooks** 2. Click the **"Test"** button next to your webhook 3. Gaffer sends a test payload to your endpoint 4. Check the result — the dashboard shows whether the delivery succeeded or failed, including the HTTP status code For local development, use a tunnel service like [ngrok](https://ngrok.com) to expose your local server to the internet. ## Plan Limits | Plan | Webhooks | |------|----------| | Free | 1 | | Pro | 5 | | Team | Unlimited | ## Troubleshooting ### Not receiving webhooks? - **Check the webhook is enabled** — disabled webhooks don't fire - **Verify branch filters** — if configured, only matching branches trigger the webhook - **Check notification trigger** — "failures only" won't fire for passing test runs - **Check event types** — make sure the relevant event type (test runs or coverage) is enabled - **Check "Last Sent" and error status** — the webhooks list shows the last delivery time and any errors ### Receiving a 401 from signature verification? - **Use the raw request body** — verify against the exact bytes received, not a re-serialized version - **Check the secret** — make sure you're using the correct secret for this webhook - **Regenerate if needed** — edit the webhook and check "Regenerate secret" to get a new one ### Endpoint URL rejected? - **HTTPS required** — webhook URLs must use HTTPS - **No private IPs** — localhost, 127.0.0.1, and private IP ranges (10.x, 172.16-31.x, 192.168.x) are blocked for security ## Security - **HTTPS only** — webhook URLs must use HTTPS to protect payloads in transit - **SSRF protection** — private IP ranges and localhost are blocked - **Encrypted secrets** — signing secrets are encrypted at rest using AES-256-GCM - **HMAC-SHA256 signing** — every payload is signed so you can verify authenticity --- ## Introduction Source: https://gaffer.sh/docs/introduction/ Gaffer wraps your test command and extracts flaky tests, failure clusters, health scores, and coverage trends from raw results. Same data in three places: - **[CLI](/docs/cli/)** — `gaffer test` enriches your terminal output after every run - **Dashboard** — your team sees history, trends, and share links at [app.gaffer.sh](https://app.gaffer.sh) - **[MCP server](/docs/mcp/)** — AI coding agents (Claude Code, Cursor, Windsurf) query your test data and act on it ## What you get - **[Flaky test detection](/solutions/flaky-test-detection/)** — flip rates, composite scores, and history for every test that alternates between pass and fail - **Failure clusters** — groups failing tests by shared error patterns so you fix root causes, not symptoms - **Health scores** — a single 0–100 number combining pass rate, flakiness, and trend direction - **[Coverage tracking](/docs/coverage/)** — line, branch, and function coverage with GitHub commit status gating - **Duration analysis** — P50/P75/P90/P95/P99 percentiles and slowest test identification - **Works with everything** — Playwright, Jest, Vitest, pytest, Go, RSpec, JUnit XML, CTRF, and any CI provider ## How it works ```bash # Install curl -fsSL https://app.gaffer.sh/install.sh | sh # Wrap your test command gaffer test -- npm test ``` Gaffer runs your tests, discovers report files, parses results, computes analytics, prints an enriched summary, and syncs to the cloud — all in one command. Your test command's exit code passes through unchanged. ## Next steps Head to the [Getting Started](/docs/getting-started/) guide to install the CLI and run your first test. --- ## MCP Server Source: https://gaffer.sh/docs/mcp/ The Gaffer MCP server (`@gaffer-sh/mcp`) connects AI coding assistants to your test history. Ask about test health, investigate flaky tests, and get context about failures without leaving your editor. ## What is MCP? The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is an open standard that lets AI assistants access external tools and data sources. By installing the Gaffer MCP server, your AI assistant gains direct access to your test analytics. ## Features - **Test health insights** - Ask about pass rates, trends, and overall test suite health - **Flaky test detection** - Identify tests with inconsistent behavior - **Test history lookup** - Check the history of specific tests to understand stability - **Cross-project access** - Query all your projects from a single API key - **Report file access** - Get links to HTML reports, coverage files, and other artifacts - **Slowest test analysis** - Find tests that are slowing down your CI pipeline ## Prerequisites 1. A [Gaffer account](https://app.gaffer.sh) with test results uploaded 2. An API Key from **Account Settings > API Keys** ## Setup ### Claude Code Add to your Claude Code settings (`~/.claude.json` or project `.claude/settings.json`): ```json { "mcpServers": { "gaffer": { "command": "npx", "args": ["-y", "@gaffer-sh/mcp"], "env": { "GAFFER_API_KEY": "gaf_your_api_key_here" } } } } ``` ### Cursor Add to `.cursor/mcp.json` in your project: ```json { "mcpServers": { "gaffer": { "command": "npx", "args": ["-y", "@gaffer-sh/mcp"], "env": { "GAFFER_API_KEY": "gaf_your_api_key_here" } } } } ``` ## What tools does the Gaffer MCP server expose? Three. The server runs in **code mode**: rather than one MCP tool per API call, it exposes three tools plus a `codemode` namespace of 17 analytics functions that you call from JavaScript. That keeps 17 tool definitions out of the context window, and it lets one execution chain several calls instead of paying a round-trip each. | MCP tool | What it does | |----------|--------------| | `execute_code` | Runs JavaScript against `codemode.Your AI agent just "fixed" a test that's been flipping between pass and fail for two weeks. It spent tokens debugging a 40% flip-rate flaky test that no human would touch. The agent doesn't know that because it has no memory of what happened before this session.
## The Problem AI coding agents operate without test history. Each session starts from zero. This leads to three failure modes that waste time and CI minutes: **Wasted cycles on flaky tests.** An agent sees a test failure and tries to fix it. But the test has a 40% flip rate: it was going to pass on the next run anyway. The agent doesn't know because it can't see historical pass/fail patterns. **Duplicate work on shared root causes.** CI reports 14 failures. The agent attempts 14 separate fixes. In reality, 11 of those failures share one root cause: a database connection timeout. Fixing one line would have resolved them all. **No before/after comparison.** After making changes, the agent can't verify whether a specific test actually improved. Did the fix work, or did a flaky test just happen to pass this time? Without comparing metrics across commits, the agent is guessing. This isn't theoretical. Google's 2025 DORA report found that [AI coding tools correlate with a 7.2% decrease in delivery stability](https://dora.dev/research/2025/ai-assistance/). The agents are fast at writing code, but they make poor decisions when they can't see test history. ## What Teams Try (and Why It Falls Short) **Piping CI logs into context.** Raw CI output is unstructured text designed for humans. An agent can't compute a flip rate from log lines. It can't cluster failures by error similarity. And it definitely can't compare results across commits when it's parsing different log formats from Playwright, Jest, and Pytest. **General-purpose memory tools.** Tools like mem0 or Letta give agents fuzzy vector recall, useful for remembering conversations but not for computing that `test_checkout_flow` has flipped 8 times in 20 runs. You can't calculate a flakiness score from semantic similarity search. **Checking CI directly.** GitHub Actions artifacts expire. Navigating workflow runs programmatically is brittle. And even if the agent finds the right artifact, it gets back HTML reports or raw XML, not structured data it can reason about. ## Structured Test Memory "Test memory" means giving agents deterministic, queryable access to test history — not fuzzy recall, but exact data: pass rates, flip counts, failure clusters, duration changes. Gaffer's [MCP server](https://www.npmjs.com/package/@gaffer-sh/mcp) exposes this as structured tool calls that any MCP-compatible agent (Claude, Cursor, Windsurf, Copilot) can use automatically. ### Identify Flaky Tests Before Wasting Time ```json // get_flaky_tests response { "flakyTests": [ { "name": "should complete checkout flow", "flipRate": 0.4, "flipCount": 8, "totalRuns": 20, "flakinessScore": 0.72 } ], "summary": { "totalFlaky": 3, "threshold": 0.1, "period": 30 } } ``` The agent sees a `flakinessScore` of 0.72 and a 40% flip rate. It skips the flaky test and focuses on real regressions. ### Group Failures by Root Cause ```json // get_failure_clusters response { "clusters": [ { "representativeError": "Connection refused: localhost:5432", "count": 11, "tests": [ { "name": "should create user", "fullName": "Auth > should create user" }, { "name": "should update profile", "fullName": "Profile > should update profile" } ] }, { "representativeError": "Expected 200, received 401", "count": 3, "tests": [ { "name": "should access dashboard", "fullName": "Dashboard > should access dashboard" } ] } ], "totalFailures": 14 } ``` 14 failures, 2 root causes. The agent fixes the database connection issue and the auth bug, not 14 individual tests. ### Track Pass/Fail Across Commits ```json // get_test_history response { "history": [ { "status": "failed", "commitSha": "a1b2c3", "durationMs": 4200 }, { "status": "passed", "commitSha": "d4e5f6", "durationMs": 3800 }, { "status": "passed", "commitSha": "g7h8i9", "durationMs": 3900 } ], "summary": { "totalRuns": 3, "passedRuns": 2, "failedRuns": 1, "passRate": 66.67 } } ``` Failed once, passed twice before: likely a regression in the latest commit, not a flaky test. ### Compare Before and After ```json // compare_test_metrics response { "before": { "status": "failed", "durationMs": 12400, "commit": "a1b2c3" }, "after": { "status": "passed", "durationMs": 3200, "commit": "d4e5f6" }, "change": { "statusChanged": true, "percentChange": -74.2 } } ``` The fix worked: status changed from failed to passed, duration dropped 74%. No guessing. ## Comparison | Capability | CI Logs | Vector Memory | Gaffer MCP | |------------|---------|---------------|------------| | Flip rate / flakiness score | No | No | Yes | | Failure clustering by root cause | No | No | Yes | | Cross-commit comparison | No | Approximate | Exact | | Structured, queryable data | No | Fuzzy | Yes | | Works across test frameworks | Varies | N/A | Yes | | Agent can use without parsing | No | Yes | Yes | ## Setup Three steps. No code changes to your test suite. ### 1. Upload test results from CI Add the [Gaffer uploader](/docs/guides/github-actions/) to your CI pipeline. Supports GitHub Actions, GitLab CI, CircleCI, and others. If your CI system isn't covered, the [CLI](/docs/cli/) uploads from any shell. ```yaml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} ``` ### 2. Add the MCP server ```bash claude mcp add gaffer -e GAFFER_API_KEY=gaf_your_key -- npx -y @gaffer-sh/mcp@latest ``` Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool. The [MCP server docs](/docs/mcp/) list every tool the agent can call and the configuration for each client. The site itself also exposes [WebMCP tools](/blog/webmcp-agent-callable-marketing-site/) for browser agents, separate from the MCP server. ### 3. Agent queries automatically Once connected, the agent calls the right tool based on context. No configuration for when to use which tool: it sees a test failure and checks history, checks flakiness, clusters failures, and compares results across commits. ## Get Started Gaffer's free tier includes test history and flaky test detection. The MCP server is [open source](https://github.com/gaffer-sh/mcp).You picked Allure for its HTML reports. Then you tried to share those reports with your team, and hit the real problem: self-hosting a server, configuring a database, managing CI integration, handling user access, and keeping everything updated. For many teams, the infrastructure tax exceeds the reporting benefit.
## Why Teams Look for an Allure Report Alternative Allure Report generates beautiful HTML reports locally. Allure TestOps (the paid product) adds history and collaboration features. But both come with trade-offs: ### Self-Hosting Complexity Running Allure TestOps means: - Setting up and maintaining a server - Configuring PostgreSQL or another database - Managing backups and storage - Handling authentication and user management - Keeping everything updated and secure For small teams, this is a lot of overhead just to share test results. ### CI Pipeline Integration Allure requires adding the Allure commandline tool to your CI environment, generating reports, then uploading them somewhere your team can access. This works, but it's another moving part to maintain. ### Report Hosting Allure generates static HTML reports, but hosting them requires additional setup. Common approaches: - GitHub Pages (works but limited for private repos) - S3 + CloudFront (more infrastructure to manage) - Self-hosted web server (even more maintenance) ## Gaffer: Hosted Test Reporting Gaffer takes a different approach: hosted test reporting with zero infrastructure. ### No Self-Hosting Gaffer is a SaaS platform. No servers to manage, no databases to configure, no updates to apply. Sign up and start uploading test results. ### Simple CI Integration One step in your CI pipeline: ```yaml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./test-results ``` Or with curl if you prefer: ```bash curl -X POST https://app.gaffer.sh/api/upload \ -H "X-API-Key: $GAFFER_PROJECT_TOKEN" \ -F "files=@test-results/report.html" ``` One YAML step is all it takes — no additional tools in CI, no database, no server to maintain. Teams migrating from Allure typically have their first report uploaded within ten minutes of creating a Gaffer project. ### Instant Team Sharing Every upload gets a shareable URL. Send it in Slack, paste it in a PR, or bookmark it. No downloads, no zip files, no "let me spin up the report server." ### Built-in Flaky Test Detection Gaffer automatically tracks test stability over time and identifies flaky tests by analyzing flip rates. No additional setup required: just upload your results. Full detail in [flaky test detection](/solutions/flaky-test-detection/). ## Feature Comparison | Feature | Allure Report | Allure TestOps | Gaffer | |---------|---------------|----------------|--------| | Test reports | Local HTML | Hosted | Hosted | | Historical trends | Manual setup | Yes | Yes | | Flaky test detection | No | Yes | Yes | | Team sharing | Manual hosting | Yes | Yes | | Self-hosting required | N/A | Yes | No | | Database required | No | Yes | No | | Pricing | Free | Per user | Flat-rate tiers | | Setup time | Minutes | Hours/Days | Minutes | ## Framework Support Both Allure and Gaffer support major test frameworks: | Framework | Allure | Gaffer | |-----------|--------|--------| | Playwright | Yes | Yes | | Jest | Yes | Yes | | Vitest | Via adapter | Yes | | pytest | Yes | Yes | | JUnit | Yes | Yes ([JUnit XML guide](/blog/junit-xml-format-guide/)) | | Cypress | Yes | Yes (via CTRF) | Gaffer also supports CTRF (Common Test Report Format), which provides adapters for 20+ test frameworks. ## When to Use Allure Allure might be a better fit if you: - Already have infrastructure and ops capacity to self-host - Need Allure's specific annotation features (@Step, @Attachment) - Have strict data residency requirements that prevent SaaS usage - Want to avoid any SaaS vendor dependency ## When to Use Gaffer Gaffer is probably a better fit if you: - Want test reporting without infrastructure overhead - Have a small team without dedicated DevOps - Need quick setup (minutes, not hours) - Want built-in Slack notifications and GitHub integration - Prefer flat-rate pricing with unlimited users over per-seat ## Migration from Allure Already using Allure? Gaffer works alongside your existing setup: 1. Keep generating Allure reports if your team is used to them 2. Add the Gaffer upload step to CI 3. Use CTRF or JUnit XML format for Gaffer analytics 4. Gradually shift to Gaffer for sharing and history No big-bang migration required. ## Other Comparisons Evaluating multiple tools? See how Gaffer compares to [ReportPortal](/solutions/reportportal-alternative/), [Currents.dev](/solutions/currents-dev-alternative/), and [TestDino](/solutions/testdino-vs-gaffer/). ## Try It Gaffer's free tier includes 500 MB storage and 7-day retention. Enough to evaluate whether it fits your workflow.A CI run goes red with 14 failed tests. You open the job log and start scrolling: stack traces, framework noise, retry output, the occasional unrelated warning. Twenty minutes later you've confirmed what you suspected on line one, that most of those failures share a single cause. Analyzing test failures shouldn't mean reading thousands of lines of console output to find the three that matter.
## What Analyzing Test Failures Actually Means for a CI/CD Team Analyzing test failures is the work of turning a list of red results into a short list of decisions: what to fix now, what to ignore, what to file. The raw CI log gives you the symptoms. The job is to find the cause, decide who owns it, and confirm the fix landed. ### Test failure vs. error: the distinction that changes your response A **test failure** is an assertion that didn't hold: the code ran, produced a result, and the result was wrong. `expected 200, received 500`. A **test error** is the test itself not completing: a thrown exception, a timeout, a missing fixture, a connection refused before any assertion ran. The distinction matters because the response differs. A failure usually points at application code or a stale expectation. An error often points at the environment, a dependency, or test setup. Treating every red result the same way is how teams waste time mocking around an assertion when the real problem is that a service never came up. ## Why Test Failures Are Hard to Triage at Scale One failed test is easy. A suite that produces dozens of failures across multiple CI runs per day is where triage breaks down, and where most teams fall into one of three traps. **The log-scanning trap.** You read the CI job output top to bottom. It works for a handful of failures and collapses past that. Logs are ordered by execution time, not by cause, so five failures from one dead database land scattered between unrelated passes and warnings. You reconstruct the pattern by hand, every single run. **The rerun-until-green trap.** Hit re-run, hope the failures were flaky, merge if they clear. Sometimes they were flaky. Sometimes you just rolled the dice on a real bug and got lucky. Either way you learned nothing about which tests are actually unreliable, so you do it again next week. **The spreadsheet trap.** Someone starts tracking failures manually: test name, date, "looking into it." It's stale within a week because nothing updates it automatically, and it answers none of the questions that matter, like whether this failure has happened before or how often. ## The Four Types of Test Failures and the Right Response to Each Every red result falls into one of four buckets, and each one calls for a different action. **Consistent failures** fail on every run with the same error. These are real regressions. The right response: fix the code or update the expectation, then confirm the test passes. **Flaky failures** pass and fail without code changes. The right response is not to re-run until green. It's to [identify the flaky test](/solutions/flaky-test-detection/) by its history and either stabilize it or quarantine it so it stops polluting triage. **New failures** are tests that started failing on a specific run. The right response: tie the failure to the change that introduced it and route it to whoever made that change. **Environment failures** are errors, not assertion failures: connection refused, missing variable, wrong runtime version. The right response is to fix the runner or the setup, not the test. These often cascade, which is exactly where clustering earns its keep. ## How Failure Clustering Changes Triage Most reporting tools show failures as a flat list ordered by test name or execution order. That ordering hides the thing you most need to see: which failures share a cause. [Failure clustering](/solutions/failure-clustering/) groups failures by their error signature instead of by test name. Gaffer extracts the error message from each failing test, strips the variable parts (timestamps, UUIDs, port numbers), and groups tests with matching signatures into a single cluster named after the shared error. Fourteen red tests become "Connection refused (9 tests)" and "expected 200, received 500 (5 tests)". You're now looking at two problems, not fourteen. The clustering runs on parsed error output, so it works the same whether the report came from Playwright, Jest, Vitest, pytest, or JUnit XML. No configuration, no per-framework rules. ## Step-by-Step: Triaging a Failure Cluster in Gaffer ### Step 1: Upload the CI report Add one step to your pipeline after the test run. GitHub Actions: ```yaml - name: Run tests run: npm test - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./test-results/results.xml ``` The `if: always()` matters: without it the upload is skipped exactly when tests fail, which is when you need the report. ### Step 2: Read the clusters Open the run. Clusters appear below the failure list, sorted by how many tests each one affects. Start at the top, because the largest cluster is usually one fix. ### Step 3: Separate flaky from consistent For each cluster, check whether those tests have failed before without code changes. A cluster of known-flaky tests gets a different owner and a different priority than a cluster of consistent regressions. This is the step the rerun-until-green habit skips. ### Step 4: Assign or file an issue Consistent cluster: file one issue for the shared cause, not one per test. New failures: route to the author of the introducing change. Flaky cluster: send it to whoever owns test stability. ### Step 5: Confirm it clears After the fix merges, the next run either shows the cluster gone or it doesn't. No manual spreadsheet update, no guessing whether the re-run was luck. ## What Gaffer Shows That Raw CI Logs Don't A CI log is a snapshot of one run. It can't answer the questions that actually drive triage decisions, because those questions span runs. - **Cross-run persistence:** has this exact failure happened before, or is it new on this commit? A log can't tell you. Gaffer keeps the failure history, so you can tell a fresh regression from a failure that has come back before. - **Frequency:** is this test failing 2% of the time or 40%? That number decides whether you stabilize it or quarantine it, and it only exists if something is tracking runs over time. - **Co-occurrence:** which failures keep showing up together? Tests that fail as a group point at a shared dependency, and the cluster makes that obvious instead of leaving you to notice it across scrolling sessions. To get failures in front of the right person without anyone watching CI, connect [test failure notifications](/solutions/test-failure-notifications/): Slack or webhook alerts with the pass/fail summary and a direct link to the hosted failure detail, filterable by branch so feature branches don't spam the channel. ## Get Started Gaffer's free tier includes 500 MB of storage with 7-day retention, enough to wire up the upload step and watch a few real CI runs cluster their own failures.Stop reconstructing failure patterns by hand
Gaffer clusters failures by root cause across runs and keeps the history a CI log throws away. Free tier, no credit card.
Start FreeYou're debugging a production issue. You remember a test failed with the same error three months ago. You go to find that test report... and it's gone. The CI artifacts expired. Sound familiar?
## Why CI Providers Delete Your Test Artifacts Every major CI/CD platform automatically deletes artifacts after a retention period: | CI Provider | Default Retention | Max Retention | |-------------|-------------------|---------------| | GitHub Actions | 90 days | 400 days (paid) | | GitLab CI | 30 days | Configurable | | CircleCI | 30 days | 30 days | | Azure DevOps | 30 days | Configurable | | Bitbucket Pipelines | 14 days | 14 days | **Why do they do this?** Storage costs money. CI providers optimize for build speed and compute, not long-term artifact storage. Your test reports are considered disposable. ## The Real Cost of Expiring Artifacts Losing [test artifacts](/blog/what-are-test-artifacts/) isn't just annoying—it has real consequences: ### 1. Debugging Becomes Harder When a bug resurfaces, historical test data is invaluable. Which tests were failing before we "fixed" this? What did the error message say? Without the original artifacts, you're starting from scratch. ### 2. Flaky Test Analysis is Impossible To understand flaky tests, you need data across many runs. If artifacts expire after 30 days, you lose the historical context needed to identify patterns. ### 3. Compliance Requirements Regulated industries (finance, healthcare, automotive) often require test evidence retention for years. CI artifact expiration violates these requirements. ### 4. Onboarding Suffers New team members can't see how the test suite evolved. They can't reference past failures to understand why certain tests exist. ### 5. Post-Mortems Lack Evidence "What went wrong?" is hard to answer when the test reports from the incident are gone. ## Workarounds (and Why They Fall Short) ### Increase CI Retention Some CI providers let you extend retention. GitHub Actions goes up to 400 days with paid plans. But: - Still not permanent - Expensive at scale (GitHub charges for storage) - Doesn't solve the access problem (finding artifacts is still tedious) ### Upload to S3/GCS You can add a CI step to upload artifacts to your own cloud storage: ```yaml - name: Upload to S3 run: aws s3 cp ./test-results s3://my-bucket/reports/${{ github.sha }} --recursive ``` Problems: - Manual setup and maintenance - No built-in access control - No UI for browsing reports - Still need to manage storage costs and cleanup ### Save Locally Some teams download important artifacts before they expire. This doesn't scale and relies on someone remembering to do it. ## The Real Solution: Dedicated Test Report Hosting Instead of fighting CI artifact limits, host your test reports on a platform designed for it: 1. **Automatic upload** from CI pipeline 2. **Configurable retention** per project (up to 90 days on paid plans) 3. **Instant access** via shareable URLs 4. **Team-based permissions** tied to your organization 5. **Search and analytics** across all historical runs ## How Gaffer Solves CI Artifact Expiration Gaffer is purpose-built for test report hosting. Here's how it works: ### Step 1: Add One Line to CI **GitHub Actions:** ```yaml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./test-results ``` **Any CI with curl:** ```bash curl -X POST https://api.gaffer.sh/upload \ -H "X-API-Key: $GAFFER_PROJECT_TOKEN" \ -F "files=@test-results/report.html" \ -F "branch=$CI_BRANCH" \ -F "commit=$CI_COMMIT_SHA" ``` ### Step 2: Choose Your Retention Every upload gets a shareable URL. You control how long reports are kept: | Plan | Default Retention | Per-Project Settings | |------|-------------------|---------------------| | Free | 7 days | Custom retention periods | | Pro | 30 days | Custom retention periods | | Team | 90 days | Custom retention periods | **Longer retention than CI providers** - Keep test data for up to 90 days with Team plans, plenty of time for debugging and analysis. **Want to keep costs low?** Enable automatic cleanup and set a shorter retention period. Test reports from feature branches might only need 7 days, while production test data could be kept longer. ### Step 3: Access Anytime - Browse reports in the Gaffer dashboard - Search by branch, commit, date range - Share links with teammates - Get Slack notifications with direct links ## Per-Project Retention Control Different projects have different needs. Gaffer lets you configure retention per-project: - **Production tests**: Keep for 90 days (Team plan) - **Feature branch tests**: Auto-delete after 7 days - **Important projects**: Use longer retention periods All plans can set custom retention periods to match your needs. ## Beyond Storage: Analytics Over Time When you keep test data longer, you unlock insights that short CI retention makes difficult: - **Pass rate trends** - Is the suite getting more stable? - **Flaky test detection** - Which tests fail intermittently? - **Regression detection** - When did this test start failing? - **Duration trends** - Are tests getting slower? ## Migration Path Already have test reports scattered across CI providers? Gaffer works alongside your existing setup: 1. Add Gaffer upload to CI (takes 5 minutes) 2. New reports go to Gaffer automatically 3. Old reports in CI continue to expire (nothing changes) 4. Over time, Gaffer becomes your source of truth No migration, no data import, just better going forward. ## Get Started Stop losing test artifacts. Gaffer gives you control over retention with per-project settings and longer retention periods than most CI providers offer.Your CI generates a coverage report every run. But by next month, that report is gone: CI artifacts expire, and with them, any record of what your coverage looked like. You can't track whether coverage is improving or regressing because there's no historical data.
## The Problem: Coverage Without History Most teams treat code coverage as a point-in-time check. CI runs, coverage is 74%, the build passes. But that number alone doesn't answer the questions that matter: - **Is coverage going up or down?** A single number doesn't tell you direction - **When did coverage drop?** Without history, you can't pinpoint the commit that removed tests - **Are new files getting tested?** You can't tell if recent code has coverage without comparing over time - **Is the team hitting coverage goals?** Progress requires a baseline and a trend line ### Why This Happens **CI artifacts expire.** GitHub Actions keeps artifacts 90 days max. Most teams set shorter retention. Once they're gone, the coverage data is gone. **No built-in trend tracking.** Coverage tools (Istanbul, coverage.py, gcov) generate reports for the current run. They don't store or compare across runs. **Context disappears.** A coverage report says `utils/parser.ts` has 45% coverage. It doesn't tell you it had 60% last week before someone deleted tests, or that it's been at 45% for six months while the file tripled in size. ## Better: Persistent Coverage Tracking The fix is to store coverage data from every run and visualize it over time. Every data point builds a picture of where your test suite is heading. ### What This Looks Like - **Trend charts** showing line, branch, and function coverage over days, weeks, or months - **Regression alerts** when coverage drops after a commit - **Per-file breakdown** showing which files have the lowest coverage - **Commit linking** connecting coverage changes to specific code changes[Currents.dev](https://currents.dev/) is a mature Cypress-era test analytics product, now oriented around enterprise workflows (SSO/SCIM, Slack Connect, 1-year retention). Its Team plan is $49/month for 10 seats and 10,000 test results per month, with $5 per additional 1,000 results on top. Teams start shopping for a Currents.dev alternative when they cross 10 users, hit the test-result cap, or simply don't want a per-usage meter on their CI output. Gaffer prices on storage, not seats or test volume: $49/month flat, unlimited users, 50 GB, 90-day retention.
## Why Teams Look for Currents.dev Alternatives ### Seat Count Ceiling on the Team Plan Currents' Team plan includes 10 seats. Past that, the path is Enterprise with custom pricing. For a 15-person engineering team that wants everyone to see test results, the Team plan doesn't fit and the next step is a sales call. ### Test-Result Metering The 10,000 test results per month included on Team sounds generous until you do the arithmetic. A team running 300 CI runs per day at 50 tests per run produces roughly 450,000 test results per month. At $5 per additional 1,000 results, that's around $2,200/month in metered overage on top of the $49 base. Usage scales with your test suite size and CI frequency, both of which tend to grow. ### No Free Tier Currents does not publish a free tier. Evaluation means a trial window, not a persistent free workspace you can leave running on a side project. ## Gaffer: Same Outcomes, Different Economics ### Flat Pricing, Unlimited Users Gaffer has three tiers. Free is $0 with 500 MB and 7-day retention. Pro is $15/month with 10 GB and 30-day retention. Team is $49/month with 50 GB and 90-day retention. Every tier includes unlimited users. Overage on Pro and Team is $0.50/GB/month. There is no test-result cap; storage is the only billed resource. ### Same CI Upload Pattern One step in your CI pipeline: ```yaml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./test-results ``` Or with curl: ```bash curl -X POST https://app.gaffer.sh/api/upload \ -H "X-API-Key: $GAFFER_PROJECT_TOKEN" \ -F "files=@test-results/report.xml" ``` JUnit XML and CTRF are both first-class. Playwright, Jest, Vitest, pytest, and Cypress all work. ### MCP Code-Mode for Agentic CI Both products expose an MCP server. The shapes differ. Currents' MCP provides roughly 25 CRUD tools for its Dashboard, GitHub PR, CLI, and Slack surfaces. Gaffer's MCP uses code-mode: three primitives (`search_tools`, `execute_code`, and a small set of named functions like `get_project_health`, `get_flaky_tests`, `get_slowest_tests`, `get_failure_clusters`, `get_test_history`, `get_coverage_summary`, `find_uncovered_failure_areas`, `compare_test_metrics`, `search_failures`, `list_test_runs`). The agent composes JavaScript that calls those functions, rather than issuing one MCP call per CRUD endpoint. A typical agent session is one `execute_code` call that does the work of 5 to 10 CRUD requests. See [GitHub Agentic Workflows with Gaffer MCP](/blog/github-agentic-workflows-test-analytics/) for the reasoning behind this shape. ### Built-in Analytics Without Extra Configuration Flaky detection, slowest-test surfacing, and failure clustering run on every upload. As a calibration point, Gaffer's own 30-day metrics (pulled via the MCP server) currently sit at a 95 health score, 99.96% pass rate over 200 runs, 4 flaky tests, and a p95 duration of 62 seconds on the slowest test. Those are the same numbers the dashboard and MCP tools would return for any project with comparable activity. ## Pricing Comparison | Tier | Currents.dev | Gaffer | |------|--------------|--------| | Free | Not offered | $0, 500 MB, 7-day retention, unlimited users | | Entry paid | Not offered | $15/month, 10 GB, 30-day retention, unlimited users | | Team | $49/month, 10 seats, 10K test results/month, +$5 per additional 1K results, up to 1-year retention | $49/month, unlimited users, 50 GB, 90-day retention | | Overage | $5 per 1,000 test results over cap | $0.50/GB/month (Pro and Team) | | Enterprise | Custom (SSO/SCIM, Slack Connect, data redaction) | Not offered | ## Feature Comparison | Feature | Currents.dev | Gaffer | |---------|--------------|--------| | Test report hosting | Yes | Yes | | Historical trends | Yes | Yes | | Flaky test detection | Yes | Yes | | Slowest-test and failure clustering | Yes | Yes | | MCP server | Yes (CRUD tools) | Yes (code-mode) | | GitHub commit status | Yes | Yes | | Slack notifications | Yes (incl. Slack Connect on Enterprise) | Yes | | Webhook notifications | Yes | Yes | | SSO / SCIM | Enterprise | Not available | | Jira / MS Teams / GitLab / BitBucket integrations | Yes | Limited | | Self-hosting | Enterprise | Not available | | Retention (paid) | Up to 1 year | Up to 90 days | | User pricing | Per seat (10 included on Team) | Unlimited users, all tiers | | Free tier | No | Yes | ## When to Use Currents.dev Currents is the better fit if you need: - SSO or SCIM today, not on a roadmap - Slack Connect, Jira, MS Teams, GitLab, or BitBucket as first-class integrations - A team larger than 10 users where per-seat pricing has already been negotiated - Retention longer than 90 days (up to 1 year on Team) - Self-hosting via an Enterprise contract ## When to Use Gaffer Gaffer is the better fit if: - Your team is approaching or past 10 users and per-seat pricing is the binding constraint - You want a free tier for evaluation or for side projects - Storage-based pricing matches your workload better than test-result metering - You want the MCP code-mode primitive for agentic CI workflows. See [Test Intelligence for AI Tools](/blog/test-intelligence-for-ai-tools/) - You prefer flat billing without a per-usage meter on CI output ## Migration from Currents.dev No big-bang cutover needed. 1. Keep uploading to Currents. Nothing breaks. 2. Add a Gaffer upload step to CI using JUnit XML or CTRF output. 3. Point the team at Gaffer for sharing and historical analytics. Retire Currents when you're ready. ## Other Comparisons Evaluating multiple tools? See how Gaffer compares to [Allure](/solutions/allure-alternative/), [ReportPortal](/solutions/reportportal-alternative/), and [TestDino](/solutions/testdino-vs-gaffer/). ## Try It Gaffer's free tier includes 500 MB storage and 7-day retention. Enough to run your CI against it for a week and decide.Cypress generates test results, but sharing them with your team means either paying for Cypress Cloud or pasting terminal output into Slack. If you want hosted results with shareable links, analytics, and notifications without Cypress Cloud pricing, there's a simpler path.
## The Cypress Reporting Problem Cypress runs tests and prints results to the console. The built-in dashboard experience requires Cypress Cloud (formerly Cypress Dashboard), which is a paid service for teams that need parallelization and analytics. ### What Teams Do Instead **1. Cypress Cloud** - Good feature set: parallelization, analytics, flaky test management - Pricing scales with recorded test runs, which can get expensive for active teams - Tightly coupled to Cypress: doesn't help if you also run Jest, Playwright, or pytest **2. CI artifacts** - Cypress screenshots and videos upload as artifacts - Recipients need CI access to download zips - Artifacts expire (90 days max on GitHub Actions) - No analytics, no historical trends **3. Mochawesome reports** - Generates HTML reports locally, but hosting them is your problem - No built-in sharing, analytics, or notification support - Another reporter to configure and maintain **4. Terminal output in Slack** - Loses all context for screenshots and videos - Truncated for large suites - No way to filter or drill into failures ## Better: CTRF + Hosted Reports Cypress doesn't produce a report format that Gaffer parses natively (like Playwright HTML or JUnit XML). Instead, the integration uses [CTRF (Common Test Report Format)](https://ctrf.io/) — a standardized JSON format with reporters for 20+ test frameworks including Cypress. The setup: 1. Add the `cypress-ctrf-json-reporter` to your project 2. Cypress generates a CTRF JSON file after tests run 3. Upload the CTRF file to Gaffer in CI 4. Get a shareable URL, Slack notification, and analytics ### What This Looks Like After CI runs: - Your team gets a Slack notification with pass/fail summary - Anyone can click through to the full hosted report - Test analytics accumulate over time: pass rates, flaky tests, slow tests - No Cypress Cloud subscription required ## Setting Up Cypress Reporting with Gaffer ### Step 1: Install the CTRF Reporter ```bash npm install cypress-ctrf-json-reporter --save-dev ``` ### Step 2: Configure Cypress Add the reporter to your `cypress.config.ts` via the `setupNodeEvents` hook: ```typescript import { defineConfig } from 'cypress'; import { GenerateCtrfReport } from 'cypress-ctrf-json-reporter'; export default defineConfig({ e2e: { setupNodeEvents(on) { new GenerateCtrfReport({ on }); }, }, }); ``` This generates a `ctrf/ctrf-report.json` file after each test run. You can customize the output: ```typescript new GenerateCtrfReport({ on, outputFile: 'ctrf-report.json', outputDir: '.', }); ``` ### Step 3: Add the Upload Step to CI **GitHub Actions:** ```yaml - name: Run Cypress tests run: npx cypress run - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./ctrf/ctrf-report.json ``` **GitLab CI:** ```yaml test: script: - npx cypress run after_script: - | curl -X POST https://app.gaffer.sh/api/upload \ -H "X-API-Key: $GAFFER_PROJECT_TOKEN" \ -F "files=@ctrf/ctrf-report.json" \ -F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'"}' ``` ### Step 4: Share the Link After upload, share the report URL in Slack, GitHub PRs, or Jira. Recipients see structured test results in their browser, no CI access needed. Need to share with someone outside your organization: a contractor, client, or partner team? Generate a [public share link](/solutions/shareable-test-report-links/) with an expiration you control (1 hour to 30 days, or never). Recipients view the results in their browser without needing a Gaffer account. ## Slack Notifications Once you're uploading results, connect Slack in project settings for automatic notifications: - Pass/fail summary with test counts - Direct link to the full report - Filter by branch: only notify on `main`, skip feature branches - Choose triggers: all runs, failures only, or consecutive failures ## Comparing Cypress Reporting Options | Feature | Cypress Cloud | Mochawesome | CI Artifacts | Gaffer (via CTRF) | |---------|--------------|-------------|--------------|-------------------| | Hosted reports | Yes | Self-host | Download zips | Yes | | Shareable links | Yes | Manual | With CI access | Yes | | Analytics/trends | Yes | No | No | Yes | | Flaky test detection | Yes | No | No | Yes | | Slack notifications | Yes | No | No | Yes | | Works with other frameworks | No | No | N/A | Yes | | Parallelization | Yes | No | No | No | | Video recording | Yes | No | Via artifacts | No | | Pricing | Paid (usage-based) | Free | Free | Free tier available | ### When Cypress Cloud Makes Sense Cypress Cloud is worth it if you need test parallelization (splitting tests across machines), video recording with cloud storage, or you're all-in on Cypress with no other frameworks. ### When Gaffer Makes Sense Gaffer is a better fit if you want a unified view across multiple frameworks (Cypress + Jest + Playwright), want test analytics without per-seat pricing, or need Slack/webhook notifications with more flexible triggers. ## Beyond Cypress: Multi-Framework Support Since Gaffer works with CTRF, you can upload results from any framework that has a CTRF reporter. One dashboard for: - **Cypress** (via `cypress-ctrf-json-reporter`) - **Playwright** (native HTML reports) - **Jest** (via `jest-ctrf-json-reporter`) - **Vitest** (native HTML reports) - **pytest** (via JUnit XML or CTRF) All results appear in the same project dashboard with unified analytics. ## Get Started Gaffer's free tier includes 500 MB of storage with 7-day retention. Enough to evaluate the CTRF workflow with your Cypress suite.Your services have dashboards, alerts, and SLOs in Datadog. Your test suite doesn't. Here's how to get test metrics into Datadog without writing custom instrumentation.
## The Problem You run hundreds of tests in CI every day, but the results live in CI logs. To answer "is our test suite healthy?" you check individual pipeline runs. There's no trend line, no alert when pass rates drop, and no way to correlate test health with deployment metrics. ### What Teams Usually Try **1. Custom metric scripts in CI** - Parse test output, POST to Datadog's API in an `after_script` block - Breaks when test runner output format changes - Every team reinvents this differently **2. Datadog CI Visibility** - Full-featured but requires Datadog's test runner instrumentation - Per-test pricing can add up at scale - Tightly coupled to Datadog: hard to switch later **3. Manual monitoring** - Someone checks CI periodically - Problems are discovered after they've compounded ## A Simpler Approach Gaffer already stores your test results and coverage reports. With OpenTelemetry export, it pushes metrics to Datadog automatically: pass rates, failure counts, test totals, and coverage percentages arrive as standard OTLP metrics. No test runner plugins. No CI script changes beyond what you already have for Gaffer uploads. ## Setup If you're already uploading test results to Gaffer, adding Datadog export takes a few steps: 1. Go to **Settings > OpenTelemetry** in the Gaffer dashboard 2. Click **"Add destination"** and select **Datadog** 3. Paste your Datadog API key and select your site region 4. Click **"Test"** to verify connectivity For full setup instructions including Grafana Cloud and generic OTLP endpoints, see the [OpenTelemetry integration guide](/docs/integrations/opentelemetry/). ## Dashboard Template We've published a Datadog dashboard template you can import directly. It includes: - **Overview gauges**: current pass rate, total tests, failed count, skipped count - **Pass rate over time**: line chart of `gaffer.tests.pass_rate` over 30 days, grouped by branch - **Test counts by result**: stacked bar chart of passed/failed/skipped per day - **Coverage trends**: line and branch coverage percentages over time ### Import the Dashboard 1. Download the [dashboard template](/templates/datadog-dashboard.json) 2. In Datadog, go to **Dashboards > New Dashboard** 3. Click the gear icon and select **Import Dashboard JSON** 4. Paste or upload the JSON file 5. Use the `$project` template variable to filter by Gaffer project ## Available Metrics | Metric | Type | Description | |--------|------|-------------| | `gaffer.tests.pass_rate` | Gauge (%) | Pass rate of the test run | | `gaffer.tests.total` | Gauge | Total test count | | `gaffer.tests.passed` | Gauge | Passed test count | | `gaffer.tests.failed` | Gauge | Failed test count | | `gaffer.tests.skipped` | Gauge | Skipped test count | | `gaffer.coverage.line_percent` | Gauge (%) | Line coverage percentage | | `gaffer.coverage.branch_percent` | Gauge (%) | Branch coverage percentage | All metrics include `gaffer.project.name`, `gaffer.branch`, and `gaffer.commit_sha` as attributes for filtering. ## What You Get - **Alerts**: set a Datadog monitor on `gaffer.tests.pass_rate` dropping below a threshold - **Correlation**: overlay test health with deploy events, error rates, and service metrics - **Trend visibility**: spot gradual test suite degradation before it becomes a problem - **Team dashboards**: share test health alongside service dashboards that your team already checks daily`dotnet test` produces results, but everyone who wants to see them has to go somewhere different: the Azure DevOps pipeline tab, a CI artifact zip, or the Visual Studio Test Explorer on whoever ran them locally. None of those work for a quick "did the suite pass?" link in a PR or Slack thread. There's a simpler path that reuses output your build already generates.
## The .NET Test Reporting Problem `dotnet test` runs MSTest, NUnit, and xUnit suites and prints results to the console. Turning that console output into something a teammate can open in a browser is left to you, and the common options each trap the results somewhere. ### What Teams Do Instead **1. Azure DevOps "Publish Test Results" task** - Solid in-pipeline view: pass/fail counts, history, failed-test drill-down - Locked to the Azure DevOps pipeline UI. There's no link you can hand to someone outside the org - Tied to the build it ran in. No unified view if some suites run elsewhere or in a different pipeline **2. Visual Studio Test Explorer** - Good for the developer at the keyboard - Local only. There's nothing to share. Results live and die in one IDE session **3. CI artifacts (.trx zips)** - The `.trx` file uploads as a build artifact - Recipients need CI access to download and a TRX viewer to read it - Artifacts expire (90 days max on GitHub Actions) and carry no historical trends **4. Pasting console output into Slack** - Loses the structure: no per-test status, no stack traces you can expand - Truncated for large suites - No way to filter failures or compare against the last run ## Better: TRX + Hosted Reports You don't need a new reporter. `dotnet test --logger:trx` writes a `.trx` file, and it does so whether the project uses MSTest, NUnit, or xUnit. The `dotnet test` CLI emits TRX for all three. Gaffer parses TRX natively, so the integration is: tell `dotnet test` to write TRX, then upload the file in CI. The setup: 1. Run `dotnet test --logger:trx` 2. Upload the `.trx` file to Gaffer in CI 3. Get a shareable URL, analytics, and flaky test detection If you prefer JUnit XML (for example to standardize across non-.NET suites), the `JUnitXml.TestLogger` NuGet package adds a `--logger:"junit"` option, and Gaffer parses JUnit XML natively too. ### What This Looks Like After CI runs: - Anyone with the link sees the full hosted report in their browser - Failures keep their structure: per-test status, duration, and stack traces - Analytics accumulate over time: pass rates, [flaky tests](/solutions/flaky-test-detection/), slow tests - No Azure DevOps access required to view results ## Setting Up .NET Reporting with Gaffer ### Step 1: Write a TRX File Add the TRX logger to your test run. This works for MSTest, NUnit, and xUnit projects without changing test code: ```bash dotnet test --logger:trx --results-directory ./TestResults ``` This writes a `.trx` file into `./TestResults`. The filename includes a timestamp, so reference the directory in the upload step rather than a fixed name. ### Step 2: Add the Upload Step to CI **GitHub Actions:** ```yaml - name: Run tests run: dotnet test --logger:trx --results-directory ./TestResults - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./TestResults/*.trx ``` **Azure Pipelines (curl):** ```yaml - script: dotnet test --logger:trx --results-directory $(Build.SourcesDirectory)/TestResults displayName: Run tests - script: | curl -X POST https://app.gaffer.sh/api/upload \ -H "X-API-Key: $(GAFFER_PROJECT_TOKEN)" \ -F "files=@$(ls TestResults/*.trx | head -n1)" \ -F 'tags={"commitSha":"$(Build.SourceVersion)","branch":"$(Build.SourceBranchName)"}' displayName: Upload to Gaffer condition: always() ``` The `if: always()` and `condition: always()` guards matter: you want the report uploaded even when tests fail, since that's when the report is most useful. ### Step 3: Share the Link After upload, drop the report URL into a PR, Slack, or your Azure DevOps work item. Recipients see structured results in their browser with no CI access needed. Need to share with someone outside your organization, like a contractor or client? Generate a [public share link](/solutions/shareable-test-report-links/) with an expiration you control (1 hour to 30 days, or never). They view the results without a Gaffer account. ## Comparing .NET Reporting Options | Feature | Azure DevOps Test Results | CI Artifacts | Gaffer | |---------|---------------------------|--------------|--------| | Hosted reports | In pipeline UI | Download zips | Yes | | Shareable links outside the org | No | With CI access | Yes | | Analytics/trends | In pipeline | No | Yes | | Flaky test detection | Limited | No | Yes | | Works across frameworks | Per pipeline | N/A | Yes | | Pricing | Included with Azure DevOps | Free | Free tier available | ### When Azure DevOps Test Results Makes Sense If your whole team lives in Azure DevOps and every suite runs in the same pipeline, the built-in "Publish Test Results" task covers the in-pipeline case well and costs nothing extra. The friction shows up when you need to share a link outside the org or unify results from suites that run in different pipelines or providers. ### When Gaffer Makes Sense Gaffer fits when you want a link anyone can open, one dashboard across multiple frameworks and pipelines, or test analytics and flaky detection without per-seat pricing. ## Beyond .NET: Multi-Framework Support TRX and JUnit XML are not .NET-specific. The same upload step handles results from any framework Gaffer parses, so a project with a .NET backend, a JavaScript frontend, and a Python service can [report all three to one dashboard](/solutions/multiple-test-frameworks/): - **.NET** (TRX via `dotnet test --logger:trx`) - **JavaScript / TypeScript** (Jest, Vitest, Playwright) - **Python** (pytest via JUnit XML) All results land in the same project with [unified hosted reports](/solutions/host-test-reports/) and shared analytics. ## Get Started Gaffer's free tier includes 500 MB of storage with 7-day retention. Enough to wire `dotnet test --logger:trx` into CI and see your suite hosted with a shareable link.Five tests fail in CI. You investigate the first one: "Connection refused." The second shows the same error. The third, fourth, and fifth match too, all the same root cause. You just spent 30 minutes discovering what should have been obvious in 3 seconds: one service is down, and every test that depends on it failed.
## What Is Failure Clustering? Failure clustering groups test failures that share the same root cause into a single cluster. Instead of showing you 5 individual failures, it shows you 1 pattern affecting 5 tests. The idea is simple: failures don't happen in isolation. When a database connection drops, every test that queries the database fails. When an API endpoint changes, every test that calls that endpoint fails. Treating each failure independently wastes time. ## Why Individual Test Failures Mislead Most test reporting tools show a flat list of failures: | Test | Status | Error | |------|--------|-------| | `auth.test.ts > login` | FAIL | Connection refused | | `auth.test.ts > signup` | FAIL | Connection refused | | `api.test.ts > get users` | FAIL | Connection refused | | `api.test.ts > create user` | FAIL | ECONNREFUSED 127.0.0.1:5432 | | `db.test.ts > migrations` | FAIL | Connection refused | A developer sees 5 failures and might think 5 things broke. In reality, one thing broke (the database connection), and it cascaded to 5 tests. The fourth failure even has a slightly different error message, making it harder to spot the pattern manually. ### The cost of debugging individually - **Time:** 5-10 minutes investigating each failure before realizing they're the same - **False fixes:** Developers sometimes "fix" individual tests (adding retries, mocking) when the real issue is upstream - **Noise:** A 5-failure CI report feels worse than a 1-pattern report, even though the same amount of work is needed - **Context switching:** Jumping between 5 test files when the fix is in one shared dependency ## How Gaffer Clusters Failures Gaffer analyzes error messages across all failing tests in a run and groups them by shared patterns: ``` Clusters: 1 pattern (5 tests) "Connection refused" — 5 tests auth.test.ts > login auth.test.ts > signup api.test.ts > get users api.test.ts > create user db.test.ts > migrations ``` The algorithm: 1. **Extract error signatures** from each failing test: the error message, stripped of variable parts (timestamps, UUIDs, port numbers) 2. **Group by similarity**, so tests with matching signatures form a cluster 3. **Name the cluster** using the most common error message in the group 4. **Sort by impact**, so clusters with more affected tests appear first This works across frameworks. Whether your errors come from Playwright, Jest, pytest, or JUnit XML, the clustering applies to the parsed error output. ## When Clustering Matters Most ### Infrastructure failures Database down, Redis unavailable, network partition: these cascade across your entire test suite. Without clustering, a single infrastructure blip looks like 50 unrelated failures. ### API contract changes Someone changes an API response format. Every test that parses that response fails with a similar error. Clustering shows you the one change that needs reverting or adapting. ### Environment issues Missing environment variables, wrong Node version, incompatible dependency: these affect multiple tests in the same way. Clustering reveals the environment problem instead of burying it in individual test errors. ### Flaky infrastructure in CI CI runners sometimes have transient issues: DNS resolution failures, disk space, rate limits. Clustering helps you distinguish "the runner had a bad day" from "our code has bugs." ## Failure Clustering vs. Flaky Test Detection These are complementary features that answer different questions: | | Failure Clustering | Flaky Test Detection | |---|---|---| | **Question** | "Why did these tests fail together?" | "Does this test sometimes fail without code changes?" | | **Scope** | Within a single test run | Across multiple runs over time | | **Action** | Fix the shared root cause | Investigate test non-determinism | | **Example** | 5 tests fail with "Connection refused" | `login` test fails 40% of runs | Used together, you get a complete picture: clustering tells you what went wrong *right now*, and flaky detection tells you what's been unreliable *over time*. Both signals matter most on a large [automated regression testing](/blog/automated-regression-testing/) suite, where a single broken dependency can produce dozens of failures from one cause. ## Getting Started Failure clustering works automatically with any test report uploaded to Gaffer. No configuration required. 1. **Upload a test report** via [CLI](/docs/getting-started/), [API](/docs/guides/github-actions/), or direct upload 2. **View clusters** on the test run detail page, below the failure list 3. **Query via CLI**: `gaffer query failures` shows clusters in your terminal 4. **Query via MCP**: AI agents access clusters through `get_test_run_details()`Stop debugging the same root cause five times
Gaffer clusters failures automatically. Free tier, no credit card.
Start FreeA fast CI/CD feedback loop is essential for developer productivity. When tests fail, you need to know immediately - and you need to access the results without friction. Gaffer delivers test results directly to Slack, cutting your feedback loop from minutes to seconds.
## The CI/CD Feedback Loop Problem Every developer knows this frustrating workflow: 1. Push code and wait for CI to run 2. Notice the build failed (eventually) 3. Navigate to GitHub/GitLab to find the workflow 4. Scroll through logs trying to find the failure 5. Download artifact zip files 6. Extract and open locally to see the actual report This process takes 5-10 minutes *every time* a test fails. Multiply that by the number of failures per day, and you're losing hours to CI archaeology. The worst part? By the time you've found the failure, you've lost your mental context. You've context-switched to Slack, email, or another task. Getting back into flow takes even more time. ## How Gaffer Improves Your CI/CD Feedback Loop Gaffer shortens your feedback loop to a single Slack notification with a direct link to the full test report.
### Instant Slack Notifications
When your CI pipeline finishes, Gaffer sends a notification to your configured Slack channel:
- **Pass/fail summary** at a glance
- **Direct link** to the full HTML report
- **Branch and commit context** so you know exactly what triggered it
- **Failure details** with test names and error snippets
A test fails on `main`, you re-run the pipeline, and it passes. You know flaky tests are eating your team's time, but you can't name the specific tests doing the damage. CI artifacts expire before patterns emerge, retries hide which tests are actually unstable, and nobody has time to keep a spreadsheet. The missing piece is persistent pass/fail history per test.
### What is a flaky test? A **flaky test** is an automated test that produces inconsistent results without any code change: it passes on one run and fails on the next under the same conditions. Flaky tests are distinct from legitimate test failures. A real failure means the code is broken. A flaky test means the test itself is unreliable, typically due to timing dependencies, shared state, or external service calls. A suite with a high proportion of flaky tests trains engineers to ignore red builds, which is how real regressions slip through. ## The Flaky Test Detection Problem Most teams experience flakiness without being able to fix it because the data needed to identify the specific bad tests is gone by the time anyone goes looking. - **Re-runs mask the signal.** A test that passes on attempt 2 looks identical to a test that passed on attempt 1. - **CI artifacts expire.** GitHub Actions defaults to 90 days, GitLab to 30. By the time a pattern is obvious, the underlying runs are gone. - **No history per test.** Build pages show "this run." Nobody shows "this test, last 50 runs." - **Retry configs lie.** `retries: 2` keeps the build green and tells you nothing about which tests needed the retry. - **Quarantine has no exit criteria.** Tests get `.skip`'d, then forgotten. Coverage rots. ### Common (Bad) Solutions **1. Manual re-run tracking** - Slack channel for "this failed but passed on rerun" - Falls apart at >2 engineers; nobody catches the trends **2. Retry everything** - `retries: 3` in CI keeps builds green - Hides root cause, doubles or triples build time, masks regressions **3. Spreadsheets** - Someone updates a sheet with flaky tests after each red build - Dies in week three when that engineer goes on vacation **4. Reading CI artifacts manually** - `actions/upload-artifact` keeps reports for 90 days - Finding the right artifact across hundreds of runs is its own job| Method | History per test | Identifies which tests | Branch-aware | Setup |
|---|---|---|---|---|
| Re-run, hope, repeat | None | ✕ | ✕ | None |
| Manual spreadsheet | Manual | ~ (whoever's tracking) | Manual | Hours/week |
| CI artifact archaeology | 30–90 days | ~ (if you click through) | ✕ | None |
| Retry configs | None | ✕ | ✕ | One line |
| Gaffer | Up to 90 days | ✓ flip rate per test | ✓ | One CI step |
Your PR has 3 green checkmarks and 1 red X. You click the red X, land on a workflow run page, expand the test job, scroll through 200 lines of output, and finally find the failing test. Why is seeing test results in GitHub so hard?
## The GitHub Actions Test Results Problem Every developer knows this workflow: 1. Push code to PR 2. Wait for CI to finish 3. See a red X on the PR 4. Click to see details 5. Land on the workflow run page 6. Click into the job 7. Expand the test step 8. Scroll through logs to find failures **Total time to see what failed: 2-5 minutes** And if you want the full test report? Download artifacts, extract the zip, open the HTML file. Good luck doing that on your phone when you're away from your desk. ## What You Actually Want
- See pass/fail on the PR (GitHub does this)
- See *which tests* failed without digging through logs
- Click directly to a readable report
- Share the report with QA who doesn't have GitHub access
Every CI provider treats test reports as disposable. Debug a regression, answer a compliance question, or work out when a test started flaking, and by the time you go looking the data is gone. GitHub Actions deletes artifacts after 90 days. GitLab defaults to 30. Bitbucket gives you 14.
## CI Artifact Retention Limits Default retention limits across major CI platforms: | CI Provider | Default Retention | Max Retention | Access Method | |-------------|-------------------|---------------|---------------| | GitHub Actions | 90 days | 400 days (paid) | Download zip, extract locally | | GitLab CI | 30 days | Configurable | Download zip or browse in UI | | CircleCI | 30 days | 30 days (hard limit) | Download from artifacts tab | | Bitbucket Pipelines | 14 days | 14 days (hard limit) | Download from pipeline page | | Azure DevOps | 30 days | Configurable | Download from build summary | | Jenkins | Until disk fills up | Manual cleanup | Browse build artifacts (if server is up) | Even when artifacts exist, accessing them is friction-heavy. Most CI providers require downloading a zip file, extracting it locally, and opening it in a browser. There's no shareable URL. There's no way for someone without CI access (a QA lead, a product manager, a contractor) to view the results directly. The underlying issue is straightforward: CI providers optimize for compute, not for report hosting. [Test artifacts](/blog/what-are-test-artifacts/) are a side effect of the build, not a first-class feature. ## Why Teams Need Persistent Test Report Hosting Persistent report hosting solves four problems that come up repeatedly in engineering teams: **Debugging regressions.** A bug resurfaces that was supposedly fixed two months ago. The original test failure had a stack trace and screenshot that would tell you exactly what happened. With CI artifacts expired, you're starting from scratch. **Flaky test analysis.** Identifying [flaky tests](/solutions/flaky-test-detection/) requires data across dozens or hundreds of runs. If your CI only retains 30 days of artifacts, you're working with a narrow window, sometimes too narrow to see the pattern. **Cross-team visibility.** QA, product, and management often need to see test results. Giving everyone CI access just to view reports is either impractical (permissions) or impossible (external stakeholders). **Audit trails.** Regulated industries need evidence that tests passed before deployment. "The CI artifact expired" doesn't satisfy an auditor. ## DIY Approaches and Their Drawbacks Most teams eventually try to solve this themselves. Each approach works up to a point, then falls apart. ### S3 or GCS Bucket The most common approach: add a CI step to upload reports to cloud storage. ```yaml - name: Upload test reports to S3 run: | aws s3 cp ./test-results s3://team-test-reports/${{ github.sha }}/ \ --recursive ``` This gives you retention control, but creates new problems: - **No browsable UI.** You need to know the exact commit SHA to find a report. There's no way to browse by branch, date, or test status. - **Access control is manual.** S3 bucket policies are notoriously difficult to get right. Too permissive and it's a security risk. Too restrictive and people can't access what they need. - **No cleanup automation.** Lifecycle rules work for simple cases, but break down when you need different retention per project or per branch. - **HTML reports don't just work.** Playwright reports reference relative assets (CSS, JS, screenshots). Serving them from S3 requires CloudFront or static hosting configuration to get Content-Type headers right. - **[Storage costs compound](/solutions/test-report-storage-costs/).** Without active management, costs grow linearly every month. A team generating 50 GB/month of test artifacts is paying real money within a quarter. ### GitHub Pages Some teams push reports to a dedicated GitHub Pages repo. ```yaml - name: Deploy report to GitHub Pages run: | git clone https://token@github.com/org/test-reports.git cp -r ./playwright-report test-reports/$(date +%Y-%m-%d)-${{ github.sha }} cd test-reports && git add . && git commit -m "Add report" && git push ``` This creates browsable HTML, but: - The repo grows unboundedly. Git wasn't designed for this - No authentication (GitHub Pages is public by default for free plans) - No search, no filtering, no analytics - Cleanup requires rewriting Git history ### Custom Server A few teams build a lightweight Express or Flask app to receive and serve reports. This gives you full control, but now you're maintaining infrastructure for a problem that isn't your core product. You need to handle uploads, storage, authentication, cleanup, monitoring, and uptime, all for a tool that your team uses but nobody's job is to maintain. ### The Common Thread Every DIY approach solves the storage problem while creating operational problems. You trade "artifacts expire" for "who maintains this," "how do I find the report from Tuesday," and "why is the S3 bill so high." ## What a Purpose-Built Solution Looks Like A proper test report hosting platform handles the full lifecycle: upload, store, browse, share, analyze, and clean up. It should be framework-agnostic (not just Playwright or just JUnit), CI-agnostic (not just GitHub Actions), and require minimal setup. Specifically, it should: 1. Accept any report format: HTML reports, JUnit XML, [CTRF JSON](/docs/guides/ctrf/), raw files 2. Provide shareable URLs for every test run 3. Parse structured data (JUnit XML, CTRF) for analytics: pass rates, flaky tests, duration trends 4. Handle retention automatically with per-project controls 5. Work with any CI provider through a standard upload mechanism ## How Gaffer Handles Test Report Hosting Gaffer is built specifically for this. Upload test reports from any CI provider, any framework, and access them through shareable URLs with configurable retention. ### Upload Any Report Format Gaffer accepts HTML reports (Playwright, pytest-html, Mochawesome), structured data (JUnit XML, CTRF JSON), and raw files (screenshots, logs, traces). Upload them together or separately. **[HTML reports](/solutions/html-test-reports/)** are served directly in the browser, so your team sees the full interactive report without downloading anything. For everything Playwright's HTML report contains and how to get the most out of it, the [Playwright reports guide](/blog/playwright-reports-guide/) is a useful reference. **Structured data** (JUnit XML, CTRF JSON) is parsed for analytics: pass/fail counts, test durations, failure messages. This feeds into trend tracking, flaky test detection, and pass rate history. ### Framework-Agnostic Setup Gaffer works with whatever your tests already produce. Here are two common patterns: **JUnit XML (works with nearly every test framework):** ```yaml # GitHub Actions - name: Run tests run: npx vitest --reporter=junit --outputFile=results.xml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./results.xml ``` **CTRF JSON (standardized format across frameworks):** ```yaml - name: Run tests run: npm i -D playwright-ctrf-json-reporter && npx playwright test --reporter=playwright-ctrf-json-reporter - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./ctrf/ctrf-report.json ``` **HTML reports with structured data (both human-readable and machine-parseable):** ```yaml - name: Run Playwright tests run: npx playwright test --reporter=html - name: Upload HTML report to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./playwright-report - name: Upload CTRF results to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./ctrf/ctrf-report.json ``` ### Non-GitHub CI Providers For [GitLab CI](/docs/guides/gitlab-ci/), [CircleCI](/docs/guides/circleci/), [Jenkins](/docs/guides/jenkins/), [Bitbucket Pipelines](/docs/guides/bitbucket-pipelines/), or [Azure DevOps](/docs/guides/azure-devops/), use the CLI upload: ```bash curl -X POST https://app.gaffer.sh/api/upload \ -H "X-API-Key: $GAFFER_PROJECT_TOKEN" \ -F "files=@./results.xml" \ -F 'tags={"branch":"'"$CI_BRANCH"'","commitSha":"'"$CI_COMMIT_SHA"'"}' ``` Three lines. Works anywhere you can run `curl`. ### Share with Anyone Every upload gets a URL in the Gaffer dashboard. Team members with access can browse reports by project, branch, or date. For people outside your organization (contractors, stakeholders, clients), generate a [share link](/solutions/shareable-test-report-links/) with a configurable expiration. Recipients view the report directly in their browser without needing a Gaffer account. ### Retention You Control | Plan | Retention | Storage | |------|-----------|---------| | Free | 7 days | 500 MB | | Pro ($15/mo) | 30 days | 10 GB | | Team ($49/mo) | 90 days | 50 GB | All plans include unlimited users, with no per-seat pricing. Configure retention per project: keep production test reports for 90 days while feature branch reports auto-delete after 7. ### Analytics from Structured Data When you upload JUnit XML or CTRF JSON alongside your HTML reports, Gaffer parses the structured data and gives you: - **Pass rate trends**: track stability across builds - **[Flaky test detection](/solutions/flaky-test-detection/)**: identify tests that flip between pass and fail - **Duration tracking**: catch tests that are getting slower - **Failure patterns**: see which tests fail most frequently These work across [multiple test frameworks](/solutions/multiple-test-frameworks/). A project with Playwright E2E tests, Cypress integration tests, and Vitest unit tests shows unified analytics in one dashboard. For teams running Cypress, the [Cypress reports guide](/blog/cypress-reports-guide/) covers how to get structured output from Cypress into Gaffer. ## Comparing Your Options | | CI Artifacts | S3 + CloudFront | GitHub Pages | Gaffer | |--|--------------|-----------------|--------------|--------| | Retention | 14-90 days | Unlimited (manual cleanup) | Unlimited (repo grows) | 7-90 days (auto cleanup) | | Shareable URLs | No (download zip) | Yes (with setup) | Yes (public only) | Yes | | HTML report viewing | Download required | Yes (with config) | Yes | Yes | | Structured data parsing | No | No | No | Yes | | Analytics | No | No | No | Yes | | Access control | CI permissions only | S3 policies | Public or private repo | Team-based + share links | | Setup time | Built-in | Hours | Hours | Minutes | | Maintenance | None | Ongoing | Ongoing | None | ## Get Started Gaffer's free tier includes 500 MB of storage with 7-day retention, enough to evaluate the workflow.HTML test reports exist because developers want more than console output. Playwright gives you screenshots, traces, and video. pytest-html gives you an interactive summary. But these reports are local files, and there's no built-in way to host them where your team can access them.
## The Hosting Problem HTML reports are designed for browsers, but they're generated on CI runners or local machines. Your team can't view them without extra steps: ### Common (Bad) Solutions **Email the zip file** Zip up the report folder, email it, hope the recipient extracts it and opens `index.html`. Works, but feels like 2005. **Upload to S3 manually** Set up a bucket, configure permissions, upload after each run, manage cleanup. That's infrastructure work just to share a test report. **Rely on CI artifacts** GitHub Actions and GitLab CI store artifacts, but: - Viewing requires downloading and extracting a zip - Finding the right artifact in dozens of workflow runs is tedious - QA might not have CI access - [Artifacts expire](/solutions/ci-test-artifacts-expiring/) (30-90 days depending on the CI system) **Share your screen** Works for live debugging. Can't reference it later. Not async-friendly. ## Better: Hosted HTML Reports with Shareable Links Upload your HTML reports after every CI run. Every report gets a permanent URL that opens directly in the browser — no downloads, no extraction, no CI access required.It starts small. A test fails, but it's probably flaky. Someone re-runs the pipeline. It passes. Everyone moves on. Over weeks and months, this pattern repeats until the team has unconsciously learned to ignore test failures. The test suite becomes decoration - green checkmarks that nobody trusts.
## How Teams Learn to Ignore Failures Test failure fatigue doesn't happen overnight. It's a gradual erosion: ### Stage 1: Friction Creates Delay Test results are buried in CI logs. Accessing them requires: - Navigating to the CI provider - Finding the right workflow run - Downloading artifact zip files - Extracting and opening locally When checking a failure takes 5 minutes, developers wait. "I'll look at it after this meeting." The failure sits. ### Stage 2: Flaky Tests Breed Skepticism Some tests fail randomly. Developers learn which ones. When they see a failure, the first thought is "is this the flaky one?" not "did I break something?" Re-running the pipeline becomes the default response. ### Stage 3: Normalization The team develops shared knowledge: "Oh, that test always fails on Mondays" or "just re-run it, it'll pass." Failures become background noise. New team members pick up these habits. The institutional knowledge is "tests fail sometimes, don't worry about it." ### Stage 4: Real Bugs Slip Through A genuine failure appears. It looks like all the other noise. Someone re-runs the pipeline. It fails again, but they're in a hurry. "Probably infrastructure, I'll check later." The bug ships to production. ## The Compounding Cost Ignoring test failures creates a vicious cycle: ### Trust Erosion When tests cry wolf too often, the team stops listening. The suite's primary value - confidence that the code works - disappears. You're paying the cost of running tests without getting the benefit. ### Technical Debt Accumulation Tests that "always fail" get skipped or deleted rather than fixed. Test coverage shrinks. The gaps compound as new code builds on untested foundations. ### Slower Debugging Without reliable test history, debugging production issues becomes harder: - "Did we have a test for this?" - "When did this behavior change?" - "Was this ever working?" These questions have no answers when test data is ignored or expired. ### Release Anxiety Teams add manual verification steps because they don't trust automated tests. QA cycles extend. Releases slow down. The promise of continuous delivery breaks. ## Breaking the Cycle The solution isn't more discipline or stricter processes. It's reducing friction and surfacing signal. ### 1. Make Results Instantly Accessible If checking a failure takes 5 minutes, it won't happen consistently. If it takes 10 seconds, it becomes habit. **What helps:** - Shareable links directly to the failure (no downloads) - Notifications in Slack where the team already works - Mobile-friendly reports for quick checks **What doesn't help:** - Telling developers to "check CI more often" - Adding Slack bots that just say "build failed" without details - Documentation on "how to download artifacts" ### 2. Separate Signal from Noise Flaky tests poison the well. Quarantine them ruthlessly: - Track flip rates to identify flaky tests automatically - Move flaky tests to a separate suite that doesn't block merges - Fix or delete - don't let them linger When failures are rare and meaningful, developers pay attention. ### 3. Make Trends Visible A single failure is easy to dismiss. A trend is harder to ignore. - Pass rate dropping from 95% to 80% over two weeks - Three new flaky tests introduced this sprint - Test duration increasing 20% month over month Dashboards and reports make patterns visible to the whole team, not just whoever happened to see the last failure. ### 4. Keep Historical Data "When did this start failing?" is only answerable with history. CI artifacts expire. If your test data disappears after 30-90 days, you lose the ability to: - Correlate failures with code changes - Track improvement over time - Debug recurring issues Persistent storage turns test runs into organizational knowledge. ### 5. Involve the Whole Team Test health isn't just a developer concern. When PMs, QA, and stakeholders can access results: - Failures get visibility beyond the person who triggered them - "Test health" becomes a team metric, not a CI detail - Accountability distributes naturally If only developers can access test results, only developers will care about them. ## How Gaffer Helps Gaffer is built around reducing friction and surfacing signal: **Instant Access** - Shareable URLs for every test run - No downloads needed - Share links in Slack, Jira, PRs **Slack Notifications** - Test results delivered where your team works - Direct links to full reports - Filter by branch to reduce noise **Flaky Test Detection** - Automatic flip rate tracking - Dashboard showing most problematic tests - Catch new flakiness before it spreads **Analytics and Trends** - Pass rate over time - Health scores per project - Historical data with configurable retention **Team Access** - Organization-based permissions - No CI login required - Works for PMs, QA, anyone who needs visibility ## The Alternative: A Healthy Test Culture Teams that maintain trust in their test suites share common traits: - Failures are rare enough to be meaningful - When failures happen, investigation is immediate - Flaky tests are quarantined quickly - Test health metrics are visible and discussed - Everyone - not just developers - can access results This doesn't require heroic effort. It requires removing friction and making signal visible. ## Get Started Stop ignoring test failures. Gaffer makes results accessible, surfaces flaky tests, and tracks trends so your team can trust the suite again.Jest runs your tests and prints results to the console. CI captures the output, but sharing it with your team means "go check the workflow run" or screenshotting terminal output into Slack. There's no built-in way to share Jest results as a link.
## The Sharing Problem Jest outputs to stdout by default. You can add reporters for JSON or HTML, but those files are stuck in CI artifacts: ### Common (Bad) Solutions **1. "Check the CI logs"** - Requires repo access and navigating through workflow runs - QA or PMs might not have CI access - Logs disappear when retention expires **2. CI artifacts** - Download a zip, extract, find the right file - GitHub Actions keeps artifacts 90 days max - Finding the right artifact in dozens of workflow runs is tedious **3. Screenshot the terminal** - Loses all structure and detail - Can't reference later - Looks terrible in Slack **4. Paste JSON output in a thread** - Nobody reads raw JSON - Truncated for large test suites - No filtering or navigation ## Better: Hosted Jest Reports with Shareable Links Upload Jest results after every CI run. Every test run gets a permanent URL that anyone on your team can open. ### What This Looks Like 1. CI runs your Jest tests 2. Results upload automatically 3. You get a URL like `https://app.gaffer.sh/reports/abc123` 4. Share in Slack, GitHub PRs, Jira, or anywhere else 5. Anyone with access sees the full report No downloads. No CI access required. No expiring artifacts. ## Setting Up Jest Report Sharing with Gaffer ### Step 1: Configure Jest Output CTRF gives the richest analytics data: ```bash npm install jest-ctrf-json-reporter --save-dev ``` ```javascript // jest.config.js module.exports = { reporters: [ 'default', ['jest-ctrf-json-reporter', { outputDir: '.', outputFile: 'ctrf-report.json' }] ] }; ``` Gaffer also accepts Jest's built-in JSON output and jest-html-reporter if you prefer those. If the reporter fails with `ENOENT`, see [why jest-ctrf-json-reporter throws ENOENT](/docs/guides/ctrf/#why-does-jest-ctrf-json-reporter-throw-enoent). Vitest's JSON reporter uses the same shape; see the [Jest/Vitest JSON schema](/blog/vitest-json-reporter-output-format/) for the field-by-field breakdown. ### Step 2: Add the Upload Step to CI **GitHub Actions:** ```yaml - name: Run tests run: npm test - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./ctrf-report.json commit_sha: ${{ github.sha }} branch: ${{ github.ref_name }} ``` **GitLab CI:** ```yaml test: script: - npm ci - npm test after_script: - | curl -X POST https://app.gaffer.sh/api/upload \ -H "X-API-Key: $GAFFER_PROJECT_TOKEN" \ -F "files=@ctrf-report.json" \ -F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'"}' ``` ### Step 3: Share the Link After upload, share the URL in Slack, GitHub PR comments, or Jira tickets. Recipients see the full report in their browser. For people outside your organization, you can generate a [public share link](/solutions/shareable-test-report-links/) with configurable expiration. They view the results without needing a Gaffer account. ## Slack and Webhook Notifications Stop checking CI manually. Get notified when Jest tests finish: **Slack integration:** ``` [FAILED] my-project - main 3 tests failed, 142 passed View report: https://app.gaffer.sh/reports/abc123 ``` - Direct link to the full report - Filter by branch: only notify on `main`, skip feature branches **Webhooks:** - Send results to any endpoint - Integrate with your existing alerting or workflow tools - Trigger custom actions on failure No more "did the tests pass?" messages in Slack. See [Test Failure Notifications](/solutions/test-failure-notifications/) for the full setup guide. ## GitHub Integration Commit statuses show test results directly on PRs. No clicking through to CI logs to find out if tests passed. ## Monorepo Support Running Jest across multiple packages? Upload results from each: ```yaml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: | ./packages/api/ctrf-report.json ./packages/web/ctrf-report.json ./packages/shared/ctrf-report.json ``` All results appear in the same project dashboard. ## Get Started Gaffer's free tier includes 500 MB of storage with 7-day retention. Paid plans offer extended retention up to 90 days.A Python service, a TypeScript frontend, and a Rust worker in one repo means four report formats in one CI run. Downloading them is annoying. The real problem shows up when you try to answer "is the suite healthier than last week?" and discover the suites disagree about what they are counting.
Gaffer's own CI uploads three report formats from three test suites into one project on a code pull request: a Vitest HTML report bundled with LCOV coverage, JUnit XML from the 331 `gaffer-parsers` test functions run under `cargo-nextest`, and a Playwright HTML report from the browser suite. Two of those three, the Vitest HTML report and the Rust JUnit XML, carry no retry count and no flaky flag, so any flakiness figure for those suites has to be computed from run history rather than read out of the file.
## Why does one repo end up with four report formats? Nobody picks four formats. Each framework ships its own reporter, and the reporter is the path of least resistance at the moment you add that framework. JUnit XML became the common denominator by accident. It started as Ant's output format for Java, has no formal specification, and every framework that emits it invented its own conventions on top. CTRF is the deliberate attempt at a shared JSON schema, but adoption is per-framework, so it arrives only where someone installed the reporter. Playwright and .NET each ship a native format that carries more than JUnit XML can express, which is a good reason to keep them and a bad reason for comparability. So the sprawl is not a discipline problem. It is the sum of five reasonable local decisions. ## What does each test report format actually carry? The formats are not interchangeable subsets of each other. Each drops something the others keep. This is the field-level picture Gaffer's parsers work from: | Format | Skip representation | Retries | Flaky | File and line | Timing detail | |---|---|---|---|---|---| | JUnit XML | `pytest runs your tests. pytest-html gives you a nice report. Then someone asks "what failed in CI?" and you're walking them through downloading artifacts from GitHub Actions. There's no built-in way to share pytest results with your team.
## The Sharing Problem pytest outputs results to the terminal or generates files (HTML, JUnit XML). Either way, those results are stuck wherever the tests ran: your machine or a CI runner. Getting them to your team means: ### Common (Bad) Solutions **1. CI artifacts** - GitHub Actions keeps artifacts 90 days max, GitLab even shorter by default - "Go to the workflow run, click artifacts, download, unzip, open in browser" - QA might not have CI access at all **2. Paste terminal output in Slack** - Loses formatting, hard to parse - Can't reference later - Doesn't scale past a handful of tests **3. Email the HTML report** - Recipient downloads, opens locally - Version confusion across multiple runs - Feels like 2005 **4. Upload to S3 manually** - Works, but it's a manual step that gets skipped - Someone has to manage the bucket and permissions ## Better: Hosted pytest Reports with Shareable Links Upload pytest results after every CI run. Every test run gets a permanent URL your team can open directly.You need test analytics, so you evaluate ReportPortal. Then you see the requirements: Elasticsearch, PostgreSQL, RabbitMQ, MinIO — five services just to view test results. For most teams, the infrastructure cost of running ReportPortal dwarfs the value of the analytics it provides.
## The ReportPortal Trade-offs ReportPortal offers a lot of features: real-time reporting, ML-based failure analysis, custom dashboards, and extensive integrations. But those features come with costs: ### Infrastructure Requirements A typical ReportPortal deployment needs: - PostgreSQL database - Elasticsearch (or OpenSearch) - RabbitMQ message queue - MinIO or S3 for binary storage - The ReportPortal services themselves That's 5+ services to deploy, monitor, and maintain. The official docs recommend 16GB+ RAM for a production setup. ### Operational Overhead Running ReportPortal means: - Managing Elasticsearch indices and storage - Monitoring RabbitMQ queues - Database backups and maintenance - Keeping all services updated - Handling scaling as test volume grows This requires dedicated DevOps attention, which smaller teams often can't spare. ### Learning Curve ReportPortal's feature set is extensive. Dashboards, widgets, filters, defect types, analysis rules - there's a lot to learn. Teams often end up using 20% of the features after spending significant time on setup. ## Gaffer: Test Reporting Without the Overhead Gaffer focuses on the core problem: getting test results out of CI and into a place where your team can actually use them. ### Zero Infrastructure Gaffer is a hosted service. No databases, no message queues, no Elasticsearch. Sign up, get an API key, upload results. ### Simple Setup Add one step to CI: ```yaml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./playwright-report ``` That's it. Reports are now hosted and shareable. ### Focus on What Matters Gaffer provides: - **Test report hosting** - Shareable URLs for every test run - **Flaky test detection** - Automatic identification of unreliable tests - **Pass rate trends** - See test health over time - **Slow test tracking** - Find tests dragging down your CI - **Slack notifications** - Get alerts when tests fail - **GitHub integration** - Commit statuses and PR comments No ML-powered failure classification, no complex dashboard builders. Just the essentials, done well. ## Feature Comparison | Feature | ReportPortal | Gaffer | |---------|--------------|--------| | Test report hosting | Yes | Yes | | Historical trends | Yes | Yes | | Flaky test detection | Yes | Yes | | ML failure analysis | Yes | No | | Custom dashboards | Yes | No | | Self-hosting required | Yes | No | | Infrastructure needs | High | None | | Setup time | Hours/Days | Minutes | | Pricing | Self-host or SaaS | SaaS | ## Framework Support Both platforms support major frameworks, though integration methods differ: | Framework | ReportPortal | Gaffer | |-----------|--------------|--------| | Playwright | Agent | Native HTML + CTRF | | Jest | Agent | JSON + CTRF | | Vitest | Agent | HTML + CTRF | | pytest | Agent | pytest-html + CTRF | | JUnit | Agent | JUnit XML | ReportPortal uses "agents" that report results in real-time. Gaffer parses standard report formats after test runs complete - no custom agents needed. ## When to Use ReportPortal ReportPortal might be a better fit if you: - Have dedicated DevOps capacity for infrastructure - Need ML-powered failure analysis - Want highly customizable dashboards - Run thousands of tests and need real-time reporting - Have enterprise compliance requirements for self-hosting ## When to Use Gaffer Gaffer is probably a better fit if you: - Want test reporting without infrastructure overhead - Have a small-to-medium team - Need quick setup that just works - Value simplicity over feature count - Want to avoid the ops burden of running 5+ services ## Migration Path If you're currently using ReportPortal and finding the maintenance burden too high: 1. Add Gaffer upload to your CI pipeline (alongside ReportPortal initially) 2. Use your existing report formats - no agent changes needed 3. Evaluate for a few weeks 4. Remove ReportPortal agents if Gaffer meets your needs 5. Decommission ReportPortal infrastructure The upload-based approach means you're not locked in - you can switch back or run both if needed. ## Other Comparisons Evaluating multiple tools? See how Gaffer compares to [Allure](/solutions/allure-alternative/) and [TestDino](/solutions/testdino-vs-gaffer/). ## Try It Gaffer's free tier gives you 500 MB storage with 7-day retention. Enough to see if the simpler approach works for your team.Playwright generates HTML test reports, but sharing them with your team is surprisingly difficult. `npx playwright show-report` works on your laptop; sending the results to QA, your PM, or a client on a Slack thread is a different problem. Here's how to share Playwright test reports properly.
## The Playwright Report Sharing Problem The local server from `npx playwright show-report` is only accessible on your machine. How do you share the report when: - **QA needs to see what failed** before approving a release - **A PM wants to check test coverage** for a feature - **You're debugging with a colleague** and need to show them the exact failure - **You need to reference an old test run** from weeks ago ### Common (Bad) Solutions **1. Email the zip file** - Clunky, fills up inboxes, version confusion - Recipient has to download, extract, open in browser **2. Upload to S3/GCS manually** - Works, but requires manual steps every time - Who manages the bucket? What about access control? **3. Rely on CI artifacts** - [GitHub Actions artifacts expire](/solutions/ci-test-artifacts-expiring/) after 90 days (configurable) - GitLab artifacts expire too - Finding the right artifact in a sea of workflow runs is tedious **4. Share your screen** - Not async-friendly - Can't reference later| Method | Retention | Shareable link | Automated | Team access |
|---|---|---|---|---|
Local show-report | None | ✕ | ✕ | ✕ |
| Email zip files | Manual | ✕ | ✕ | ✓ |
| CI artifacts | 30–90 days | ✓ | ✓ | ✓ (if CI access) |
| S3 manual upload | Configurable | ✓ | ✕ | Depends |
| Gaffer | Up to 90 days | ✓ | ✓ | ✓ (public share links) |
Your test results live behind a login. That works for your team, but not for the contractor debugging a regression, the PM checking release status, or the partner team blaming your API. Share links give you a read-only public URL for any test run. Recipients open the full interactive HTML report in their browser without a Gaffer account.
## The Problem: Test Results Are Locked Behind Logins Test reports sit in your CI dashboard or your Gaffer organization. When someone outside your team needs to see them, you're stuck with bad options: - **Screenshots in Slack**: Loses detail, can't drill into failures, no interactivity - **Screen sharing**: Requires both people online at the same time - **Exporting to PDF**: Static, loses the interactive HTML report - **Adding them to your org**: Overkill for a contractor who needs one report You shouldn't need to add someone to your organization just to show them a test result. ## Share Links: Public URLs with Controls From any test run in Gaffer, open the share modal and get a public URL. The link gives read-only access to the test run summary and the HTML report, no login required. ### How It Works 1. Open a test run and click **Share** 2. Gaffer generates a unique URL for that test run 3. Send the link to anyone, they see the results in their browser 4. Set an expiration (1 hour, 24 hours, 7 days, 30 days, or never) 5. Revoke the link at any time Each test run gets one share link. Creating a link is idempotent — opening the share modal again returns the same URL. ### What Recipients See The share page shows the same data your team sees on the test run detail page: - **Test counts**: Passed, failed, and skipped - **Branch and commit info** - **Interactive HTML report**: The full Playwright, Jest, or other framework report, viewable in the browser Recipients who don't have a Gaffer account see a sign-up prompt. Members of your organization see a link back to the dashboard. Authenticated users from other organizations see an option to request access. ## Use Cases ### Sharing with Contractors A freelance QA engineer is testing your app. They don't need full dashboard access, they need to see what failed in the latest run. Send them a share link with a 7-day expiration. ### Stakeholder Updates Your PM asks "did the regression suite pass before release?" Instead of a screenshot or a verbal "yes," send them the link. They see the actual results. ### Cross-Team Debugging Another team's API change broke your integration tests. Share the link in their Slack channel. They can see exactly which tests failed and what the errors were. ### Client Reporting If you're an agency running tests for a client, share links let you send test results without provisioning accounts. Set the link to expire after 30 days so it doesn't linger. ## Expiration and Revocation Share links aren't permanent by default. You control how long they're active: | Setting | Duration | |---------|----------| | 1 hour | Quick review, then auto-expires | | 24 hours | End-of-day sharing | | 7 days | Sprint-length access | | 30 days | Monthly reporting | | Never | Link stays active until manually revoked | You can update the expiration after creating the link, or revoke it entirely. Revoked links return a "not found" page. ## How It Fits with the Rest of Gaffer Share links complement the existing workflow: - **CI uploads test results** to Gaffer automatically - **Your team** views results in the dashboard and gets [Slack notifications](/solutions/test-failure-notifications/) - **External collaborators** get share links when they need access to specific runs - **[HTML reports](/solutions/html-test-reports/)** are viewable through share links, no downloads required Share links are read-only. Recipients can view results but can't modify anything in your organization. ## Get Started Share links are available on all Gaffer plans. Open any test run, click Share, and send the link.Your Playwright report is in GitHub Actions. The screenshot is in a Slack thread from last Tuesday. The coverage data is in an S3 bucket that two people have access to. When a test fails, the first ten minutes aren't spent debugging. They're spent finding the evidence.
For definitions and file types, read [What Are Test Artifacts in Software Testing?](/blog/what-are-test-artifacts/). This page covers where the files go once CI has produced them.
Your CI pipeline runs tests on every push. When tests fail, how does your team find out? If the answer is "someone checks the GitHub Actions tab," you have a notification problem. Failures sit unnoticed for minutes or hours while developers context-switch to other work.
## The Problem: Nobody Checks CI Proactively Here's what typically happens when tests fail in CI: 1. Developer pushes code and moves on to the next task 2. CI runs tests in the background 3. Tests fail, but nobody's watching 4. Eventually someone notices the red X on the PR (or doesn't) 5. By the time they investigate, they've lost the mental context The feedback loop is broken. CI runs tests, but the results sit in a dashboard nobody monitors in real-time. ### What Teams Do Instead **1. Check CI manually** - Requires remembering to check after every push - Developers context-switch away while waiting for CI - Easy to miss failures on shared branches like `main` **2. GitHub email notifications** - Buried in inbox noise alongside PR reviews, mentions, and issue updates - No summary of *which* tests failed or why - No link to the full report **3. Custom scripts with `curl` and Slack webhooks** - Brittle, hard to maintain, lives in CI config - Usually just sends "build failed" with no detail - No filtering by branch or failure severity - Every team reinvents this from scratch **4. CI provider's built-in notifications** - GitHub Actions has no native Slack integration for test results - GitLab notifications are noisy and lack test-level detail - No structured test result data, just "job passed/failed" ## Better: Structured Test Notifications What you actually want is a notification that tells you: - **Which tests failed**: not just "the build broke" - **A link to the full report**: one click to see errors, traces, screenshots - **Branch and commit context**: what triggered the failure - **Filtering**: only notify on branches you care about ## How Gaffer Handles Test Notifications Gaffer sends test results to Slack, webhooks, and GitHub commit status after every upload. No custom scripts, no maintenance. ### Slack Integration Connect your Slack workspace in project settings. After each test run upload, Gaffer posts a summary: - Pass/fail counts and overall status - Direct link to the hosted test report - Branch, commit SHA, and trigger context - Failed test names for quick triage ### Notification Triggers Choose when to get notified: - **All runs**: every test upload sends a notification - **Failures only**: only hear about problems - **Consecutive failures**: alert when a test fails multiple runs in a row, filtering out one-off noise ### Branch Filtering Don't get spammed by feature branch noise. Configure notifications per branch pattern: - Only notify on `main` and `release/*` - Skip `dependabot/*` and draft PRs - Different channels for different branches ### Webhooks For custom integrations, Gaffer sends HMAC-SHA256 signed HTTP POST payloads to any endpoint. Payloads include test run results, project context, and report links. See the [webhook docs](/docs/integrations/webhooks/) for the full payload schema. Use webhooks to: - Post to Microsoft Teams, Discord, or any chat tool - Trigger PagerDuty alerts for critical test suites - Update a status page or internal dashboard - Feed into your existing alerting pipeline ### GitHub Commit Status Gaffer sets commit status checks on your PRs automatically. Developers see test results without leaving GitHub: green check or red X directly on the commit. ## Setting Up Test Failure Notifications ### Step 1: Upload Test Results from CI ```yaml # GitHub Actions example - name: Run tests run: npm test - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./test-results ``` ### Step 2: Connect Slack In your Gaffer project settings, click "Connect to Slack" and authorize the integration. Select a channel and choose your notification trigger. ### Step 3: Configure Triggers Pick the trigger that matches your workflow: | Trigger | Best For | |---------|----------| | All runs | Small teams that want full visibility | | Failures only | Teams that only want to hear about problems | | Consecutive failures | Filtering out flaky one-off failures | That's it. Your next CI run will notify your team automatically. ## Comparing Test Notification Approaches | Approach | Setup | Maintenance | Test Detail | Filtering | |----------|-------|-------------|-------------|-----------| | Check CI manually | None | None | Full (if you find it) | None | | GitHub email notifications | Built-in | None | Minimal | Limited | | Custom Slack webhook script | High | High | Whatever you build | Whatever you build | | CI provider notifications | Built-in | Low | Job-level only | Limited | | Gaffer | Low | None | Full report link + summary | Branch, trigger type | ## Beyond Notifications Notifications are the entry point. Once you're uploading test results, you also get: - **[Hosted test reports](/solutions/html-test-reports/)**: shareable URLs for every run - **[Flaky test detection](/solutions/flaky-test-detection/)**: automatic identification of unreliable tests - **[Test analytics dashboard](/solutions/test-results-dashboard/)**: pass rate trends, duration tracking, failure patterns ## Get Started Gaffer's free tier includes Slack integration, webhook support, and GitHub commit status with 500 MB of storage and 7-day retention.You set up an S3 bucket to store test reports. Smart move. Now you have historical data. Six months later, the storage bill is 10x what you expected. Sound familiar?
## The Hidden Cost of "Just Store It" When teams outgrow CI artifact storage, the natural solution is cloud storage: ```yaml - name: Upload to S3 run: aws s3 cp ./test-results s3://test-reports/${{ github.sha }} --recursive ``` Simple. Effective. And a ticking time bomb for your cloud bill. ### Why Storage Costs Compound Unlike compute (which stops billing when idle), storage bills accumulate: | Month | New Reports | Total Stored | Monthly Cost* | |-------|-------------|--------------|---------------| | 1 | 50 GB | 50 GB | $1.15 | | 3 | 50 GB | 150 GB | $3.45 | | 6 | 50 GB | 300 GB | $6.90 | | 12 | 50 GB | 600 GB | $13.80 | | 24 | 50 GB | 1.2 TB | $27.60 | *S3 Standard pricing at $0.023/GB That's assuming steady growth. Teams that scale up CI runs or add more test suites see exponential increases. ### The Playwright Problem E2E testing frameworks like Playwright make storage costs explode. A typical Playwright setup generates: - **Screenshots on failure**: 200KB–2MB each - **Video recordings**: 5–50MB per test - **Trace files**: 10–100MB per test run A team running 100 E2E tests across 3 browsers, 4 times per day: | Artifact Type | Size per Run | Daily | Monthly | |---------------|--------------|-------|---------| | HTML Report | 5 MB | 20 MB | 600 MB | | Screenshots | 50 MB | 200 MB | 6 GB | | Videos | 500 MB | 2 GB | 60 GB | | Traces | 200 MB | 800 MB | 24 GB | | **Total** | **755 MB** | **~3 GB** | **~90 GB** | That's 90 GB per month from *one* project. Most teams have multiple. And nobody wants to disable recordings: they're essential for debugging flaky tests. ## DIY Cleanup: Harder Than It Looks "We'll just write a cleanup script." Famous last words. ### Lifecycle Rules Look Simple ```xmlQA needs to see test results. But test results live in CI: behind GitHub logins, buried in workflow runs, locked in expiring artifacts. QA shouldn't need to learn developer tooling just to find out what failed.
## The QA Access Problem Automated test results typically live in CI: - **GitHub Actions** - Requires repo access, navigation through workflow runs - **Jenkins** - Another dashboard to learn, often requires VPN - **GitLab CI** - Mixed with pipelines, deployments, and other dev stuff QA engineers end up asking developers "can you send me the test results?" or learning CI tools they shouldn't need to know. And when CI artifacts expire (30-90 days), historical data disappears. Good luck comparing this week's test health to last month. ## What QA Teams Need ### Direct Report Access A URL that shows test results. No CI login, no navigating pipelines, no downloading zip files. Share it in Slack: ``` Here's the regression suite from this morning: https://app.gaffer.sh/reports/abc123 ``` Anyone with the link sees the results immediately. ### Historical View QA needs to answer questions like: - "Is this test suite getting more stable?" - "When did these tests start failing?" - "How often does this test actually pass?" This requires data across time, not just the latest run. ### Flaky Test Tracking Flaky tests are a QA nightmare. "It passed locally" vs "It failed in CI" wastes hours of back-and-forth. Knowing which tests are legitimately unreliable vs which failures are real bugs is essential. ### Non-Technical Interface QA shouldn't need to learn git, CI pipelines, or command-line tools to check test results. A web interface that shows pass/fail counts, trends, and failures is enough. ## How Gaffer Helps QA Teams ### Shareable Report URLs Every test run gets a URL in the Gaffer dashboard. QA accesses results directly — no CI login required. Share links in: - Slack channels - Bug tickets - Test management tools - Emails to stakeholders For people outside your organization (external QA consultants, clients, or partner teams), you can generate [public share links](/solutions/shareable-test-report-links/) with configurable expiration. Recipients view the full report in their browser without needing a Gaffer account. ### Dashboard View See all projects and recent test runs in one place: - Pass/fail counts - Trend direction (improving or degrading) - Recent failures No navigating through CI menus. Just a list of projects and their test health. ### Historical Trends Track test suite health over time: - **Pass rate trends** - Is quality improving? - **Failure patterns** - Which tests fail most often? - **Duration trends** - Is the suite getting slower? Gaffer retains data for up to 90 days (depending on plan), longer than most CI artifact retention. ### Flaky Test Reports Automatically identify tests with inconsistent results: - **Flip rate** - How often does this test change between pass and fail? - **Last seen** - When did the flaky behavior last occur? - **Run count** - How many executions are in the analysis? QA can take this list to developers: "These 5 tests are flaky. Can we fix or quarantine them?" ### Slack Notifications Get notified when tests fail: ``` [FAILED] regression-suite - main 3 tests failed, 142 passed View report: https://app.gaffer.sh/reports/xyz789 ``` QA sees failures as they happen, with direct links to investigate. ## QA + Dev Collaboration Test results become a shared artifact: 1. **CI runs tests** - Results upload to Gaffer automatically 2. **QA monitors** - Sees failures in Slack or dashboard 3. **QA investigates** - Opens report, reviews failures 4. **QA files bugs** - Includes report link for context 5. **Dev fixes** - Uses the same report link for debugging 6. **QA verifies** - Checks next test run No "can you send me the output?" conversations. Everyone looks at the same report. ## For QA Leads and Managers ### Test Health Reporting Need to report on test suite health to stakeholders? Gaffer provides: - Overall pass rates - Trend direction - Flaky test counts - Test count over time Export data or share dashboard views in status meetings. ### Coverage Across Projects If your team tests multiple projects, see them all in one dashboard. Compare health across projects, identify which need attention. ## Getting Started QA doesn't need to set up Gaffer - developers add one CI step and results start flowing. But QA should be involved in: 1. **Getting access** - Request a Gaffer account from whoever set it up 2. **Connecting Slack** - Ensure notifications go to the right channels 3. **Learning the dashboard** - It's simple, but a quick walkthrough helps Once set up, QA has direct access to test results without touching CI tools.Small teams ship fast. You don't have dedicated DevOps. You don't have time to set up and maintain test infrastructure. You just need to see what failed in CI without digging through logs.
## The Small Team Testing Reality You're probably dealing with some combination of: - **Limited CI artifacts** - GitHub Actions gives you 90 days, then reports disappear - **No central dashboard** - Test results scattered across branches and workflow runs - **Slack is your command center** - That's where work happens - **Nobody has time for infrastructure** - You're writing features, not managing servers Enterprise test reporting tools want you to set up databases, deploy services, and manage infrastructure. That's not realistic when you have 3-10 engineers and a hundred things to ship. ## What Small Teams Actually Need ### Shareable Test Results When a test fails, you need to share it with the person who can fix it: - Not "go to GitHub, find the workflow, click through three menus, download artifacts" - Just a URL that shows the failure Need to share results with someone outside your org — a contractor or a stakeholder? Generate a [public share link](/solutions/shareable-test-report-links/) with an expiration you control. They see the results in their browser, no account needed. ### Slack Integration You're already in Slack. Test failures should show up there: ``` [FAILED] api-tests - main 2 tests failed, 47 passed View report: https://app.gaffer.sh/reports/abc123 ``` Click the link, see the failure, fix it. No context switching to CI dashboards. ### Flaky Test Detection Small teams can't afford to waste time re-running "that flaky test." You need to know which tests are unreliable so you can fix or quarantine them. ### Zero Infrastructure If it requires running a server, maintaining a database, or configuring Kubernetes, it's not for small teams. You need something that works with a single CI step. ## How Gaffer Works for Small Teams ### 5-Minute Setup 1. Sign up (free tier available) 2. Get an API key 3. Add one step to CI **GitHub Actions:** ```yaml - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./test-results ``` That's it. Test reports are now hosted and shareable. ### Connect Slack Enable Slack notifications and choose which branches trigger alerts. Most teams only want notifications for `main` - you don't need noise from every feature branch. ### Connect GitHub Enable commit statuses to see test results directly on PRs. No more "CI passed" that actually means "tests passed" - you see the test count right on the commit. ### Track Flaky Tests Gaffer automatically identifies tests that flip between pass and fail. The analytics page shows: - Which tests are most unreliable - How often they flip - Whether they're getting better or worse ## Pricing That Makes Sense Small teams don't have enterprise budgets. Gaffer pricing is flat-rate, not per seat: - **Free** - 500 MB storage, 7-day retention - **Pro** - $15/mo, 10 GB storage, 30-day retention - **Team** - $49/mo, 50 GB storage, 90-day retention All plans include unlimited users and unlimited projects. One price whether you have 3 engineers or 15. ## Real Small Team Workflow Here's how it typically works: 1. **Developer pushes code** - Tests run in CI 2. **Tests fail** - Slack gets a notification with a report link 3. **Developer clicks link** - Sees exactly what failed, with screenshots and stack traces 4. **Developer fixes it** - Or asks for help by sharing the same link 5. **Tests pass** - Commit status turns green No digging through CI logs. No downloading artifacts. No "can you check the test output?" conversations. ## What You Don't Need Gaffer skips features that small teams don't use: - No complex dashboard builder - No ML-powered failure analysis - No role-based access control tiers - No enterprise SSO requirements Just test reports, sharing, and analytics. The basics, done well. ## Try It Start with the free tier. One project, 7-day retention. Enough to see if it fits your workflow before paying anything.Your tests finished 10 minutes ago. You click into GitHub Actions, find the workflow run, expand the test step, scroll through 500 lines of output looking for the failure. By the time you find it, you've forgotten why you were looking. Sound familiar?
## Where CI Test Results Hide Every CI platform buries test results differently, but they all make you work for it: - **GitHub Actions** - Workflow run → Job → Step → Expand logs → Scroll - **GitLab CI** - Pipeline → Job → Scroll through output - **Jenkins** - Build → Console Output → Search for "FAILED" - **CircleCI** - Workflow → Job → Steps → Find the test step Even when you find the results, you're looking at raw terminal output. Pass counts mixed with stack traces mixed with timing info. Good luck sharing that with your PM. ## The Real Pain Points ### No Single View Test results are scattered across CI runs, branches, and PRs. Want to know the overall health of your test suite? You'd need to manually check multiple workflow runs and piece it together. ### Context Switching Every failed test means: leave your IDE → open CI → find the right run → expand the right step → scroll to the failure. Multiply by 5 failures and 10 minutes are gone. ### No Historical Context Was this test flaky last week? Has the pass rate been declining? CI logs don't tell you. Each run exists in isolation with no connection to the past. ### Team Visibility When QA asks "did the tests pass?" you have to go check. When the PM wants to see test coverage for a feature, you export a CSV. When a colleague debugs the same flaky test you fixed last month, there's no shared knowledge. ## What Teams Actually Need A test results dashboard should answer these questions instantly: | Question | Dashboard Answer | |----------|------------------| | Did the tests pass? | Green checkmark or red X, visible in 2 seconds | | Which tests failed? | Click to see the specific failures | | Is this test flaky? | Historical pass rate shows the pattern | | Is the suite healthy? | Trend line shows improvement or decline | | Can I share this with QA? | Send a link, no CI access required | ## CI Logs vs. Test Dashboard | Aspect | CI Logs | Test Dashboard | |--------|---------|----------------| | Time to find failure | 2-5 minutes | 10 seconds | | Historical data | None | Full run history | | Shareable | Requires CI access | Simple link | | Cross-framework view | Separate per tool | Unified | | Flaky test detection | Manual observation | Automatic | | Team access | Developer-only | Anyone with link | ## How Gaffer Works as Your Test Dashboard Gaffer collects test results from your CI pipeline and presents them in a clean, searchable dashboard.Gaffer and TestDino both host and analyze CI test reports, and they bet on different things. [TestDino](https://testdino.com/) goes deep on Playwright: inline trace, DOM, and network debugging, a built-in test-case management layer, and an agent-writable MCP server. Gaffer stays framework-agnostic and prices on storage instead of executions or seats. The clearest split is the bill. TestDino's tiers gate on a monthly execution quota and a hard user cap (1 user on Free, 3 on Pro, 20 on Team), while Gaffer bills on stored bytes with unlimited users on every tier. A 25-person team running roughly 100,000 Playwright results a month clears TestDino's Team tier on two axes at once, the 75,000-result quota and the 20-seat cap, while the same workload sits inside Gaffer's $49/month Team plan, which never caps users.
## What Gaffer and TestDino Actually Are TestDino is a Playwright-native test reporting and analytics platform. It ingests Playwright output and adds inline debugging (trace, DOM snapshots, network logs), root-cause failure categorization, test-case management, and an MCP server that coding agents can both read from and write to. The full feature reference is in [TestDino's docs](https://docs.testdino.com/). Gaffer hosts and analyzes test reports from many frameworks. It ingests JUnit XML, CTRF, Jest and Vitest JSON, native Playwright JSON, and TRX/.NET, plus four coverage formats, then runs [flaky detection](/solutions/flaky-test-detection/), pass-rate trends, and [failure clustering](/solutions/failure-clustering/) on every upload. It is a reporting and analytics surface, not a test-management tool. ## Where TestDino Is Stronger If your stack is Playwright and you want depth over breadth, TestDino has real advantages Gaffer does not match: - **Playwright-native inline debugging:** trace, DOM snapshots, and network logs attached to each failure, viewable without leaving the report. - **Write-capable MCP:** its public MCP server (`testdino-hq/testdino-mcp`) exposes 33 tools, including mutations like `create_manual_test_case` and `create_release`, so a coding agent can manage test artifacts directly. - **Built-in test-case management:** suites, cycles, and imports from TestRail or CSV. - **Root-cause failure grouping:** failures are grouped by error message and a root-cause category (timing, environment, network, assertion), on top of message matching. - **Failed-only reruns and flakiness merge-gates:** rerun only the failures, and block merges on a flaky signal. - **Published compliance posture:** TestDino publicly claims SOC 2 Type II and ISO 27001 certification and GDPR compliance on its security page. Gaffer claims no certifications, which is a real gap for teams with a security-review checklist. ## Where Gaffer Is Stronger Gaffer's advantages are pricing shape and framework breadth, plus a smaller surface to adopt. - **Flat, storage-based pricing with unlimited users:** no per-tier execution quota and no seat cap. Everyone on the team can have an account on any tier, including Free. - **Multi-framework reporting:** one dashboard for JUnit XML, CTRF, Jest, Vitest, native Playwright JSON, and TRX/.NET, which transitively covers pytest, Go, Cypress, and anything that emits JUnit XML or CTRF. See [multiple test frameworks](/solutions/multiple-test-frameworks/). - **[Coverage tracking](/solutions/code-coverage/) alongside test results:** four coverage formats parsed and trended in the same view as pass rate. - **No test-management layer to adopt:** Gaffer hosts reports and runs analytics. There is nothing to configure beyond a CI upload step. ## Pricing: Execution Quotas vs Flat Storage The two products meter different things. TestDino charges a flat monthly subscription but gates each tier on a monthly execution quota and a hard user cap. Gaffer charges a flat monthly subscription that meters only storage, and never caps users. TestDino counts each test-case run as one result: a single test run across three browsers counts as three results, and each retry counts again. Cross-browser matrices and retries consume the quota quickly. Gaffer counts nothing per-execution. It bills on stored bytes, with $0.50/GB/month overage on Pro and Team, and puts no cap on users on any tier including Free. Which is cheaper depends on the shape of your workload. A small Playwright-only team may fit comfortably inside TestDino's Free or Pro tier. A CI-heavy team, or one past 20 people, usually pays less on Gaffer, because neither execution volume nor headcount moves the bill. The above-quota overage price on TestDino is not published, so a team near its quota should confirm it before scaling up. ## Pricing Comparison | Tier | TestDino | Gaffer | |------|----------|--------| | Free | $0, 5,000 results/mo, 1 user, 1 project | $0, 500 MB, 7-day retention, unlimited users | | Entry paid | $39/mo billed annually ($49/mo monthly), 25,000 results/mo, 3 users, 3 projects | $15/month, 10 GB, 30-day retention, unlimited users | | Team | $79/mo (annual-effective), 75,000 results/mo, 20 users, 5 projects | $49/month, 50 GB, 90-day retention, unlimited users | | Overage | Above-quota price not published | $0.50/GB/month (Pro and Team) | | Enterprise | Custom | Not offered | | Billing basis | Execution quota + user cap per tier | Storage, unlimited users | A "result" on TestDino is one test-case run, so cross-browser runs and retries multiply against the quota. Gaffer's bill moves only with stored bytes. ## Feature Comparison | Feature | TestDino | Gaffer | |---------|----------|--------| | Test report hosting | Yes | Yes | | Framework scope | Playwright-native | Multi-framework (JUnit XML, CTRF, Jest, Vitest, Playwright JSON, TRX/.NET; transitively pytest, Go, Cypress) | | Inline trace / DOM / network debugging | Yes (Playwright-native) | No | | Historical trends | Yes | Yes | | Flaky test detection | Yes | Yes | | Failure grouping | Error message + root-cause category | Error-message similarity + file-path fallback | | Test-case management (suites, cycles, imports) | Yes | No (by design) | | Failed-only reruns / flaky merge-gates | Yes | No | | MCP server | Yes, write-capable (33 tools) | Yes, read-only (16 GET functions) | | Coverage tracking | Not advertised | Yes (4 formats) | | Published compliance posture | SOC 2 Type II and ISO 27001 claimed; GDPR compliance stated | None | | User pricing | Capped per tier (1 / 3 / 20) | Unlimited users, all tiers | Both MCP servers are real. TestDino's is write-capable so an agent can create test cases and releases. Gaffer's is read-only code-mode analytics (16 GET functions, zero mutations), a deliberate choice that matches Gaffer not being a test-management tool: query your results, do not manage test cases from the editor. ## When to Use TestDino TestDino is the better fit if: - Your suite is Playwright and you want inline trace, DOM, and network debugging on every failure. - You want a coding agent that can write test artifacts (manual test cases, releases) through MCP, beyond reading metrics. - You need built-in test-case management: suites, cycles, and imports from TestRail or CSV. - Failed-only reruns and flakiness merge-gates are part of your CI workflow. - Your security review needs a vendor that publishes SOC 2 Type II and ISO 27001 claims. - Your team is small enough to sit inside the 1 / 3 / 20 user caps and the per-tier result quota. ## When to Use Gaffer Teams shopping for a TestDino alternative usually hit one of two walls first: the seat cap or the execution quota. Gaffer is the better fit if: - You run more than one test framework and want them in a single dashboard. - Your team is at or past 20 people, or growing, and a seat cap is the binding constraint. - Your CI produces high result volume (cross-browser matrices, frequent retries) and you would rather not track it against a quota. - You want coverage tracking trended next to pass rate for the same upload. - You want a plain hosted-report and analytics surface with no test-management layer to adopt. - You want read-only MCP analytics for agentic CI without giving an agent write access to your test data. ## Using Both You can run both without a migration. Standard report formats are portable, so a common split is to upload Playwright output to TestDino for deep, inline debugging and send JUnit XML or CTRF to Gaffer for multi-framework hosting and analytics. Neither side requires a big-bang cutover, and JUnit XML and CTRF are first-class inputs on Gaffer, so adding a second upload step in CI is the whole cost. ## FAQ ### Is Gaffer or TestDino cheaper? It depends on the shape of your workload. TestDino gates each tier on a monthly execution quota and a hard user cap (1 user on Free, 3 on Pro, 20 on Team), while Gaffer bills on storage with unlimited users. A CI-heavy team, or one larger than 20 people, usually pays less on Gaffer, because neither execution volume nor headcount changes the bill. A small Playwright-only team may fit inside TestDino's free or Pro tier. ### Does Gaffer support Playwright? Yes. Gaffer ingests native Playwright JSON, and also Jest, Vitest, pytest, Go, Cypress, and .NET through JUnit XML, CTRF, and native JSON formats, plus coverage formats. TestDino is Playwright-native and does not claim other frameworks. ### Does Gaffer have test-case management? No, by design. Gaffer hosts and analyzes test results; it is not a test-management tool and has no suites, cycles, or manual test plans. TestDino includes test-case management with suites, cycles, and imports from TestRail or CSV. ### Is TestDino only for Playwright? TestDino is Playwright-native and does not claim support for other frameworks. Its inline trace, DOM, and network debugging are built on Playwright's own artifacts. If you run multiple frameworks and want them in one dashboard, Gaffer's multi-framework ingestion is the better fit. ### Can I use both Gaffer and TestDino? Yes. Standard report formats are portable, so a team can upload Playwright output to TestDino for deep debugging and send JUnit XML or CTRF to Gaffer for multi-framework hosting and analytics. There is no big-bang migration required in either direction. ## Other Comparisons Evaluating more tools? See how Gaffer compares to [Currents.dev](/solutions/currents-dev-alternative/), [Allure](/solutions/allure-alternative/), and [ReportPortal](/solutions/reportportal-alternative/). ## Try It Gaffer's free tier includes 500 MB storage, 7-day retention, and unlimited users. Enough to point your CI at it for a week and decide.Vitest's HTML reporter is excellent: interactive, fast, and genuinely useful for debugging. But it generates local files. When your teammate asks "what failed?", you can't just send them a link. You end up screenshotting terminal output or walking them through downloading CI artifacts.
## The Sharing Problem After `vitest run`, your HTML report lands in a local directory. To share it with your team, you need to get that file to them somehow. Here's what most teams do: ### Common (Bad) Solutions **1. CI artifacts** - GitHub Actions keeps artifacts for 90 days max, other CI systems are shorter - Finding the right artifact means clicking through workflow runs - Recipient downloads a zip, extracts it, opens `index.html` locally - QA might not even have CI access **2. Email or Slack the file** - "Here's the test report" with a zip attachment - Recipient has to download, extract, open in browser - Version confusion when multiple runs are in flight **3. Upload to S3 manually** - Works, but someone has to set up the bucket, permissions, and cleanup - Manual step every time, so it gets skipped when things are busy **4. Paste terminal output** - Loses all the interactive HTML goodness - Hard to parse in a Slack thread - Can't reference it later ## Better: Hosted Vitest Reports with Shareable Links The fix is straightforward: upload your Vitest reports to a hosting service after every CI run. Every test run gets a permanent URL that anyone on your team can open. ### What This Looks Like 1. CI runs your Vitest tests 2. Reports upload automatically 3. You get a URL like `https://app.gaffer.sh/reports/abc123` 4. Share it in Slack, GitHub PRs, Jira, or anywhere else 5. Anyone with access sees the full interactive report No downloads. No manual steps. No expiring artifacts. ## Setting Up Vitest Report Sharing with Gaffer ### Step 1: Configure Vitest Reporters Make sure you're generating output Gaffer can parse. CTRF gives the richest analytics: ```bash npm install vitest-ctrf-json-reporter --save-dev ``` ```typescript // vitest.config.ts import { defineConfig } from 'vitest/config' export default defineConfig({ test: { reporters: [ 'default', ['vitest-ctrf-json-reporter', { outputDir: '.', outputFile: 'ctrf-report.json' }], ], } }) ``` If the reporter fails with `ENOENT`, see [why jest-ctrf-json-reporter throws ENOENT](/docs/guides/ctrf/#why-does-jest-ctrf-json-reporter-throw-enoent). Gaffer also accepts Vitest's built-in HTML and JSON reporters if you prefer those; the [Vitest JSON reporter output format](/blog/vitest-json-reporter-output-format/) is documented field by field. If you also want to [set up a Vitest coverage report and track it across runs](/blog/vitest-coverage-reports/), that's a separate upload step covered in the coverage guide. ### Step 2: Add the Upload Step to CI **GitHub Actions:** ```yaml - name: Run tests run: npm test - name: Upload to Gaffer if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: ./ctrf-report.json commit_sha: ${{ github.sha }} branch: ${{ github.ref_name }} ``` **GitLab CI:** ```yaml test: script: - npm ci - npm test after_script: - | curl -X POST https://app.gaffer.sh/api/upload \ -H "X-API-Key: $GAFFER_PROJECT_TOKEN" \ -F "files=@ctrf-report.json" \ -F 'tags={"commitSha":"'"$CI_COMMIT_SHA"'","branch":"'"$CI_COMMIT_REF_NAME"'"}' ``` ### Step 3: Share the Link After upload, Gaffer returns a URL. Share it anywhere: Slack, GitHub PR comments, Jira tickets. Recipients see the full report in their browser. For people outside your organization, you can generate a [public share link](/solutions/shareable-test-report-links/) with configurable expiration. They view the full interactive report without needing a Gaffer account. ## Slack and Webhook Notifications Instead of checking CI manually, get notified when tests finish: **Slack integration:** - Pass/fail summary posted to your channel - Direct link to the full report - Filter by branch: only notify on `main`, skip feature branches **Webhooks:** - Send test results to any endpoint - Integrate with your existing alerting or workflow tools - Trigger custom actions on failure No more "did the tests pass?" messages. Your team sees failures as they happen, with a link to investigate. ## GitHub Integration Commit statuses show test results directly on PRs. No clicking through to CI logs. You see pass/fail right on the commit. ## One Dashboard for Every Suite Most JavaScript teams run Vitest for unit tests and Playwright or Cypress for end-to-end. Gaffer aggregates [multiple test frameworks](/solutions/multiple-test-frameworks/) into one dashboard, so unit and e2e results land in the same place instead of separate report folders. ## What About Flaky Tests? Vitest's `--retry` flag handles flaky tests within a single run. Gaffer tracks flakiness *across* runs, identifying tests that sometimes pass and sometimes fail over time. Different problem, complementary solution. ## Get Started Gaffer's free tier includes 500 MB of storage with 7-day retention. Paid plans offer extended retention up to 90 days.