How to Analyze Test Failures Across CI Runs (Without Drowning in Logs)

A CI run goes red with 14 failed tests. You open the job log and start scrolling: stack traces, framework noise, retry output, the occasional unrelated warning. Twenty minutes later you’ve confirmed what you suspected on line one, that most of those failures share a single cause. Analyzing test failures shouldn’t mean reading thousands of lines of console output to find the three that matter.

What Analyzing Test Failures Actually Means for a CI/CD Team

Analyzing test failures is the work of turning a list of red results into a short list of decisions: what to fix now, what to ignore, what to file. The raw CI log gives you the symptoms. The job is to find the cause, decide who owns it, and confirm the fix landed.

Test failure vs. error: the distinction that changes your response

A test failure is an assertion that didn’t hold: the code ran, produced a result, and the result was wrong. expected 200, received 500. A test error is the test itself not completing: a thrown exception, a timeout, a missing fixture, a connection refused before any assertion ran.

The distinction matters because the response differs. A failure usually points at application code or a stale expectation. An error often points at the environment, a dependency, or test setup. Treating every red result the same way is how teams waste time mocking around an assertion when the real problem is that a service never came up.

Why Test Failures Are Hard to Triage at Scale

One failed test is easy. A suite that produces dozens of failures across multiple CI runs per day is where triage breaks down, and where most teams fall into one of three traps.

The log-scanning trap. You read the CI job output top to bottom. It works for a handful of failures and collapses past that. Logs are ordered by execution time, not by cause, so five failures from one dead database land scattered between unrelated passes and warnings. You reconstruct the pattern by hand, every single run.

The rerun-until-green trap. Hit re-run, hope the failures were flaky, merge if they clear. Sometimes they were flaky. Sometimes you just rolled the dice on a real bug and got lucky. Either way you learned nothing about which tests are actually unreliable, so you do it again next week.

The spreadsheet trap. Someone starts tracking failures manually: test name, date, “looking into it.” It’s stale within a week because nothing updates it automatically, and it answers none of the questions that matter, like whether this failure has happened before or how often.

The Four Types of Test Failures and the Right Response to Each

Every red result falls into one of four buckets, and each one calls for a different action.

Consistent failures fail on every run with the same error. These are real regressions. The right response: fix the code or update the expectation, then confirm the test passes.

Flaky failures pass and fail without code changes. The right response is not to re-run until green. It’s to identify the flaky test by its history and either stabilize it or quarantine it so it stops polluting triage.

New failures are tests that started failing on a specific run. The right response: tie the failure to the change that introduced it and route it to whoever made that change.

Environment failures are errors, not assertion failures: connection refused, missing variable, wrong runtime version. The right response is to fix the runner or the setup, not the test. These often cascade, which is exactly where clustering earns its keep.

How Failure Clustering Changes Triage

Most reporting tools show failures as a flat list ordered by test name or execution order. That ordering hides the thing you most need to see: which failures share a cause.

Failure clustering groups failures by their error signature instead of by test name. Gaffer extracts the error message from each failing test, strips the variable parts (timestamps, UUIDs, port numbers), and groups tests with matching signatures into a single cluster named after the shared error. Fourteen red tests become “Connection refused (9 tests)” and “expected 200, received 500 (5 tests)”. You’re now looking at two problems, not fourteen.

The clustering runs on parsed error output, so it works the same whether the report came from Playwright, Jest, Vitest, pytest, or JUnit XML. No configuration, no per-framework rules.

Step-by-Step: Triaging a Failure Cluster in Gaffer

Step 1: Upload the CI report

Add one step to your pipeline after the test run. GitHub Actions:

- name: Run tests
  run: npm test

- name: Upload to Gaffer
  if: always()
  uses: gaffer-sh/gaffer-uploader@v2
  with:
    api-key: ${{ secrets.GAFFER_PROJECT_TOKEN }}
    report-path: ./test-results/results.xml

The if: always() matters: without it the upload is skipped exactly when tests fail, which is when you need the report.

Step 2: Read the clusters

Open the run. Clusters appear below the failure list, sorted by how many tests each one affects. Start at the top, because the largest cluster is usually one fix.

Step 3: Separate flaky from consistent

For each cluster, check whether those tests have failed before without code changes. A cluster of known-flaky tests gets a different owner and a different priority than a cluster of consistent regressions. This is the step the rerun-until-green habit skips.

Step 4: Assign or file an issue

Consistent cluster: file one issue for the shared cause, not one per test. New failures: route to the author of the introducing change. Flaky cluster: send it to whoever owns test stability.

Step 5: Confirm it clears

After the fix merges, the next run either shows the cluster gone or it doesn’t. No manual spreadsheet update, no guessing whether the re-run was luck.

What Gaffer Shows That Raw CI Logs Don’t

A CI log is a snapshot of one run. It can’t answer the questions that actually drive triage decisions, because those questions span runs.

• Cross-run persistence: has this exact failure happened before, or is it new on this commit? A log can’t tell you. Gaffer keeps the failure history, so you can tell a fresh regression from a failure that has come back before.
• Frequency: is this test failing 2% of the time or 40%? That number decides whether you stabilize it or quarantine it, and it only exists if something is tracking runs over time.
• Co-occurrence: which failures keep showing up together? Tests that fail as a group point at a shared dependency, and the cluster makes that obvious instead of leaving you to notice it across scrolling sessions.

To get failures in front of the right person without anyone watching CI, connect test failure notifications: Slack or webhook alerts with the pass/fail summary and a direct link to the hosted failure detail, filterable by branch so feature branches don’t spam the channel.

Get Started

Gaffer’s free tier includes 500 MB of storage with 7-day retention, enough to wire up the upload step and watch a few real CI runs cluster their own failures.