Flaky tests: causes, how to detect them and a quarantine strategy

Short answer: a flaky test is one that sometimes passes and sometimes fails without the code having changed. It's almost always caused by timing, shared data between tests, environment differences or animations. The strategy that works is to detect them with retries and traces, fix the root cause and, in the meantime, quarantine them with a deadline instead of ignoring them.

Why they matter more than they seem

The damage of a flaky test isn't the time it costs to rerun: it's that the team stops believing the suite. When "red is usually a false positive", people retry until it passes, and a real failure ends up slipping through.

The most common causes

1. Race conditions: the test moves faster than the application (for example, it clicks before the element is ready).
2. Shared state: one test leaves data that breaks the next, or tests depend on execution order.
3. Environment differences: different browser versions, limited CI resources, time zones or changing database data.
4. Animations and transitions that cover an element right when the test interacts with it.
5. External dependencies (third-party APIs, email, payments) that respond slowly or fail.
6. Fixed waits (waitForTimeout) instead of waiting for a real condition.

How to detect them

Playwright marks a test as flaky when it fails on the first attempt and passes on retry. With a minimal configuration you already get detection with no extra infrastructure:

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  fullyParallel: true,
  retries: process.env.CI ? 2 : 0,
  use: {
    trace: "on-first-retry", // saves DOM, network and console when something fails
  },
});

With trace: "on-first-retry" you capture the evidence at the exact moment the test misbehaves, which is the hardest thing to reproduce later. To confirm whether a test is unstable, run it many times in a row:

npx playwright test tests/checkout.spec.ts --repeat-each=20

How to really fix them

• Replace fixed waits with condition-based waits: use Playwright's assertions (expect(locator).toBeVisible()), which wait and retry on their own.
• Isolate data: every test creates its own and doesn't depend on another.
• Control external dependencies: mock third-party responses in tests that don't need a real integration.
• Pin the environment: browser and dependency versions, and the same image in CI and locally.

A quarantine strategy (so you don't block the team)

If an unstable test blocks Pull Requests, don't delete it or silently ignore it: quarantine it with clear rules.

test("checkout with coupon", { tag: "@quarantine" }, async ({ page }) => {
  // ...
});

# Pull Requests: exclude the quarantine
npx playwright test --grep-invert @quarantine

# Nightly run: only the quarantine, to keep measuring it
npx playwright test --grep @quarantine

Rules that make it work:

1. Every quarantined test has an owner and a deadline (for example, five business days).
2. When the deadline passes, it gets fixed or deleted. A quarantine without a date is just a graveyard.
3. Publish the list on the team's board so it stays visible.
4. Track the flakiness rate (percentage of unstable runs) and watch it go down.

A sign of maturity

A team with a healthy suite isn't the one that "never has unstable tests", but the one that detects them quickly, treats them as defects and doesn't let them pile up.

To go deeper on building a reliable suite from the start, see what to automate first and Playwright with AI: what works and what doesn't.

Does your test suite lose credibility because of false failures? Let's talk and we'll review it together.