Fixing Flaky Playwright Tests Without More Retries

Every Playwright suite gets flaky tests eventually. The common reaction is to set retries: 2 and move on, and the suite turns green again. But a test that passes on the second try is still telling you something: either the test is wrong, or the app sometimes is. Retries hide both.

What Playwright calls flaky

Playwright's documentation defines a flaky test as one that failed on the first run but passed when retried. By default, failing tests aren't retried at all, so you only see "flaky" in reports once you've turned retries on. That's useful: retries are a way to detect flakiness, not a fix for it.

Reproduce it first

Before changing any code, make the failure happen on demand:

npx playwright test checkout.spec.ts --repeat-each=20 --retries=0 --trace=retain-on-failure

--repeat-each runs each test that many times, --retries=0 stops retries hiding failures, and --trace=retain-on-failure keeps a trace for every failing run. Run it with your CI's worker count too, because many flaky tests only fail in parallel.

Then open the trace. Playwright's own advice is to use the trace viewer for CI failures rather than videos and screenshots. It shows each action, the DOM at that moment, network calls and console output, which is usually enough to see what the test was waiting for.

The usual causes

Most flaky tests we fix come from four places:

1. Timing. The test checks something before the app has finished. Fixed waits like waitForTimeout(2000) make this worse, because they're too short on a slow CI machine and waste time on a fast one.
2. Locators. A selector matches two elements, or matches an element that's replaced during rendering. Text and CSS classes that change between builds cause this too.
3. Shared state. Tests that depend on each other's data, or run in parallel against the same account, cart or record.
4. Things you don't control. Third-party scripts, payment providers, analytics, chat widgets, or a staging environment that's slow at certain times of day.

Fixes that last

• Use web-first assertions. await expect(page.getByRole('button', { name: 'Pay' })).toBeEnabled() waits until the condition is true or the timeout runs out. Playwright's best practices recommend these over manual checks.
• Use role and label locators. getByRole and getByLabel match what the user sees and survive markup changes better than CSS selectors.
• Isolate each test. Playwright recommends each test be completely isolated. Give every test its own user or data, created in a fixture, so order and parallelism don't matter.
• Mock what you don't control. Playwright's guidance is not to test third-party sites or servers. Route those requests and return fixed responses, and test the integration separately.

Gate CI on flakiness

Once the suite is clean, keep it that way. Playwright has a --fail-on-flaky-tests flag that fails the run if any test needed a retry. Use it on the main branch, so a test that turns flaky is fixed the day it happens, not after it's been ignored for a month.

For pull requests, --only-changed runs tests related to files changed since a git ref, and --repeat-each on new tests catches flakiness before they're merged.

Quarantine, but with a date

Sometimes a flaky test can't be fixed today. Move it to a quarantine tag that runs separately and doesn't block the build, with an owner and a date in the test annotation. Review the quarantine list every week. A test that sits there for a month should either be fixed or deleted.

Checkout tests

Checkout flows are the hardest to keep stable, because they depend on payment gateways, discount rules and stock. Our post on automated checkout tests on Shopify covers where to run them and what to mock. If you're using Playwright's newer AI test agents, the same rules apply to the tests they generate. See our notes on the planner, generator and healer.