AgentAlley
Submit

Fix Failing or Flaky Tests

Fix failing or flaky Playwright tests.

How to use it

Claude Code
  1. Run the line below. It pulls the whole folder into ~/.claude/skills/fix, including the files SKILL.md points to.
  2. Describe your job in plain words. Claude Code follows the skill from there.
Claude Code — installs the whole folder, not just SKILL.md
npx degit alirezarezvani/claude-skills/engineering-team/playwright-pro/skills/fix#main ~/.claude/skills/fix

For one project only, change the path to .claude/skills/fix. This skill also uses flaky-taxonomy.md — copying SKILL.md alone won't be enough. See the folder on GitHub.

Claude (web or desktop app)
  1. On this page open ⋯ → Download .md.
  2. Save it as SKILL.md in a folder, zip the folder, then Customize → Skills → + → Create skill → Upload a skill.
  3. Pick the file and Save. Claude shows the name and description and runs a security scan.
  4. Check the skill is switched on.
  5. Start a new chat and describe your job in plain words. The AI follows the skill from there.
ChatGPT or another app
  1. ChatGPT: make a Project and paste it into Instructions.
  2. Neither? Paste it at the top of a new chat — it works for that chat.
Not working?
  • Check which app you pasted it into — the steps above name the right one.
  • Some skills need the paid tier of Claude or ChatGPT.
Step-by-step guide with screenshots · Ask in the forum

Paste into Claude, ChatGPT or Cursor.

Source of Fix Failing or Flaky Tests

Show the full text114 lines
fix/SKILL.md114 lines · 3.0 KB
Outline
RawView on GitHub
namedescription
fix>- Fix failing or flaky Playwright tests. Use when user says "fix test", flaky test", "test failing", "debug test", "test broken", "test passes sometimes", or "intermittent failure".

Fix Failing or Flaky Tests

Diagnose and fix a Playwright test that fails or passes intermittently using a systematic taxonomy.

Input

$ARGUMENTS contains:

  • A test file path: e2e/login.spec.ts
  • A test name: ""should redirect after login"`
  • A description: "the checkout test fails in CI but passes locally"

Steps

1. Reproduce the Failure

Run the test to capture the error:

npx playwright test <file> --reporter=list

If the test passes, it's likely flaky. Run burn-in:

npx playwright test <file> --repeat-each=10 --reporter=list

If it still passes, try with parallel workers:

npx playwright test --fully-parallel --workers=4 --repeat-each=5
2. Capture Trace

Run with full tracing:

npx playwright test <file> --trace=on --retries=0

Read the trace output. Use /debug to analyze trace files if available.

3. Categorize the Failure

Load flaky-taxonomy.md from this skill directory.

Every failing test falls into one of four categories:

Category Symptom Diagnosis
Timing/Async Fails intermittently everywhere --repeat-each=20 reproduces locally
Test Isolation Fails in suite, passes alone --workers=1 --grep "test name" passes
Environment Fails in CI, passes locally Compare CI vs local screenshots/traces
Infrastructure Random, no pattern Error references browser internals
4. Apply Targeted Fix

Timing/Async:

  • Replace waitForTimeout() with web-first assertions
  • Add await to missing Playwright calls
  • Wait for specific network responses before asserting
  • Use toBeVisible() before interacting with elements

Test Isolation:

  • Remove shared mutable state between tests
  • Create test data per-test via API or fixtures
  • Use unique identifiers (timestamps, random strings) for test data
  • Check for database state leaks

Environment:

  • Match viewport sizes between local and CI
  • Account for font rendering differences in screenshots
  • Use docker locally to match CI environment
  • Check for timezone-dependent assertions

Infrastructure:

  • Increase timeout for slow CI runners
  • Add retries in CI config (retries: 2)
  • Check for browser OOM (reduce parallel workers)
  • Ensure browser dependencies are installed
5. Verify the Fix

Run the test 10 times to confirm stability:

npx playwright test <file> --repeat-each=10 --reporter=list

All 10 must pass. If any fail, go back to step 3.

6. Prevent Recurrence

Suggest:

  • Add to CI with retries: 2 if not already
  • Enable trace: 'on-first-retry' in config
  • Add the fix pattern to project's test conventions doc

Output

  • Root cause category and specific issue
  • The fix applied (with diff)
  • Verification result (10/10 passes)
  • Prevention recommendation
1---
2name: "fix"
3description: >-
4 Fix failing or flaky Playwright tests. Use when user says "fix test",
5 "flaky test", "test failing", "debug test", "test broken", "test passes
6 sometimes", or "intermittent failure".
7---
8 
9# Fix Failing or Flaky Tests
10 
11Diagnose and fix a Playwright test that fails or passes intermittently using a systematic taxonomy.
12 
13## Input
14 
15`$ARGUMENTS` contains:
16- A test file path: `e2e/login.spec.ts`
17- A test name: ""should redirect after login"`
18- A description: `"the checkout test fails in CI but passes locally"`
19 
20## Steps
21 
22### 1. Reproduce the Failure
23 
24Run the test to capture the error:
25 
26```bash
27npx playwright test <file> --reporter=list
28```
29 
30If the test passes, it's likely flaky. Run burn-in:
31 
32```bash
33npx playwright test <file> --repeat-each=10 --reporter=list
34```
35 
36If it still passes, try with parallel workers:
37 
38```bash
39npx playwright test --fully-parallel --workers=4 --repeat-each=5
40```
41 
42### 2. Capture Trace
43 
44Run with full tracing:
45 
46```bash
47npx playwright test <file> --trace=on --retries=0
48```
49 
50Read the trace output. Use `/debug` to analyze trace files if available.
51 
52### 3. Categorize the Failure
53 
54Load `flaky-taxonomy.md` from this skill directory.
55 
56Every failing test falls into one of four categories:
57 
58| Category | Symptom | Diagnosis |
59|---|---|---|
60| **Timing/Async** | Fails intermittently everywhere | `--repeat-each=20` reproduces locally |
61| **Test Isolation** | Fails in suite, passes alone | `--workers=1 --grep "test name"` passes |
62| **Environment** | Fails in CI, passes locally | Compare CI vs local screenshots/traces |
63| **Infrastructure** | Random, no pattern | Error references browser internals |
64 
65### 4. Apply Targeted Fix
66 
67**Timing/Async:**
68- Replace `waitForTimeout()` with web-first assertions
69- Add `await` to missing Playwright calls
70- Wait for specific network responses before asserting
71- Use `toBeVisible()` before interacting with elements
72 
73**Test Isolation:**
74- Remove shared mutable state between tests
75- Create test data per-test via API or fixtures
76- Use unique identifiers (timestamps, random strings) for test data
77- Check for database state leaks
78 
79**Environment:**
80- Match viewport sizes between local and CI
81- Account for font rendering differences in screenshots
82- Use `docker` locally to match CI environment
83- Check for timezone-dependent assertions
84 
85**Infrastructure:**
86- Increase timeout for slow CI runners
87- Add retries in CI config (`retries: 2`)
88- Check for browser OOM (reduce parallel workers)
89- Ensure browser dependencies are installed
90 
91### 5. Verify the Fix
92 
93Run the test 10 times to confirm stability:
94 
95```bash
96npx playwright test <file> --repeat-each=10 --reporter=list
97```
98 
99All 10 must pass. If any fail, go back to step 3.
100 
101### 6. Prevent Recurrence
102 
103Suggest:
104- Add to CI with `retries: 2` if not already
105- Enable `trace: 'on-first-retry'` in config
106- Add the fix pattern to project's test conventions doc
107 
108## Output
109 
110- Root cause category and specific issue
111- The fix applied (with diff)
112- Verification result (10/10 passes)
113- Prevention recommendation
114 

Discussion

Browse more free Claude skills.