← All work
AI QAPlaywrightAgentsTest automationCost control
AI-powered QA: 3,300 tests, every night
A regression suite of over 3,000 tests kept alive by AI agents. Agents tell flaky or outdated tests apart from real product bugs, repair the tests, and file evidence-backed bug reports.
- Context
- Internal platform at a B2B marketplace
- My role
- Owner of the QA phase and bug triage
- When
- 2026
- end-to-end tests run every night
- 3,300+
- real bugs filed in the first ~5 weeks
- 300+
- suite growth in its first month
- ~4×
The challenge
End-to-end tests are valuable but expensive to keep alive. They break whenever the UI changes, so teams stop trusting them, and eventually stop running them.
How it works
- Fresh environments every night. Sandboxes are built from the latest code on dedicated runners, then the full Playwright suite runs.
- Flakiness filter. Every failure is re-run on an idle environment before anyone looks at it.
- AI only where judgement is needed. A headless agent session looks at each confirmed failure and decides: is the test outdated, or is the product broken? Outdated tests get a fix PR. Real bugs go to Jira with screenshots, steps and severity, and the test is quarantined until the fix lands.
- Bug hunting and coverage. Separate nightly agents explore the product across several “lanes”, such as permissions, money flows and sessions, and extend coverage where it’s thin. A change-impact hook maps merged PRs to the test groups they affect.
- Morning summary in Microsoft Teams before 9:00.
Design principle: deterministic execution costs no AI — the model only enters per failure. That keeps the nightly cost low and the results reproducible.
Results
The suite grew about 4× in its first month and now runs over 3,300 tests nightly. In the first five weeks it filed 300+ genuine bugs. More than a third were rated High or Blocker. They included permission and money-handling issues that humans had missed.