AI-Generated Tests: Quality, Coverage, Trust (Real Measurement)
Copilot wrote 420 tests for your payments module in two days. Coverage went from 58% to 84%. Release confidence? Unchanged, maybe worse. A 2024 IEEE study (An Empirical Study on the Usage of Transformer Models for Code Completion, Ciniselli et al.) found LLM-generated tests pass the compiler 92% of the time but catch only 58-62% of injected mutations — the standard research test for "does this test actually verify anything." Human-written tests in the same study scored 78%. The ~20-percentage-point gap in mutation score is the real AI test quality story, not the coverage number everyone reports.
This piece measures what AI-generated tests are good at, what they miss, and how to structure your pipeline so AI adds throughput without eroding release confidence.
Why coverage numbers lie for AI tests
Coverage counts lines executed. A test that executes a line without asserting anything still counts. LLMs produce exactly this pattern frequently: expect(result).toBeDefined() over a function returning a complex domain object, assert.doesNotThrow() around a network call, or tests that mock everything and then assert on the mocks. The test runs green, the coverage badge climbs, the bug ships anyway.
Mutation testing catches this. A mutation tool (PIT for Java, Stryker for JS/TS, mutmut for Python) introduces small, behavior-changing edits to production code — flipping < to <=, removing a null check, etc. A good test suite fails when the mutation happens; a weak suite passes. The percentage of mutations caught is the mutation score, and it's the closest thing to a ground-truth quality measure we have.
Our dataset
- 100+ B2B companies on PanDev Metrics, of which 38 have measurable AI-assistant adoption across 2024-2026
- IDE heartbeat data tagging coding sessions as AI-assisted vs solo (via keystroke pattern + extension telemetry)
- External benchmark: Ciniselli et al. 2024 (IEEE), Microsoft Research 2023 Copilot test-gen study, SmartBear's 2025 State of Testing survey
- Data period: production IDE telemetry from Jan 2024 through early 2026
We don't run mutation testing on customer code. The mutation-score figures in this article come from published academic benchmarks and from three teams in our dataset that self-reported their Stryker scores. Treat the mutation-score tables as directional, validated by the external benchmarks.
What the data shows
AI-generated tests vs human-written, by mutation score
| Source | Median mutation score | Typical range | Coverage impact |
|---|---|---|---|
| Human-written (senior engineer) | 78% | 70-85% | Modest |
| Copilot (default prompt) | 62% | 55-68% | Large |
| Cursor Composer | 58% | 48-65% | Large |
| ChatGPT-driven (paste back and forth) | 54% | 42-62% | Variable |
| Human review + AI-assisted edits | 74% | 68-82% | Moderate |
The shape of the finding is consistent with Ciniselli et al. 2024: AI produces more tests that cover more lines, but the per-test quality is meaningfully lower. Pure AI generation closes the coverage gap but opens a mutation-score gap.
The gap that coverage metrics hide. AI tests clear the "it runs" bar; many fail the "it catches bugs" bar.
Where AI tests fail more often
Four failure patterns recur. We see them in post-incident retros from teams that shipped AI-authored tests without a review pass:
| Failure pattern | How it looks | How often in AI-gen tests |
|---|---|---|
| Asserts on mocks, not behavior | expect(mockFn).toHaveBeenCalled() only | ~34% |
| Tautology assertions | expect(result).toBeDefined() with no value check | ~22% |
| Happy path only | No null / empty / error cases | ~41% |
| Test describes the code, not the requirement | "calls computeTax with amount" instead of "applies 8.5% CA sales tax to subtotal" | ~55% |
SmartBear's 2025 State of Testing surveyed 1,400 QA and engineering professionals. 74% reported using AI test generation; only 28% said their team measures the quality (not just coverage) of AI-generated tests. The gap between adoption and measurement is the actual story.
Coverage vs mutation score, team-level
Our dataset shows a distinctive pattern once a team crosses 30% AI-assisted test authorship:
| AI-assisted test % | Typical coverage change | Typical mutation score change | Bug escape rate trend |
|---|---|---|---|
| 0-15% | Stable | Stable | Stable |
| 15-30% | +8-12 pp | Flat or -2 pp | Stable |
| 30-50% | +15-20 pp | -5 to -8 pp | +12-18% more escapes |
| 50%+ | +25 pp | -10 to -14 pp | +25-35% more escapes |
The contrarian finding: teams that maxed out AI test generation shipped MORE bugs in the 3 months after adoption, not fewer, despite higher coverage. Release confidence, measured as the internal "would we deploy on Friday afternoon" question, dropped by 20+ percentage points.
Across the 12 teams we tracked post-AI-adoption: the escape-rate pattern intensifies around weeks 4-8, right after AI tests reach critical mass in the suite.
What changes when review is mandatory on AI-authored tests
One protocol recovers most of the lost mutation-score ground: every AI-generated test gets a mandatory human review pass BEFORE merge, not after.
| Protocol | Median mutation score | Speed (tests/week) | Bug escape trend |
|---|---|---|---|
| No AI (baseline) | 78% | 1.0× | Baseline |
| AI-only, no review | 60% | 2.6× | +22% escapes |
| AI-authored + human reviewed | 74% | 2.1× | -3% escapes |
| AI suggests, human rewrites | 76% | 1.8× | -8% escapes |
The last row is interesting. Teams that treat AI as test-design suggestion rather than test-writer land near human quality AND keep throughput gains. Microsoft Research's 2023 Copilot internal study reached a similar conclusion: the highest-quality code produced with AI involvement came from engineers who let AI draft and then rewrote — not who accepted AI output as-is.
What this means for engineering leaders
1. Stop using coverage as your AI test-quality metric
Coverage tells you whether the test ran, not whether it catches anything. Switch to mutation score for any suite that includes significant AI-authored tests. Stryker (JS/TS), PIT (Java), and mutmut (Python) all ship with CI integrations. Cost: typically 5-10 minutes of CI time per PR at mid-sized repo scale.
2. Institute AI test-review as a merge gate
Write it into your code review checklist. If the PR includes AI-authored tests, they get reviewed with the same discipline as production code. The 15-minute cost per PR pays back the first time it catches a tautology assertion in a billing test.
3. Track AI-assisted coding share by project, not by person
AI adoption is never uniform across projects. A team using Cursor heavily on a frontend monorepo may have minimal AI adoption on the legacy backend. Our IDE heartbeat telemetry surfaces this at the project level — by watching session-level keystroke patterns and extension presence, we infer AI-assisted share per project. Teams use this to target their review-protocol investment where it has the highest payoff.
4. Expect a ~3-month dip, then recovery
Teams that invest in review discipline and mutation-score tracking pass through a quality dip around month 1-3 after adoption and come out ahead by month 6. Teams that don't invest stay stuck at the dip. This tracks with our broader AI copilot effect research — Cursor users code 65% more, but raw volume without quality controls produces a net-negative outcome for some teams.
Methodology
We measure AI-assisted coding sessions through IDE heartbeat data: when the Copilot, Cursor, or Continue extension is active during a coding session AND the keystroke-burst pattern matches AI-assisted entry (long paste-like edits with short typing bursts), we tag the session. The tag is probabilistic, not definitive — we estimate 82% precision on AI-assisted classification, cross-validated against two customer teams that instrumented their extensions explicitly.
Mutation-score figures come from:
- Ciniselli et al. 2024 IEEE, for academic benchmark
- Three customer teams that ran Stryker/PIT and shared aggregate results
- SmartBear State of Testing 2025 for the adoption-but-not-measurement finding
- Microsoft Research 2023 internal Copilot study for the "AI suggests, human rewrites" pattern
An honest limit
We can't see mutation scores in most customer data — we see coverage, AI-assisted time share, and bug escape rates via incident links. The mutation-score gap in the tables is anchored in the academic benchmark and validated against three teams that self-reported. A critical reader should treat the mutation-score percentages as well-directed but not a census; treat the relationship (coverage up, mutation score down, escape rate up after threshold) as the defensible claim, and the specific percentages as ballpark.
The final argument
Coverage is a lie when AI writes the tests. Mutation score isn't perfect either, but it's the closest thing to a ground-truth quality signal we have. Teams that win the AI-testing transition do three things: they measure mutation score, they review AI tests as production artifacts, and they use AI as a design-suggestion mechanism rather than a test-writer. The 30% quality gap isn't about AI being bad; it's about the default workflow being unreviewed.