Back to the blogAI Testing Tools: Why AI Can't Test the Code It Just Wrote
By WonyoungJuly 25, 20266 minOn this page
Can AI test its own code? If you asked yours to and your users still hit errors, you did not prompt it wrong. When the side that built something also tests it, "passed" stops meaning passed. This is not a question of how good the model is — it is a question of who does the testing. That distinction decides how far AI testing tools can take you. Here is why that happens, and what to change without switching the tools you already use.
- When the AI that built your app also tests it, the same side sets the bar and calls the result — and models favoring their own output is a measured effect, not a hunch.
- A human should approve what counts as passing, and the verdict should come from a real browser actually clicking through, not from text.
- A better model does not fix this. You don't need to switch tools — just move the testing out.
You asked for a test and got a pass
You have probably typed something like this into Cursor or Claude Code.
"Write a test for signup and run it."
A moment later: "I wrote the tests and ran them. All passing." You ship. The next morning a user writes — "I can't sign up."
So what went wrong? Usually not your prompt. A more detailed instruction produces the same outcome, because the problem is who did the testing.
Why AI testing tools stall when the builder also judges
When the side that built it also decides what counts as passing, the verdict comes out generous — no bad intent required. The bar was set by the same hand.
Handing testing to the AI that wrote the code creates exactly that arrangement. The builder defines what passing means and then judges its own output against that definition.
And this is measured, not felt. The 2024 paper LLM Evaluators Recognize and Favor Their Own Generations showed that models like GPT-4 rate their own output higher than other work.
The sharper finding is the 2026 paper Self-Preference Bias in Rubric-Based Evaluation. The researchers handed judges rubrics that are verified programmatically — pass or fail, no interpretation. The bias held anyway: among items the generator actually failed, judges were more than 50% more likely to mark them "satisfied" when the output was their own. That held on a code benchmark too.
However clear you make the bar, a builder testing its own work tilts the call.
![AI test results showed all passed at 14:20, and the next morning a user reported they could not sign up](https://specnote.io/api/content/images/3edf3930-ff38-48d1-929c-27bf3c95784c.png "A full day sat between "passed" and what users actually got")
Why AI written test code still doesn't reassure you
"Then have the AI write the test code." For a developer that works — you can read it and fix it.
If you can't read code, it only adds one more thing you can't check. Handed a test file, you cannot tell:
- what it actually checks
- how far "passed" really goes
- when it breaks, whether the app is wrong or the test is wrong
The third one is the dangerous one. Tests written badly enough to always pass give you a green light that means nothing — and it still feels like reassurance. A check you can't read isn't verification. It's one more thing to take on faith.
Test automation without clicking through everything yourself
No. What changes is not your tools but one arrangement.
Keep building with AI. Move only the testing out. Two things change.
First, a human approves the bar. Steps written in ordinary words — "open the login page → enter email and password → the dashboard loads" — are steps you can read and decide on yourself.
Second, the verdict comes from execution, not from prose. Instead of an AI telling you it works, a real Chromium browser walks those steps like a person and marks each one passed or failed. The whole run is recorded.
You are not switching tools — only one box changes hands
Specnote sits in that testing seat. Connect it to the AI coding tool you already use, keep talking the way you already talk, and the verdict comes from a real browser. You are not switching tools. You are moving where the testing happens.
Five lines tell you exactly what was checked
Won't better models fix this?
It's a fair question. Models do keep improving.
But what we're describing is an arrangement, not a capability. However strong the model, putting it in a position to judge its own output produces the bias those papers measured. A better model does not pull building and testing apart on its own.
The cost stays too. If every verdict runs through an AI, each run is slow, expensive, and slightly different from the last even on identical input. Regression checks are a game you repeat hundreds of times a year, so per-run cost and verdict stability decide it.
Try it now
Move just the testing out
Keep the AI tools you use today. Take the one flow that matters most — signup, checkout, whichever — and hand over only its testing. You never read a line of code.
Frequently asked questions
Aren't there AI testing tools that already handle this?
There are, and they help. The question to ask about any of them is a structural one: does the thing that wrote the code also decide whether it passed? Most test automation tools now advertise AI features, but the feature list is not what decides the outcome. Test automation that runs AI-written checks against AI-written code inherits the same blind spot, however good the model gets. What changes the outcome is moving the verdict outside the builder — the AI still writes the app, but a real browser walks the user's path and reports what it saw.
Can AI test its own code?
It can run tests, but testing its own work tilts the verdict. Even with objective, machine-checked rubrics, judges were more likely to mark their own failures as passing. Let the AI build; move the testing out.
So should I stop using Cursor or Claude Code?
No — keep using them. Specnote connects to those tools and takes only the testing seat: you ask in plain language, and a real browser returns the verdict.
What about test code the AI writes for me?
Good option if you can read it. If you can't, you can't tell what it checks, how far "passed" goes, or whether a failure means the app or the test is broken. A green light you can't read only gives comfort.
What does "a real browser" mean here?
Not analyzing code and inferring an outcome — an actual Chromium browser opens your app, clicks, types, and checks what appears. It walks the same path your users walk.
Does it review code quality too?
No. It does one thing: confirm that the flow a user actually walks actually works. Code quality and security scanning belong to other tools.
Do I need to read the first post to follow this one?
This one stands alone. If you want the background on why checking is hard for non-coders, start with You Built an App with Vibe Coding. Does It Actually Work?.
