Automated Regression Testing: When an AI Fix Breaks Another Feature

You asked AI to fix one thing, and a different feature stopped working. You did not prompt it wrong. The part you changed is fine; a screen that looked unrelated is the one that broke. The problem is not how well you can fix things — it is that what you checked once is never checked again on the next deploy. Re-running those earlier checks to confirm that what used to work still works is called regression testing. This post is about why that gap opens, and how automated regression testing lets you write a check once and have every deploy repeat it.

The part you fixed is fine. A different screen stopped.

You have probably lived this.

"The checkout button doesn't do anything."

You hand the fix to your AI tool and it lands quickly. You check checkout yourself. It works. You ship.

A few days later a different message arrives — "I can't sign up." You changed checkout, and signup stopped working.

This is not rare. Changing one screen does not only change that screen; anything sharing the same code moves with it. You check what you changed. What breaks is usually what you didn't.

Working code quietly stopping later is common enough that the industry gave it a name, and large teams run a separate pass just for it. You don't need the vocabulary. What matters is why it keeps slipping past you.

The previous post asked who does the testing. This one asks when you check again. If you haven't read Web App Testing Without Code yet, that is a reasonable place to start.

Keep readingAI Testing Tools: Why AI Can't Test the Code It Just WroteAI testing toolsno-code testing6 min

Why AI fixes keep breaking other features

The bigger the app, the more places there are to check. Ten screens means ten places; thirty means thirty. Meanwhile you deploy more often, because AI made changes cheap.

That is where the arithmetic fails. You would need to click through places to check × number of deploys, and the hours available never grow.

Deploys causing damage is not an edge case either. Google's DORA research tracks "what share of deploys caused an incident, rollback, or hotfix" as a headline metric — the assumption baked in is that something can go wrong every time you ship.

So here is what actually happens.

• At first you click through everything
• Then only what you changed and what sits near it
• Then only what you changed
• Eventually you ship without checking

The more you changed, the less you end up checking. That is not laziness. It is arithmetic.

One fix, and the impact reaches screens you never looked at

Automated regression testing, running on every deploy

Turn the problem around and the answer shows up. If the flaw is that a person has to remember each time, then write the checks down outside the person.

Once they are written, three things change.

You stop having to remember. "What did I check last time?" is no longer a question you answer from memory. It is already written.

Every deploy repeats the same checks. Not only what you changed this time — the flows you confirmed earlier run again too. People get tired. A written check doesn't.

You get a record of what was looked at. Passed or failed, which step it stopped at, kept where you can read it later.

What test scenarios look like in practice

Specnote writes those checks down for you and repeats them. You never read code.

1. Connect it to the AI tool you already use. With Claude Code it is two lines pasted in and one approval. No tokens, no config files.

2. Say what you want checked. "Make me a scenario that checks signup works" — the way you already talk — and the steps come back as ordinary English. You can hand over a spec document instead.

3. You approve the pass conditions. The AI does not get to decide what counts as passing. You read the generated steps and approve them before they take effect. That is where this breaks the loop from the last post: when the builder also renders the verdict, the verdict tilts.

4. It runs again on every deploy. An approved scenario stays put, so the next deploy sends a real Chromium browser through the same flow, clicking like a person. You get pass or fail, plus a video of the screen.

Write it down once, and every deploy walks the same path again

Failures are where the value shows up

A check earns its keep when something fails. A check that only ever says "passed" is just reassurance.

When it fails you get a report: which step it stopped at, and what was on the screen. Your AI tool reads that report directly. It fixes the code, and the check runs again.

The loop looks like this.

Specnote does not modify your code. Your AI tool keeps doing that. Specnote only handles pointing at exactly where the problem is.

A failure isn't the end — it leads to the next move

Some places break more often than others

You cannot know in advance how far a single fix will reach. What is clear is that code shared across many screens carries the widest blast radius.

Usually these:

• Anything checking whether a user is signed in — nearly every screen leans on it. Break it and login works while one specific page refuses to open.
• Anything wired to payments — the checkout screen is not the only reader. Order history, receipts, and notification emails read the same values.
• Routing rules — change how addresses work and navigation stops somewhere you never touched.
• Shared interface pieces — buttons, inputs, dialogs reused all over.

If your change lands on that list, it is worth not stopping at the screen you edited.

A history makes causes easy to find

Repeating the checks gives you something extra: you can see when the problem started.

Because each run leaves a pass or fail, a report from a user turns into "fine through run 3, failing from run 4." Check what changed in between and the search narrows to a handful of things.

Without that record the order reverses. You start from the moment a user told you and work backwards, guessing when it first broke. Guess wrong and you keep inspecting the wrong place.

Failed runs keep a video too. You can watch the screen the user actually got, which cuts short the "it works on my machine" exchange.

What should you write down first

If you built with vibe coding, decide how far your pre-deploy check should reach. Each item you write down is a test case, and a connected series of them is a test scenario. You don't need all of them. Pick by this standard.

• Flows where a stop costs revenue — checkout, starting a subscription
• Flows where a stop locks users out — signup, login
• Flows that already broke once — the same spot tends to come back

Three is enough. Start there and add more when you need to.

Try it now