Connect CI for automatic verification

What this is

Think of it as running an automatic rehearsal once before you ship. Until now, you ran verification by pressing a button yourself. Set up CI integration, and every time your code changes (for example, when you push new code or open a review request), Specnote verifies your tests on its own.

CI here means the thing that automatically runs a set of checks every time you push code. If you use GitHub, GitHub Actions plays that role. Specnote slips "test verification" into one of those automatic checks.

One thing you can rest easy about: Specnote's server is what actually runs the verification. The place where your code runs (CI) doesn't open a browser or do any heavy lifting. CI just pulls the trigger — "verify now" — waits for the result, and picks it up. So your side stays light, and there's no chance of production passwords or keys being handed over.

When verification finishes, the results land in two places.

• The review request (PR) screen — a pass/fail summary is attached in plain language, so you can see what broke before merging.
• The Run tests tab — this verification is stacked as one run group, so you can look back on it later.

What Specnote's server opens must be a publicly reachable site address. A localhost address that only runs on your own PC can't be reached by the server, so it can't be verified. If you don't have a deploy or preview address, see "Does it work if the app only runs locally?" below.

Choosing which tests go into CI

You don't have to run every test in CI. Just pick the ones you feel "this must pass before every deploy." Flip the Include in CI switch on a test and it becomes a CI verification target.

This on/off takes effect right away. From the next verification, newly switched-on tests are included automatically and switched-off ones drop out — there's nothing extra to save or reconnect. When CI runs on a push, it verifies exactly the tests that are switched on at that moment.

This toggle is something only a person can do. The connected AI can create and run tests, but it never decides what goes into CI verification. The principle: a bar that can block a deploy is set by a human.

Tests you've switched on gather on the CI tab of the workspace board. There you can see at a glance how each one has trended.

• Last result — whether the most recent round passed or failed.
• Last 5 verdicts — the last five, pass or fail, side by side.
• Pass rate — what percentage has passed so far.
• Flaky badge — attached to a test that flip-flops between pass and fail (unstable). A test with this badge is more often "the test itself is shaky" than "a real bug," so it's worth smoothing out first.

Let AI do it (recommended)

This is the easiest way. Copy the text below and paste it into the AI coding tool you usually use (Claude Code, Cursor, and the like), and AI handles what's needed to connect — issuing the token, creating the workflow file, and committing, all of it.

Connect Specnote's automatic verification to this repository's GitHub Actions. 1) Use the specnote_get_ci_workflow tool to get the workflow file contents and save them to .github/workflows/specnote-verify.yml (specId is ""; use standard mode if there's a deploy/staging address, or local mode if there's only a local dev server). 2) Use the specnote_issue_ci_token tool to issue a CI-only token. 3) Register the issued token as the GitHub repository secret SPECNOTE_TOKEN — if the gh CLI is available, use gh secret set SPECNOTE_TOKEN; if not, guide me to the repository → Settings → Secrets and variables → Actions path. 4) If a verification target address (staging/preview) is needed, ask me, then register it with gh variable set SPECNOTE_TARGET_URL. 5) Commit and push the workflow file, and tell me what you connected.

Replace <my workspace ID> with this workspace's ID. If AI asks, just tell it. (If you copy from the "CI integration" card on the Run tests tab, the workspace ID is already filled in.)

Connect it by hand

If you'd rather connect it yourself, press the [Connect it yourself] button on the "CI integration" card in the Test runs tab and finish in three steps right in the browser — issue a token, copy or download the pre-filled workflow file, and confirm the connection, all in one place. Even the commands come with your values already filled in, so you can copy them as-is.

If you'd rather do it by hand without the button, here are the steps.

1. Issue a token — issue a CI-only token from My Page → MCP connection. This token is the key that starts verification.
2. Create the workflow file — create a .github/workflows/specnote-verify.yml file in your repository. Its contents come straight from the specnote_get_ci_workflow tool in the AI prompt above, already filled with your workspace's values.
3. Register the secret — in GitHub repository → Settings → Secrets and variables → Actions, register the token you just issued as a secret named SPECNOTE_TOKEN.
4. Register the target address — on the Variables tab of the same screen, put the address of the site to verify (a staging or preview address) into a variable named SPECNOTE_TARGET_URL.

Once these four are done, the workflow runs on its own every time you push code and starts verification.

It doesn't have to be GitHub Actions — connect directly via API

GitLab, Jenkins, CircleCI, Bitbucket — even a short shell script — any CI can plug in. No special plugin is needed; you just need three things.

1. A CI token — issue one from My Page, the "Connect it yourself" wizard, or your connected AI (MCP), and store it as a CI secret. Send it as an Authorization: Bearer <token> header.
2. Start verification — POST /api/ci/runs with your workspace ID (specId) and the public address to verify (targetUrl). Verification starts right away and you get back a run-group ID.
3. Wait for the result — poll GET /api/ci/runs/{run-group ID} every 10 seconds. When status is COMPLETED, ERROR, or CANCELED, it's done. The response's summaryMarkdown is a human-readable summary, and if failCount is greater than 0, treat it as a failure.

If you use GitLab, there's a dedicated template (.gitlab-ci.yml). Pick GitLab CI in the "Connect it yourself" wizard, or ask your connected AI for specnote_get_ci_workflow with provider set to gitlab, and you'll get the file pre-filled.

A script you can drop into any CI

Here's a minimal example with no GitHub/GitLab-specific values, so it runs anywhere (all you need is curl and jq). Put your workspace ID in <workspace ID>, and provide SPECNOTE_TOKEN and TARGET_URL as CI secret/variable.

#!/usr/bin/env bash
set -euo pipefail

API="https://specnote.io"
SPEC_ID="<workspace ID>"
TARGET_URL="${TARGET_URL:-<public address to verify>}"
: "${SPECNOTE_TOKEN:?SPECNOTE_TOKEN is required}"

# 1) Start verification (trigger)
BODY=$(jq -nc --arg specId "$SPEC_ID" --arg targetUrl "$TARGET_URL" \
  '{specId: $specId, targetUrl: $targetUrl, meta: {provider: "generic"}}')
CREATE=$(curl -sS --fail-with-body -X POST "$API/api/ci/runs" \
  -H "Authorization: Bearer $SPECNOTE_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$BODY")
RUN_ID=$(echo "$CREATE" | jq -r ".runGroupId // empty")
if [ -z "$RUN_ID" ]; then
  echo "Could not start verification:"; echo "$CREATE"
  exit 1
fi

# 2) Poll until done (every 10s)
for i in $(seq 1 180); do
  RES=$(curl -sS --fail-with-body "$API/api/ci/runs/$RUN_ID" \
    -H "Authorization: Bearer $SPECNOTE_TOKEN")
  STATUS=$(echo "$RES" | jq -r ".status // empty")
  case "$STATUS" in COMPLETED|ERROR|CANCELED) break ;; esac
  sleep 10
done

# 3) Human-readable summary + exit 1 on failure
echo "$RES" | jq -r ".summaryMarkdown // \"Could not fetch the result.\""
FAIL=$(echo "$RES" | jq -r ".failCount // 0")
[ "$FAIL" -gt 0 ] && exit 1 || exit 0

A few things to keep in mind — these are the same no matter which CI you use.

• A public address is required. It must be reachable by the Specnote server. A localhost address that only runs on your PC won't work; in that case, open a temporary tunnel to make a public address and pass it as TARGET_URL.
• Start in report mode. The example above exits with code 1 on failure. Until you're comfortable, remove that line so a failure doesn't block your pipeline.
• Only switched-on tests run. Only scenarios you've turned on with [Include in CI] are verified. This rule is the same for GitHub, GitLab, or a direct API call.

Report mode ↔ Blocking mode

When you connect it, it starts in report mode. Even if verification fails, it doesn't block your code from merging — it just tells you the result. The single line continue-on-error: true in the workflow file does this.

Once you're used to it and your pass rate is stable, you can delete that line to switch to blocking mode. In blocking mode, a failed verification blocks the merge. Leaving it in blocking mode from the start can tie you up over a momentary false alarm, so we recommend switching to blocking mode only after your pass rate is stable.

Frequently asked questions

Does it work if the app only runs locally? Yes. You don't need a deployed address. In that case, CI briefly starts the app and opens a temporary passage (a tunnel) so it can be reached like a public address for that time. Tell the AI prompt to use "local mode" and it'll build it that way. That said, the temporary passage isn't guaranteed for speed or lifespan, so if you have a deploy or preview address, that one is more stable.

Does it cost anything? Verification through CI costs credits, the same as a normal verification. Each replay accrues a tiny amount of credit in proportion to how long the run took. Running several tests at once accrues that much more, and if you're short on credits that round won't run (just top up and run it again). It doesn't call the AI anew, so it isn't as pricey as auto-extraction — but it isn't "completely free" either, so keep that in mind.

What happens if I re-issue the token? The CI token has a single fixed name, so re-issuing it automatically retires the previous token and keeps only the new one alive. Feel free to re-issue whenever you've lost the token or want to rotate it. (After re-issuing, update the GitHub secret SPECNOTE_TOKEN to the new value too.)

Verification isn't running. The most common cause is the target address. Check that the site address you put in the SPECNOTE_TARGET_URL variable is a publicly reachable one right now. If the address has changed, just update that value to the new address.

For the basic way of running verification by pressing a button, see Running verification and reading results; if you haven't connected your code yet, see Connect your code first.