# Run QA in CI in 5 minutes

Source: https://docs.lookover.ai/webagent/guides/qa-in-ci

> Run your .webagent/tests.yaml on every pull request with the Lookover QA GitHub Action. The job passes or fails on the result and one comment on the pull request shows what broke.

The Lookover QA GitHub Action runs the test cases in your `.webagent/tests.yaml` in a cloud browser against a URL, usually a preview deploy. The job passes or fails on the result. One comment on the pull request shows what broke and is updated on every push. The job summary has the full report.

> **Warning: Beta**
>
> The Action is bundled JavaScript that runs on the `node24` runtime, with no npm install. It is not published yet: the `uses:` line below works once the plugins repository is public and tagged `v1`. Until then, the [Smoke tests in CI](/guides/ci) page shows the same idea with plain API calls, and those work today.

## 1. Set up [#1-set-up]

1. Create an API key on the **API keys** page of the dashboard. The **member** role is enough. Choose the **CI** scope: a CI key can run tests and read results, and nothing else, so a leaked key cannot touch your team, billing or settings. Add an expiry if you want the key to stop on its own. See [API keys](/concepts/api-keys). Save it as the repository secret `WEBAGENT_API_KEY` (Settings, Secrets and variables, Actions). The key never goes in a file.
2. Commit a `.webagent/tests.yaml`. In Claude Code, `/webagent:qa-init` writes one from your code. Put a `key:` in it, for example `key: checkout-flow`. The key names the workflow in the dashboard and the pull request comment. See [the tests file](#the-tests-file).
3. Add the workflow file below.

## 2. The workflow [#2-the-workflow]

```yaml title=".github/workflows/lookover-qa.yml"
name: Lookover QA
on:
  pull_request:
  push:
    branches: [main]

concurrency:                       # a new push cancels the old run (the action cancels the cloud run too)
  group: lookover-qa-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write             # for the PR comment (without it the check still works, just no comment)

jobs:
  qa:
    name: Lookover QA              # the status check you can mark "required" later
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: lookover-ai/webagent-plugins/github-action@v1
        with:
          api-key: ${{ secrets.WEBAGENT_API_KEY }}
          url: https://staging.example.com   # the site to test
```

The URL must be public https. The cloud browser never opens `localhost` or a private address. See [Test a local app](/guides/local-apps).

`sync` turns on by itself for a push to the default branch: the file is saved as the workflow (by `key`), so the dashboard, its history and its schedule use the `main` definition. A pull request run never changes the saved workflow. It runs the file from the pull request ad hoc and groups under the same workflow in the dashboard. An edited file in a pull request cannot redefine what `main` tests. Set `sync: true` or `sync: false` to override.

A workflow synced from a repository shows as "Managed by owner/repo" in the dashboard. Edit `.webagent/tests.yaml` in the repository, or duplicate the workflow to edit it in the dashboard.

## The tests file [#the-tests-file]

`.webagent/tests.yaml` lists the cases to run. Each case has a title, plain-language steps and what must be true afterwards.

```yaml title=".webagent/tests.yaml"
version: 1
name: Acme checkout
key: checkout-flow
url: https://staging.example.com
retries: 1
cases:
  - id: login-ok
    title: A user can sign in
    path: /login
    steps:
      - type "qa@example.com" into Email
      - type "<TEST_PASSWORD>" into Password
      - click "Sign in"
    checks: { text: Welcome, url: /dashboard }
    tags: [smoke, auth]
    priority: 1
  - id: login-wrong-password
    title: A wrong password is refused
    path: /login
    steps:
      - type "qa@example.com" into Email
      - type "nope" into Password
      - click "Sign in"
    expect: shows "Wrong email or password."
    required: false
```

| Field | What it does |
|---|---|
| `key` | Slug (lower-case letters, digits, `-` and `_`). Names the workflow and the pull request comment. |
| `url` or `targets` | The site to test. Use `url` for one site, or `targets` with names and `defaultTarget` for several. Never both. The Action's `url` input wins over both. |
| `engine` | Optional. An engine's short name or id. Without it the saved workflow's engine runs, then your project's default engine. A name that matches no engine in your project stops the job before any run starts (exit code 3). |
| `retries` | Optional, 0 to 2 extra tries for a failed case. |
| `cases` | Required, 1 to 100 cases. |
| `cases[].steps` | What a user does, one action per item. No selectors, no secrets. Without steps the case only loads the page. |
| `cases[].expect` | What a user sees, in words: `shows "X"`, `address contains Y`, `does not show "Z"`. Any other wording is reported as "not checked". |
| `cases[].checks` | The exact form of `expect`: `text`, `url`, `noText`. When set, it wins over `expect`. |
| `cases[].required` | Default `true`. With `false`, a failure is a warning and never fails the job. |
| `cases[].tags`, `priority` | Labels for choosing a subset (`tags` input) and for order. Priority runs from 1 (most important) to 5. |

Rules that save you a failed run:

- **Quote every value a step types**: `type "qa@example.com" into Email`. A step that describes the value, such as `type a valid email into Email`, is typed as written.
- **Quote a whole YAML value that contains a colon and a space**: `expect: 'shows "Total: $19.00"'`. Unquoted, the file does not load.
- Never put a password, API key or token in the file. The text of steps is stored with the run, so use a throwaway test account.
- A case marked `signedIn: true` is for the plugin only. A workflow run in CI skips it and says so in a warning.

## 3. Preview URLs [#3-preview-urls]

The `url` input wins over the target in `tests.yaml`. The Action waits up to 120 seconds for the URL to answer with 2xx or 3xx before it starts (`wait-for-url`), so a preview that is still building does not become a failed test.

**A host that sends a `deployment_status` event, such as Vercel:**

```yaml title=".github/workflows/lookover-qa.yml"
name: Lookover QA
on: deployment_status

permissions:
  contents: read
  pull-requests: write

jobs:
  qa:
    name: Lookover QA
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: lookover-ai/webagent-plugins/github-action@v1
        with:
          api-key: ${{ secrets.WEBAGENT_API_KEY }}
          url: ${{ github.event.deployment_status.environment_url || github.event.deployment_status.target_url }}
```

The comment goes to the pull request that owns the deployed commit. If the preview is behind a login page such as Vercel Deployment Protection, the cloud browser sees the login page, not your app. Use a staging URL, or turn protection off for the previews you test.

**A host with a fixed preview address, such as Netlify:** `pull_request` is enough.

```yaml title="Inside the job"
      - uses: lookover-ai/webagent-plugins/github-action@v1
        with:
          api-key: ${{ secrets.WEBAGENT_API_KEY }}
          url: https://deploy-preview-${{ github.event.number }}--YOUR-SITE.netlify.app
          wait-for-url: 180   # the most the action waits
```

If the build takes longer than 180 seconds, start the job after the deploy.

**Deploy gate:** order the jobs, no special integration needed: `deploy-preview`, then `qa` (this Action), then `promote` with `needs: [qa]`. A red check stops `promote`.

## 4. What the job does [#4-what-the-job-does]

| State | Job | Pull request comment |
|---|---|---|
| Running | In progress | `Lookover QA · RUNNING · testing <url>`, replaced when the run ends |
| Pass (flaky passes included) | Exit code 0 | `PASS`, counts and a case table. A case that passed on a retry reads `PASS (flaky: passed on try 2 of 2)` |
| A required case failed | Exit code **1** | `FAIL`, the failed step, expected against actual, the page, the console error, the failed request and a replay link |
| Could not test (the preview never answered, no capacity, timeout). Not an app failure | Exit code **2** | `COULD NOT TEST`, in plain words why |
| Setup problem (no key, key rejected, revoked or expired, bad file, bad input, `engine:` not found) | Exit code **3** | `NOT RUN · setup problem`, and what to fix, for example "API key rejected: make a new one in API keys" |
| Fork pull request or Dependabot (GitHub gives no secrets) | Exit code 0 with one notice | None |
| No `pull-requests: write` permission | Unchanged | One notice in the log. The job summary still has the report |
| Job cancelled (a new push) | The cloud run is cancelled | Updated to could not test (best effort) |

### Exit codes [#exit-codes]

| Exit code | Meaning |
|---|---|
| 0 | Pass. Also: could not test with `on-error: neutral`, any result with `fail-on: never`, and a fork pull request skip |
| 1 | A required case failed: your app |
| 2 | Could not test: not an app failure |
| 3 | Setup problem: your configuration. `fail-on: never` does not hide these |

Keep 1 and 2 apart. A 1 is a finding about your app. A 2 means the test could not finish, so nobody should hunt a bug that is not there. A busy platform at the start is retried twice before it counts as a 2.

A failed case that passes on a retry is **flaky**. It passes the job and is flagged. The `retries` input (0 to 2) sets the extra tries. Left empty, the API default applies, which is 1 for CI runs.

### Make it a required check [#make-it-a-required-check]

GitHub only blocks a merge on checks you mark **required** (Settings, Branches). The status check is the job, so name it `Lookover QA`. To try it first, add `fail-on: never`. The job then reports but never fails because of a result. Run it that way for a week as a non-required check, then remove the line and require it. Setup problems still fail the job.

Replay links in the comment appear only when `share-links` allows it. Anyone with a link can watch the replay, so the default is off on public repositories and on for private ones. The dashboard link works for every member of your project.

## 5. Inputs and outputs [#5-inputs-and-outputs]

| Input | Default | What it does |
|---|---|---|
| `api-key` | (required) | Your API key, from a secret. Empty on a fork pull request: the Action then skips with a notice |
| `api-url` | The host built into the CLI | API origin, without a path |
| `url` | | The site to test. Wins over the file's target |
| `file` | `.webagent/tests.yaml` | The tests file. If this default is missing and `workflow` is set, the saved workflow of that key runs (needs `url`) |
| `workflow` | `key` in the file | Workflow key. Also names the pull request comment |
| `target` | `defaultTarget` of the file | A target name from the file |
| `tags` | All cases | Comma-separated. Only cases with one of these tags run |
| `case-ids` | All cases | Comma-separated. Only these cases run |
| `sync` | On for a push to the default branch, else off | Save the file as the workflow before the run |
| `wait-for-url` | `120` | Seconds, 1 to 180, to wait for the URL to answer. `0` or `false` skips the wait |
| `timeout-minutes` | `15` | Give up and cancel the run after this long |
| `retries` | The API default (1 for CI runs) | Extra tries for a failed case, 0 to 2 |
| `on-error` | `block` | `block`: could not test fails the job with exit code 2. `neutral`: it passes with a warning |
| `fail-on` | `fail` | `fail`: a failed required case fails the job. `never`: report only |
| `comment` | `true` | Keep one comment on the pull request, updated on every push |
| `share-links` | On for private repositories, off for public | A 7-day replay link per failed case in the comment |
| `github-token` | `${{ github.token }}` | Token for the comment |

| Output | Value |
|---|---|
| `verdict` | `pass`, `fail`, `error` (could not test), `setup` (not run: setup problem) or `skipped` (fork pull request without the secret) |
| `run-id` | The workflow run id. Empty when no run started |
| `run-url` | The dashboard page of the run |
| `flaky` | How many cases passed only on a retry |

Use an output in a later step. Give the Action step `id: qa`, then write `if: steps.qa.outputs.verdict == 'fail'`.

## Fork pull requests [#fork-pull-requests]

GitHub gives no secrets to a workflow started by a fork or by Dependabot, so there is no API key. The Action prints one notice, leaves the comment alone and exits with 0: nothing was tested and nothing failed. It runs when a maintainer pushes the branch to your repository. Do not switch to `pull_request_target` to get the secret. That runs untrusted code with your key.

## Other CI systems [#other-ci-systems]

The Action carries the `webagent` command line tool and runs `webagent test`. The CLI is not published to npm yet, so the Action is the supported way today. For another CI system, the [Smoke tests in CI](/guides/ci) page shows the API calls. The options of `webagent test`, for reference:

| Option | What it does |
|---|---|
| `--file <path>` | The tests file. Default `.webagent/tests.yaml` |
| `--url <url>` | The site to test. Wins over the file's target |
| `--target <name>` | A target of the file |
| `--tags a,b`, `--case-ids a,b` | Run a subset |
| `--workflow <key>` | The workflow key. With no file, runs the saved workflow of that key |
| `--sync` | Save the workflow by key first, then run it. Use on pushes to the default branch only |
| `--retries 0..2` | Extra tries for a failed case |
| `--timeout-minutes 15` | On timeout or Ctrl-C the run is cancelled |
| `--wait-for-url`, `--wait-for-url-timeout <s>` | Poll the URL until it answers 2xx or 3xx (up to 180 seconds, default 120) before starting |
| `--on-error block\|neutral` | Could not test exits with 2 (default) or 0 |
| `--junit <path>`, `--markdown <path>`, `--json` | Write a JUnit file, a Markdown report, or print the run as JSON |

It reads the API key from the environment variable `WEBAGENT_API_KEY`. The exit codes are the ones above.

## Next steps [#next-steps]

- Write your first cases: [QA testing](/qa-testing) and [Recipes](/qa-testing/recipes).
- Read a failed report: [Reading results](/qa-testing/reading-results).
- Get a call when a run ends: the `workflow_run.completed` and `workflow_run.failed` [webhook events](/reference/webhooks#workflow-run-events).
- Limit what a pipeline key can do, and let it expire: [API keys](/concepts/api-keys).
- Billing is by browser compute, and a retry runs the case again. See [Billing](/concepts/billing).