Run QA in CI in 5 minutes

The Lookover QA GitHub Action runs the test cases in your .webagent/tests.yaml in a cloud browser against a URL, usually a preview deploy. The job passes or fails on the result. One comment on the pull request shows what broke and is updated on every push. The job summary has the full report.

1. Set upCopy Anchor Link

1. Create an API key on the API keys page of the dashboard. The member role is enough. Choose the CI scope: a CI key can run tests and read results, and nothing else, so a leaked key cannot touch your team, billing or settings. Add an expiry if you want the key to stop on its own. See API keys. Save it as the repository secret WEBAGENT_API_KEY (Settings, Secrets and variables, Actions). The key never goes in a file.
2. Commit a .webagent/tests.yaml. In Claude Code, /webagent:qa-init writes one from your code. Put a key: in it, for example key: checkout-flow. The key names the workflow in the dashboard and the pull request comment. See the tests file.
3. Add the workflow file below.

2. The workflowCopy Anchor Link

name: Lookover QA
on:
  pull_request:
  push:
    branches: [main]

concurrency:                       # a new push cancels the old run (the action cancels the cloud run too)
  group: lookover-qa-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write             # for the PR comment (without it the check still works, just no comment)

jobs:
  qa:
    name: Lookover QA              # the status check you can mark "required" later
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: lookover-ai/webagent-plugins/github-action@v1
        with:
          api-key: ${{ secrets.WEBAGENT_API_KEY }}
          url: https://staging.example.com   # the site to test

The URL must be public https. The cloud browser never opens localhost or a private address. See Test a local app.

sync turns on by itself for a push to the default branch: the file is saved as the workflow (by key), so the dashboard, its history and its schedule use the main definition. A pull request run never changes the saved workflow. It runs the file from the pull request ad hoc and groups under the same workflow in the dashboard. An edited file in a pull request cannot redefine what main tests. Set sync: true or sync: false to override.

A workflow synced from a repository shows as "Managed by owner/repo" in the dashboard. Edit .webagent/tests.yaml in the repository, or duplicate the workflow to edit it in the dashboard.

The tests fileCopy Anchor Link

.webagent/tests.yaml lists the cases to run. Each case has a title, plain-language steps and what must be true afterwards.

version: 1
name: Acme checkout
key: checkout-flow
url: https://staging.example.com
retries: 1
cases:
  - id: login-ok
    title: A user can sign in
    path: /login
    steps:
      - type "qa@example.com" into Email
      - type "<TEST_PASSWORD>" into Password
      - click "Sign in"
    checks: { text: Welcome, url: /dashboard }
    tags: [smoke, auth]
    priority: 1
  - id: login-wrong-password
    title: A wrong password is refused
    path: /login
    steps:
      - type "qa@example.com" into Email
      - type "nope" into Password
      - click "Sign in"
    expect: shows "Wrong email or password."
    required: false

Rules that save you a failed run:

• Quote every value a step types: type "qa@example.com" into Email. A step that describes the value, such as type a valid email into Email, is typed as written.
• Quote a whole YAML value that contains a colon and a space: expect: 'shows "Total: $19.00"'. Unquoted, the file does not load.
• Never put a password, API key or token in the file. The text of steps is stored with the run, so use a throwaway test account.
• A case marked signedIn: true is for the plugin only. A workflow run in CI skips it and says so in a warning.

3. Preview URLsCopy Anchor Link

The url input wins over the target in tests.yaml. The Action waits up to 120 seconds for the URL to answer with 2xx or 3xx before it starts (wait-for-url), so a preview that is still building does not become a failed test.

A host that sends a deployment_status event, such as Vercel:

name: Lookover QA
on: deployment_status

permissions:
  contents: read
  pull-requests: write

jobs:
  qa:
    name: Lookover QA
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: lookover-ai/webagent-plugins/github-action@v1
        with:
          api-key: ${{ secrets.WEBAGENT_API_KEY }}
          url: ${{ github.event.deployment_status.environment_url || github.event.deployment_status.target_url }}

The comment goes to the pull request that owns the deployed commit. If the preview is behind a login page such as Vercel Deployment Protection, the cloud browser sees the login page, not your app. Use a staging URL, or turn protection off for the previews you test.

A host with a fixed preview address, such as Netlify: pull_request is enough.

      - uses: lookover-ai/webagent-plugins/github-action@v1
        with:
          api-key: ${{ secrets.WEBAGENT_API_KEY }}
          url: https://deploy-preview-${{ github.event.number }}--YOUR-SITE.netlify.app
          wait-for-url: 180   # the most the action waits

If the build takes longer than 180 seconds, start the job after the deploy.

Deploy gate: order the jobs, no special integration needed: deploy-preview, then qa (this Action), then promote with needs: [qa]. A red check stops promote.

4. What the job doesCopy Anchor Link

Exit codesCopy Anchor Link

Keep 1 and 2 apart. A 1 is a finding about your app. A 2 means the test could not finish, so nobody should hunt a bug that is not there. A busy platform at the start is retried twice before it counts as a 2.

A failed case that passes on a retry is flaky. It passes the job and is flagged. The retries input (0 to 2) sets the extra tries. Left empty, the API default applies, which is 1 for CI runs.

Make it a required checkCopy Anchor Link

GitHub only blocks a merge on checks you mark required (Settings, Branches). The status check is the job, so name it Lookover QA. To try it first, add fail-on: never. The job then reports but never fails because of a result. Run it that way for a week as a non-required check, then remove the line and require it. Setup problems still fail the job.

Replay links in the comment appear only when share-links allows it. Anyone with a link can watch the replay, so the default is off on public repositories and on for private ones. The dashboard link works for every member of your project.

5. Inputs and outputsCopy Anchor Link

Use an output in a later step. Give the Action step id: qa, then write if: steps.qa.outputs.verdict == 'fail'.

Fork pull requestsCopy Anchor Link

GitHub gives no secrets to a workflow started by a fork or by Dependabot, so there is no API key. The Action prints one notice, leaves the comment alone and exits with 0: nothing was tested and nothing failed. It runs when a maintainer pushes the branch to your repository. Do not switch to pull_request_target to get the secret. That runs untrusted code with your key.

Other CI systemsCopy Anchor Link

The Action carries the webagent command line tool and runs webagent test. The CLI is not published to npm yet, so the Action is the supported way today. For another CI system, the Smoke tests in CI page shows the API calls. The options of webagent test, for reference:

It reads the API key from the environment variable WEBAGENT_API_KEY. The exit codes are the ones above.

Next stepsCopy Anchor Link

• Write your first cases: QA testing and Recipes.
• Read a failed report: Reading results.
• Get a call when a run ends: the workflow_run.completed and workflow_run.failed webhook events.
• Limit what a pipeline key can do, and let it expire: API keys.
• Billing is by browser compute, and a retry runs the case again. See Billing.