UI Test Runner Server

Run one manual test through sync-then-UI in the cloud and return pass or fail with video

Overview

@zooly/ui-test-runner-srv runs one adaptive UI test end to end in the cloud. It first checks the test against the latest code, then runs it on a fresh box. A run started from the test console records its result in the app database, so the console can list runs and open a report. A run started with the plain trigger is recorded only in Upstash. The nightly QA batch builds on the same package: one call per test, one result per test.

This package is server-only. It creates Upstash Boxes and talks to QStash, so it must not be imported from client code. The thin routes live in zooly-app.

Package Details

Package Name: @zooly/ui-test-runner-srv
Location: packages/ui-test-runner/srv
Type: Server-side test-run orchestration

What you pass, what comes back

You pass one test path, such as tests/manual/onboarding-no-chat-email.md. The test must already be on dev, otherwise the run stops at once.

Pass runItemId as well when the run belongs to the test console: that is the report row the workflow writes its stages back to.

Pass skipVideoOnPass to make the walkthrough video a failure artifact: the HTML report is still produced, but only a failing test records and uploads a video — a passing test ends with no video, which keeps large regression runs fast and cheap. Two kinds of passing test are the exception and still record a video: a test the sync stage changed, so the walkthrough of the file that just moved is kept, and a test whose run changed files, because that run opens a pull request and the fix has to be watched. Without the flag every run records a video, which is what the laptop box flow during development does.

You get back one result for the whole run:

Status — pass, fail, or skipped. Skipped means the sync step opened a pull request, so the UI test did not run.
Script updated — yes when the sync step changed the test file, either by pushing to dev or by opening a pull request.
Test passed — yes or no from the UI run, or not-run when the UI test never started.
Summaries — one sentence per stage, plus the reason the run ended.
Video — public walkthrough link, when the UI test produced one.
Failure log — public OpenCode log bundle, uploaded when a stage failed so the failure can still be read after the box is gone.
Box IDs and cleanup state — both boxes are deleted when their stage ends.

How a run flows

Confirm the test exists on dev.
Sync stage: fresh box, compare the test with the code, push the test when it is stale, fall back to a pull request when the push fails twice.
UI stage (only when the test on dev is the updated one): fresh box with apps and database, run the test, record and upload the walkthrough video. With skipVideoOnPass, a passing test skips the video and a failing test still records one. A test the sync stage pushed still records a video even when it passes, and so does a passing run that changed files and opened a pull request.
Delete each box when its stage ends. A stage that failed uploads its OpenCode log first, so the evidence of the failure outlives the box.

Each box has up to 10 minutes to boot. Each agent has up to four hours. The workflow waits between checks, so idle time holds no function open.

When runItemId is set, the workflow also records the run in the database: queued → running when the first box starts, the sync fields when the sync stage ends, and the final result, video link, failure log, and end time when the run finishes. A database problem never fails the run itself — it only costs a report field. When a run fails after QStash has used up its retries, the workflow's failure handler marks the row failed with the error, so a crash between stages does not leave it running.

Public API

import { startTestRun } from "@zooly/ui-test-runner-srv";

const workflowRunId = await startTestRun(
  { test: "tests/manual/onboarding-no-chat-email.md" },
  process.env.NEXT_PUBLIC_APP_URL as string,
);

// Console run: records the result against a report row.
// skipVideoOnPass makes the video a failure artifact (see above).
const result = await startConsoleRun(
  { suiteId: "...", skipVideoOnPass: true }, // or { test: "tests/manual/....md" }
  process.env.NEXT_PUBLIC_APP_URL as string,
);

Export	Purpose
`startTestRun({ test, runItemId?, skipVideoOnPass? }, baseUrl)`	Queue one cloud run, returns the workflow run ID
`startConsoleRun({ suiteId \| test, skipVideoOnPass? }, baseUrl)`	Create the run rows and queue one workflow per test, in parallel
`runTestWorkflow`	The workflow QStash calls back; wired to the zooly-app route
`recordTestWorkflowFailure`	Failure handler for that route; marks the console row failed
`validateTestInput`	Shared test-path check used by trigger and workflow

Environment

Variable	Required	Purpose
`UPSTASH_BOX_API_KEY`	yes	Create and delete boxes
`GITHUB_TOKEN`	yes	Box checkout, dev existence check, sync push
`AI_GATEWAY_API_KEY`	yes	Coding agent model on the box
`QSTASH_TOKEN`	yes	Queue the workflow run
`QSTASH_CURRENT_SIGNING_KEY`	yes	Verify QStash callbacks
`QSTASH_NEXT_SIGNING_KEY`	yes	Verify QStash callbacks during rotation
`NEXT_PUBLIC_APP_URL`	yes	Base URL the trigger calls back (dev deployment)
`CRON_SECRET`	for the curl route	Bearer auth on `/api/ui-test-runner/runs` (the console routes use the admin session instead)
`DATABASE_URL`	for the console	The report rows the workflow writes when `runItemId` is set

Only key names belong in notes and tickets. Never paste values.

Out of scope

No schedule yet. A suite runs when a person clicks Run; the nightly batch comes later.
No Jira post from the cloud run. The older laptop script still posts it.
No per-test lock. A second run of the same test is not refused yet.
Box concurrency limits on the account are unverified.
A workflow that never calls back at all (for example a lost QStash message) leaves its row "running" — there is no sweeper yet.

MCP Client Auth Overview

On This Page

Overview Package Details What you pass, what comes back How a run flows Public API Environment Out of scope