Cloud Test Runs

Overview

A cloud test run takes one manual test prompt and does two things in order:

1. It checks the test against the latest code on dev and updates the test if a step is out of date.
2. It runs the updated test on a fresh box and records a walkthrough video.

The run happens in the cloud. A laptop does not need to stay on, and the run does not depend on one person's machine.

A box is a remote Linux machine. It holds the Zooly apps, a local Postgres, headless Chromium, ffmpeg, and a coding agent. The agent does the check and runs the test on the box.

Start a run

A run needs two things:

• The name of one manual test prompt, which must already be on dev. If it is not on dev, the run stops at once.
• The team's shared secret for the test runner.

The call returns a run ID. Use that ID to find the run in the Upstash console.

What a run does

Stage 1: check the test

1. A fresh box starts and fetches the latest dev.
2. The agent compares each claim in the test with the code: screens, labels, messages, and the order of steps.
3. If a claim is out of date, the agent changes only the test and pushes the change to dev. If dev moved in the meantime, it tries once more.
4. If the push still fails, the agent opens a pull request with the change instead. The UI test then does not run, because the test on dev is not the updated one.
5. If nothing is out of date, nothing is pushed.

Stage 2: run the test

1. A fresh box starts the Zooly apps and their database, applies pending migrations, and waits until both apps respond. If a migration fails, the run stops.
2. The agent runs the test in the browser, following the adaptive test prompt: fresh browsers, fresh accounts, and fresh inboxes.
3. The agent records the walkthrough video and uploads it.

Time limits

• Each box has up to 10 minutes to start.
• Each agent has up to four hours to finish.

Cleanup

Each box is deleted when its stage ends, whether it passed or failed. Boxes are never paused or resumed, so every run starts from a clean box.

What the result shows

The walkthrough video is the only artifact that survives the run. The HTML report the agent builds on the box is deleted with the box.

Known gaps

• A run started from the command line exists only in the Upstash console. Runs started in the test console are stored in the app database with their report.
• A cloud run does not post its result to the Jira issue yet. The older laptop-driven run still does.
• If a run stops before its box is deleted, the box keeps running. Delete it by its ID, and check the account for leftover boxes.
• How many boxes can run at the same time has not been checked yet.
• Resuming a paused box has not been checked. Runs never resume a box.

Older laptop run

A laptop can still start the older run from the command line. That run stays on the laptop until it ends, creates the box itself, and posts the Jira comment. If the laptop sleeps or the command stops, the box must be deleted by hand.

Environment and access

• Only key names belong in notes, tickets, and chat. Never paste values.
• Upstash makes every box environment value visible to all code on the box, including the coding agent. Run only trusted test prompts.
• The box uses the company GitHub account. Its only database is the box's own Postgres. It never talks to the laptop database, staging, or production.
• Use a fresh browser and a fresh MailSlurp inbox for each run. Leave inboxes active until someone decides they can be deleted.

Reference

• Test Console
• Adaptive AI UI Testing
• Helper endpoints