Image Generation Benchmark

What is the Benchmark?

The Image Generation Benchmark is an admin-only tool inside Merch Admin for running controlled experiments on merch AI art generation. Each benchmark session sweeps a cartesian product of:

selfies × design text × prompts × models × quality × input fidelity

Design text is optional. A run with no names stays selfies × prompts × models × quality × input fidelity, the same as before.

Every generation runs through the same production pipeline as live merch:

1. AI generation — GPT Image 2 gateway model call
2. Crop, mask, overlay — center-crop, optional mask cut-out, then the template overlay

Background removal is deliberately never applied in the benchmark — sessions compare raw model output, so a remover would obscure what the prompt and model actually produced.

Results are stored in PostgreSQL and uploaded to S3 so admins can compare outputs side-by-side, like/dislike individual cells, and rerun past sessions.

Where to Access

The Benchmark tab is admin-only. Supplier users are routed to /orders and cannot access it.

Routes:

---

Architecture

apps/zooly-app/app/(pages)/admin/merch/ (Host — mounts the admin UI)
  [[...path]]/page.tsx   Lazy-imports @zooly/merch-admin-client App (basename /admin/merch)
  Benchmark tab mounts pages from @zooly/merch-benchmark-client
  BenchmarkPathsProvider base="/benchmark"

packages/merch-benchmark/client/ (@zooly/merch-benchmark-client)
  dashboard-page.tsx       Setup form + sticky Run Benchmark bar
  results-page.tsx         Session list
  session-detail-page.tsx  Comparison grid, config, rerun
  components/              Image upload, selfie groups modal, results grid

packages/merch-benchmark/srv/ (@zooly/merch-benchmark-srv)
  create-session.ts        Cartesian product → DB rows
  run-generation.ts        Full pipeline: GPT Image → mask → overlay → S3
  selfies.ts               Western / Japan selfie corpus helpers

packages/merch/img-gen/ (@zooly/merch-img-gen)
  generateAIArt, removeBackgroundWithBria, downloadAndProcessAIImage, applyTemplateMask
  Shared with production /api/merch/ai/generate and render steps

apps/zooly-app/app/api/merch/admin/benchmark/
  sessions/                GET list, POST create
  sessions/[id]/           GET detail, PATCH rename
  generate/                POST run one generation
  generations/[id]/like/   PATCH like/dislike
  selfies/                 GET selfie corpus

packages/db/
  schema/merchBenchmarkTables.ts
  access/merch/merch-benchmark.ts

Auth

Same pattern as the rest of merch admin:

• Client calls GET /api/merch/admin/* with credentials: "include"
• Server routes use requireAdmin() from @zooly/merch-admin-srv
• Unauthenticated users redirect to zooly-auth SSO (VITE_AUTH_URL)

Client-side generation loop

After creating a session, the dashboard fires a client-side job queue (run-loop.ts) that POSTs to /api/merch/admin/benchmark/generate with bounded concurrency. The session detail page polls every 3 seconds while status is running.

---

Setup Page

Session name

Auto-generated timestamp name; editable. Required to run.

Selfies

Two corpora:

Selection modes:

Selfie groups are shared with the Simulation tab. Use Manage groups to create, edit, or delete groups from the full pool (Western + Japan + uploaded selfies). The group editor also supports uploading new selfies — they are stored on S3 and become available to every group.

Prompts

One or more named prompt variants. At least one prompt with non-empty text is required.

Design text

Some designs ask the shopper for a name or other words and place that writing in the picture. A benchmark can test several of those words against the same selfies.

Add one or more lines. Each line has a label (what you see on the results) and the words that go into the picture. The run multiplies: every selfie is generated once for each line.

Leave the section empty when the design does not collect words. Older runs without this section keep working.

The prompt must include the same placeholder the shop uses, written as <USER-INPUT>. A named placeholder such as <USER-INPUT:company> receives the same words. The model never sees the placeholder itself.

You can turn on the same text check the shop uses. Each result then shows whether those words would have been allowed. The picture is still generated, and the check is recorded the same way Issue Catcher records pass or fail.

Issue Catcher checks can name the words for that result. For example: "the company name ACME appears above ACQUIRED in the same font." To write one check that follows every line, put <USER-INPUT> in the check where the name should be.

Template, overlay, and mask

Drag-and-drop image upload fields (or paste a URL). These map to production design assets:

• Template — AI template image passed to the model
• Overlay — composited on cropped art after generation
• Mask — optional PNG whose transparent areas are cut away from the art during generation
• Model pipeline — GPT Image 1.5, GPT Image 2, or GPT Image 2.5 (Sunburst / Flare), always opaque (no background removal)

Models and hyperparameters

Models come from GET /api/merch/admin/models (same catalog as production). Only active models appear.

Run summary bar

A sticky bar at the bottom shows total generations, estimated cost, and a Run Benchmark button. The button is disabled with a hint when required fields are missing.

---

Results

Run history (/benchmark/results)

Lists past sessions with name, date, status, and models used. New Benchmark returns to the setup page.

Session detail (/benchmark/results/:id)

Header

• Click session name to rename
• Status badge: running, completed, or failed
• Rerun with these settings — opens setup pre-filled from this session (?from=sessionId)

Stats

Progress, failed count, cost, average duration. Progress bar while running.

Configuration

Expandable section showing prompts (read-only textarea, max ~500px height), template/overlay/mask thumbnails, and run parameters.

Results grid

• Rows = selfies (and, when the run tests design text, one row per selfie and name); columns = model / prompt / quality / fidelity (selectable via column field dropdown)
• Each result that used design text shows that name. Open Prompt on the result to read the words already written in
• Column filters — checkboxes to show/hide specific column values
• Click an image for fullscreen view
• Thumbs up / down on each completed cell (persisted via API)

---

Database Schema

Defined in packages/db/src/schema/merchBenchmarkTables.ts. Migration: 0128_merch_benchmark_tables.sql.

merch_benchmark_session

One row per benchmark run.

merch_benchmark_generation

One row per (selfie, design text, prompt, model, quality, fidelity) combination. Design text is omitted when the run has no lines.

Access layer: packages/db/src/access/merch/merch-benchmark.ts

---

API Reference

All routes under apps/zooly-app/app/api/merch/admin/benchmark/. All require admin auth + CORS.

GET /api/merch/admin/benchmark/sessions

List all sessions with aggregated model names.

POST /api/merch/admin/benchmark/sessions

Create a session and bulk-insert pending generation rows.

Body (key fields):

{
  "name": "Benchmark 2026-06-10",
  "prompts": [{ "name": "Default", "text": "..." }],
  "texts": [{ "name": "ACME", "value": "ACME" }],
  "screenDesignText": false,
  "selfies": [{ "index": 1, "url": "https://..." }],
  "models": [{ "endpointId": "gpt-image", "displayName": "GPT" }],
  "templateImageUrl": null,
  "overlayUrl": null,
  "maskImageUrl": null,
  "concurrency": 5,
  "selfieTestSet": "western",
  "qualities": ["low"],
  "inputFidelities": ["high"],
  "selfieFilter": null
}

GET /api/merch/admin/benchmark/sessions/[id]

Session + all generations.

PATCH /api/merch/admin/benchmark/sessions/[id]

Rename session: { "name": "..." }

POST /api/merch/admin/benchmark/generate

Run one generation: { "generationId": "..." }. maxDuration = 300.

PATCH /api/merch/admin/benchmark/generations/[id]/like

Set review: { "liked": true | false | null }

GET /api/merch/admin/benchmark/selfies?set=western|japan

Returns the selfie corpus for the chosen test set.

---

Environment

Merch admin is served by zooly-app at /admin/merch, same-origin with the API. Open it at http://localhost:3004/admin/merch/benchmark.

Run migration 0128_merch_benchmark_tables against your local database before first use.

---

Related Documentation

• Merch Overview — system components
• Architecture — stack and package layout
• Database Schema — all merch tables
• Template Studio (v2) — reusable templates that prefill benchmark via ?templateV2=
• Environment Setup — local dev prerequisites