ci(cli): add a full benchmark for integration-heavy commands by Prashansa-K · Pull Request #7412 · supabase/cli

Prashansa-K · 2026-10-30T11:38:14Z

Adds the second tier of the CLI benchmark: the commands that need a running local stack or
database. It stays report only; it never blocks a merge and is not a required check.

/bench full on a pull request (same permission rules as /bench) dispatches the full
workflow on the pull request's branch and re-measures the integration-heavy benchmark. It
refreshes the lightweight benchmark comment only when that comment is behind the head commit. It
replies when a run is already in progress, when the branch has no full benchmark workflow yet, or
when the pull request is a draft, from a fork, or not into develop.
Actions UI: workflow_dispatch on any branch with workloads, runtime, and base
(the merge base, or a release tag). The job summary always has the report; the pull request
comments are updated when the base is the merge base and the branch has exactly one open pull
request into develop.
Weekly schedule (Sunday, off-peak) on develop against the latest stable release, resolved at
run time. The report goes to the job summary and artifacts. Slack notifications come later, once
the data proves valid and useful.
Comments: the integration-heavy benchmark posts its own comment, separate from the existing
lightweight benchmark comment. Each comment shows the commit it measured, keeps its own previous
runs, and ends with a note on which command re-runs it. Writes to each comment are serialized per
pull request.
Workloads: stack lifecycle (start, status, stop, db start, and stack start|status|stop), database work against the local stack (db reset, db query, test db,
gen types at three schema sizes, db diff cold and warm, db dump, db lint), and migrations
(migration up, db push --local, migration list). Migration workloads add their migrations
after the stack starts, so the measured command applies them.
Runtimes: Docker and the native backend on Linux x64. Workloads are packed into shards of
roughly ten minutes; base and branch alternate on one runner, and one workload runs at a time.
Metrics: wall time, phase deltas, Management API counts, container operation counts, and
binary size. Benchmark runs never send product telemetry.

Changed: apps/cli/scripts/bench-cli.ts, bench-report.ts, bench-summary.ts,
.github/workflows/cli-bench.yml, .github/scripts/cli-bench/*, and
apps/cli/docs/tracing-monitoring.md. Added: .github/workflows/cli-bench-full.yml,
.github/scripts/cli-bench/full-plan.ts, apps/cli/scripts/bench-pack.ts, bench-merge.ts,
bench-fixtures.ts, and apps/cli/bench/ (the full benchmark manifest and fixtures).

GitHub runs dispatch, schedule, and comment-triggered workflows only from the default branch, so the
workflow's actual runs (the Actions UI dispatch, /bench full, and the schedule) cannot be tested
before this merges to develop. They get their first real run after the merge.

🤖 Generated with Claude Code

The benchmark comment holds a Lightweight Commands and an Integration Heavy Commands section. Each writer replaces only its own section, verifies it landed, and retries when another writer overwrote it. Previous runs record the result each section replaced. /bench full dispatches the stack benchmark workflow on the pull request's branch and refreshes the lightweight section only when it is behind the head commit. Co-Authored-By: Claude Opus 5.5 <[email protected]>

…ner counts The benchmark harness runs workloads that need a local stack: untimed setup and teardown shared per run or per block of launches, a stack switch, a runtime, fixtures, a shared native artifact cache, per-workload run counts, and tracing limits. Reports add container operation counts, a runtime column, a section name, shard packing, and a report merge. Every launch disables product telemetry. The stack manifest and fixtures are generated and committed. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Runs from the Actions UI, the /bench full command, and a weekly schedule against the latest stable release. Plans shards, builds both binaries once, measures each shard on one runner, and reports to the job summary and the pull request comment. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-actions · 2026-10-30T11:41:51Z

CLI benchmark

Lightweight Commands: no changes found

Commit 7179549 against develop at db5ee12 · 10 runs each · report only, never blocks merging

Command	Before	After	Change
`supabase --version`	284 ms	285 ms	no real change
`supabase --help`	368 ms	466 ms	no real change (+27%, within normal variation)
`supabase init`	425 ms	392 ms	no real change (-8%, within normal variation)
`supabase migration new bench`	381 ms	364 ms	no real change
`supabase gen types --lang typescript --project-id replayfixtureprojref`	387 ms	370 ms	no real change
`supabase projects list`	358 ms	379 ms	no real change (+6%, within normal variation)
`supabase functions list --project-ref replayfixtureprojref`	619 ms	544 ms	no real change (-12%, within normal variation)

API calls: unchanged for all 7 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command	Before range	After range	Change	Verdict
`supabase --version`	235-379 ms	246-448 ms	+1 ms (+0%)	within noise
`supabase --help`	288-503 ms	321-622 ms	+98 ms (+27%)	within noise
`supabase init`	314-730 ms	298-901 ms	-33 ms (-8%)	within noise
`supabase migration new bench`	296-436 ms	285-528 ms	-18 ms (-5%)	within noise
`supabase gen types --lang typescript --project-id replayfixtureprojref`	318-562 ms	342-491 ms	-17 ms (-4%)	within noise
`supabase projects list`	301-483 ms	300-508 ms	+21 ms (+6%)	within noise
`supabase functions list --project-ref replayfixtureprojref`	536-1265 ms	454-882 ms	-75 ms (-12%)	within noise

API calls per command

Command	Before	After
`supabase --version`	0	0
`supabase --help`	0	0
`supabase init`	0	0
`supabase migration new bench`	0	0
`supabase gen types --lang typescript --project-id replayfixtureprojref`	2	2
`supabase projects list`	1	1
`supabase functions list --project-ref replayfixtureprojref`	2	2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench (only includes lightweight benchmarks) or /bench full to run it again on the latest commit.

github-actions

🤖 AI Review

Verified all nine findings against the checked-out code: two major, five minor, and two nits are confirmed, with no duplicates. Controlled reproductions demonstrated lost concurrent updates, broken table rendering, discarded history, and skipped refresh of failed results. All 89 existing benchmark-comment tests pass but miss these cases. Both independent reviews were available; no branch-sync or next-context files were present.

Findings

Severity	Location	Category	Sources	Claim
🟠 MAJOR	`.github/scripts/cli-bench/post-comment.ts:290`	`concurrency`	codex	Concurrent section writers can both report success while a stale write permanently removes the other writer's result and history.
🟠 MAJOR	`apps/cli/scripts/bench-cli.ts:413`	`error-handling`	codex	Failed setup commands do not invalidate benchmark measurements, allowing incomplete workloads to produce apparently valid comparisons.
🟡 MINOR	`.github/scripts/cli-bench/post-comment.ts:68`	`correctness`	claude	Doubling all backslashes during sanitization breaks the escaped pipes produced for benchmark table cells.
🟡 MINOR	`.github/scripts/cli-bench/rerun.ts:211`	`correctness`	claude	/bench full treats a failed Lightweight Commands section at the head SHA as current and does not refresh it.
🟡 MINOR	`.github/scripts/cli-bench/post-comment.ts:168`	`data-loss`	claude	Oversized comment sections can erase all stored run history, and equal section budgets can unnecessarily truncate the larger section.
🟡 MINOR	`apps/cli/bench/workloads.stack.json:205`	`benchmark-correctness`	codex	The migration-up and db-push-local workloads measure migrations that setup has already applied.
🟡 MINOR	`apps/cli/docs/tracing-monitoring.md:269`	`documentation`	codex	The documentation incorrectly states that dispatched workflows execute only from the default branch.
⚪ NIT	`.github/scripts/cli-bench/post-comment.ts:61`	`readability`	claude	The zero-width-space constant and related test expressions use an invisible literal instead of an explicit Unicode escape, obscuring their intent.
⚪ NIT	`.github/scripts/cli-bench/post-comment.test.ts:132`	`test-quality`	codex	The test claiming the benchmark marker appears exactly once does not assert uniqueness and passes with duplicate markers.

Stats

Claude findings: 4 · Codex findings: 5 · Confirmed: 9 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

… outputs Co-Authored-By: Claude Opus 5.5 <[email protected]>

Keep escaped table pipes through sanitizing, rebuild a comment another writer changed before writing it, keep each section's newest run and reallocate unused space when truncating, refresh a failed lightweight section on /bench full, skip launches when a setup step fails, and stage the migration fixture after setup for the migration workloads. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Co-Authored-By: Claude Opus 5.5 <[email protected]>

The lightweight and integration-heavy benchmarks each keep their own pull request comment, identified by its own hidden marker and carrying its own history and re-run note. This removes the shared-comment section markers, cross-section size budgeting and the read-back retry loop. Co-Authored-By: Claude Opus 5.5 <[email protected]>

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Prashansa-K · 2026-10-30T14:37:02Z

Created a temporary PR to show how the command output for /bench full look like:
#7415

Workflow run: https://github.com/supabase/cli/actions/runs/41058498633/attempts/1

Coly010

nice, the dry run on #7415 made this a lot easier to review, thanks for setting that up

a few things, none blocking since it's report-only:

#7415 is basically an A/A run (no cli source changes, binary size unchanged) but the comment still flags 5 of 27 commands, e.g. status +71% and db lint -23%. i think it's the block-scoped workloads: with runs: 10 and BLOCK_SIZE 5 each build only gets two stack sessions, so session-to-session noise lands outside the base min-max. could we alternate in smaller blocks, or only call something slower/faster when it shifts in every block? otherwise folk will end up chasing regressions that aren't there
the lightweight comment concurrency in cli-bench.yml says a waiting writer is never cancelled, but with the default queue: single github cancels the pending one when a newer one queues, so its result never gets written. adding queue: max to both groups should sort it (slim-release-published.yml already uses it, so jactionlint is happy with it)
teardown runs with the build under test and the exit code is dropped (bench-cli.ts:521). if a branch breaks stop, the next unit (usually base) fails start on the port and the comment ends up blaming base. could teardown failures go into the report, or a hard clean by project label after teardown?
small one: rows are labelled by command only, so gen types small/medium/large and db diff cold/warm show up as identical rows. the workload id next to the command would help

happy for any of these to be follow-ups if you'd rather, otherwise looks good to me!

Prashansa-K and others added 6 commits October 30, 2026 15:33

docs(cli): document the stack benchmark

339eb13

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ci(cli): harden benchmark comment sanitizing and dispatch ref

1abdfce

Co-Authored-By: Claude Opus 5.5 <[email protected]>

chore: merge develop

7179549

Prashansa-K requested review from a team and supabase-oss as code owners October 30, 2026 11:38

Prashansa-K mentioned this pull request Oct 30, 2026

test(cli): temporary stack benchmark dry run (do not merge) #7413

Closed

github-actions Bot reviewed Oct 30, 2026

View reviewed changes

Prashansa-K and others added 2 commits October 30, 2026 17:28

ci(cli): rename the stack benchmark tier to full and drop unused plan…

9b05727

… outputs Co-Authored-By: Claude Opus 5.5 <[email protected]>

Prashansa-K changed the title ~~ci(cli): add a stack benchmark for integration-heavy commands~~ ci(cli): add a full benchmark for integration-heavy commands Oct 30, 2026

Prashansa-K and others added 2 commits October 30, 2026 17:43

chore(cli): format benchmark files

691bf23

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Prashansa-K mentioned this pull request Oct 30, 2026

test(cli): temporary full benchmark dry run (do not merge) #7415

Draft

Prashansa-K and others added 3 commits October 30, 2026 19:37

ci(cli): label the lightweight benchmark comment

8c858cd

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ci(cli): count API calls in the full benchmark

c1855b4

Co-Authored-By: Claude Opus 5.5 <[email protected]>

chore: merge develop

16b0c34

Coly010 approved these changes Oct 30, 2026

View reviewed changes

Navigation Menu

Repository navigation

ci(cli): add a full benchmark for integration-heavy commands - #7412

ci(cli): add a full benchmark for integration-heavy commands#7412
Prashansa-K wants to merge 13 commits into
developfrom
ci/cli-bench-stack

Prashansa-K commented Oct 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 30, 2026

Timing ranges

API calls per command

Slowest phases

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Prashansa-K commented Oct 30, 2026

Uh oh!

Coly010 left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Prashansa-K commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 30, 2026

CLI benchmark

Lightweight Commands: no changes found

Timing ranges

API calls per command

Slowest phases

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

🤖 AI Review

Findings

Stats

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Prashansa-K commented Oct 30, 2026

Uh oh!

Coly010 left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Prashansa-K commented Oct 30, 2026 •

edited

Loading