[joseph-isaacs]

Summary

Benchmarking one SQL suite on a PR (e.g. PolarSignals) currently means running a whole preset via action/bench-sql*. This adds a per-benchmark label and a workflow input so you can run just the benchmarks you need.

Changes

• vx-bench matrix <preset> --only <ids>: narrows a preset to the given benchmark IDs (comma- or space-separated), keeping the preset's order and coverage. Empty input keeps the whole preset. An ID the preset doesn't schedule is an error that lists the IDs it does.
• action/bench-sql/<id> labels (pr-bench-dispatch.yml): one generic job matches any label with this prefix, removes it, strips the prefix and runs that benchmark at its pr-all coverage, e.g. action/bench-sql/polarsignals. The label reaches the shell only through an env var.
• benchmarks input on pr-bench-sql.yml (workflow_call + workflow_dispatch) and sql-bench-matrix.yml, passed to --only. The concurrency group now includes it, so a single-benchmark run doesn't cancel a full run of the same preset.
• Docs in docs/developer-guide/benchmarking.md and bench-orchestrator/README.md.

The labels must exist before they can be added. Valid IDs are the ones vx-bench matrix pr-all prints: clickbench-nvme clickbench-sorted-nvme tpch-nvme tpch-s3 tpch-nvme-10 tpch-s3-10 tpcds-nvme statpopgen fineweb fineweb-s3 polarsignals appian-nvme job-nvme vortex-queries. To create them:

for id in clickbench-nvme clickbench-sorted-nvme tpch-nvme tpch-s3 tpch-nvme-10 tpch-s3-10 tpcds-nvme statpopgen fineweb fineweb-s3 polarsignals appian-nvme job-nvme vortex-queries; do gh label create "action/bench-sql/$id" --repo vortex-data/vortex --color 5319e7 --description "Run the $id SQL benchmark"; done

Testing: new tests in bench-orchestrator/tests/test_matrix.py check that every ID in every preset can be selected on its own, plus ordering, empty selection, unknown-ID rejection and the CLI separators. pytest bench-orchestrator/tests (47 passed), ruff format --check, ruff check, yamllint --strict and typos are clean. actionlint was not run, and the label path hasn't run in real CI yet.

---

Generated by Claude Code

[codspeed]

Merging this PR will improve performance by 13.15%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 1 improved benchmark
✅ 2305 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Walltime	threshold_avx2[80]	3 µs	2.7 µs	+13.15%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/nice-bohr-7bkxmv (e9ec2d5) with develop (c6d6bad)}