Repository navigation
Compare PR benchmarks against the benchmarks database instead of S3 - #10987
Conversation
PR benchmark comments read their develop baseline from the 353 MB `data.json.gz` S3 dump. Read it from the benchmarks Postgres instead, which develop runs already write to. - `scripts/fetch-benchmark-baseline.py` reads the PR run's ingest records, finds the newest commit reachable from HEAD that recorded the same measurements, and writes that commit's rows. It connects as the read-only `bench_read` role. - `compare-benchmark-jsons.py` compares ingest records on both sides. File sizes and suite metadata still come from the PR's results JSON. - Random access, compression and GPU compression PR runs now emit ingest records. String Encoding and SQL file sizes are not in the database, so their PR comments show the PR measurements without a baseline. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UeniBBtX1DUban9Fda2Edh Signed-off-by: Joe Isaacs <[email protected]>
|
Generated by Claude Code |
Merging this PR will not alter performance1 warning
Comparing |
Polar Signals Profiling ResultsLatest Run
Previous Runs (59)
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.991x ➖, 2↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.988x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.965x ➖, 2↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR The baseline could not be read from the benchmarks database; PR measurements are shown without comparison. datafusion / vortex-compact / ns (no group data, 0↑ 0↓)
datafusion / parquet / ns (no group data, 0↑ 0↓)
duckdb / vortex-compact / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR The baseline could not be read from the benchmarks database; PR measurements are shown without comparison. datafusion / vortex-compact / ns (no group data, 0↑ 0↓)
datafusion / parquet / ns (no group data, 0↑ 0↓)
duckdb / vortex-compact / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
|
Benchmarks: Appian on NVME 📖Commits: PR The baseline could not be read from the benchmarks database; PR measurements are shown without comparison. datafusion / vortex-compact / ns (no group data, 0↑ 0↓)
datafusion / parquet / ns (no group data, 0↑ 0↓)
duckdb / vortex-compact / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
|
Benchmarks: FineWeb S3 📖Commits: PR The baseline could not be read from the benchmarks database; PR measurements are shown without comparison. datafusion / vortex-compact / ns (no group data, 0↑ 0↓)
datafusion / parquet / ns (no group data, 0↑ 0↓)
duckdb / vortex-compact / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.009x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 1↓)
|
Benchmarks: Compression 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. vortex / vortex-file-compressed / ns (no group data, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (no group data, 0↑ 0↓)
vortex / parquet / ns (no group data, 0↑ 0↓)
vortex / parquet / bytes (no group data, 0↑ 0↓)
vortex / arrow-ipc / ns (no group data, 0↑ 0↓)
vortex / arrow-ipc / bytes (no group data, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR The baseline could not be read from the benchmarks database; PR measurements are shown without comparison. datafusion / vortex-compact / ns (no group data, 0↑ 0↓)
datafusion / parquet / ns (no group data, 0↑ 0↓)
duckdb / vortex-compact / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
|
Benchmarks: String Encoding 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. vortex / vortex-file-compressed / ms (no group data, 0↑ 0↓)
vortex / vortex-file-compressed / % (no group data, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR The baseline could not be read from the benchmarks database; PR measurements are shown without comparison. datafusion / vortex-compact / ns (no group data, 0↑ 0↓)
datafusion / parquet / ns (no group data, 0↑ 0↓)
duckdb / vortex-compact / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
|
- SQL rows keep the `tpch_q01/engine:format` names of the results JSON; the scale factor and variant move to the dataset join key. - When the baseline cannot be read (no database credential), the comment says so instead of claiming the benchmark has no baseline yet. - File sizes have no baseline in the database, so the comment no longer prints a file-size section that can never compare anything. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UeniBBtX1DUban9Fda2Edh Signed-off-by: Joe Isaacs <[email protected]>
#10997 set exclude-newer = "7 days" for the workspace. NVIDIA's index publishes no upload dates, so uv filtered out every cudf-cu12 wheel and the GPU compression benchmark failed to install cuDF. The cuDF venv is standalone, so install into it with --no-config. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UeniBBtX1DUban9Fda2Edh Signed-off-by: Joe Isaacs <[email protected]>
Composite actions cannot read the vars context, so GitHub rejected the action file. Callers pass the role, region, endpoint and database in; the record paths keep their defaults. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UeniBBtX1DUban9Fda2Edh Signed-off-by: Joe Isaacs <[email protected]>
Keep develop's branch argument to build_commit (#10987) and this branch's retried write around connect and ingest. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Will Manning <[email protected]>
Summary
PR benchmark comments get their develop baseline by downloading
data.json.gzfrom S3 (353 MB compressed). Develop runs also append to that file, and that append is the slowest step of each develop benchmark job. Develop runs already write every result to the benchmarks Postgres, so this PR reads the PR baseline from Postgres instead. After this PR, nothing reads the S3 file. The next PR in the stack (#10988) stops writing it.CI reads the database the same way the ingest does, through an AWS role over OIDC with an RDS IAM token. It uses the read-only pair added in vortex-data/vortex.dev#149: the role
GitHubBenchmarkReadRole(vars.GH_BENCH_READ_ROLE_ARN) and the Postgres userbench_ci_read(migration 011, SELECT only).Merge order: apply vortex.dev migration 012 (
commits.branch) before merging this PR, because the ingest now writes that column.Changes
.github/actions/fetch-bench-baseline(new):bench_ci_read.scripts/fetch-benchmark-baseline.py(new):--ingest-jsonlrecords.HEAD.scripts/compare-benchmark-jsons.py:tpch_q01/datafusion:parquet), and the scale factor and variant stay in the dataset key.--emit-ingest-records; the SQL runs already did. GPU Compression also getsid-token: write.scripts/post-ingest.pytakes a required--branchand writes it tocommits.branch; the ingest workflows passgithub.ref_name.What the PR comments lose:
What they gain: compression timings get hot/cold columns, because the records carry every run.
Testing
scripts/tests/test_benchmark_reporting.py(28 passed) covers:IS NULLmatch for absent fields, and the shallow-checkout refusal.scripts/tests/test_post_ingest.pyandtest_measurement_id.pypass.ruff format,ruff checkandyamllint --strictare clean.Baseline query against a local Postgres 16, built from the vortex.dev migrations with 4.2M synthetic query rows; the new query returns the same records as the old one:
The old query had hit the 30s statement timeout on TPC-H in CI.
In CI, the SQL benchmark read a real baseline through the read role (
PR 2e69732a vs base c6d6bada). One run of each benchmark kind is going on86d3c3e4.🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBBtX1DUban9Fda2Edh