Skip to content

Stop writing benchmark results to S3 - #10988

Merged
joseph-isaacs merged 1 commit into
developfrom
ji/bench-stop-s3-results
Oct 30, 2026
Merged

joseph-isaacs merged 1 commit into
developfrom
ji/bench-stop-s3-results

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

After #10987, nothing reads s3://vortex-ci-benchmark-results/data.json.gz or commits.json. Appending to the 353 MB data.json.gz (download, gunzip, append, gzip, upload, with ETag retries across parallel jobs) was the slowest step of every develop benchmark job. All results are already ingested into the benchmarks Postgres.

@codspeed

codspeed Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 10.24%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 1 improved benchmark
✅ 2317 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime threshold_avx2[80] 3 µs 2.7 µs +10.24%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/bench-stop-s3-results (3e8ffa7) with develop (05fd248)

Open in CodSpeed

Copy link
Copy Markdown
Contributor Author

The CodSpeed failure on 66141e3d (bitpack_compare_sweep compare[1] −15.9%, starts_with_avx512[64, VarBin] −14.9%, alongside a +18% improvement elsewhere) is benchmark noise, not this PR. This stack only changes GitHub workflows and Python scripts under scripts/; no Rust code changed, so nothing CodSpeed measures can move. The same kind of swing showed up on #10987 earlier and cleared on the next head. A newer head has already been pushed, so CodSpeed will run again there.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

CUDA build & lint failed on a20275f3 before any step ran. The runner couldn't download its setup action: Unable to resolve action runs-on/action@v2, unable to find version v2. That's a transient GitHub/runner problem, not this PR. A newer head has been pushed since, so the job runs again there.


Generated by Claude Code

Base automatically changed from ji/bench-baseline-postgres to develop October 30, 2026 13:19
Nothing reads `data.json.gz` or `commits.json` once PR comparisons read from
the benchmarks database, and appending to the 353 MB `data.json.gz` was the
slowest step of every develop benchmark job. Remove the S3 uploads, the
`commit-metadata` job that only fed `commits.json` (`commit-metadata.yml`
already records commits in Postgres), and `cat-s3.sh` and `commit-json.sh`.

String Encoding has no database table yet, so develop no longer stores its
results.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01UeniBBtX1DUban9Fda2Edh
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs
joseph-isaacs force-pushed the ji/bench-stop-s3-results branch from 10c83ef to 3e8ffa7 Compare October 30, 2026 13:19
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review October 30, 2026 13:19
@joseph-isaacs
joseph-isaacs enabled stacked auto-merge (squash) October 30, 2026 13:20
@joseph-isaacs
joseph-isaacs merged commit 996bb93 into develop Oct 30, 2026
79 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/bench-stop-s3-results branch October 30, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants