From ab73b5ba084c01eaa76a55a3fca6b3d79877ab3c Mon Sep 17 00:00:00 2001 From: Will Manning Date: Wed, 28 Oct 2026 21:32:03 +0100 Subject: [PATCH 01/19] Plan bench's upgrade onto the site's toolchain Co-Authored-By: Claude Opus 5.5 Signed-off-by: Will Manning --- .../plans/2026-10-28-toolchain-and-cleanup.md | 226 ++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 bench/docs/plans/2026-10-28-toolchain-and-cleanup.md diff --git a/bench/docs/plans/2026-10-28-toolchain-and-cleanup.md b/bench/docs/plans/2026-10-28-toolchain-and-cleanup.md new file mode 100644 index 0000000..2db63f0 --- /dev/null +++ b/bench/docs/plans/2026-10-28-toolchain-and-cleanup.md @@ -0,0 +1,226 @@ + + +# bench: upgrade, share the site's toolchain, clean up + +Status: design, 2026-10-28. Step 1 of 3. + +## Why + +bench (`bench/web`) came from the archived `vortex-data/benchmarks-website` +repository. It runs on a second toolchain beside the marketing site: pnpm, +ESLint, Prettier, TypeScript 5.9 and Next.js 15.5, against the site's Bun, +Biome, TypeScript 7 and Next.js 16. Its packages are behind, and two of its +largest files (`components/Chart.tsx`, 2,200 lines; `lib/chart-store.ts`, 632 +lines) are hard to change safely. + +The goal is maintainability: one toolchain and one set of supply-chain rules +for both apps, current dependencies, and smaller, clearer code. The two apps +keep separate deployments. + +This is the first of three steps: + +1. **This document.** Upgrade and clean up bench with no change in behavior. +2. A new landing page: one page, every group folded with its headline numbers, + and opening a group loads its charts in place. +3. One visual redesign across vortex.dev, bench, explore and docs. Open PR + vortex-data/vortex.dev#120 is held for that step. + +## Goals + +- bench and the site share one Bun workspace, one lockfile, one Biome config + and one TypeScript base config. +- bench is on Next.js 16, React 19.3, TypeScript 7 and current versions of its + other packages, inside the 14-day cooldown. +- The code is smaller and clearer where that is cheap: dead paths removed, the + two largest client files split into modules, and a group's charts fetched + with one query instead of one per chart. +- Behavior is exactly the same, and each PR proves it. + +## Non-goals + +- Any change a visitor can see. The landing page is step 2; the visual + redesign is step 3. +- Replacing Vitest. It stays: its 8,000 lines of tests are the main evidence + that behavior did not change. +- Porting `bench/scripts/migrate-schema.py` to Bun. That comes later and is + not urgent. +- Database schema, ingest contract (`bench/CONTRACT.md`) or AWS changes. +- Merging the two Vercel projects or changing either app's domain. + +## Anti-goals + +The work fails, even if it builds, if any of these happen: + +- A page, API response, response header, cache rule or URL changes. +- The supply-chain gates get weaker: the 14-day cooldown, explicit overrides, + the lifecycle-script allowlist, SHA-pinned actions, `ubuntu-24.04` runners. +- The marketing site's build, behavior or deploy changes, beyond installing + from a shared lockfile. +- The work lands as a long stack of small PRs. Will smoke-tests each PR by hand + in its Vercel preview, so fewer, larger PRs are better. + +## Constraints + +- **Order.** vortex-data/vortex.dev#124 (database pool lifecycle) merged on + 2026-10-28; build on it. PR 1 deletes `bench/web/pnpm-workspace.yaml`, so its + exact-version cooldown exemptions go with it. A version still inside the + window then needs a `bunfig.toml` exemption, with a removal date. +- **Supply chain.** CLAUDE.md "Supply chain hardening" is the policy. Bun + enforces the cooldown through `bunfig.toml`, so the pnpm settings move as + follows: + - The `overrides` in `bench/web/pnpm-workspace.yaml` (postcss, sharp and + zod) move to the root `package.json` `overrides`. Each entry keeps its + advisory and removal trigger. Drop postcss if the version Next.js 16 pins + is already patched. Bun applies overrides to the whole workspace, so the + zod override (`^4.6.7`, for `@vercel/functions` > `@vercel/oidc` > + `@vercel/cli-config`, which pins 4.1.11) also moves Lighthouse CI's + `chromium-bidi` from zod 3 to zod 4. That reverses a CLAUDE.md decision on + purpose: the Lighthouse job must still pass, and it also clears the site's + accepted zod advisory. If Lighthouse CI breaks, drop the override and + record bench's zod 4.1.11 as an accepted advisory instead (it parses only + local Vercel CLI config files). + - GitHub's dependency graph doesn't read `bun.lock`. Once bench leaves + `pnpm-lock.yaml`, `dependency-review` would no longer see its transitive + packages; the site already has that gap. A workflow submits a dependency + snapshot built from `bun.lock` to the Dependency Submission API on pushes + to `main` and on pull requests, so `dependency-review` sees every resolved + package for both apps. It must run before `dependency-review` reads the + pull request's dependencies. + - `allowBuilds` moves to `trustedDependencies`. Today only `sharp` and + `unrs-resolver` may build, and `unrs-resolver` goes away with ESLint. + - The bench parts of CLAUDE.md ("bench/ (pnpm)", "bench advisories") are + rewritten to match. +- **Required checks.** The `main` ruleset requires a check named + `Bench: format, lint, build, test`. Keep that job name. Renaming it needs a + ruleset change. +- **Settings changes need Will's confirmation first.** That covers GitHub, + Vercel and AWS settings. The `benchmarks-website` Vercel project builds on + the GitHub runner (`vercel build` in `.github/workflows/bench-deploy.yml`), + so the install command probably changes only in the workflow. If a project + setting must change, ask first. +- **The site's Vercel project** (`vortex`) builds through the git + integration. Adding a workspace must not make it build bench, and must not + noticeably slow its install. `scripts/vercel-ignore-build.sh` keeps skipping + bench-only commits. +- **Node 24** stays the runtime (CLAUDE.md "Node version"). +- **DCO sign-off** on every commit. Squash merges only. +- **Licensing.** New bench files carry SPDX headers: Apache-2.0 for code, + CC-BY-4.0 for Markdown. + +## Decided by Will + +- Share the toolchain and code, and keep separate deployments. +- Fewer, larger PRs, each smoke-tested in its Vercel preview. +- Keep Vitest for now. +- The Python migration runner moves to Bun eventually, not in this work. +- Close the `bun.lock` gap in dependency review with a submitted dependency + snapshot. +- Force zod 4 across the workspace and let Lighthouse CI prove it still works. + +## The work: two PRs + +### PR 1: upgrade bench and move it onto the site's toolchain + +Mechanical changes only. Commits are ordered so a reviewer can read each one +alone. + +1. **Parity check first.** Add the parity script (see "Proving nothing + changed") and run it with `main` on both sides, to show it reports no + difference. Later commits must keep it clean. +2. **Bun workspace.** The root `package.json` lists `bench/web` as a workspace. + One `bun.lock` replaces `bench/web/pnpm-lock.yaml`. `pnpm-workspace.yaml` + goes, and its settings move as described under Constraints. + `.github/workflows/bench-ci.yml`, `bench-deploy.yml`, the + `.github/actions/bench-checks` composite and Renovate's bench rule switch + from pnpm to Bun. The dependency-snapshot workflow lands in the same + commit, so `dependency-review` never loses sight of bench's packages. +3. **Output file tracing.** `bench/web/next.config.js` sets + `outputFileTracingRoot: __dirname`. Bun hoists workspace packages to the + root `node_modules`, so tracing must start at the repository root, or the + deployed functions can miss dependencies. Check the traced files of a + preview function, not only the build log. +4. **Next.js 16 and React 19.3**, with the other dependency updates + (`pg`, `chart.js`, `jsdom` 30, `@types/*`). Read the Next.js 16 upgrade + guide against bench's use of `unstable_cache`, `revalidateTag`, + `force-dynamic` pages, route handlers and `vercel.json` headers. Turbopack + becomes the default builder. +5. **Biome replaces ESLint and Prettier.** One root `biome.json` covers both + apps: remove the `!bench` exclusion. An `overrides` entry for `bench/**` + keeps bench's current Prettier style (single quotes, trailing commas, + 100 columns), so the switch reformats almost nothing. One style for both + apps can come later; `main` takes squash merges only, so a mass reformat + can't be hidden from `git blame` without also hiding the real changes in + the same squashed commit. ESLint's `next/core-web-vitals` rules have no + Biome equivalent; the site already accepts that. Map any rule bench relies + on, such as the React hooks rules, to its Biome counterpart. +6. **TypeScript 7.** bench's `tsconfig.json` extends a base shared with the + site and keeps its current strictness. Stricter flags wait for PR 2. + +### PR 2: clean up bench's code + +Logic stays the same. The parity check and the tests prove it. + +- **Remove the IAM sign-in path.** Production signs in with the `bench_read` + password; IAM auth is not available for that role as shipped (see + `bench/web/README.md`). Remove `@aws-sdk/rds-signer`, `BENCH_DB_REGION` and + the signer code, and make `BENCH_DB_PASSWORD` required. Update the README + and runbooks. +- **One query per group.** `collectGroupCharts` now runs two queries for each + chart (a seed window and the data), about 200 for a TPC-DS group of 99 + charts. Fetch a group's rows in one or two queries, keeping each chart's own + last-N-commits window (for example, with a window function partitioned by + chart). The JSON must stay byte-identical. This removes the fan-out that + `QUERY_CONCURRENCY` and the pool-wait discussion on #124 exist for. Keep the + concurrency limit unless the new code no longer needs it. +- **Split `components/Chart.tsx` and `lib/chart-store.ts`** into smaller + modules along their existing seams (fetching, formatting, zoom, the loading + queues). Move code; don't rewrite it. +- **Adopt the site's stricter TypeScript flags** + (`noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`), with the code + changes they need. +- Remove dead code that the stricter flags and Biome expose. + +## Proving nothing changed + +Each PR passes these checks, in order: + +1. **CI**: `Bench: format, lint, build, test`, including the Postgres + integration suite. +2. **Parity, run locally.** Restore a production `pg_dump` into a local + Postgres container. Build `main` and the PR branch, serve both with + `next start` on two ports against that one database, and compare: + - JSON from `/api/groups`, `/api/group/{slug}` for every group at `n=100` + and one other window, and `/api/chart/{slug}` for every chart at `n=100` + and a sample at `n=all`. The bodies must match exactly. + - The response headers the app sets: status, `content-type`, + `cache-control`. + - The `/` and `/chart/{slug}` pages: screenshots after the charts load, at + desktop and phone widths, compared pixel by pixel with Playwright (already + a dev dependency of the site). + + Comparing against production or a Vercel preview doesn't work: the Data + Cache (`unstable_cache`, with a 24-hour backstop, flushed only by + production's `/api/revalidate`) makes a preview's cached routes lag + production after any ingest, and a shared preview cache could serve + payloads built by older code. Two local servers start with empty caches + and read the same frozen data, so any difference comes from the code. + This needs no token and no Vercel setting. Will's Mac holds a dump from + 2026-10-28 in `~/Backups/vortex-bench/`. +3. **The deployed preview.** `bench-deploy.yml` already runs + `bench/scripts/verify-web-deploy.mjs` against each preview's public read + paths. It catches what only a Vercel build shows, such as file tracing. + `vercel.json`, which carries the CDN cache headers, must not change. +4. **Will's smoke test** in the preview. + +## Left to the builder + +- The parity script's exact sample, and how it waits for charts before a + screenshot. +- The Biome rule mapping, and how to apply the site's style. +- Whether TypeScript 7 or Next.js 16 needs code changes, and what they are. +- The module boundaries for the `Chart.tsx` and `chart-store.ts` split. +- The SQL for the group query, as long as the output is byte-identical. +- Whether to install the whole workspace or only one app in each workflow. From 45e667a8680b22a5ff5e95b658ef13b4f84c7de7 Mon Sep 17 00:00:00 2001 From: Will Manning Date: Wed, 28 Oct 2026 21:37:08 +0100 Subject: [PATCH 02/19] Plan bench PR 1: the toolchain move Co-Authored-By: Claude Opus 5.5 Signed-off-by: Will Manning --- .../plans/2026-10-28-toolchain-pr1-plan.md | 239 ++++++++++++++++++ 1 file changed, 239 insertions(+) create mode 100644 bench/docs/plans/2026-10-28-toolchain-pr1-plan.md diff --git a/bench/docs/plans/2026-10-28-toolchain-pr1-plan.md b/bench/docs/plans/2026-10-28-toolchain-pr1-plan.md new file mode 100644 index 0000000..bbd07e4 --- /dev/null +++ b/bench/docs/plans/2026-10-28-toolchain-pr1-plan.md @@ -0,0 +1,239 @@ + + +# bench PR 1: upgrade and move onto the site's toolchain — implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** bench builds, tests, lints and deploys with the site's Bun, Biome and TypeScript 7 toolchain on Next.js 16, and nothing a visitor or API client sees changes. + +**Architecture:** The repository root becomes a Bun workspace whose members are the site (root) and `bench/web`, with one `bun.lock`. A local parity harness serves `main` and the branch side by side against a restored production dump and diffs every route and screenshot. A submitted dependency snapshot keeps `dependency-review` able to see every package in `bun.lock`. + +**Tech Stack:** Bun 1.4.3, Next.js 16.4.1, React 19.3.1, TypeScript 7, Biome 2.5, Vitest 4 (stays), Playwright 1.64, Postgres 16 in Docker, GitHub Actions. + +**Spec:** `bench/docs/plans/2026-10-28-toolchain-and-cleanup.md` (read it first; this plan covers its "PR 1"). + +## Global Constraints + +- Behavior is exactly the same: pages, API JSON, response headers, cache rules, URLs. `bench/web/vercel.json` headers and crons do not change. +- Cooldown: `bunfig.toml` `minimumReleaseAge = 1209600` applies to every install. A version still inside the window needs a `minimumReleaseAgeExcludes` entry with an advisory reason and a removal date. +- Root `package.json` `overrides`: keep `"sharp": "^0.35.5"`; add `"zod": "^4.6.7"`; add `"postcss": "^8.5.30"` only if the postcss that Next.js 16.4.1 resolves to is below a patched version for GHSA-6g55-p6wh-862q and GHSA-r28c-9q8g-f849. +- `trustedDependencies` stays `["esbuild", "sharp"]` unless `bun pm untrusted` shows a bench package that must build; justify any addition in the commit message. +- The job name `Bench: format, lint, build, test` does not change (the `main` ruleset requires it). +- Actions are SHA-pinned with a `# vX.Y.Z` comment; runners are `ubuntu-24.04`; Node comes from `.nvmrc` (24). Reuse `oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0` with `bun-version: 1.4.3`. +- No GitHub, Vercel or AWS **settings** change without asking Will first. Files in the repository (workflows, `vercel.json`) are fine. +- Vitest stays the bench test runner; the Python migration runner and its uv-based tests stay as they are. +- bench's code style stays: single quotes, trailing commas everywhere, 100 columns, 2-space indent. +- Every commit: `git commit -s`, ending with `Co-Authored-By: Claude Opus 5.5 `. New code files under `bench/` start with `// SPDX-License-Identifier: Apache-2.0` and `// SPDX-FileCopyrightText: Copyright the Vortex contributors`. +- Root `tsconfig.json` and `biome.json` stop excluding `bench/` only where a task says so. + +## Review Focus + +1. **A deployed function misses a hoisted package.** With Bun hoisting `pg` and `@vercel/functions` to the root `node_modules`, a preview whose file tracing starts at `bench/web` returns 500 on every API route. Task 3 asserts the traced files include them. +2. **Ingest stops flushing the cache.** Next.js 16 changes `revalidateTag`; the emitter's `POST /api/revalidate` must still expire cached data at once. Task 4 pins the second argument in a test. +3. **The site's deploy gets slower or breaks.** The `vortex` Vercel project runs `bun install --frozen-lockfile` at the root, which now installs bench too. Task 3 times it and checks the site still builds. +4. **Dependency review passes because it saw nothing.** If the snapshot isn't submitted before `dependency-review` reads the pull request, the check passes blind. Task 2 makes it wait, and Task 7 reads its log. +5. **A root-level change skips bench CI.** `changed-paths` treats `package.json` and `bun.lock` as site-only today; after the move they change bench too. Task 3 reclassifies them and checks sample paths. + +--- + +### Task 1: Parity harness + +**Files:** +- Create: `bench/scripts/parity/run.sh` +- Create: `bench/scripts/parity/compare-api.ts` +- Create: `bench/scripts/parity/compare-api.test.ts` +- Create: `bench/scripts/parity/screenshots.spec.ts` +- Create: `bench/scripts/parity/playwright.config.ts` +- Create: `bench/scripts/parity/README.md` (CC-BY-4.0 header) + +**Interfaces:** +- Produces: `bash bench/scripts/parity/run.sh ` — exits 0 when the two builds match, non-zero with a printed diff otherwise. Base is a git ref; head is the current working tree. +- Produces: `diffJson(base: string, head: string): string | null` and `compareApi(baseUrl: string, headUrl: string): Promise` in `compare-api.ts` (the array holds one line per mismatch). + +- [ ] **Step 1: Write the failing test** in `compare-api.test.ts` (run with `bun test`): + +```ts +import { expect, test } from 'bun:test'; +import { diffJson } from './compare-api'; + +test('identical bodies match', () => { + expect(diffJson('{"a":[1,2]}', '{"a":[1,2]}')).toBeNull(); +}); +test('a changed value is reported with its path', () => { + expect(diffJson('{"a":[1,2]}', '{"a":[1,3]}')).toContain('a[1]'); +}); +test('key order counts, because clients see bytes', () => { + expect(diffJson('{"a":1,"b":2}', '{"b":2,"a":1}')).not.toBeNull(); +}); +``` + +- [ ] **Step 2: Run it:** `bun test bench/scripts/parity/compare-api.test.ts` — expect FAIL (module missing). +- [ ] **Step 3: Implement `diffJson`** (byte comparison first; on mismatch, parse both and return the first differing path) and **`compareApi`**. `compareApi` reads `/api/groups` from the base, then requests from both servers: `/api/groups`; `/api/group/{slug}?n=100` and `?n=50` for every group; `/api/chart/{slug}?n=100` for every chart; `/api/chart/{slug}?n=all` for every 25th chart. It compares status, `content-type`, `cache-control` and the body. +- [ ] **Step 4: Run the test again** — expect PASS. +- [ ] **Step 5: Write `screenshots.spec.ts` and its config.** Pages: `/` and `/chart/{slug}` for five slugs taken from `/api/groups` (the first chart of the first five groups). Projects: desktop 1440×900 and phone 390×844, Chromium only. The config reads `BASE_URL` and writes snapshots under `PARITY_OUT`. The run takes baselines from the base server with `--update-snapshots`, then compares the head server with `maxDiffPixels: 0`. How to wait for charts, and what to mask, are the implementer's call. +- [ ] **Step 6: Write `run.sh`.** + - Start `postgres:16-alpine` (the image the integration suite uses) as container `bench-parity-pg` on port 55432. + - Restore with `pg_restore --no-owner --no-privileges -j 4` from the dump directory, mounted read-only. Skip the restore if the container already holds the `commits` table. + - Export the base with `git archive | tar -x -C "$TMPDIR/bench-parity/base"`; no worktree. + - Build each side with its own toolchain: `pnpm install --frozen-lockfile && pnpm build` when `bench/web/pnpm-lock.yaml` exists, otherwise `bun install --frozen-lockfile` at the root and `bun run --cwd bench/web build`. + - Serve base on 3101 and head on 3102 with `next start`, with `BENCH_DB_HOST=127.0.0.1`, `BENCH_DB_PORT=55432`, `BENCH_DB_NAME=vortex_bench`, `BENCH_DB_USER=postgres`, `BENCH_DB_PASSWORD=parity` and `BENCH_DB_SSL=disable`. + - Run `compareApi`, then the screenshots, and stop the servers on exit (`trap`). +- [ ] **Step 7: Prove it reports no difference on equal code:** `bash bench/scripts/parity/run.sh origin/main ~/Backups/vortex-bench/vortex_bench-20261028T171815Z` from a clean checkout of `main`'s tree — expect exit 0 and `0 mismatches`. +- [ ] **Step 8: Prove it catches a difference:** temporarily change one number in a `/api/groups` response (for example, a summary explanation string in `bench/web/lib/summary.ts`), rerun, expect a non-zero exit naming that path; revert. +- [ ] **Step 9: Commit:** `Add a local parity check for bench builds`. + +### Task 2: Dependency snapshot from `bun.lock` + +**Files:** +- Create: `scripts/dependency-snapshot.ts` +- Create: `scripts/dependency-snapshot.test.ts` +- Create: `.github/workflows/dependency-snapshot.yml` +- Modify: `.github/workflows/ci.yml` (the `dependency-review` job) +- Modify: `CLAUDE.md` (supply-chain layer 5) + +**Interfaces:** +- Produces: `parseBunLock(text: string): BunLock`, `buildSnapshot(lock: BunLock, meta: SnapshotMeta): Snapshot`, with `SnapshotMeta = { sha: string; ref: string; correlator: string; jobId: string; scanned: string }`. `Snapshot` follows GitHub's Dependency Submission API (`version: 0`, `detector`, `manifests[*].resolved[purl] = { package_url, relationship, scope, dependencies }`). + +- [ ] **Step 1: Write the failing tests.** A fixture lock string with trailing commas holds: + - root workspace `vortex`: dependency `a`, devDependency `d`; + - workspace `bench/web`, named `benchmarks-web`: dependency `@s/x`; + - packages: `a@1.0.0` depends on `b` and `c`; nested `a/c@2.0.0`; `d@1.0.0` depends on `b` and `c`; `b@1.0.0`; `c@1.0.0`; `@s/x@1.0.0`. + + Assertions: + +```ts +const snap = buildSnapshot(parseBunLock(FIXTURE), META); +const root = snap.manifests['bun.lock (vortex)'].resolved; +expect(root['pkg:npm/a@1.0.0']).toMatchObject({ relationship: 'direct', scope: 'runtime' }); +expect(root['pkg:npm/b@1.0.0'].scope).toBe('runtime'); // reachable from a +expect(root['pkg:npm/c@2.0.0'].scope).toBe('runtime'); // a resolves the nested a/c +expect(root['pkg:npm/c@1.0.0'].scope).toBe('development'); // only d reaches top-level c +expect(root['pkg:npm/a@1.0.0'].dependencies).toEqual(['pkg:npm/b@1.0.0', 'pkg:npm/c@2.0.0']); +const bench = snap.manifests['bun.lock (bench/web)'].resolved; +expect(bench['pkg:npm/%40s/x@1.0.0']).toMatchObject({ relationship: 'direct', scope: 'runtime' }); +expect(snap.manifests['bun.lock (bench/web)'].file.source_location).toBe('bun.lock'); +``` + +- [ ] **Step 2: Run:** `bun test scripts/dependency-snapshot.test.ts` — expect FAIL. +- [ ] **Step 3: Implement.** + - Parse by removing trailing commas before `}` or `]`, then `JSON.parse`. + - Resolve dependency `N` of package key `K` by trying `K/N`, then `N` under each shorter ancestor of `K`, then `N`. A key is a chain of package names, where a scoped name is two segments. + - Resolve a workspace's own dependency `N` by trying `/N`, then `N`. + - Follow `dependencies`, `optionalDependencies` and `peerDependencies`, and skip names absent from the lock (platform-specific optional packages). + - A package is `runtime` if a workspace's `dependencies` or `optionalDependencies` reach it, and `development` otherwise. It is `direct` if the workspace lists it. + - The CLI (`import.meta.main`) reads `bun.lock` and `POST`s to `/repos/$GITHUB_REPOSITORY/dependency-graph/snapshots`, using `GITHUB_TOKEN`, `SNAPSHOT_SHA` and `SNAPSHOT_REF`. Anything but 201 fails. +- [ ] **Step 4: Run the tests** — expect PASS. Then run against the real lock without submitting (`DRY_RUN=1 bun scripts/dependency-snapshot.ts | head`) and check that `lighthouse`, `puppeteer-core` and `chromium-bidi` appear with scope `development`, and `next` with scope `runtime`. +- [ ] **Step 5: Write the workflow.** + - Triggers: `push` to `main`, and `pull_request` from the same repository only. Fork PRs have no write token. + - `permissions: contents: write`. + - Steps: checkout (`persist-credentials: false`), setup-bun, run the tests, then submit. + - `SNAPSHOT_SHA` and `SNAPSHOT_REF` are `github.sha` and `github.ref` on push, and `github.event.pull_request.head.sha` and `refs/heads/${{ github.head_ref }}` on pull requests. +- [ ] **Step 6: Make `dependency-review` wait for the snapshot:** add `retry-on-snapshot-warnings: true` and `retry-on-snapshot-warnings-timeout: 300` to its `with:`. Leave `fail-on-severity: high` and the default `fail-on-scopes` (runtime) as they are. +- [ ] **Step 7: Update CLAUDE.md** layer 5: dependency review now sees `bun.lock` through the submitted snapshot. It fails on runtime-scope advisories, and dev-only advisories stay in `bun audit` and the "Audit advisories" list. Rewrite the "bench advisories" sentence about pnpm lockfiles to match. +- [ ] **Step 8: Commit:** `Submit bun.lock to the dependency graph`. + +### Task 3: One Bun workspace for the site and bench + +**Files:** +- Modify: `package.json` (`workspaces: ["bench/web"]`, `overrides`), `bun.lock` (regenerated) +- Delete: `bench/web/pnpm-lock.yaml`, `bench/web/pnpm-workspace.yaml` +- Modify: `bench/web/package.json` (drop `packageManager`; scripts call Bun-installed binaries) +- Modify: `bench/web/next.config.js` (`outputFileTracingRoot`) +- Modify: `.github/actions/bench-checks/action.yml`, `.github/workflows/bench-ci.yml`, `.github/workflows/bench-deploy.yml`, `.github/actions/changed-paths/action.yml`, `renovate.json` +- Modify: `CLAUDE.md` ("bench/ (pnpm)" → Bun; "bench advisories"; "Repository layout" toolchain line), `bench/web/README.md`, `bench/AGENTS.md`, `bench/docs/architecture/deploy-and-infra.md` + +**Interfaces:** +- Produces: `bun install --frozen-lockfile` at the root installs both apps; `bun run --cwd bench/web `, { + headers: { 'content-type': 'text/html; charset=utf-8', 'cache-control': 'no-store' }, + ...init, + }); +} + +type Override = (request: Request) => Response | undefined | Promise; + +/** + * A fake bench server. It answers `/api/groups` with `groups`; every other URL goes to `override` + * first and, when that returns nothing, gets the answer the app gives: 200 for the data and the + * pages, 404 for the unknown chart, 400 from the API and 404 from the page for the malformed slug. + */ +function api(override?: Override, groups: unknown = GROUPS): Handler { + return async (request) => { + const { pathname } = new URL(request.url); + if (pathname === '/api/groups') { + return json(groups); + } + const answer = await override?.(request); + if (answer !== undefined) { + return answer; + } + const isApi = pathname.startsWith('/api/'); + if (pathname.endsWith(`/${MALFORMED_SLUG}`)) { + return isApi + ? json({ error: 'bad_request' }, { status: 400 }) + : html(NOT_FOUND_TITLE, { status: 404 }); + } + if (pathname.endsWith(`/${UNKNOWN_CHART_SLUG}`)) { + return isApi + ? json({ error: 'not_found' }, { status: 404 }) + : html(NOT_FOUND_TITLE, { status: 404 }); + } + return isApi ? json({ value: 1 }) : html(TITLE); + }; } +/** The data URLs, the two pages and the five error probes planned for {@link GROUPS}. */ +const PLANNED = 5 + 2 + 5; + test('servers that answer alike have no mismatches', async () => { const result = await compareApiDetailed(serve(api()), serve(api())); expect(result.mismatches).toEqual([]); - // /api/groups, two windows for the group, and the chart at n=100 and n=all. - expect(result.requests).toBe(5); + // /api/groups, two windows for the group, the chart at n=100 and n=all, the landing page, the + // chart page, and the five error probes. + expect(result.requests).toBe(PLANNED); }); test('the result tallies the statuses and the body bytes that were compared', async () => { const result = await compareApiDetailed(serve(api()), serve(api())); - expect(result.statuses).toEqual({ '200': 10 }); - const groupsBytes = JSON.stringify(GROUPS).length; - const valueBytes = JSON.stringify({ value: 1 }).length; - expect(result.bytes).toEqual({ - base: groupsBytes + 4 * valueBytes, - head: groupsBytes + 4 * valueBytes, - }); + expect(result.statuses).toEqual({ '200': 14, '404': 6, '400': 4 }); + const length = async (response: Response) => (await response.text()).length; + const perSide = + JSON.stringify(GROUPS).length + + 4 * JSON.stringify({ value: 1 }).length + + 2 * (await length(html(TITLE))) + + 2 * (await length(html(NOT_FOUND_TITLE))) + + JSON.stringify({ error: 'not_found' }).length + + 2 * JSON.stringify({ error: 'bad_request' }).length; + expect(result.bytes).toEqual({ base: perSide, head: perSide }); }); test('a different body, status, content-type and cache-control are each reported', async () => { const base = serve(api()); const head = serve( - api((request) => - new URL(request.url).pathname === '/api/group/g' - ? json({ value: 2 }, { status: 404 }) - : new Response(JSON.stringify({ value: 1 }), { + api((request) => { + const { pathname } = new URL(request.url); + if (pathname === '/api/group/g') { + return json({ value: 2 }, { status: 404 }); + } + return pathname === '/api/chart/c' + ? new Response(JSON.stringify({ value: 1 }), { headers: { 'content-type': 'text/plain', 'cache-control': 'no-store' }, - }), - ), + }) + : undefined; + }), ); const lines = await compareApi(base, head); expect(lines).toContain('/api/group/g?n=100: status 200 != 404'); @@ -153,7 +266,7 @@ test('a 404 on both sides fails, because every planned URL comes from the base', const missing = api((request) => new URL(request.url).pathname === '/api/chart/c' ? json({ error: 'not_found' }, { status: 404 }) - : json({ value: 1 }), + : undefined, ); const result = await compareApiDetailed(serve(missing), serve(missing)); expect(result.mismatches).toContain( @@ -163,7 +276,57 @@ test('a 404 on both sides fails, because every planned URL comes from the base', `/api/chart/c?n=100: head answered 404 instead of 200, ${NOTHING_COMPARED}`, ); expect(result.mismatches.filter((line) => !line.startsWith('/api/chart/c?n='))).toEqual([]); - expect(result.statuses).toEqual({ '200': 6, '404': 4 }); + expect(result.statuses).toEqual({ '200': 10, '404': 10, '400': 4 }); +}); + +test('an expected error status passes, and any other status on that probe fails', async () => { + const page = `/chart/${UNKNOWN_CHART_SLUG}`; + const found = api((request) => + new URL(request.url).pathname === page ? html(TITLE) : undefined, + ); + const result = await compareApiDetailed(serve(api()), serve(found)); + expect(result.mismatches).toEqual([ + `${page}: head answered 200 instead of 404, ${NOTHING_COMPARED}`, + `${page}: status 404 != 200`, + `${page}: title ${JSON.stringify(NOT_FOUND_TITLE)} != ${JSON.stringify(TITLE)}`, + ]); +}); + +test('an API error probe compares its body too', async () => { + const path = `/api/group/${MALFORMED_SLUG}`; + const other = api((request) => + new URL(request.url).pathname === path + ? json({ error: 'invalid' }, { status: 400 }) + : undefined, + ); + const lines = await compareApi(serve(api()), serve(other)); + expect(lines).toEqual([`${path}?n=100: body differs $.error: "bad_request" != "invalid"`]); +}); + +test('a page compares its title, not the rest of its HTML', async () => { + const rebuilt = api((request) => + ['/', '/chart/c'].includes(new URL(request.url).pathname) + ? html(TITLE, {}, 'build-2') + : undefined, + ); + expect(await compareApi(serve(api()), serve(rebuilt))).toEqual([]); + + const retitled = api((request) => + new URL(request.url).pathname === '/' ? html('Benchmarks') : undefined, + ); + expect(await compareApi(serve(api()), serve(retitled))).toEqual([ + `/: title ${JSON.stringify(TITLE)} != "Benchmarks"`, + ]); +}); + +test('a page without a title fails, because nothing was compared', async () => { + const untitled = api((request) => + new URL(request.url).pathname === '/chart/c' ? html(null) : undefined, + ); + expect(await compareApi(serve(untitled), serve(untitled))).toEqual([ + `/chart/c: base sent no , ${NOTHING_COMPARED}`, + `/chart/c: head sent no <title>, ${NOTHING_COMPARED}`, + ]); }); test('a redirect or an empty 200 on both sides fails too', async () => { @@ -184,15 +347,17 @@ test('a redirect or an empty 200 on both sides fails too', async () => { test('a slug that needs encoding reaches the servers encoded', async () => { const groups = { groups: [{ slug: 'we ird/slug?x', charts: [{ slug: 'c d#e' }] }] }; const seen: string[] = []; + const app = api(undefined, groups); const handler: Handler = (request) => { const { pathname, search } = new URL(request.url); seen.push(pathname + search); - return pathname === '/api/groups' ? json(groups) : json({ value: 1 }); + return app(request); }; const result = await compareApi(serve(handler), serve(handler)); expect(result).toEqual([]); expect(seen).toContain('/api/group/we%20ird%2Fslug%3Fx?n=100'); expect(seen).toContain('/api/chart/c%20d%23e?n=all'); + expect(seen).toContain('/chart/c%20d%23e'); expect(seen.filter((path) => path.includes(' ') || path.includes('#'))).toEqual([]); }); @@ -201,7 +366,7 @@ test('an unreachable head is reported for every URL', async () => { const head = serve(api()); await servers[servers.length - 1].stop(true); const lines = await compareApi(base, head); - expect(lines).toHaveLength(5); + expect(lines).toHaveLength(PLANNED); expect(lines[0]).toContain('head request failed'); }); @@ -209,25 +374,29 @@ test('requests that fill the group cache never overlap, even across the two serv const groups = { groups: ['g0', 'g1', 'g2'].map((slug) => ({ slug, charts: [{ slug: `${slug}-chart` }] })), }; + const app = api(undefined, groups); let fills = 0; let mostFills = 0; let mostRequests = 0; let requests = 0; const handler: Handler = async (request) => { const url = new URL(request.url); - const fill = url.pathname === '/api/groups' || url.search === '?n=100'; - const isGroupPath = url.pathname.startsWith('/api/group'); + // The landing page reads every group through the same cache as /api/groups. + const fill = + url.pathname === '/' || + url.pathname === '/api/groups' || + (url.pathname.startsWith('/api/group/') && url.search === '?n=100'); requests++; mostRequests = Math.max(mostRequests, requests); - if (fill && isGroupPath) { + if (fill) { mostFills = Math.max(mostFills, ++fills); } await Bun.sleep(5); - if (fill && isGroupPath) { + if (fill) { fills--; } requests--; - return url.pathname === '/api/groups' ? json(groups) : json({ value: 1 }); + return app(request); }; const result = await compareApiDetailed(serve(handler), serve(handler)); expect(result.mismatches).toEqual([]); diff --git a/bench/scripts/parity/compare-api.ts b/bench/scripts/parity/compare-api.ts index c3159ff..c7eee50 100644 --- a/bench/scripts/parity/compare-api.ts +++ b/bench/scripts/parity/compare-api.ts @@ -2,7 +2,8 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors /** - * Compare the read API of two bench servers that share one database. + * Compare the responses of two bench servers that share one database: the read API, and the + * status, headers and title of the pages. * * Run as `bun compare-api.ts <base-url> <head-url>`; it prints one line per mismatch and a * summary, and exits non-zero when anything differs. `run.sh` calls it after it starts both @@ -15,6 +16,22 @@ const DEFAULT_CONCURRENCY = 4; const FULL_HISTORY_EVERY = 25; /** `?n=all` can read the whole history of a chart, so the deadline is generous. */ const REQUEST_TIMEOUT_MS = 180_000; +/** Chart pages to load: the first chart of each of the first this many groups, as the screenshots. */ +const CHART_PAGES = 5; + +/** + * A chart slug that the app parses (a compression-time chart, in the slug format of + * `bench/web/lib/slug.ts`) but that names no chart, so the API and the page answer 404. + */ +export const UNKNOWN_CHART_SLUG = `ct.${Buffer.from( + JSON.stringify({ + k: 'CompressionTime', + dataset: 'parity-check-no-such-dataset', + dataset_variant: null, + }), +).toString('base64url')}`; +/** A slug the app cannot parse: the API answers 400 and the chart page 404. */ +export const MALFORMED_SLUG = 'parity-check-malformed'; /** The parts of `GET /api/groups` that decide which URLs to request. */ interface GroupLinks { @@ -22,6 +39,18 @@ interface GroupLinks { charts: { slug: string }[]; } +/** One URL to request from both servers, and what both must answer. */ +export interface PlannedRequest { + path: string; + /** The status both servers must answer. */ + status: number; + /** + * What to compare besides the status, `content-type` and `cache-control`: the body byte for + * byte, or a page's `<title>` only, because its HTML names the build's own script files. + */ + compare: 'body' | 'title'; +} + const NOTHING_COMPARED = 'so nothing was compared for this route'; /** The parts of a response the parity check compares. */ @@ -141,6 +170,33 @@ export function diffJson(base: string, head: string): string | null { ); } +const ENTITIES: Record<string, string> = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'" }; + +function decodeEntities(text: string): string { + return text.replace(/&(#x[0-9a-f]+|#[0-9]+|[a-z]+);/gi, (entity, name: string) => { + if (name[0] === '#') { + const hex = name[1] === 'x' || name[1] === 'X'; + return String.fromCodePoint(Number.parseInt(name.slice(hex ? 2 : 1), hex ? 16 : 10)); + } + return ENTITIES[name.toLowerCase()] ?? entity; + }); +} + +/** + * The page's title as a browser shows it: the first `<title>` outside inline SVG (an SVG's title + * labels the icon), with entities decoded and whitespace collapsed. + * + * @returns null when the page has no title. + */ +export function pageTitle(html: string): string | null { + const outsideSvg = html.replace(/<svg[\s>][\s\S]*?<\/svg>/gi, ''); + const match = /<title(?:\s[^>]*)?>([\s\S]*?)<\/title>/i.exec(outsideSvg); + if (match === null) { + return null; + } + return decodeEntities(match[1]).replace(/\s+/g, ' ').trim(); +} + async function capture(origin: string, path: string): Promise<Captured> { const response = await fetch(new URL(path, origin), { redirect: 'manual', @@ -162,7 +218,12 @@ async function captureOrError(origin: string, path: string): Promise<Captured | } } -function compareCaptured(path: string, base: Captured | Error, head: Captured | Error): string[] { +function compareCaptured( + planned: PlannedRequest, + base: Captured | Error, + head: Captured | Error, +): string[] { + const { path } = planned; const lines: string[] = []; for (const [side, result] of [ ['base', base], @@ -170,12 +231,19 @@ function compareCaptured(path: string, base: Captured | Error, head: Captured | ] as const) { if (result instanceof Error) { lines.push(`${path}: ${side} request failed: ${result.message}`); - } else if (result.status !== 200) { - // Every planned URL comes from the base's own /api/groups, so it must answer 200. Two equal - // 404s, 500s or redirects would otherwise match each other and hide that no data was read. - lines.push(`${path}: ${side} answered ${result.status} instead of 200, ${NOTHING_COMPARED}`); + } else if (result.status !== planned.status) { + // Every data URL comes from the base's own /api/groups, so it must answer 200, and each + // error probe must answer its own status. Two equal 404s, 500s or redirects would otherwise + // match each other and hide that nothing was read. + lines.push( + `${path}: ${side} answered ${result.status} instead of ${planned.status}, ${NOTHING_COMPARED}`, + ); } else if (result.body.length === 0) { - lines.push(`${path}: ${side} answered 200 with an empty body, ${NOTHING_COMPARED}`); + lines.push( + `${path}: ${side} answered ${result.status} with an empty body, ${NOTHING_COMPARED}`, + ); + } else if (planned.compare === 'title' && pageTitle(result.body) === null) { + lines.push(`${path}: ${side} sent no <title>, ${NOTHING_COMPARED}`); } } if (base instanceof Error || head instanceof Error) { @@ -190,9 +258,17 @@ function compareCaptured(path: string, base: Captured | Error, head: Captured | if (base.cacheControl !== head.cacheControl) { lines.push(`${path}: cache-control ${show(base.cacheControl)} != ${show(head.cacheControl)}`); } - const bodyDifference = diffJson(base.body, head.body); - if (bodyDifference !== null) { - lines.push(`${path}: body differs ${bodyDifference}`); + if (planned.compare === 'title') { + const baseTitle = pageTitle(base.body); + const headTitle = pageTitle(head.body); + if (baseTitle !== headTitle) { + lines.push(`${path}: title ${show(baseTitle)} != ${show(headTitle)}`); + } + } else { + const bodyDifference = diffJson(base.body, head.body); + if (bodyDifference !== null) { + lines.push(`${path}: body differs ${bodyDifference}`); + } } return lines; } @@ -210,22 +286,41 @@ async function readGroups(baseUrl: string): Promise<GroupLinks[]> { return groups as GroupLinks[]; } -/** The URLs to request from each server, in a stable order. */ -export function planRequests(groups: GroupLinks[]): string[] { - const paths = ['/api/groups']; +/** + * The URLs to request from each server, in a stable order: the read API for every group and chart, + * the landing page and the chart pages the screenshots use, then the error probes. + */ +export function planRequests(groups: GroupLinks[]): PlannedRequest[] { + const data = (path: string): PlannedRequest => ({ path, status: 200, compare: 'body' }); + const page = (path: string, status = 200): PlannedRequest => ({ path, status, compare: 'title' }); + const plan = [data('/api/groups')]; for (const group of groups) { const slug = encodeURIComponent(group.slug); - paths.push(`/api/group/${slug}?n=100`, `/api/group/${slug}?n=50`); + plan.push(data(`/api/group/${slug}?n=100`), data(`/api/group/${slug}?n=50`)); } const chartSlugs = [...new Set(groups.flatMap((group) => group.charts.map((c) => c.slug)))]; for (const [index, chartSlug] of chartSlugs.entries()) { const slug = encodeURIComponent(chartSlug); - paths.push(`/api/chart/${slug}?n=100`); + plan.push(data(`/api/chart/${slug}?n=100`)); if (index % FULL_HISTORY_EVERY === 0) { - paths.push(`/api/chart/${slug}?n=all`); + plan.push(data(`/api/chart/${slug}?n=all`)); } } - return paths; + plan.push(page('/')); + for (const group of groups.slice(0, CHART_PAGES)) { + const first = group.charts[0]; + if (first !== undefined) { + plan.push(page(`/chart/${encodeURIComponent(first.slug)}`)); + } + } + plan.push( + page(`/chart/${UNKNOWN_CHART_SLUG}`, 404), + { path: `/api/chart/${UNKNOWN_CHART_SLUG}?n=100`, status: 404, compare: 'body' }, + page(`/chart/${MALFORMED_SLUG}`, 404), + { path: `/api/chart/${MALFORMED_SLUG}?n=100`, status: 400, compare: 'body' }, + { path: `/api/group/${MALFORMED_SLUG}?n=100`, status: 400, compare: 'body' }, + ); + return plan; } async function mapPool<T, R>( @@ -249,10 +344,11 @@ async function mapPool<T, R>( * Whether a request fills the group cache. Those fills (`cachedGroups` and * `cachedDefaultGroupCharts` in `lib/data-cache.ts`) take one PostgreSQL advisory lock for the whole * database, and a fill that finds the lock taken fails with a 500 after a few retries. Both servers - * share one database, so two fills at once, even on different servers, can fail. + * share one database, so two fills at once, even on different servers, can fail. The landing page + * reads `cachedGroups` too. */ function fillsGroupCache(path: string): boolean { - return path === '/api/groups' || /^\/api\/group\/[^?]+\?n=100$/.test(path); + return path === '/' || path === '/api/groups' || /^\/api\/group\/[^?]+\?n=100$/.test(path); } /** Runs one piece of work at a time, in the order requested. */ @@ -271,12 +367,13 @@ export async function compareApiDetailed( headUrl: string, options: CompareOptions = {}, ): Promise<ApiComparison> { - const paths = planRequests(await readGroups(baseUrl)); + const plan = planRequests(await readGroups(baseUrl)); const oneFillAtATime = serialQueue(); const statuses: Record<string, number> = {}; const bytes = { base: 0, head: 0 }; let done = 0; - const lines = await mapPool(paths, options.concurrency ?? DEFAULT_CONCURRENCY, async (path) => { + const lines = await mapPool(plan, options.concurrency ?? DEFAULT_CONCURRENCY, async (planned) => { + const { path } = planned; const fetchBoth = async (): Promise<[Captured | Error, Captured | Error]> => fillsGroupCache(path) ? oneFillAtATime(async () => [ @@ -293,18 +390,20 @@ export async function compareApiDetailed( statuses[status] = (statuses[status] ?? 0) + 1; bytes[side] += result instanceof Error ? 0 : Buffer.byteLength(result.body); } - options.onProgress?.(++done, paths.length); - return compareCaptured(path, base, head); + options.onProgress?.(++done, plan.length); + return compareCaptured(planned, base, head); }); - return { requests: paths.length, mismatches: lines.flat(), statuses, bytes }; + return { requests: plan.length, mismatches: lines.flat(), statuses, bytes }; } /** - * Request the read API from both servers and compare status, `content-type`, `cache-control` and - * body: `/api/groups`; `/api/group/{slug}` at `?n=100` and `?n=50` for every group; - * `/api/chart/{slug}?n=100` for every chart; and `/api/chart/{slug}?n=all` for every 25th chart. - * The group and chart slugs come from the base server's `/api/groups`, so every URL must answer 200 - * with a body on both servers: anything else is a mismatch, even when both servers agree on it. + * Request the planned URLs ({@link planRequests}) from both servers and compare status, + * `content-type`, `cache-control`, and the body (API) or the `<title>` (pages): `/api/groups`; + * `/api/group/{slug}` at `?n=100` and `?n=50` for every group; `/api/chart/{slug}?n=100` for every + * chart; `/api/chart/{slug}?n=all` for every 25th chart; `/` and the first chart page of each of the + * first five groups; and an unknown and a malformed slug. The group and chart slugs come from the + * base server's `/api/groups`, so each of those URLs must answer 200 with a body on both servers, + * and each error probe its own status: anything else is a mismatch, even when both servers agree. * * @returns one line per mismatch; empty when the servers agree. */ @@ -335,7 +434,7 @@ if (import.meta.main) { .join(', '); const megabytes = (count: number) => `${(count / 1_000_000).toFixed(1)} MB`; console.log( - `API: ${requests} URLs on each server, ${mismatches.length} mismatches; ` + + `HTTP: ${requests} URLs on each server, ${mismatches.length} mismatches; ` + `statuses ${tally}; bodies ${megabytes(bytes.base)} from base, ${megabytes(bytes.head)} from head; ${seconds}s`, ); process.exit(mismatches.length === 0 ? 0 : 1); diff --git a/bench/scripts/parity/playwright.config.ts b/bench/scripts/parity/playwright.config.ts index d219cbe..1302b0f 100644 --- a/bench/scripts/parity/playwright.config.ts +++ b/bench/scripts/parity/playwright.config.ts @@ -14,6 +14,8 @@ function requiredEnv(name: string): string { } const out = requiredEnv('PARITY_OUT'); +const DESKTOP = { viewport: { width: 1440, height: 900 } }; +const PHONE = { viewport: { width: 390, height: 844 }, isMobile: true, hasTouch: true }; /** * Pixel comparison of the pages of one bench server. `run.sh` runs it twice: once against the base @@ -38,17 +40,17 @@ export default defineConfig({ use: { baseURL: requiredEnv('BASE_URL'), browserName: 'chromium', - colorScheme: 'light', deviceScaleFactor: 1, locale: 'en-US', reducedMotion: 'reduce', timezoneId: 'UTC', }, + // Each page in both themes. bench follows `prefers-color-scheme` when no theme is stored, and + // the screenshots store none. projects: [ - { name: 'desktop', use: { viewport: { width: 1440, height: 900 } } }, - { - name: 'phone', - use: { viewport: { width: 390, height: 844 }, isMobile: true, hasTouch: true }, - }, + { name: 'desktop', use: { ...DESKTOP, colorScheme: 'light' } }, + { name: 'phone', use: { ...PHONE, colorScheme: 'light' } }, + { name: 'desktop-dark', use: { ...DESKTOP, colorScheme: 'dark' } }, + { name: 'phone-dark', use: { ...PHONE, colorScheme: 'dark' } }, ], }); diff --git a/bench/scripts/parity/run.sh b/bench/scripts/parity/run.sh index 91aa114..44d70a0 100755 --- a/bench/scripts/parity/run.sh +++ b/bench/scripts/parity/run.sh @@ -3,7 +3,8 @@ # SPDX-FileCopyrightText: Copyright the Vortex contributors # # Serve two builds of bench against one restored production database, then compare their API -# responses and their pages. Exits 0 when nothing differs. +# responses, their pages' status, headers and titles, and screenshots of the pages in the light and +# dark themes. Exits 0 when nothing differs. # # bash bench/scripts/parity/run.sh <base-ref> <dump-dir> # @@ -262,10 +263,10 @@ wait_ready head "$HEAD_URL" "${SERVER_PIDS[1]}" rm -rf "$OUT" mkdir -p "$OUT" -log "comparing the API" +log "comparing the API and the pages' status, headers and titles" api_status=0 bun "$HERE/compare-api.ts" "$BASE_URL" "$HEAD_URL" 2>&1 | tee "$OUT/api.log" || api_status=$? -api_mismatches=$(sed -n 's/^API: .* \([0-9][0-9]*\) mismatches.*/\1/p' "$OUT/api.log") +api_mismatches=$(sed -n 's/^HTTP: .* \([0-9][0-9]*\) mismatches.*/\1/p' "$OUT/api.log") api_mismatches=${api_mismatches:-"unknown"} run_playwright() { @@ -295,7 +296,7 @@ echo "parity: $(tail -n 1 "$OUT/api.log")" echo "parity: $shots_total screenshots, $shots_mismatches differ" echo "parity: wall time $((SECONDS / 60))m $((SECONDS % 60))s" if [[ $api_status -ne 0 ]]; then - echo "parity: API differences are listed above and in $OUT/api.log" + echo "parity: response differences are listed above and in $OUT/api.log" fi if [[ $shots_status -ne 0 ]]; then echo "parity: screenshots that differ; each folder holds the expected, actual and diff images:" @@ -307,5 +308,5 @@ if [[ $api_status -eq 0 && $shots_status -eq 0 ]]; then echo "parity: 0 mismatches" exit 0 fi -echo "parity: FAILED (API mismatches: $api_mismatches, screenshots that differ: $shots_mismatches)" +echo "parity: FAILED (response mismatches: $api_mismatches, screenshots that differ: $shots_mismatches)" exit 1 From a810be660c8075324967d103f55f88d346264e6a Mon Sep 17 00:00:00 2001 From: Will Manning <will@willmanning.io> Date: Thu, 29 Oct 2026 00:50:09 +0100 Subject: [PATCH 16/19] Correct the linker and JSX comments for the shared toolchain bunfig.toml gave a reason for the isolated linker that no longer holds, now that both apps share TypeScript and ESLint is gone. State the reason that does hold. vitest.config.ts said Next.js owns tsconfig's jsx setting; it now comes from tsconfig.base.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Will Manning <will@willmanning.io> --- bench/web/vitest.config.ts | 9 +++++---- bunfig.toml | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/bench/web/vitest.config.ts b/bench/web/vitest.config.ts index 23f342d..8eaf4bb 100644 --- a/bench/web/vitest.config.ts +++ b/bench/web/vitest.config.ts @@ -5,10 +5,11 @@ import { fileURLToPath } from 'node:url'; import { defineConfig } from 'vitest/config'; export default defineConfig({ - // Next.js owns tsconfig's `jsx` setting (it was `"preserve"` before Next.js 16, - // and Vite (rolldown/oxc) leaves preserved JSX untransformed, which fails the - // component tests). Transform JSX with the React automatic runtime for tests - // explicitly (no `import React` needed; pulls from `react/jsx-runtime`). + // Transform JSX with the React automatic runtime for tests explicitly (no + // `import React` needed; pulls from `react/jsx-runtime`), so the component + // tests do not depend on tsconfig's `jsx` setting. That comes from the shared + // `tsconfig.base.json` (`"react-jsx"`); it was `"preserve"` before Next.js 16, + // and Vite (rolldown/oxc) leaves preserved JSX untransformed. // Vite's transform option is `oxc`, not `esbuild`, under rolldown-vite; the // typed form of the automatic runtime is the `{ runtime: 'automatic' }` // object (the bare `'react-jsx'` string is runtime-valid but not in the type). diff --git a/bunfig.toml b/bunfig.toml index d577972..aa8dee9 100644 --- a/bunfig.toml +++ b/bunfig.toml @@ -1,10 +1,11 @@ # Bun configuration. Project-scoped — Bun merges this with $HOME/.bunfig.toml. [install] -# Give every package its own resolved dependencies, as pnpm did for bench, -# instead of one hoisted tree, so one app's tools never load the other's copy -# of a package. The site and bench/web share Next.js, React and TypeScript at -# the same versions. This is also Bun's default for a workspace. +# Link only the dependencies each package declares, as pnpm did for bench, +# instead of hoisting every package to the top of node_modules: no code can +# load a transitive dependency that its own package does not declare. This is +# also Bun's default for a workspace; it is set here so that the install +# layout does not change with that default. linker = "isolated" # Refuse to install package versions younger than 14 days. Mirrors the From 83d27f95c63dfb8653fc932472db9e42a79614b3 Mon Sep 17 00:00:00 2001 From: Will Manning <will@willmanning.io> Date: Thu, 29 Oct 2026 00:54:39 +0100 Subject: [PATCH 17/19] Cache Bun's packages and test the parity check in bench CI bench-checks lost its dependency cache when it moved from pnpm to Bun. Restore it with the same cache step as ci.yml. Run the parity check's own tests there too, by explicit path, so its planning and comparison logic is checked on every bench change. Correct a comment that said Biome checks bench's imports; import sorting is off for bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Will Manning <will@willmanning.io> --- .github/actions/bench-checks/action.yml | 34 +++++++++++++++++++------ 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/.github/actions/bench-checks/action.yml b/.github/actions/bench-checks/action.yml index 65e50ef..695bda0 100644 --- a/.github/actions/bench-checks/action.yml +++ b/.github/actions/bench-checks/action.yml @@ -1,8 +1,8 @@ name: Bench checks description: >- - bench's full check suite: infrastructure trust policies, format, lint, build, - the migration-runner and deployment-verification tests, and the web tests with - their Postgres integration suite. Bench CI runs it on every change to bench, + bench's full check suite: infrastructure trust policies, the parity check's + tests, format, lint, build, the migration-runner and deployment-verification + tests, and the web tests with their Postgres integration suite. Bench CI runs it on every change to bench, and bench-deploy runs it on the exact commit it deploys. Run it after a checkout. @@ -31,17 +31,35 @@ runs: # @types/node in bench/web/package.json on the same major. node-version-file: .nvmrc + # Bun's package store, keyed on the lockfile, as in ci.yml. The restore + # key falls back to any earlier cache from this OS. + - name: Cache Bun install cache + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: ~/.bun/install/cache + key: ${{ runner.os }}-bun-${{ hashFiles('bun.lock') }} + restore-keys: | + ${{ runner.os }}-bun- + # At the root, where bun.lock and bunfig.toml live. This installs both apps. - name: Install dependencies shell: bash run: bun install --frozen-lockfile + # The parity check's own logic: request planning, the expected statuses + # and the comparisons. The explicit path keeps `bun test` from also + # collecting bench's Vitest files. + - name: Test the parity check + shell: bash + run: bun test bench/scripts/parity/compare-api.test.ts + # Biome over all of bench (web, scripts, parity harness) with the root - # biome.json, in CI mode: format, lint and import checks, no writes. It - # replaces the bench/web `format:check` and `lint` scripts, which cover only - # bench/web and would leave bench/scripts unchecked on bench-only changes, - # where ci.yml skips the root `check:ci`. `bun run` uses the workspace's - # pinned Biome and never downloads one. + # biome.json, in CI mode: format and lint, no writes. Import sorting is off + # for bench (see bench/AGENTS.md). It replaces the bench/web `format:check` + # and `lint` scripts, which cover only bench/web and would leave + # bench/scripts unchecked on bench-only changes, where ci.yml skips the + # root `check:ci`. `bun run` uses the workspace's pinned Biome and never + # downloads one. - name: Format and lint shell: bash run: bun run biome ci bench From bfd712c02ef49a206c1a14d33afdaf0e1b30aa68 Mon Sep 17 00:00:00 2001 From: Will Manning <will@willmanning.io> Date: Thu, 29 Oct 2026 00:54:55 +0100 Subject: [PATCH 18/19] Fail dependency review when the bun.lock snapshot is missing Dependency review passed without bun.lock's packages when the snapshot job failed, was skipped for a fork, or finished after the action's retry window. A guard now runs before the action. For a pull request from this repository, it waits up to 10 minutes for the snapshot job on the head commit and fails unless that job succeeded. For a fork, which gets no snapshot, it fails when the pull request changes a bun.lock. The job gains checks: read, and its checkout no longer keeps the token. The snapshot job now checks out the pull request's head commit, so the graph it files under that commit holds the head's own bun.lock. CLAUDE.md describes the guard, says that development-scope advisories (bench's included) no longer block merge and show in bun audit, and gives the real fallback for the zod override. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Will Manning <will@willmanning.io> --- .github/workflows/ci.yml | 76 +++++++++++++++++++++-- .github/workflows/dependency-snapshot.yml | 14 +++-- CLAUDE.md | 9 +-- 3 files changed, 86 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5645d65..d2956fe 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -228,19 +228,87 @@ jobs: # (severity `high` and above) rather than the whole pre-existing tree. # # GitHub's dependency graph does not read bun.lock. The dependency-snapshot - # workflow submits it, and that run races this job, so the retry inputs - # make this one wait up to five minutes for the snapshot instead of - # reviewing a graph without the transitive packages. `fail-on-scopes` stays - # at the action's default, `runtime`: a development-only advisory does not + # workflow submits it, and the action would pass without it, so a guard + # runs first (see its comment). The retry inputs then cover the short time + # GitHub takes to process a submitted snapshot. `fail-on-scopes` stays at + # the action's default, `runtime`: a development-only advisory does not # block merge and stays with `bun audit` and CLAUDE.md "Audit advisories". dependency-review: if: github.event_name == 'pull_request' runs-on: ubuntu-24.04 permissions: contents: read + # Implies read, which the guard uses to list a fork's changed files. pull-requests: write + # The guard reads the snapshot job's check run. + checks: read steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + # The action reviews whatever graph it finds. If the snapshot job + # failed, was skipped or finished late, that graph lacks bun.lock's + # packages and the review passes without them. So: for a pull request + # from this repository, wait for that job on the head commit and fail + # unless it succeeded. A fork's pull request gets no snapshot (its token + # cannot submit one), so fail it if it changes a bun.lock; if it does + # not, it adds no locked package. + - name: Require the bun.lock snapshot + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + PR_NUMBER: ${{ github.event.pull_request.number }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + CHANGED_FILES: ${{ github.event.pull_request.changed_files }} + FROM_FORK: ${{ github.event.pull_request.head.repo.full_name != github.repository }} + # The `name:` of the job in dependency-snapshot.yml. + SNAPSHOT_CHECK: Submit bun.lock + WAIT_SECONDS: 600 + run: | + if [ "$FROM_FORK" = true ]; then + files=$(gh api --paginate "repos/$REPO/pulls/$PR_NUMBER/files?per_page=100" --jq '.[].filename') + # The API lists at most 3000 files; a longer pull request could hide a bun.lock. + listed=$(grep -c . <<<"$files" || true) + if [ "$listed" -lt "$CHANGED_FILES" ]; then + echo "::error::Listed $listed of the pull request's $CHANGED_FILES changed files, so a bun.lock change could go unseen." + exit 1 + fi + if grep -Eq '(^|/)bun\.lock$' <<<"$files"; then + echo "::error::This pull request comes from a fork and changes bun.lock. A fork gets no dependency snapshot, so dependency review cannot see the locked packages. Make the lock change from a branch in this repository." + exit 1 + fi + echo "Fork pull request without a bun.lock change: it adds no locked package." + exit 0 + fi + + name=$(jq -rn --arg name "$SNAPSHOT_CHECK" '$name | @uri') + deadline=$((SECONDS + WAIT_SECONDS)) + while :; do + # The newest run of the job (highest id), so a successful rerun counts. + if run=$(gh api "repos/$REPO/commits/$HEAD_SHA/check-runs?check_name=$name&per_page=100" \ + --jq '(.check_runs | max_by(.id)) // {} | "\(.status // "missing") \(.conclusion // "none")"'); then + read -r status conclusion <<<"$run" + if [ "$status" = completed ]; then + if [ "$conclusion" = success ]; then + echo "The '$SNAPSHOT_CHECK' job submitted the snapshot for $HEAD_SHA." + exit 0 + fi + echo "::error::The '$SNAPSHOT_CHECK' job concluded '$conclusion', so dependency review would not see bun.lock's packages. Rerun it, then rerun this job." + exit 1 + fi + echo "'$SNAPSHOT_CHECK' on $HEAD_SHA: $status" + else + echo "::warning::Could not read the check runs of $HEAD_SHA; trying again." + fi + if [ "$SECONDS" -ge "$deadline" ]; then + echo "::error::The '$SNAPSHOT_CHECK' job did not finish on $HEAD_SHA within $WAIT_SECONDS seconds." + exit 1 + fi + sleep 15 + done + - name: Dependency Review uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0 with: diff --git a/.github/workflows/dependency-snapshot.yml b/.github/workflows/dependency-snapshot.yml index 6c627ad..4d0686a 100644 --- a/.github/workflows/dependency-snapshot.yml +++ b/.github/workflows/dependency-snapshot.yml @@ -28,8 +28,11 @@ jobs: snapshot: name: Submit bun.lock # Pull requests from forks get a read-only token, which the submission API - # rejects, so they are skipped. Dependency review waits out its retry - # window for such a pull request, then proceeds with a snapshot warning. + # rejects, so they are skipped. Dependency review then fails such a pull + # request if it changes bun.lock; with bun.lock unchanged, it adds no + # locked package. For every other pull request, dependency review waits + # for this job by its name and fails unless it succeeded, so rename both + # together. if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }} runs-on: ubuntu-24.04 timeout-minutes: 5 @@ -37,11 +40,12 @@ jobs: # The submission API wants `contents: write`; nothing else. contents: write steps: - # A pull request run checks out the merge commit: the lock that would - # land, filed below under the head commit that dependency-review - # compares. + # A pull request run checks out the head commit, not the merge commit, + # so the snapshot it files under that head commit (below) holds the + # head's own bun.lock, the one dependency-review compares with main's. - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} persist-credentials: false - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 diff --git a/CLAUDE.md b/CLAUDE.md index bbe4ad2..6e7d894 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -83,8 +83,9 @@ Defense in depth against the npm-worm class (Shai-Hulud, mini-Shai-Hulud, the Ma - **Document and accept** — if no upstream fix exists yet, add an entry under "Audit advisories" below with: GHSA ID, vulnerable range, package, why exposure is acceptable (e.g. dev-only, not in client bundle), and a removal trigger. The job carries no `--ignore` flags: with nothing to keep green, suppressing an advisory only hides it, so this file is where an accepted advisory is recorded. - When a parent dep eventually patches its own transitive, drop the corresponding `overrides` entry — leaving stale overrides means we keep deduping a fix that was already merged upstream. 5. **`actions/dependency-review-action`** runs on every PR and **hard-fails** the build if the PR introduces a new advisory at severity `high` or above in a runtime-scope package. With `bun audit` informational, this is the only advisory check that blocks merge — deliberately scoped to what the PR itself adds, not the whole pre-existing tree. - - **It sees `bun.lock` through a submitted snapshot.** GitHub's dependency graph does not read `bun.lock`, so `.github/workflows/dependency-snapshot.yml` runs `scripts/dependency-snapshot.ts` on every push to main and every same-repository PR. It posts one manifest per workspace to the Dependency Submission API, listing every package that workspace reaches with a scope: `runtime` if a workspace's `dependencies` or `optionalDependencies` reach it, `development` otherwise. An optional peer, like Next.js's Playwright, does not pull its package into `runtime`. `dependency-review` waits up to 300 seconds for the snapshot (`retry-on-snapshot-warnings`). A fork PR gets no snapshot, because its token cannot write, so its review proceeds with a snapshot warning. - - **It fails on runtime scope only.** `fail-on-scopes` is left at the action's default, `runtime` (`action.yml` declares no default; the action's schema and README give `runtime`). A development-only advisory, such as the `@lhci/cli` tree, does not block merge: it stays in `bun audit` and the "Audit advisories" list below. Runtime means reachable from `dependencies`, which includes build-time tools the site lists there (velite, tailwindcss, `@tailwindcss/typography`). + - **It sees `bun.lock` through a submitted snapshot.** GitHub's dependency graph does not read `bun.lock`, so `.github/workflows/dependency-snapshot.yml` runs `scripts/dependency-snapshot.ts` on every push to main and on the head commit of every same-repository PR. It posts one manifest per workspace to the Dependency Submission API, listing every package that workspace reaches with a scope: `runtime` if a workspace's `dependencies` or `optionalDependencies` reach it, `development` otherwise. An optional peer, like Next.js's Playwright, does not pull its package into `runtime`. + - **A guard fails the review when the snapshot is missing.** The action reviews whatever graph it finds, so without the snapshot it would pass without `bun.lock`'s packages. Before the action, the job's "Require the bun.lock snapshot" step waits up to 10 minutes for the snapshot job (`Submit bun.lock`) on the PR's head commit, and fails unless that job succeeded. A fork PR gets no snapshot, because its token cannot write, so the guard fails a fork PR that changes `bun.lock` (make that change from a branch in this repository) and passes one that does not, because it adds no locked package. After the guard, the action still waits up to 300 seconds (`retry-on-snapshot-warnings`) while GitHub processes the snapshot. If you rename the snapshot job, change the guard's `SNAPSHOT_CHECK` with it. + - **It fails on runtime scope only.** `fail-on-scopes` is left at the action's default, `runtime` (`action.yml` declares no default; the action's schema and README give `runtime`). An advisory in a `development`-scope package does not block merge. That includes bench's development tools (Vitest, testcontainers, jsdom and what they pull in), whose advisories blocked bench PRs under pnpm. `bun audit` is where those advisories show: it reports both apps' whole trees on every run, and the "Audit advisories" list below records the accepted ones. Runtime means reachable from `dependencies`, which includes build-time tools the site lists there (velite, tailwindcss, `@tailwindcss/typography`). 6. **`trufflesecurity/trufflehog`** secret scan runs on every PR and push to main. Defense-in-depth on top of GitHub's push protection; catches verified secrets that slipped past push protection (low-entropy formats, detector patterns added after the secret was committed, or push-protection bypass). 7. **OpenSSF Scorecard** (`.github/workflows/scorecard.yml`) grades the repo weekly on supply-chain hygiene (pinned actions, branch protection, token permissions, dangerous workflow patterns). SARIF posts to the Security tab; aggregate score publishes to https://scorecard.dev. 8. **All third-party GitHub Actions are SHA-pinned**, not tag-pinned. A tag can be moved to point at a malicious commit (and has been, in prior supply-chain incidents); a SHA can't. Renovate's `helpers:pinGitHubActionDigests` preset enforces this on auto-bump PRs. When bumping an action manually, update both the SHA and the trailing `# vX.Y.Z` comment in the same diff. @@ -121,7 +122,7 @@ Keep them in lockstep when bumping. Raising `.nvmrc` alone makes CI validate aga - **sharp `<0.35.5`** (GHSA-wq5f-xc86-pv6w, high — librsvg CVE-2026-96889; also GHSA-rgj7-g3m4-5g8c `<0.35.4` and GHSA-f88m-g3jw-g9cj `<0.35.0`). `next@16.4.1` asks for `sharp@^0.35.5`, but `velite@0.4.0` declares `sharp@^0.34.5`, a range that cannot reach 0.35.x, so the tree would carry a vulnerable second copy. `overrides.sharp = "^0.35.5"` pulls velite onto the patched line and dedupes to one copy. Crossing 0.34 → 0.35 is a breaking bump on sharp's 0.x line; verified safe because velite's image pipeline runs during `bun run build` and the build passes. Drop the override once velite ships a release declaring `sharp@^0.35.5` or later — 0.4.0 is still the latest. -- **zod `<4.6.6`** (GHSA-mrhx-c7xc-m3w5, high memory exhaustion on oversized constrained collections). Two paths: bench's `@vercel/functions > @vercel/oidc > @vercel/cli-config` pins `zod@4.1.11` exactly, and the site's `@lhci/cli > lighthouse > puppeteer-core > chromium-bidi` declares `zod@^3.24.1`, a line with no fix. `overrides.zod = "^4.6.7"` puts both on 4.6.7. bench only imports `attachDatabasePool`, which does not load the OIDC code that uses zod. For chromium-bidi the override crosses zod 3 → 4 under a library we do not control, on purpose, and the Lighthouse CI job must keep passing. chromium-bidi still parses valid WebDriver BiDi messages under zod 4, but its error path reads zod 3's `errors` field, so an invalid message throws a TypeError instead of a protocol error. puppeteer-core loads chromium-bidi only for the BiDi protocol; lhci drives Chrome over CDP. If it breaks, drop the override and accept bench's zod 4.1.11 instead (it parses only local Vercel CLI config files). Drop the override once `@vercel/cli-config` and chromium-bidi both accept zod 4.6.6 or later. +- **zod `<4.6.6`** (GHSA-mrhx-c7xc-m3w5, high memory exhaustion on oversized constrained collections). Two paths: bench's `@vercel/functions > @vercel/oidc > @vercel/cli-config` pins `zod@4.1.11` exactly, and the site's `@lhci/cli > lighthouse > puppeteer-core > chromium-bidi` declares `zod@^3.24.1`, a line with no fix. `overrides.zod = "^4.6.7"` puts both on 4.6.7. bench only imports `attachDatabasePool`, which does not load the OIDC code that uses zod. For chromium-bidi the override crosses zod 3 → 4 under a library we do not control, on purpose, and the Lighthouse CI job must keep passing. chromium-bidi still parses valid WebDriver BiDi messages under zod 4, but its error path reads zod 3's `errors` field, so an invalid message throws a TypeError instead of a protocol error. puppeteer-core loads chromium-bidi only for the BiDi protocol; lhci drives Chrome over CDP. If it breaks, drop the override, accept bench's zod 4.1.11 (it parses only local Vercel CLI config files), and record it under "Accepted" here. The snapshot scopes bench's zod `runtime`, because `@vercel/functions` is in bench's `dependencies`, so dependency review would block the PR that drops the override: add `allow-ghsas: GHSA-mrhx-c7xc-m3w5` to the Dependency Review step in `ci.yml` in the same PR. Drop the override once `@vercel/cli-config` and chromium-bidi both accept zod 4.6.6 or later. **Accepted — fix exists but is inside the 14-day cooldown.** These clear on their own: Renovate's `lockFileMaintenance` (or a manual `bun update`) picks up the patched version once it is 14 days old, because each parent's range already admits it. @@ -143,7 +144,7 @@ The `audit` job in `.github/workflows/ci.yml` runs a bare `bun audit` and never #### bench advisories -`bun audit` at the root reports bench's advisories with the site's. Dependency review sees bench's packages through the snapshot's `bun.lock (bench/web)` manifest, scoped like the site's (layer 5). State as of 2026-10-28: `bun audit` reports two for bench, both moderate. bench's move to Next.js 16.4.1 (postcss 8.5.23) and vitest 5.0.4 cleared the other six. +`bun audit` at the root reports bench's advisories with the site's. Dependency review sees bench's packages through the snapshot's `bun.lock (bench/web)` manifest, scoped like the site's (layer 5), so an advisory in one of bench's development tools no longer blocks a PR. State as of 2026-10-28: `bun audit` reports two for bench, both moderate. bench's move to Next.js 16.4.1 (postcss 8.5.23) and vitest 5.0.4 cleared the other six. - **Accepted, dev-only — protobufjs 7.6.2** (GHSA-f38q-mgvj-vph7, fixed in 7.6.3, and GHSA-j3f2-48v5-ccww, fixed in 7.6.5; both moderate; via testcontainers > dockerode > @grpc/proto-loader). The move to Bun kept pnpm's version. It runs only in bench's tests. Drop this entry when a lockfile refresh reaches protobufjs 7.6.5. - **Resolved by `overrides` in the root `package.json`:** zod (see "Audit advisories"). sharp needs no override for bench: next 16.4.1 asks for `^0.35.5` itself. From f92f33abfeef338d4c82a00136a34618339eedc5 Mon Sep 17 00:00:00 2001 From: Will Manning <will@willmanning.io> Date: Thu, 29 Oct 2026 00:54:55 +0100 Subject: [PATCH 19/19] Correct two bench notes in CLAUDE.md Give the publish times behind the statement that next 16.4.1 and react 19.3.1 were past the cooldown when bench moved to them, and say how the "13 days old" for the site differs. Rewrite the vite note, which still described pnpm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Will Manning <will@willmanning.io> --- CLAUDE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6e7d894..23ecbcf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -148,6 +148,6 @@ The `audit` job in `.github/workflows/ci.yml` runs a bare `bun audit` and never - **Accepted, dev-only — protobufjs 7.6.2** (GHSA-f38q-mgvj-vph7, fixed in 7.6.3, and GHSA-j3f2-48v5-ccww, fixed in 7.6.5; both moderate; via testcontainers > dockerode > @grpc/proto-loader). The move to Bun kept pnpm's version. It runs only in bench's tests. Drop this entry when a lockfile refresh reaches protobufjs 7.6.5. - **Resolved by `overrides` in the root `package.json`:** zod (see "Audit advisories"). sharp needs no override for bench: next 16.4.1 asks for `^0.35.5` itself. -- **Resolved in the lockfile:** @grpc/grpc-js, brace-expansion, js-yaml and source-map-js, by refreshing within their parents' ranges. vite is a direct dev dependency (`^8.3.4`): it is vitest's peer, and pnpm kept the vulnerable locked 8.0.16 even with an override. -- **Cooldown exemptions:** `undici` in `bunfig.toml`, until 2026-10-29 15:06 UTC (see "Cooldown exemptions" above). next 16.4.1, its `@next/*` packages and react 19.3.1 were already past the window when bench moved to them. +- **Resolved in the lockfile:** @grpc/grpc-js, brace-expansion, js-yaml and source-map-js, by refreshing within their parents' ranges. vite is a direct dev dependency of bench (`^8.3.4`), because it is vitest's peer: as a direct dependency its range sets the version the lockfile holds, so the vulnerable 8.0.16 cannot come back. (Under pnpm, an override alone left 8.0.16 locked.) +- **Cooldown exemptions:** `undici` in `bunfig.toml`, until 2026-10-29 15:06 UTC (see "Cooldown exemptions" above). next 16.4.1, its `@next/*` packages and react 19.3.1 were already past the window when bench moved to them, at 2026-10-28 22:31 UTC: npm published next and its `@next/*` packages on 2026-10-14 between 17:28 and 17:43 UTC, and react and react-dom at 16:47 and 16:49 UTC. The "13 days old" in "Cooldown exemptions" above was their age when the site adopted them. - **Accepted, the Vercel CLI (64.0.2) in `bench-deploy.yml`:** tar (one critical), undici, path-to-regexp, minimatch, js-yaml, smol-toml, zod and @fastify/busboy. The CLI and its framework builders pin these exactly; fixing undici alone means a 5 → 6 major inside the CLI's HTTP client. It runs only in CI, on our own build output. It is installed without a lockfile, so dependency review never sees it, and these advisories live only here. Re-audit when Renovate bumps `VERCEL_CLI_VERSION`: install it with `npm install --package-lock-only vercel@<version>` in a scratch directory and run `npm audit`.