[pyramation]

Summary

Fixes #170: release builds now record exactly what they were built from, and nothing floats. Build config and scripts only: no wrapper code, proto or generated-type changes.

What was floating before

• LIBPG_QUERY_TAG for v15/17/18 named a branch of constructive-io/libpg_query, so the source could change without any commit here.
• wasm:make ran emscripten/emsdk with no tag. Today that pulls 6.0.13, not the 6.0.3 the releases were actually built with.
• copy:templates hard-coded the pganalyze repo URL. Running it switched 15/17/18 back to pganalyze sources and removed v18's WASM_* memory knobs (REPO_NOTES.md warned about this).

Change

• x-publish in each versions/*/package.json now carries libpgQueryRepo, libpgQueryTag and libpgQueryCommit (a full SHA, enforced by copy:templates).
• Makefiles are generated from x-publish again. The build clones the branch, runs git checkout $(LIBPG_QUERY_COMMIT), and keys the source cache by commit, so a stale cached clone can't be reused.
• wasm:make now uses emscripten/emsdk:6.0.3 in all versions.
• The WASM_INITIAL_MEMORY / WASM_MAXIMUM_MEMORY / WASM_STACK_SIZE overrides move into the template, so every version gets them. Defaults are unchanged (128 MiB / 1 GiB / 32 MiB).
• x-publish ships in the published package.json, so the source commit is recorded in npm metadata next to gitHead.

Build-script hardening

• fetch:protos downloads pg_query.proto from libpgQueryRepo@libpgQueryCommit, the same source the WASM builds from. Before, it built a pganalyze tag URL, which 404s for the fork branch names. It now writes the file only after a complete 200 and exits 1 on failure. On main, a 404 truncated the committed .proto to 0 bytes and the script still exited 0.
• The clone, the checkout and (v13) the spinlock patch run as one recipe step, and the clone directory is deleted if any of them fails:

  git clone ... && git -C $(LIBPG_QUERY_DIR) checkout $(LIBPG_QUERY_COMMIT) $(LIBPG_QUERY_PATCH) || \
    { rm -rf $(LIBPG_QUERY_DIR); exit 1; }
  # v13 only, under EMSCRIPTEN:
  LIBPG_QUERY_PATCH := && patch -d $(LIBPG_QUERY_DIR) -p1 < $(CURDIR)/patches/emscripten_disable_spinlocks.patch

  Before, the patch was a separate line after the directory existed, so a failed or interrupted patch left an unpatched clone. Every later build reused it and failed with unknown type name 'slock_t', which happened on a local v13 build.
• copy:templates no longer swallows x-publish validation errors, so a bad pin exits 1.

Pins are the current branch heads, so a build today produces the same thing it did before this PR:

v	repo	branch/tag	commit
13	pganalyze	13-2.2.0	1097b2c
14	pganalyze	14-3.0.0	397cbb9
15	constructive-io	fix/negative-int-pg15	ae10504
16	pganalyze	16-5.2.0	fce106a
17	constructive-io	17-constructive	1ade880
18	constructive-io	18-constructive	b0f5f33

Existing local caches are keyed by tag, so after this PR every version clones fresh once.

WASM hashes

• publish-single-version.js writes x-publish.wasmSha256 (the sha256 of wasm/libpg-query.wasm and wasm/libpg-query.js) into the package.json it publishes, so anyone can rebuild from the pins and compare. If wasm/libpg-query.wasm is missing it exits 1. Building stays optional. Tested with npm pack on v17: the hashes in the packed package.json match sha256sum of the files in the tarball.
• Rebuilding all 6 versions from main and from this branch with emsdk 6.0.3 gives byte-identical .wasm and .js output, so this PR changes no binary or memory setting.

Docs

• PUBLISH.md gets a "Keeping the C source in sync" section: how to move a pin, what "skip build" means for the pins, and how to check x-publish after publishing. It also drops the latest/legacy parser build types, which no longer exist, and the root build:parser:legacy script, which failed with "Invalid build type".

Not changed here: the committed protos/17/pg_query.proto predates the v17 pin. Running fetch:protos adds upstream's SummaryResult message, and regenerating types from it would add stray Table/Function/FilterColumn node types to @pgsql/types. So the proto is left as it is.

Evidence

• Running pnpm run copy:templates on this branch produces no diff.
• pnpm run wasm:build on this branch reproduces the published npm packages byte-for-byte:
  • v18 vs [email protected]: wasm 378b64e6…, js 219217cc…
  • v17 vs [email protected]: wasm 06fbfc24…
• fetch:protos on this branch: v13/14/15/16/18 come out byte-identical to the committed protos. v17 only gains the fork's SummaryResult message. That file is not committed here, and no types are regenerated.
• Each failure path was triggered by hand:
  • a short pin makes copy:templates exit 1;
  • an all-zero commit makes fetch:protos exit 1 with the committed proto intact;
  • a bad LIBPG_QUERY_COMMIT makes make exit non-zero, and the clone directory is removed.
• Patch failure, using a v13 patch that doesn't apply and make EMSCRIPTEN=1 <clone dir>:
  • before: make exits 2, the clone is left behind, and the next make says "is up to date" (exit 0) without patching;
  • after: make exits 2 and the clone is removed. With the real patch, make exits 0 and s_lock.h has the __sync_lock_test_and_set fallback.
• After merging main (fix(wasm): replace the module after an exception escapes a WASM call #175): a clean v13 build with emsdk:6.0.3 passes pnpm test (72 pass, 0 fail, 20 skipped). v17 and v18 tests pass locally.

Link to Devin session: https://app.devin.ai/sessions/bcd8b2f791164247acb7a659b481efc2
Open in Devin Desktop: https://app.devin.ai/desktop/session/bcd8b2f791164247acb7a659b481efc2?variant=devin
Requested by: @pyramation

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[tenki-reviewer]

Review complete. 🟡 3 medium

💬 Inline comments (1)

• 🟡 Failed pin checkout leaves poisoned, unpinned cache dir — Makefile.template:50

📍 Findings outside the diff (2) — 🟡 2 medium — defects on lines GitHub can't attach comments to

🟡 Medium — Pin validation throw is swallowed; script exits 0 · copy-templates.js:39–56 · unchanged line

// scripts/copy-templates.js
39	      if (version && libpgQueryTag) {
40	        if (!libpgQueryRepo || !/^[0-9a-f]{40}$/.test(libpgQueryCommit || '')) {
41	          throw new Error(`x-publish needs libpgQueryRepo and a full 40-char libpgQueryCommit`);
42	        }
43	        configs[version] = {
44	          tag: libpgQueryTag,
45	          repo: libpgQueryRepo,
46	          commit: libpgQueryCommit,
47	          hasEmscriptenPatch: version === '13', // Only version 13 needs the patch
48	          fullApi
49	        };
50	        console.log(`  Version ${version}: ${libpgQueryTag} @ ${libpgQueryCommit.slice(0, 7)}${fullApi ? ' (full API)' : ''}`);

The new validation throw at scripts/copy-templates.js:41 sits inside the per-file try whose catch at line 54 only logs Error reading <file> and continues, so a versions/*/package.json missing libpgQueryRepo or with a malformed libpgQueryCommit is silently dropped from VERSION_CONFIGS and the script still reports success with exit code 0. That version's checked-in generated Makefile keeps its previous (stale or absent) repo/tag/commit pin under the "DO NOT MODIFY MANUALLY" header. CI and release tooling that check only the exit code will ship artifacts built from an unpinned or outdated source without any signal.

---

🟡 Medium — fetch-protos 404s for repointed pg15/pg17 refs · fetch-protos.js:73 · file not in diff

// scripts/fetch-protos.js
73	    const url = `https://raw.githubusercontent.com/pganalyze/libpg_query/refs/tags/${libpgQueryTag}/protobuf/pg_query.proto`;

This PR repoints versions 15 and 17 to https://github.com/constructive-io/libpg_query.git with branch refs fix/negative-int-pg15 and 17-constructive (versions/15/package.json:17, versions/17/package.json:17), but scripts/fetch-protos.js:73 still builds https://raw.githubusercontent.com/pganalyze/libpg_query/refs/tags/${libpgQueryTag}/protobuf/pg_query.proto and never reads the new libpgQueryRepo. For pg15/pg17 both the host and the refs/tags/ prefix are wrong, so the download 404s and the script only logs ❌ Failed to fetch protobuf... while exiting 0, leaving stale protos in protos/15 and protos/17.

---

The change adds {{LIBPG_QUERY_REPO}}/{{LIBPG_QUERY_COMMIT}} placeholders to the Makefile template, makes copy-templates.js validate and substitute per-version repo/tag/commit triples from x-publish, and regenerates all six per-version Makefiles and package.json files with explicit pins (constructive-io fork commits for versions 15/17/18). It also documents the new pinning scheme in REPO_NOTES.md and templates/README.md. A few robustness gaps remain around failure paths of the new validation and checkout steps.

Files	Change
scripts/copy-templates.js, templates/Makefile.template, templates/README.md	Adds repo/commit placeholders, validation, and substitution to the template-copy pipeline, plus updated docs
versions/*/Makefile, versions/*/package.json	Regenerates all six versions with explicit fork repo/tag/commit source pins
REPO_NOTES.md	Documents the new commit-pinning scheme and regeneration workflow

_{Reviewed commit: 0b8713c}

[tenki-reviewer]

This PR switches libpg_query source pinning from tags to explicit fork repo/commit pins across all six version Makefiles and package.json files, with template regeneration support.

Key findings

• 🟡 Pin validation throw is swallowed; script exits 0 — copy-templates.js:39
• 🟡 Failed pin checkout leaves poisoned, unpinned cache dir — Makefile.template:50
• 🟡 fetch-protos 404s for repointed pg15/pg17 refs — fetch-protos.js:73

[tenki-reviewer]

🟡 bug · medium

Failed pin checkout leaves poisoned, unpinned cache dir

The $(LIBPG_QUERY_DIR) rule stamps the target by directory existence while the new git checkout $(LIBPG_QUERY_COMMIT) line (templates/Makefile.template:53) can fail — e.g. the commit is not reachable from the single ref fetched by git clone -b $(LIBPG_QUERY_TAG) --single-branch, or the run is interrupted. When that happens the directory already exists, so the next make considers the target up to date and silently skips clone, checkout, and the v13 emscripten patch, building against whatever revision the clone was left at instead of the pin. This defeats the byte-for-byte reproducibility the pin was added for.

📋 Prompt for AI Agents

In templates/Makefile.template lines 51-53 (then regenerate all versions/*/Makefile via npm run copy:templates), the $(LIBPG_QUERY_DIR) rule runs mkdir, git clone -b $(LIBPG_QUERY_TAG) --single-branch ..., and git checkout $(LIBPG_QUERY_COMMIT) as separate recipe lines against a directory-existence target: if the checkout fails, the directory remains and later make runs skip the whole recipe, silently building from the branch tip instead of the pin. Combine the clone and checkout into one chained recipe that removes the directory on failure (git clone ... && git -C ... checkout ... || { rm -rf $(LIBPG_QUERY_DIR); exit 1; }), and consider fetching the pinned commit explicitly (git -C $(LIBPG_QUERY_DIR) fetch origin $(LIBPG_QUERY_COMMIT)) so a commit not reachable from the single-branch ref can still be checked out.