Skip to content

build: 40 MiB WASM initial memory default, retire low-memory variants (patch 13.8.1–18.2.1) - #179

Merged
pyramation merged 1 commit into
mainfrom
feat/wasm-initial-40mib
Oct 25, 2026
Merged

pyramation merged 1 commit into
mainfrom
feat/wasm-initial-40mib

Conversation

@pyramation

Copy link
Copy Markdown
Collaborator

Summary

Drops the WASM initial memory floor from 128 MiB to 40 MiB for every major (13–18), keeps the 32 MiB stack and 1 GiB maximum, and retires the separate low-memory variants. Patch releases: 13.8.1, 14.5.1, 15.7.1, 16.8.1, 17.8.1, 18.2.1.

Why. Today every parser instance reports ~130 MiB of V8 external memory before it has parsed anything (the Supabase worker-pool report behind #165, and #158). That floor was set in Aug 2025 (#136/#137) on the assumption that a memory access out of bounds crash on constructive-db's generated SQL meant "heap too small". It was actually the 64 KiB C stack overflowing; STACK_SIZE fixed that, INITIAL_MEMORY never mattered because ALLOW_MEMORY_GROWTH=1 already grows the heap on demand to 1 GiB. So INITIAL_MEMORY only decides the idle footprint, not the largest input.

Change.

  • templates/Makefile.template: WASM_INITIAL_MEMORY ?= 41943040 (was 134217728); WASM_STACK_SIZE/WASM_MAXIMUM_MEMORY unchanged. versions/*/Makefile regenerated with pnpm run copy:templates.
  • PUBLISH.md: the "Low-Memory Variants" publish flow is replaced by a note on the defaults, a one-line ad-hoc override (pnpm wasm:make build WASM_INITIAL_MEMORY=…, never a stack below 32 MiB), and the npm dist-tag rm pg18-lowmem-* commands as a later, separate step after downstream users are notified (not part of the release).
  • Wrapper message: when libpg_query returns no tree and no error (its strdup of the JSON failed), callers used to get No parse tree generated / Parse result is null; it now says libpg_query returned no parse tree: out of WASM memory (heap limit is WASM_MAXIMUM_MEMORY, 1 GiB); parse large inputs in smaller pieces. No behaviour change otherwise.

Effect. Idle external per instance 129.8 β†’ 41.8 MiB (βˆ’88 MiB); process RSS unchanged (the untouched floor was never resident: 128 MiB mapped, 0.9 MiB RSS). Large inputs unchanged: the heap grows to the same size either way.

Expected wasm/libpg-query.wasm sha256 (Docker emscripten/emsdk:6.0.3, regenerated Makefiles, current pins)

major sha256
13 e2682fe2a6f74b5921ae43cae281c45592ed9ce659e12f1714992cc14114d1fc
14 611aaf534627a50a3355725f20dcf7448ef2d1ad7110ccf6ee20f95f8d2a8878
15 51080181c37e2e299b56dd9bf1bb48fcfd44c707f2e17d896ec3dbf36b2141c8
16 4a5223daa5958979f4af19c7a954bb09ad1b1340f074e8693cb7b140a04307c0
17 27a3fb059946aa5940d65981c540ccb58b5671cc04bdc35c25fc14824a0be30e
18 355c489d8b0bf2bac28b48408f656e818852d7000e1e66a67503ad2bc7cc6062

(The 128 MiB baseline rebuilt from main is byte-identical to npm [email protected], 378b64e6…, so the toolchain is reproducible.)

Evidence (v18 unless noted; 40/32 vs the current 128/32)

  • pnpm test: 143/143 on v18, 72/72 on v13–v17, for every major built with the new Makefiles. pnpm test:stress 18: 4/4 (1,000 stack overflows, 200-in-one-tick Γ—20, 20 inputs over the 1 GiB limit, 1,000 normal parses).
  • Memory (Node 24, gc();gc()): idle external 41.8 vs 129.8 MiB; after the 8.4 MB constructive-db bundle 160 vs 186 MiB, RSS 311 vs 302 MiB, parse time 0.53 vs 0.48 s. Growth cost on large inputs is within noise: 25 MB 1.5–1.7 s vs 1.4–2.0 s, 50 MB 3.0–4.1 s vs 3.2–3.7 s, 76 MB 3.9–4.3 s vs 5.1–5.4 s.
  • Over the 1 GiB maximum: on 40/32 a 101 MB input ends in Error: libpg_query terminated (Program terminated with exit(1)), most likely out of WASM memory… (libpg_query also prints its memory-context dump to stderr); on 128/32 a 67 MB input returned the old No parse tree generated (renamed above) because growth failed earlier in that heap layout. In every case, for parse and parseSync, the next call succeeds (fix(wasm): replace the module after an exception escapes a WASM callΒ #175 re-instantiation).
  • Adversarial nesting (nested parens/subqueries 10k, CASE 5k, 1+1+… 20k–500k, -> 50k, 3k nested PL/pgSQL IF/BEGIN, 8.8 MB and 26 MB files) under default Node stack and --stack-size=8000: identical verdicts on 40/32 and 128/32, no traps.
  • Why the stack stays 32 MiB: on a 32/16 build, two back-to-back parseSync calls of a 100k-term expression under --stack-size=8000 trap (memory access out of bounds) on the second call because the stale instance's shadow-stack pointer is not unwound after a RangeError; 40/32 and 128/32 throw a clean RangeError instead. Untouched stack pages cost no resident memory, so there is no reason to lower it.
  • constructive-db downstream gate (global pgpm 5.38.7, wasm swapped in place): pgpm package of application/constructive (8.8 MB SQL + bundle) produces byte-identical artifacts in 4.7 s vs 6.3 s; pgpm deploy --fast --yes --package app into a scratch PG 18 deploys all 26,512 changes in 28.6 s vs 32.7 s. No traps.

Link to Devin session: https://app.devin.ai/sessions/2a9e06519f424ebab14772c74c05dd83
Open in Devin Desktop: https://app.devin.ai/desktop/session/2a9e06519f424ebab14772c74c05dd83?variant=devin
Requested by: @pyramation

INITIAL_MEMORY is only the floor (the heap grows on demand to 1 GiB), so
128 MiB bought nothing but ~88 MiB of idle external memory per instance.
Keep the 32 MiB stack and 1 GiB maximum. Patch releases 13.8.1, 14.5.1,
15.7.1, 16.8.1, 17.8.1, 18.2.1. The no-tree/no-error wrapper path now
names out-of-memory instead of 'No parse tree generated'.
@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@tenki-reviewer

tenki-reviewer Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Review complete. No blocking issues β€” approved βœ…; 1 nitpick below.

🧹 Nitpicks (1) β€” 🟒 1 low
  • 🟒 Null parse-tree error hardcodes OOM cause and 1 GiB (index.ts:291) β€” The null parse-tree branch now throws a fixed message asserting out-of-WASM-memory with a hardcoded "1 GiB" limit, replacing the previous generic text (templates/index.ts:291).

This PR synchronizes the six versioned packages (PG 13 through 18) and their shared templates with a reworded null-parse-tree error message and a reduced initial WASM memory setting, plus documentation of the publishing flow.

Files Change
templates/index.ts, templates/full/index.ts, versions/*/src/index.ts Reword the error thrown when libpg_query returns no parse tree in both the async and parseSync paths.
templates/Makefile.template, versions/*/Makefile Lower the initial WASM memory floor while keeping the 1 GiB overridable maximum and on-demand growth.
versions/*/package.json, PUBLISH.md Patch-version bumps for each versioned package and updated publish instructions.

Reviewed commit: dc71875

@pyramation
pyramation merged commit 55c0963 into main Oct 25, 2026
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant