Repository navigation
build: 40 MiB WASM initial memory default, retire low-memory variants (patch 13.8.1β18.2.1) - #179
Merged
Merged
Conversation
INITIAL_MEMORY is only the floor (the heap grows on demand to 1 GiB), so 128 MiB bought nothing but ~88 MiB of idle external memory per instance. Keep the 32 MiB stack and 1 GiB maximum. Patch releases 13.8.1, 14.5.1, 15.7.1, 16.8.1, 17.8.1, 18.2.1. The no-tree/no-error wrapper path now names out-of-memory instead of 'No parse tree generated'.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
Review complete. No blocking issues β approved β ; 1 nitpick below. π§Ή Nitpicks (1) β π’ 1 low
This PR synchronizes the six versioned packages (PG 13 through 18) and their shared templates with a reworded null-parse-tree error message and a reduced initial WASM memory setting, plus documentation of the publishing flow.
Reviewed commit: dc71875 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Drops the WASM initial memory floor from 128 MiB to 40 MiB for every major (13β18), keeps the 32 MiB stack and 1 GiB maximum, and retires the separate low-memory variants. Patch releases: 13.8.1, 14.5.1, 15.7.1, 16.8.1, 17.8.1, 18.2.1.
Why. Today every parser instance reports ~130 MiB of V8
externalmemory before it has parsed anything (the Supabase worker-pool report behind #165, and #158). That floor was set in Aug 2025 (#136/#137) on the assumption that amemory access out of boundscrash on constructive-db's generated SQL meant "heap too small". It was actually the 64 KiB C stack overflowing;STACK_SIZEfixed that,INITIAL_MEMORYnever mattered becauseALLOW_MEMORY_GROWTH=1already grows the heap on demand to 1 GiB. SoINITIAL_MEMORYonly decides the idle footprint, not the largest input.Change.
templates/Makefile.template:WASM_INITIAL_MEMORY ?= 41943040(was134217728);WASM_STACK_SIZE/WASM_MAXIMUM_MEMORYunchanged.versions/*/Makefileregenerated withpnpm run copy:templates.PUBLISH.md: the "Low-Memory Variants" publish flow is replaced by a note on the defaults, a one-line ad-hoc override (pnpm wasm:make build WASM_INITIAL_MEMORY=β¦, never a stack below 32 MiB), and thenpm dist-tag rm pg18-lowmem-*commands as a later, separate step after downstream users are notified (not part of the release).strdupof the JSON failed), callers used to getNo parse tree generated/Parse result is null; it now sayslibpg_query returned no parse tree: out of WASM memory (heap limit is WASM_MAXIMUM_MEMORY, 1 GiB); parse large inputs in smaller pieces. No behaviour change otherwise.Effect. Idle
externalper instance 129.8 β 41.8 MiB (β88 MiB); process RSS unchanged (the untouched floor was never resident: 128 MiB mapped, 0.9 MiB RSS). Large inputs unchanged: the heap grows to the same size either way.Expected
wasm/libpg-query.wasmsha256 (Dockeremscripten/emsdk:6.0.3, regenerated Makefiles, current pins)e2682fe2a6f74b5921ae43cae281c45592ed9ce659e12f1714992cc14114d1fc611aaf534627a50a3355725f20dcf7448ef2d1ad7110ccf6ee20f95f8d2a887851080181c37e2e299b56dd9bf1bb48fcfd44c707f2e17d896ec3dbf36b2141c84a5223daa5958979f4af19c7a954bb09ad1b1340f074e8693cb7b140a04307c027a3fb059946aa5940d65981c540ccb58b5671cc04bdc35c25fc14824a0be30e355c489d8b0bf2bac28b48408f656e818852d7000e1e66a67503ad2bc7cc6062(The 128 MiB baseline rebuilt from
mainis byte-identical to npm[email protected],378b64e6β¦, so the toolchain is reproducible.)Evidence (v18 unless noted; 40/32 vs the current 128/32)
pnpm test: 143/143 on v18, 72/72 on v13βv17, for every major built with the new Makefiles.pnpm test:stress 18: 4/4 (1,000 stack overflows, 200-in-one-tick Γ20, 20 inputs over the 1 GiB limit, 1,000 normal parses).gc();gc()): idleexternal41.8 vs 129.8 MiB; after the 8.4 MB constructive-db bundle 160 vs 186 MiB, RSS 311 vs 302 MiB, parse time 0.53 vs 0.48 s. Growth cost on large inputs is within noise: 25 MB 1.5β1.7 s vs 1.4β2.0 s, 50 MB 3.0β4.1 s vs 3.2β3.7 s, 76 MB 3.9β4.3 s vs 5.1β5.4 s.Error: libpg_query terminated (Program terminated with exit(1)), most likely out of WASM memoryβ¦(libpg_query also prints its memory-context dump to stderr); on 128/32 a 67 MB input returned the oldNo parse tree generated(renamed above) because growth failed earlier in that heap layout. In every case, forparseandparseSync, the next call succeeds (fix(wasm): replace the module after an exception escapes a WASM callΒ #175 re-instantiation).1+1+β¦20kβ500k,->50k, 3k nested PL/pgSQL IF/BEGIN, 8.8 MB and 26 MB files) under default Node stack and--stack-size=8000: identical verdicts on 40/32 and 128/32, no traps.parseSynccalls of a 100k-term expression under--stack-size=8000trap (memory access out of bounds) on the second call because the stale instance's shadow-stack pointer is not unwound after aRangeError; 40/32 and 128/32 throw a cleanRangeErrorinstead. Untouched stack pages cost no resident memory, so there is no reason to lower it.pgpm 5.38.7, wasm swapped in place):pgpm packageofapplication/constructive(8.8 MB SQL + bundle) produces byte-identical artifacts in 4.7 s vs 6.3 s;pgpm deploy --fast --yes --package appinto a scratch PG 18 deploys all 26,512 changes in 28.6 s vs 32.7 s. No traps.Link to Devin session: https://app.devin.ai/sessions/2a9e06519f424ebab14772c74c05dd83
Open in Devin Desktop: https://app.devin.ai/desktop/session/2a9e06519f424ebab14772c74c05dd83?variant=devin
Requested by: @pyramation