Repository navigation
fix(wasm): replace the module after an exception escapes a WASM call - #175
Merged
Merged
Conversation
A RangeError (V8 stack overflow on deeply nested SQL), a trap, or exit() when the heap limit is hit unwinds out of libpg_query without running its cleanup, leaking the shadow stack and the parse memory context. After ~35 such errors the instance trapped on every call, even SELECT 1. The FATAL path also threw a non-Error ExitStatus and set process.exitCode=1. Discard the instance and re-instantiate it, restore process.exitCode, and turn non-Error throws into an Error.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
Review complete. 🟡 2 medium 💬 Inline comments (2)
This PR wires a recovery path into the WASM wrappers:
Reviewed commit: 01c4459 |
There was a problem hiding this comment.
Adds WASM crash-recovery (re-instantiation on OOM/abort) and recovery tests across the six pg_query versions, with two medium-severity gaps in the recovery window left to address.
Key findings
- 🟡 Concurrent callers fail spuriously during WASM recovery window — index.ts:144
- 🟡 Recovery
instantiate()rejection is left unhandled — index.ts:145
…a failed reload Async entry points now loop until an instance exists instead of awaiting initPromise once, so a call queued behind a failing one waits for the fresh instance. A rejected load clears initPromise (and is handled), so it no longer surfaces as an unhandled rejection and the next call retries.
…o's wrapper On main the v13 job runs the wrapper bundled in @libpg-query/[email protected], so the recovery test exercised the old wrapper and hung. Also bound the recovery suite with a 120 s timeout so a regression fails instead of hanging.
…r-throw # Conflicts: # .github/workflows/ci.yml
After a WASM call throws, mark the instance stale instead of dropping it: async calls wait for the replacement, sync calls keep using the stale instance (as before this PR) until it is ready, up to 10 failures on it. One replacement loads at a time. Name non-Error throws without a message by their constructor.
…tion; add test:stress and test:mutations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Before: in every published build (17.7.4, 18.1.5 and the lowmem tags), one bad input could permanently break the parser for the rest of the process.
RangeError(stack overflow on deeply nested SQL), a trap, or Emscripten'sexit()when the 1 GiB heap limit is hit.SELECT 1+1+…, 100k terms) and the instance traps on every later call, includingSELECT 1. Past about 45 failures in one synchronous stretch, the next overflow can spin forever.ExitStatus, which is not anError, and silently setsprocess.exitCode = 1in the host process.Change: JS wrapper only (
templates/index.ts,templates/full/index.ts, regenerated intoversions/*/src). No WASM rebuild, no memory-flag change, no C/fork change.loadModule()first" error instead, rather than risk the hang. AwaitingloadModule()(or any async call) switches to the fresh instance.loading, so the next call (or next failure, for sync-only callers) retries. The load promise always has a handler, so a failed load is never an unhandled rejection.SqlErrorfrom a normal parse error is built in JS after the call returns.Regression found in review and fixed here: the first version of this PR set
wasmModule = undefinedon every failure, so after one crash every later*Synccall in the same synchronous loop threw "CallloadModule()first": 10/200 statements parsed vs 200/200 on 18.1.5 (found by a red-team pass, reproduced on v18/v17/v13). The test keeps sync calls in the same synchronous loop working after a failure fails on that version and passes now.Tests (
test/wasm-recovery.test.js, identical in every version 13–18; v13–17 skip the tests for functions their build lacks):SELECT 1; 60 real overflows in one sync loop (the first 10 areRangeError, the rest the stale-module error, and the loop finishes); 50 MB input rejects with anErrorand leavesprocess.exitCodealone, onparseand (v18)parsePlPgSQL/fingerprint/normalize/scan, each followed by a normal call; queued async call; the sync-loop regression above; failed reload retried both by an async caller and by a sync-only caller; a child process with--expose-gcforcing 30 deep failures plus 2 over-1-GiB inputs must keep V8externalgrowth under 32 MB and exit with code 0 (18.1.5: +443 MB from the deep failures alone).Error, non-Error): original error rethrown or wrapped withcause, exit code restored, no_freeon the instance that threw, a sync call during the reload runs on the stale instance and frees its memory, the next async call runs on the fresh instance. Plus: many failures in one sync loop start one load; the 10-failure cap;loadModule()and queued calls waiting again when a queued call breaks the fresh instance; constructor-name message for a thrown object without.message.pnpm test:mutations [versions](scripts/test-mutations.js, local only, about 25 min): breaks one branch or condition of the builtwasm/index.cjsat a time and runs this test file against it. v18: 36/36 caught; v13: 24/24 caught. It doesn't generate the!stalecleanup-guard mutation for async functions, because async calls only start oncestaleis false, so that mutation can't change behavior.pnpm test:stress [versions](scripts/test-stress.js, local only, about 2 min per version): in a fresh process each, 1,000 async overflows, 1,000 sync overflows, 20 × 200 overflows in one sync loop, 20 over-1-GiB inputs, and 1,000 normal parses as a control. Each fails if V8externalgrows by 32 MB or more,SELECT 1breaks, orprocess.exitCodechanges. v13 and v18: all pass, growth ≤ 0.3 MB.Memory over time (real WASM, v18, a fresh process per run,
gc()before each reading):externalstart → endSELECT 1afterprocess.exitCode = 1Rerun on the final commit. v13 gives the same result. Calls that don't throw never reload, so normal use is the same as today.
Also checked: Chrome (no
process; bundled ESM): a deepparseSyncfailure, aparseSyncright after it on the stale instance, 50 MBfingerprint/parsePlPgSQLfailures and 20 more failures all recover. Red-team pass on the previous commit: 1–100 MB inputs across all 10 APIs, concurrency with strict unhandled-rejection mode, worker threads, ESM, v13/v17/v18 memory: no other issue caused by this PR.Not changed (pre-existing, tracked separately): 50 MB+ realistic SQL can exceed the 1 GiB WASM heap (now a normal error instead of a broken module);
scanon ~20 MB+ can return{version:0,tokens:[]}; libpg_query prints a memory-context dump to stdout on the heap-limit path.Link to Devin session: https://app.devin.ai/sessions/bcd8b2f791164247acb7a659b481efc2
Open in Devin Desktop: https://app.devin.ai/desktop/session/bcd8b2f791164247acb7a659b481efc2?variant=devin
Requested by: @pyramation