Skip to content

fix(wasm): replace the module after an exception escapes a WASM call - #175

Merged
pyramation merged 9 commits into
mainfrom
fix/wasm-recover-after-throw
Oct 23, 2026
Merged

pyramation merged 9 commits into
mainfrom
fix/wasm-recover-after-throw

Conversation

@pyramation

@pyramation pyramation commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Before: in every published build (17.7.4, 18.1.5 and the lowmem tags), one bad input could permanently break the parser for the rest of the process.

  • If an exception escapes a WASM call, libpg_query's cleanup never runs. The exception can be a V8 RangeError (stack overflow on deeply nested SQL), a trap, or Emscripten's exit() when the 1 GiB heap limit is hit.
  • Each such exception leaks the C shadow stack and the whole parse memory context.
  • Repeat a deep input about 35–45 times (SELECT 1+1+…, 100k terms) and the instance traps on every later call, including SELECT 1. Past about 45 failures in one synchronous stretch, the next overflow can spin forever.
  • The heap-limit path is worse: it throws ExitStatus, which is not an Error, and silently sets process.exitCode = 1 in the host process.

Change: JS wrapper only (templates/index.ts, templates/full/index.ts, regenerated into versions/*/src). No WASM rebuild, no memory-flag change, no C/fork change.

resultPtr = callWasm(() => wasmModule._wasm_parse_query_raw(queryPtr));

function callWasm(fn) {
  const exitCode = process?.exitCode;
  try { return fn(); }
  catch (error) {
    failures++;                       // this call's finally skips _free on the instance that threw
    stale = true;                     // async calls now wait for a replacement
    if (++staleFailures >= MAX_STALE_FAILURES) wasmModule = undefined;  // 10: stop sync calls before the ~45-failure hang
    instantiate();                    // single-flight: reuses the in-flight load if there is one
    process.exitCode = exitCode;      // undo exit()'s side effect
    if (error instanceof Error) throw error;
    throw new Error(`libpg_query terminated (${error.message ?? error.constructor?.name}), most likely out of WASM memory; …`, { cause: error });
  }
}
// loadModule() and every async entry point:
while (!wasmModule || stale) await instantiate();
  • Async calls (including ones already queued behind the failing call) wait for the fresh instance, so they never run on one that threw.
  • Sync calls cannot wait, so they keep using the stale instance until the replacement is ready, exactly as today's wrapper does. A sync loop with one deep statement in the middle parses the other 199 fine. After 10 failures on the same stale instance, sync calls throw the existing "Call loadModule() first" error instead, rather than risk the hang. Awaiting loadModule() (or any async call) switches to the fresh instance.
  • When the fresh instance loads, the stale one is unreferenced and gets garbage-collected; only one instance is kept after the swap.
  • A failed reload clears loading, so the next call (or next failure, for sync-only callers) retries. The load promise always has a handler, so a failed load is never an unhandled rejection.
  • Only throws that unwind out of WASM trigger this. A SqlError from a normal parse error is built in JS after the call returns.

Regression found in review and fixed here: the first version of this PR set wasmModule = undefined on every failure, so after one crash every later *Sync call in the same synchronous loop threw "Call loadModule() first": 10/200 statements parsed vs 200/200 on 18.1.5 (found by a red-team pass, reproduced on v18/v17/v13). The test keeps sync calls in the same synchronous loop working after a failure fails on that version and passes now.

Tests (test/wasm-recovery.test.js, identical in every version 13–18; v13–17 skip the tests for functions their build lacks):

  • Real WASM: 60 deep failures then SELECT 1; 60 real overflows in one sync loop (the first 10 are RangeError, the rest the stale-module error, and the loop finishes); 50 MB input rejects with an Error and leaves process.exitCode alone, on parse and (v18) parsePlPgSQL/fingerprint/normalize/scan, each followed by a normal call; queued async call; the sync-loop regression above; failed reload retried both by an async caller and by a sync-only caller; a child process with --expose-gc forcing 30 deep failures plus 2 over-1-GiB inputs must keep V8 external growth under 32 MB and exit with code 0 (18.1.5: +443 MB from the deep failures alone).
  • Injected throws on an isolated copy of the wrapper, for each of the 10 entry points × (Error, non-Error): original error rethrown or wrapped with cause, exit code restored, no _free on the instance that threw, a sync call during the reload runs on the stale instance and frees its memory, the next async call runs on the fresh instance. Plus: many failures in one sync loop start one load; the 10-failure cap; loadModule() and queued calls waiting again when a queued call breaks the fresh instance; constructor-name message for a thrown object without .message.
  • Against the previous commit of this PR the new file fails 24/32. Locally: v18 143/143; v17 and v13 pass. CI: the new checks make the v18 test job about 15 s longer (45 s on Ubuntu, 67 s on Windows); no new jobs.
  • Mutation check, pnpm test:mutations [versions] (scripts/test-mutations.js, local only, about 25 min): breaks one branch or condition of the built wasm/index.cjs at a time and runs this test file against it. v18: 36/36 caught; v13: 24/24 caught. It doesn't generate the !stale cleanup-guard mutation for async functions, because async calls only start once stale is false, so that mutation can't change behavior.
  • Stress check, pnpm test:stress [versions] (scripts/test-stress.js, local only, about 2 min per version): in a fresh process each, 1,000 async overflows, 1,000 sync overflows, 20 × 200 overflows in one sync loop, 20 over-1-GiB inputs, and 1,000 normal parses as a control. Each fails if V8 external grows by 32 MB or more, SELECT 1 breaks, or process.exitCode changes. v13 and v18: all pass, growth ≤ 0.3 MB.

Memory over time (real WASM, v18, a fresh process per run, gc() before each reading):

scenario V8 external start → end RSS end SELECT 1 after
18.1.5, 100 deep failures 130 → 745 MB 678 MB broken from failure 40
this PR, 1,000–2,000 deep failures 130 → 130 MB 117–126 MB works
18.1.5, 5 over-1-GiB inputs 130 → 1,026 MB (never shrinks) 1,124 MB process.exitCode = 1
this PR, 20 over-1-GiB inputs 130 → 130 MB 150 MB works
this PR, 1,000–2,000 normal parses (control) 130 → 130 MB 136–168 MB works

Rerun on the final commit. v13 gives the same result. Calls that don't throw never reload, so normal use is the same as today.

Also checked: Chrome (no process; bundled ESM): a deep parseSync failure, a parseSync right after it on the stale instance, 50 MB fingerprint/parsePlPgSQL failures and 20 more failures all recover. Red-team pass on the previous commit: 1–100 MB inputs across all 10 APIs, concurrency with strict unhandled-rejection mode, worker threads, ESM, v13/v17/v18 memory: no other issue caused by this PR.

Not changed (pre-existing, tracked separately): 50 MB+ realistic SQL can exceed the 1 GiB WASM heap (now a normal error instead of a broken module); scan on ~20 MB+ can return {version:0,tokens:[]}; libpg_query prints a memory-context dump to stdout on the heap-limit path.

Link to Devin session: https://app.devin.ai/sessions/bcd8b2f791164247acb7a659b481efc2
Open in Devin Desktop: https://app.devin.ai/desktop/session/bcd8b2f791164247acb7a659b481efc2?variant=devin
Requested by: @pyramation

A RangeError (V8 stack overflow on deeply nested SQL), a trap, or exit()
when the heap limit is hit unwinds out of libpg_query without running its
cleanup, leaking the shadow stack and the parse memory context. After ~35
such errors the instance trapped on every call, even SELECT 1. The FATAL
path also threw a non-Error ExitStatus and set process.exitCode=1.

Discard the instance and re-instantiate it, restore process.exitCode, and
turn non-Error throws into an Error.
@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@tenki-reviewer

tenki-reviewer Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Review complete. 🟡 2 medium

💬 Inline comments (2)

  • 🟡 Concurrent callers fail spuriously during WASM recovery window — index.ts:144
  • 🟡 Recovery instantiate() rejection is left unhandled — index.ts:145

This PR wires a recovery path into the WASM wrappers: callWasm catches native aborts/OOM, nulls the module, re-instantiates it, and retries or rethrows, and each version gains a wasm-recovery test suite exercising OOM and stack-overflow recovery.

Files Change
templates/index.ts, templates/full/index.ts Add callWasm guard/recovery logic used to regenerate the per-version wrappers.
versions/{13..18}/src/index.ts Regenerated from templates; identical recovery semantics across versions.
versions/{13..18}/package.json Append wasm-recovery suite to test scripts (18 keeps its extra suites).
versions/{13..18}/test/wasm-recovery.test.js New resource-exhaustion tests: large multi-statement OOM input and repeated stack-overflow loop asserting recovery via loadModule().

Reviewed commit: 01c4459

@tenki-reviewer tenki-reviewer Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds WASM crash-recovery (re-instantiation on OOM/abort) and recovery tests across the six pg_query versions, with two medium-severity gaps in the recovery window left to address.

Key findings

  • 🟡 Concurrent callers fail spuriously during WASM recovery window — index.ts:144
  • 🟡 Recovery instantiate() rejection is left unhandled — index.ts:145

Comment thread templates/index.ts Outdated
Comment thread templates/index.ts Outdated
…a failed reload

Async entry points now loop until an instance exists instead of awaiting
initPromise once, so a call queued behind a failing one waits for the fresh
instance. A rejected load clears initPromise (and is handled), so it no
longer surfaces as an unhandled rejection and the next call retries.
…o's wrapper

On main the v13 job runs the wrapper bundled in @libpg-query/[email protected], so
the recovery test exercised the old wrapper and hung. Also bound the
recovery suite with a 120 s timeout so a regression fails instead of hanging.
…r-throw

# Conflicts:
#	.github/workflows/ci.yml
After a WASM call throws, mark the instance stale instead of dropping it:
async calls wait for the replacement, sync calls keep using the stale
instance (as before this PR) until it is ready, up to 10 failures on it.
One replacement loads at a time. Name non-Error throws without a message
by their constructor.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant