You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
pstack orch: stale-lock takeover can be lost (unlink-then-create TOCTOU); frontier SHAs use local branches #730
acquireLock in the pstack orchestrator store takes over a stale lock by unlink-then-create, which has a TOCTOU window: two processes that both see the same dead holder can end up both holding the lock.
consttakeOver=async(): Promise<void>=>{awaitunlink(path);// (1) both processes can reach heretry{awaitcreate();// (2) one wins; the loser's unlink may land later}catch(retryError){ ... }};
Interleaving that loses mutual exclusion
step
P1
P2
1
reads holder, decides it is dead
2
reads holder, decides it is dead
3
unlink(path)
4
create() → holds lock
5
unlink(path) → deletes P1's fresh lock
6
create() → also "holds" lock
Both processes then proceed to write the store, so one writer's update is lost.
Suggested fix
Make takeover atomic rather than unlink-then-create. Either:
Lock directory — mkdir the lock path (atomic; EEXIST is the contention signal) and rmdir on release. Changes the lock artifact from a file to a directory, and the reader of holderIsDead needs to read the pid from a file inside it.
Rename-based takeover — rename(path, path + ".stale." + pid): only one process can win the rename; then read the renamed file to report the previous holder, and create() the new lock. On EEXIST, report the current holder.
A two-process test (spawn two children that both observe a dead pid and assert exactly one acquireLock resolves) would pin the behaviour.
Related finding in the same file: frontier SHAs come from local branches
returnchain.map((row)=>({pr: row.pr,branches: row.branch,sha: branchSha({branch: row.branch, repo }),// local branch, not PR headstate: row.state,}));
The frontier's verification identity is taken from the local branch, while the thing being verified is the PR head. If a local branch is stale or missing, the recorded SHA silently describes a commit that is not under review — a false verification identity rather than an error.
Suggested fix: source the SHA from the PR head OID already present in the snapshot, and represent "no local branch" as an explicit unknown instead of substituting a branch tip.
Happy to send PRs for either; filing first in case the lock artifact shape (file vs directory) is a compatibility constraint I cannot see.
Hey @BowmanStephen, confirmed on main: create() is an atomic wx open, but takeOver() is unlink then create, so two processes that both read the same dead holder can interleave exactly as in your table, and the second unlink removes the first process's fresh lock. The existing test only covers one taker. Tracked internally with a fix sketch: serialize the takeover with a second wx file (<lock>.takeover), so the loser falls back to create(), hits EEXIST, re-reads the holder and gets held by pid <winner>, plus a two-taker test. PR to follow unless the pstack owners prefer the rename-and-verify variant.
This can be closed: #1043 removed scripts/orch/ entirely, so the lock and its takeover path are gone from main. #733 is closed for the same reason. The same race was re-reported against the pre-#1043 commit as #1055.
Activity
deanrie commentedon Oct 20, 2026
Hey @BowmanStephen, confirmed on
main:create()is an atomicwxopen, buttakeOver()isunlinkthencreate, so two processes that both read the same dead holder can interleave exactly as in your table, and the secondunlinkremoves the first process's fresh lock. The existing test only covers one taker. Tracked internally with a fix sketch: serialize the takeover with a secondwxfile (<lock>.takeover), so the loser falls back tocreate(), hitsEEXIST, re-reads the holder and getsheld by pid <winner>, plus a two-taker test. PR to follow unless the pstack owners prefer the rename-and-verify variant.test(pstack): model-check the orch store lock with TLA+ (#29)
deanrie commentedon Oct 28, 2026
This can be closed: #1043 removed
scripts/orch/entirely, so the lock and its takeover path are gone frommain. #733 is closed for the same reason. The same race was re-reported against the pre-#1043 commit as #1055.