Repository navigation
Epic: prove bounded full-schema Dirty Runs A → B correctness #699
Description
Outcome and authorization
Prove the first real Dirty Runs settlement/reconciliation boundary on pgflow's actual schemas, constraints, PGMQ queues, and worker executor:
start an internal v2 A -> B run
-> claim A
-> commit A's fenced result + durable pending work, with no reconciler
-> lose the acknowledgement response and replay that same result
-> reconcile in later, separately committed calls
-> create and queue B exactly once in its stored queue
-> settle B, then finalize the run through reconciliation
The user authorized this bounded correctness slice and the shared worker-result fixes on 2026-10-14. This epic coordinates the slice; it is not authorization for a deployable beta, unrestricted DAG support, production migrations, releases, or cutover. Creating these issues is planning, not completed implementation.
Full-schema means schema-first changes under pkgs/core/schemas/, real runtime rows and constraints, real PGMQ, generated migration/types, and a narrow typed bridge to the existing executor. It does not mean full graph-feature parity. Do not substitute a toy schema or a processor that merely invokes the current cascade.
First proof envelope
The contract subissue freezes numeric limits and transitions before stateful implementation. Initial scope is exactly two ordinary single-task steps, A as the sole root and B depending only on A. Use both the existing flow queue mode and private per-step queue mode in tests.
No maps, conditions, skips, retryable handler failure, provider-selected retry delay, or structural mutation while a v2 run exists. Require effective maxAttempts=1 and whenExhausted='fail'; a real handler failure therefore fails the run synchronously. B's startDelay and the actual effective claim timeout remain supported within declared bounds. Reject unsupported definitions/options before a v2 run, work row, or message exists. V1 keeps its existing supported behavior.
Each internal v2 claim/recovery call targets one explicit task/run with at most one message ID; the worker bridge uses singleton reservations/claims. #700 also freezes acknowledgement attempts and an absolute reporting deadline including in-flight SQL waits before stateful work.
Prove one controlled lease-loss/takeover per task; automatic worker-loss exhaustion and #689 policy are later work. Durable engine-error/backoff recording and poison isolation are in scope. For this internal proof, attention leaves messages intact, blocks new claims, and permits valid current claims to settle. Attention repair/reconstruction and beta runtime liveness are not in scope. The resulting proof cannot be described as deployable.
Subissues and order
| Subissue | Deliverable | Blocked by |
|---|---|---|
| #700 | Dirty Runs: freeze the bounded A → B protocol and proof envelope | None; can start now |
| #701 | Fix shared worker result boundaries, falsy outputs, and shutdown classification | None; can start now |
| #702 | Dirty Runs: add real schema state and isolate v2 from legacy writers | #700 |
| #703 | Dirty Runs: implement fenced A → B claims and controlled lease takeover | #702 |
| #704 | Dirty Runs: persist replayable settlements without downstream orchestration | #703 |
| #705 | Dirty Runs: reconcile A → B with absolute state and committed chunks | #704 |
| #706 | Dirty Runs: connect the typed worker path and close the A → B evidence gate | #701, #705 |
The shared worker fixes can start independently. The SQL path follows contract → storage/isolation → claims → settlement → reconciliation. The final worker bridge/evidence gate needs all of them. #703 closes on standalone claims/takeover and a race-test handoff; #704 owns the real settlement/takeover race, so predecessor acceptance does not depend on its blocked successor. Add focused tests with each implementation; the final subissue assembles evidence rather than postpones test design.
One editing/database-mutating implementer at a time under the current test lifecycle. Read-only review can run in parallel. Do not wait for #653, #689, #694, #696, or #697 before beginning SQL exploration.
Non-negotiable invariants
- Ownership is a per-task claim, not a worker ID. Settled replay returns the recorded result; conflict/stale/terminal submissions never mutate task or queue state.
- An accepted terminal task result, its owned-message archive, and durable orchestration request commit atomically. No descendant rule or run finalization occurs in success acknowledgement.
- Stable per-run work survives for the run's lifetime. Generations never decrease or wrap; partial chunks keep work pending. Only a locked no-immediate-action check marks a generation processed.
- All v2 writers use
run -> work -> ordered step states -> ordered tasks -> ordered (queue_name,message_id). Definition/maintenance locks precede that sequence when needed. Legacy recovery, callbacks, dispatch, and pruning cannot write v2 state. - Graph progress derives from authoritative rows and guarded same-transaction queue effects, not replayed dependency decrements. External effects remain at least once; no exactly-once handler promise.
Scope cuts and follow-ons
#271 still owns unrestricted retry arithmetic; this proof never calls exponential backoff on a retryable handler error. #689's business worker-loss terminalization is not orchestration attention. #678's broader identity enforcement is not closed by narrow v2 guards. #696/#697 remain startup/worker-level guarantees, not task fencing. #698's external-effect uncertainty remains unsolved.
Later work must add full graph semantics, retry budgets/deadlines, infrastructure-loss policy, attention repair, resumable large workloads, a supervised driver, operational diagnostics, deployment/cutover, and load gates. Do not create a persistent service, event outbox, generic effects ledger, graph snapshot system, repair UI, aliases, shared queues, manual tasks, or resume as prerequisites.
Epic acceptance
- All linked subissues meet their gates against one exact candidate revision and migration artifact.
- A's result commits with no reconciler; B remains undispatched until later reconciliation. Response-loss replay does not execute A or apply effects twice.
- Double-claim, settlement/takeover, two-processor, and settlement/generation races pass with real separate PostgreSQL sessions and deterministic barriers.
- Reconciliation rollback, committed-progress restart, re-dirtying, poison isolation, and private-queue message-ID collision tests pass.
- Shared worker fixes preserve falsy results and distinguish handler, transport, shutdown, and logging errors. The v2 worker resubmits only the same settlement, never the handler.
- Unsupported v2 shapes/options are rejected atomically; live definition/identity and v1 writer isolation checks pass; baseline v1 execution remains intact.
- Publish focused/batch/final evidence and explicit limitations. An open applicable required test gate stays open; Diagnose and fix intermittent Supabase concurrency stall in portable-runtime E2E #694 is not permission to skip it or rerun until green.
Baseline and sources
Original source review: 9160db62009f9535264824a7e76593c2e09b1769. On the 2026-10-20 review resumption, local and GitHub main are cee3636f9bc28e8dfe1a923a5fdfe7f269e0a046; the intervening #707 changes docs/glossary only, with no core/worker source change. No runtime tests ran for this planning review. Recheck the implementation base and repository instructions before implementation.
- Current callback executor and SQL client.
- Runtime tables, claims, completion cascade, and recovery.
- Existing separate-session race test.
- Current public behavior: execution model, data model, failure/termination.
The private 2026-09-08 architecture review and 2026-10-14 critical-path recommendation are background. This issue set must contain the implementation contract without requiring access to private notes.
Activity