Skip to content

Epic: prove bounded full-schema Dirty Runs A → B correctness #699

Open
1 of 7 issues completed
Open

Description

@jumski

Outcome and authorization

Prove the first real Dirty Runs settlement/reconciliation boundary on pgflow's actual schemas, constraints, PGMQ queues, and worker executor:

start an internal v2 A -> B run
  -> claim A
  -> commit A's fenced result + durable pending work, with no reconciler
  -> lose the acknowledgement response and replay that same result
  -> reconcile in later, separately committed calls
  -> create and queue B exactly once in its stored queue
  -> settle B, then finalize the run through reconciliation

The user authorized this bounded correctness slice and the shared worker-result fixes on 2026-10-14. This epic coordinates the slice; it is not authorization for a deployable beta, unrestricted DAG support, production migrations, releases, or cutover. Creating these issues is planning, not completed implementation.

Full-schema means schema-first changes under pkgs/core/schemas/, real runtime rows and constraints, real PGMQ, generated migration/types, and a narrow typed bridge to the existing executor. It does not mean full graph-feature parity. Do not substitute a toy schema or a processor that merely invokes the current cascade.

First proof envelope

The contract subissue freezes numeric limits and transitions before stateful implementation. Initial scope is exactly two ordinary single-task steps, A as the sole root and B depending only on A. Use both the existing flow queue mode and private per-step queue mode in tests.

No maps, conditions, skips, retryable handler failure, provider-selected retry delay, or structural mutation while a v2 run exists. Require effective maxAttempts=1 and whenExhausted='fail'; a real handler failure therefore fails the run synchronously. B's startDelay and the actual effective claim timeout remain supported within declared bounds. Reject unsupported definitions/options before a v2 run, work row, or message exists. V1 keeps its existing supported behavior.

Each internal v2 claim/recovery call targets one explicit task/run with at most one message ID; the worker bridge uses singleton reservations/claims. #700 also freezes acknowledgement attempts and an absolute reporting deadline including in-flight SQL waits before stateful work.

Prove one controlled lease-loss/takeover per task; automatic worker-loss exhaustion and #689 policy are later work. Durable engine-error/backoff recording and poison isolation are in scope. For this internal proof, attention leaves messages intact, blocks new claims, and permits valid current claims to settle. Attention repair/reconstruction and beta runtime liveness are not in scope. The resulting proof cannot be described as deployable.

Subissues and order

Subissue Deliverable Blocked by
#700 Dirty Runs: freeze the bounded A → B protocol and proof envelope None; can start now
#701 Fix shared worker result boundaries, falsy outputs, and shutdown classification None; can start now
#702 Dirty Runs: add real schema state and isolate v2 from legacy writers #700
#703 Dirty Runs: implement fenced A → B claims and controlled lease takeover #702
#704 Dirty Runs: persist replayable settlements without downstream orchestration #703
#705 Dirty Runs: reconcile A → B with absolute state and committed chunks #704
#706 Dirty Runs: connect the typed worker path and close the A → B evidence gate #701, #705

The shared worker fixes can start independently. The SQL path follows contract → storage/isolation → claims → settlement → reconciliation. The final worker bridge/evidence gate needs all of them. #703 closes on standalone claims/takeover and a race-test handoff; #704 owns the real settlement/takeover race, so predecessor acceptance does not depend on its blocked successor. Add focused tests with each implementation; the final subissue assembles evidence rather than postpones test design.

One editing/database-mutating implementer at a time under the current test lifecycle. Read-only review can run in parallel. Do not wait for #653, #689, #694, #696, or #697 before beginning SQL exploration.

Non-negotiable invariants

  • Ownership is a per-task claim, not a worker ID. Settled replay returns the recorded result; conflict/stale/terminal submissions never mutate task or queue state.
  • An accepted terminal task result, its owned-message archive, and durable orchestration request commit atomically. No descendant rule or run finalization occurs in success acknowledgement.
  • Stable per-run work survives for the run's lifetime. Generations never decrease or wrap; partial chunks keep work pending. Only a locked no-immediate-action check marks a generation processed.
  • All v2 writers use run -> work -> ordered step states -> ordered tasks -> ordered (queue_name,message_id). Definition/maintenance locks precede that sequence when needed. Legacy recovery, callbacks, dispatch, and pruning cannot write v2 state.
  • Graph progress derives from authoritative rows and guarded same-transaction queue effects, not replayed dependency decrements. External effects remain at least once; no exactly-once handler promise.

Scope cuts and follow-ons

#271 still owns unrestricted retry arithmetic; this proof never calls exponential backoff on a retryable handler error. #689's business worker-loss terminalization is not orchestration attention. #678's broader identity enforcement is not closed by narrow v2 guards. #696/#697 remain startup/worker-level guarantees, not task fencing. #698's external-effect uncertainty remains unsolved.

Later work must add full graph semantics, retry budgets/deadlines, infrastructure-loss policy, attention repair, resumable large workloads, a supervised driver, operational diagnostics, deployment/cutover, and load gates. Do not create a persistent service, event outbox, generic effects ledger, graph snapshot system, repair UI, aliases, shared queues, manual tasks, or resume as prerequisites.

Epic acceptance

  • All linked subissues meet their gates against one exact candidate revision and migration artifact.
  • A's result commits with no reconciler; B remains undispatched until later reconciliation. Response-loss replay does not execute A or apply effects twice.
  • Double-claim, settlement/takeover, two-processor, and settlement/generation races pass with real separate PostgreSQL sessions and deterministic barriers.
  • Reconciliation rollback, committed-progress restart, re-dirtying, poison isolation, and private-queue message-ID collision tests pass.
  • Shared worker fixes preserve falsy results and distinguish handler, transport, shutdown, and logging errors. The v2 worker resubmits only the same settlement, never the handler.
  • Unsupported v2 shapes/options are rejected atomically; live definition/identity and v1 writer isolation checks pass; baseline v1 execution remains intact.
  • Publish focused/batch/final evidence and explicit limitations. An open applicable required test gate stays open; Diagnose and fix intermittent Supabase concurrency stall in portable-runtime E2E #694 is not permission to skip it or rerun until green.

Baseline and sources

Original source review: 9160db62009f9535264824a7e76593c2e09b1769. On the 2026-10-20 review resumption, local and GitHub main are cee3636f9bc28e8dfe1a923a5fdfe7f269e0a046; the intervening #707 changes docs/glossary only, with no core/worker source change. No runtime tests ran for this planning review. Recheck the implementation base and repository instructions before implementation.

The private 2026-09-08 architecture review and 2026-10-14 critical-path recommendation are background. This issue set must contain the implementation contract without requiring access to private notes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions