Skip to content

Dirty Runs: reconcile A → B with absolute state and committed chunks #705

Open

Description

@jumski

Parent epic: #699. Depends on #704; contract: #700.

Deliverable

One internal reconcile_one_run operation: at most one run and one bounded chunk per caller transaction. The two-step envelope is enough to prove real dispatch, finalization, durable generation handling, and committed restart. Do not reuse the old callback cascade or replay dependency decrements.

Locking and generation handling

Read candidate IDs without taking work-row ownership. Lock the run first (nonblocking skip/return-busy for a locked candidate), then its stable work row and required ordered step/task rows. Recheck versions, run status, health/backoff, requested/processed generations, and authoritative current rows under lock.

Derive remaining dependencies/tasks/steps from real state. Guard B's transition and its task + PGMQ message insertion/pointer in the SAME transaction. Dispatch through stored route; apply B's startDelay once at creation, not on replay. Never revive a terminal task/step/run.

Use the contract's small transition budget. A dispatch chunk commits B's task/message and keeps processed behind requested. A later locked fixed-point check marks caught up while B is queued; only after that clean commit may B claim and execute. A physical reservation before clean is not permission to run. After B settles, a later chunk finalizes only a started run with all steps terminal and bounded leaf output. Mark processed only for the locked captured generation when no immediate action remains. New settlement between chunks must not lose its generation.

No generation bump merely for partial progress. No delete/recreate of the work row. Re-read state on every transaction; no in-memory continuation list, recursive cascade, or clean-row early-return masquerading as rule replay coverage.

Errors and independent progress

Acquire run/work locks outside the inner graph-mutation exception block. Roll back the failed chunk's mutations, then durably record SQLSTATE/error/failure count/next retry outside that block in the SAME outer transaction. Earlier committed chunks remain intact. Preserve pending generations/backoff on new settlement; do not auto-clear attention. Honor the reviewed error/attention bound and leave messages intact for this proof.

A second client/test driver chooses another eligible run in a separate call/transaction. One poison or locked run must not abort or monopolize another run. No SQL function that loops over runs and claims per-run commits. No cron, Edge loop, notification dependency, repair, or production service in this subissue.

Acceptance

  • A settlement followed by later reconciliation creates one B task/message with correct dependency input, queue, VT/delay, and absolute counters.
  • Rollback during dispatch removes task/message/transition together; replay produces one committed effect.
  • Commit dispatch, close the processor connection, resume from another connection, and reach fixed point without duplicate queue effects. Processed stays behind while the partial chunk is pending.
  • Between dispatch-pending and fixed-point-clean commits, an actual worker/claim attempt cannot execute B or create claim history. After clean, B may claim once; startDelay still measures from dispatch, not the clean commit.
  • Explicitly re-dirty already-processed state and execute rules again; no counter drift, repeated dispatch, or shifted startDelay.
  • Separate-session settlement-before-processor and processor-before-settlement schedules leave the newest generation pending or processed legitimately; no lost request.
  • Two processors for one run never double-dispatch; a locked/poison run permits another run to progress. Error metadata commits even though the graph subtransaction rolls back.
  • B's accepted result survives processor loss; later finalization yields the correct leaf output once. Failed/attention/unsupported-version runs never finalize or newly dispatch.

Use real separate connections with explicit barriers. Commit boundaries are actual client statements, not savepoints alone. Tests own fault injection; avoid a production fault-control API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpkgs/corepriority:p1Next batch: current correctness, user blocker, or operational safety

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions