feat(mothership): run scheduled jobs as durable unattended chat turns by waleedlatif1 · Pull Request #9331 · simstudioai/sim

waleedlatif1 · 2026-10-23T02:59:25Z

Summary

Restores the runtime for scheduled agent jobs on the TypeScript worker. The schedule tick claims each due job occurrence and runs it as a durable, leased chat turn in the job's own chat. There is no creation surface yet (no API, UI or agent command), so this PR ships the runtime only, and the lane has nothing to run until that surface lands.

Unattended turns are durable leased chat turns (request/lifecycle/unattended.ts, chat/application/admit-unattended-turn.ts). They are admitted like a typed send: the user message, the run and its recovery intent commit together under the chat lock. The new recovery kind is unattended_stream, and abandoned-run takeover drives one on after its Sim process dies, keeping it non-interactive and mounting only the secrets the job recorded. The headless wake lifecycle cannot do this: it holds no controller lease, so nothing can continue its run once the pod is gone.
Occurrence = run. In one transaction the claim records the occurrence (job_execution_logs, queued) and advances the schedule. The occurrence id is a UUIDv5 of the schedule and the instant, and it is the turn's message, stream and run id. Two ticks, or a reconciler retry, therefore admit the turn at most once. A reconciler re-dispatches lost dispatches, finalizes ended turns exactly once under a row lock, and stops a run at its six-hour limit through the same Stop delivery a person's Stop uses.
Budgets:
- a per-run spend cap: settled cost is tracked per billing request, which survives takeovers, and is answered with a job_run_budget card;
- three failed runs in a row turn the job off and notify the job's people;
- max_runs;
- a 15-minute minimum cadence;
- an owner who loses access turns the job off.
Outcome is a field on the run record (met, not_met, blocked), not a tool call scanned out of a response.
Schema: migration 0437 adds the occurrence columns plus three indexes, built CONCURRENTLY. Script migration 0036 archives (never deletes) every job row the retired Go runner wrote. It commits per batch and is idempotent.

Why this runs without a flag

Nothing can create a job. No route, Copilot tool, CLI command or UI writes a source_type = 'job' row: schedule deploys insert workflow schedules only, and the job create paths were removed earlier.
Legacy rows are archived before the new tick runs. Script migration 0036 runs in the migrations step, after the SQL migrations and before the app starts. The claim only reads rows that are unarchived and not disabled or completed.
An idle tick costs one indexed probe. With no live jobs, the lane runs one query, jobLaneWorkQuery. Its two EXISTS legs probe the partial indexes workflow_schedule_due_job_idx and job_execution_logs_open_idx. It claims nothing and reconciles nothing. An integration test seeds 2,000 archived jobs and 5,000 finished logs, then asserts no claim, no reconcile, no sequential scan and fewer than 20 buffer reads.
Missing prerequisites mean no work at all. Without COPILOT_API_KEY or REDIS_URL the lane returns before any query (unit test).
Why approval holds can come later. Production-effect approval holds for unattended runs are not part of this PR. Until they exist, an unattended run could do anything its owner can. That is safe only because no job row can exist. The holds must land before any creation surface.

Companion: simstudioai/mothership#818

Type of Change

Testing

lib/mothership/request/lifecycle/unattended.integration.ts (real Postgres and Redis): one unattended turn survives three controller deaths mid-run and finishes once. It ends with one user message, one assistant message and one run row, and every recovered leg stays non-interactive with the job's secrets.
lib/mothership/jobs/lane.integration.ts covers:
- concurrent claims recording one occurrence;
- a lost dispatch retried once;
- the death of the Sim process mid-run, finalized exactly once;
- one-shot jobs and max_runs;
- busy-chat skips and lost owner access;
- until_complete;
- the three-failure streak;
- the spend cap;
- the time-limit Stop, whose test fails on the previous stop path because the run never settles;
- the idle-tick probe.
lib/mothership/jobs/lane.test.ts checks that the lane makes no DB calls when its prerequisites are missing.
packages/db/script-migrations/0036_archive_legacy_scheduled_jobs.integration.ts covers batching, idempotency, and keeping completed jobs completed.
Reviewers should focus on the shared-path edits in start.ts, recovery-config.ts, recover-stream.ts and admit-turn.ts: the recovery kind is now explicit, and the chat-move step is shared with typed sends.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

A scheduled job's occurrence must outlive the Sim process driving it. The headless lifecycle the wake door uses takes no controller lease, so a pod death leaves its run unrecoverable. An unattended turn now admits like a typed send (user message, run and recovery intent in one commit under the chat lock), runs detached from any HTTP response, and records a new unattended_stream recovery kind. Takeover restores it non-interactive with the secret authority it was admitted with, instead of the interactive default, and the abandoned-run sweep, stream liveness and billing-admission recording treat both durable kinds alike. Durable start now also forwards a caller's billing-admission hook, so a job can key its per-run budget on the run's billing request. The integration test kills the turn's controller three times mid-run; it fails on a headless implementation, on a sweep blind to the new kind, and on a takeover that restores the interactive default.

Scheduled jobs return as Sim-owned schedules whose occurrences run as durable chat turns. A job's run record (job_execution_logs) gains the occurrence it runs (unique per schedule), its outcome and summary, and the billing request its charges settle under; a schedule gains a per-run spend cap. Indexes are built concurrently. Every job row written for the retired Go runner is archived, never deleted, so none can start once the tick claims jobs again. The archive is a script migration committing per keyed batch, so it finishes on any row count and resumes after a crash.

Behind MOTHERSHIP_JOBS_ENABLED (off by default, and to stay off in production until unattended runs hold production effects for approval), the schedule tick gains a job lane: - Claim: each due job's occurrence is recorded (queued) and the schedule advances past it in one transaction, skipping locked rows, so two ticks record it once and a crash never leaves a stale claim. Missed occurrences collapse into one; cadence is never under 15 minutes; a one-shot job (no cron) runs once and completes; max_runs completes. - Dispatch: the occurrence starts as an unattended durable turn in the job's chat under its occurrence id (UUIDv5 of schedule and instant), so every retry finds the turn it admitted. A busy chat skips the occurrence; a payer over its plan limit skips it; an owner who lost access turns the job off. Only secrets a person named for the job mount. - Reconcile: a queued occurrence whose dispatch was lost is dispatched again after two minutes; a running one whose turn ended is finalized; one past six hours is stopped. - Finalize: exactly once under a row lock. A reported outcome is kept; a turn that ended well without one is not_met, summarized by its last reply. Three failed or interrupted runs in a row turn the job off and email its people; a met until_complete job completes. - Budget: each billing callback records the run's cumulative settled cost on its record (keyed by the run's billing request), and a run at its cap pauses at its next step with a job_run_budget card. The cost is cumulative per billing request, which recovered legs reuse, so the cap survives worker hand-offs and Sim takeovers.

…d probe

…the shared Stop delivery

vercel · 2026-10-23T02:59:31Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 23, 2026 3:41am UTC

github-actions · 2026-10-23T03:00:12Z

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

❌ simstudioai/mothership#818 — OPEN, not merged (targets staging) — feat(worker): run scheduled job turns as automated chat turns

cubic-dev-ai

1 issue found and verified against the latest diff

Confidence score: 3/5

In apps/sim/lib/mothership/jobs/dispatch.ts, a crash after the run becomes recoverable but before billingAdmission is persisted leaves hosted recovery unable to resume the occurrence. Persist the admission before marking the run recoverable.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/mothership/jobs/dispatch.ts">

<violation number="1" location="apps/sim/lib/mothership/jobs/dispatch.ts:90">
P2: The job run becomes recoverable before its billing admission is persisted. A crash in that gap leaves hosted recovery without `billingAdmission`, which `runCopilotLifecycle` rejects instead of resuming the occurrence; persist the admission before exposing the run to recovery or make this pre-dispatch recovery retryable.</violation>
</file>

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-23T03:06:31Z

Confidence Score: 5/5

[Medium impact] The changes appear safe to merge; no new actionable issue was found.

Summary

The latest changes close running job occurrences whose chat run was removed. They now finish as interrupted instead of staying open forever.

The schedule tick runs each due job occurrence as one tracked run.
Unattended job turns can recover in their chat without a person present.
Each scheduled job run stops when its own spend cap is reached.
Legacy scheduled-job rows are archived before the new lane runs.
The unused-export baseline drops a chat-context entry.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Running job occurrence] --> B{Chat run exists?}
  B -->|No| C[Record interrupted outcome]
  B -->|Yes, still working| D[Keep occurrence running]
  B -->|Yes, ended| E[Record final outcome]
  C --> F[Update job failure count]
  E --> F

_{Reviews (4) · Last reviewed commit: "fix(mothership): end a running job recor..." · Reviewed by Greptile}

…hat protocol graph

…s and failure counts durable

waleedlatif1 · 2026-10-23T03:18:29Z

@greptile

waleedlatif1 · 2026-10-23T03:18:30Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T03:18:35Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 49 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

…umber of deleted occurrences

waleedlatif1 · 2026-10-23T03:30:40Z

@greptile

waleedlatif1 · 2026-10-23T03:30:41Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T03:30:47Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 49 files

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-23T03:41:42Z

@greptile

waleedlatif1 · 2026-10-23T03:41:43Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-23T03:41:50Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 49 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 added 5 commits October 22, 2026 19:39

improvement(mothership): run the job lane unflagged behind one indexe…

7ed96b5

…d probe

improvement(mothership): simplify the job lane and stop jobs through …

1049e99

…the shared Stop delivery

github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Oct 23, 2026

chore(db): format the scheduled job migration metadata

a0e1f41

cubic-dev-ai Bot reviewed Oct 23, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

improvement(mothership): keep the durable-run predicates out of the c…

1fd25b6

…hat protocol graph

vercel Bot temporarily deployed to Preview October 23, 2026 03:09 Inactive

fix(mothership): settle job runs before claiming, keep spend-cap link…

8ec6dd4

…s and failure counts durable

vercel Bot deployed to Preview October 23, 2026 03:21

View deployment

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread ...space/[workspaceId]/home/components/message-content/components/special-tags/special-tags.tsx Outdated

Comment thread .../[workspaceId]/home/components/message-content/components/special-tags/special-tags.test.tsx Outdated

cubic-dev-ai Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/jobs/reconcile.ts

Comment thread apps/sim/lib/mothership/jobs/occurrence.ts Outdated

fix(mothership): stop orphaned job runs at their limit and skip any n…

cb11851

…umber of deleted occurrences

vercel Bot temporarily deployed to Preview October 23, 2026 03:30 Inactive

cubic-dev-ai Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/jobs/reconcile.ts Outdated

fix(mothership): end a running job record whose run was removed

0cb2042

vercel Bot temporarily deployed to Preview October 23, 2026 03:41 Inactive

cubic-dev-ai Bot reviewed Oct 23, 2026

View reviewed changes

waleedlatif1 merged commit bec4639 into staging Oct 23, 2026
67 checks passed

waleedlatif1 deleted the feat/scheduled-jobs branch October 23, 2026 06:43

Conversation

waleedlatif1 commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why this runs without a flag

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2026

⚠️ Cross-repo companion check

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 23, 2026

Uh oh!

waleedlatif1 commented Oct 23, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 23, 2026

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 23, 2026

Uh oh!

waleedlatif1 commented Oct 23, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 23, 2026

Uh oh!

waleedlatif1 commented Oct 23, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

waleedlatif1 commented Oct 23, 2026 •

edited

Loading

vercel Bot commented Oct 23, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading