[ksarangmath]

Summary

Stabilize the live Anthropic compaction-recall fixture implicated in this failing main job. Its first padded prompt can exceed the configured input budget once provider-reported usage replaces the initial estimate. Locally, the original fixture reported 13,044 input tokens against a 10,880-token budget, triggering compaction before the tool exchange finished. That checkpoint covered only the user prompt, while the test later required it to contain the tool-generated sample ID. Whether another compaction occurred depended on the summary size.

Separate setup from context pressure for every provider. Complete the original padded tool turn with a 128,000-token context window and assert that no checkpoint was needed. Lower the agent's context window to 16,000 through the public config API, verify the effective configuration, and enqueue the original bridge prompt on the same worker. This makes the complete tool exchange eligible for compaction without relying on provider-specific padding adjustments or changing the original prompts.

Retain the requirements for an actual budget-admission failure, a successful checkpoint covering both the tool call and its result, removal of raw padding, and exact recall of the project label and generated sample ID. Count all checkpoints separately from checkpoints containing the expected facts, and fail promptly with both counts when the bridge turn finishes without satisfying the contract instead of polling an idle agent for five minutes. Read output before its supporting evidence to avoid observing a newly finished turn alongside stale counts.

Also:

• Process-tools prompt uses tail -f /dev/null, not sleep 20
• Fix process diagnostics query for execution_spec
• Retry each live step once, with a warning

Testing

• Passed all four live compaction-recall tests on the final change: Anthropic, OpenAI Responses, OpenAI Chat Completions, and OpenRouter. Also passed the existing deterministic E2E test that verifies compaction during an active tool exchange preserves the raw call/result group:
  go test -count=1 -v -timeout=12m -tags='integration servicee2e live' ./internal/e2e -run '^TestServiceE2E(Live(Anthropic|OpenAIResponses|OpenAIChatCompletions|OpenRouter)CompactionRecall|DeterministicCompactionKeepsToolGroupRaw)$'
• Passed the repository's custom golangci-lint for ./internal/e2e with integration,servicee2e,webe2e,live,blackbox tags (0 issues).
• gofmt and git diff --check passed.

[ommybot]

This is still a draft, so I’ll hold off until it’s ready for review. If you’d like a review sooner, mention @ommybot and ask for one.

[asherdale]

@ommybot do a very thorough review now

[ommybot]

Taking a look now; I’ll post my review here when I’m done.

[ommybot]

I’m already reviewing the current head. I’ll make this a thorough pass, including the new CI retries and daemon-process fixtures.

[ommybot]

The fixture fixes look sound, but the blanket CI retries should be removed before merging. The workflow comment is the one to address: those retries turn intermittent contract failures into green checks. The worker-restart comment is a non-blocking simplification.

Separating setup from context pressure preserves the budget-admission, complete tool-result summary, and exact recall checks. I traced the checkpoint/output commit ordering, config activation, worker cleanup, and daemon signal handling; the fixture changes address the causes of the flakes without weakening those assertions.

[ommybot]

These wrappers turn any first-attempt failure into a green check when the rerun passes, including failed product assertions. For example, if assertLiveToolResultSummarized detects that compaction left the tool result outside the checkpoint, rerunning make test-live-anthropic in a fresh test environment can erase that failure from the job result. The warning doesn't preserve the failed status, and Show database logs on failure is skipped if all reruns succeed.

Remove the blanket retries and keep the fixture fixes, which address the actual causes. Any additional retry for a transient provider error should be scoped to that classified failure rather than rerunning failed assertions. This applies to all seven live targets.

[asherdale]

@ommybot keeping these as a deliberate, temporary tradeoff so unrelated merges stop turning main red. a fail-then-pass still emits a ::warning::, a second failure still fails the job, and we'll drop the retries once the remaining flaky causes are fixed. what do you think?

[ommybot]

I understand wanting to keep unrelated merges moving, but I’d still narrow this before merging. The warning makes the retry visible; it doesn’t distinguish a known flake from a new regression. If a new compaction bug loses the sample ID on its first run and passes on its second, it gets the same green check as the unrelated flake you intended to tolerate.

For a temporary exception, I’d limit retries to specific tracked flaky tests with an explicit removal condition, so other assertion failures remain failures. Whether to accept the broader risk temporarily is a maintainer decision; my recommendation remains to narrow or remove these wrappers.

[ommybot]

I think your reasoning is sound for a temporary mitigation: a frequently red main check becomes easy to ignore, and one retry can make it useful again. Repeatable regressions still fail, and the warning preserves visibility into the first failure. That is a real operational benefit.

The tradeoff is reduced sensitivity to intermittent regressions, which includes product bugs as well as test flakes. My concern is the scope: the current wrappers grant that exception to every failure across all seven targets, including tests that weren’t flaky. “Until the remaining causes are fixed” also leaves the removal point open-ended.

So I agree with the motivation, but I’d apply the temporary exception to specific tracked flakes with a concrete removal condition. Keeping the blanket version is a defensible maintainer risk decision; it just remains broader than I recommend.

[asherdale]

going with the blanket retries as a maintainer call. intermittent live failures are mostly model and provider noise, and the regular CI gates stay strict.

[ommybot]

Understood. I’ll treat the blanket retries as the agreed CI tradeoff for this PR. I have no further findings on the fixture changes.

[ommybot]

nit: this stop/restart isn't needed to separate the two phases. The setup turn is already idle, and a config-only turn without unanswered content stays idle (CaptureExecutionOpening in agentexecution/queries/inputs.sql and selectModel in agentexecution/selection.go). Each new attempt reads agents.current_config_id in ExecutionPreparationBoundary, so keeping the worker alive also checks that it picks up the changed budget. Remove this stop and the second env.startWorker at line 900 to avoid the extra rebuild and restart.

[asherdale]

@ommybot done, removed the stop/restart in 2271fb7. does this look right to you?

[ommybot]

Yes, this addresses it. The same worker now handles the setup and bridge turns, with the config update and effective-budget assertion between them. The phase boundary is preserved without the restart, and I found no new issue in the change.

[ommybot]

The worker-restart comment is addressed, and I found no new issues in this update. The same worker now picks up the reduced budget while the setup and bridge checks remain intact.

The blanket CI retries remain my only pre-merge concern, as discussed in the workflow thread.