[diwu-sf]

What changed?

With external storage, the workflow worker no longer downloads stored payloads it never reads: the inputs of activities, child workflows, signals and Nexus operations that the workflow itself started. Activity workers, Nexus workers and clients still download everything as before.

Why?

Each large activity argument was downloaded twice: once by the workflow worker, which ignores it, and once by the activity worker, which needs it. When a workflow was replayed after a cache miss, the workflow worker downloaded every past argument again.

Replay matches these events by IDs and types only, so their payloads are never needed by the workflow worker.

Our workload

We run an AI agent on Temporal. On every turn, the workflow passes the whole conversation so far to an LLM activity. A conversation grows to 500k tokens, which is about 2M characters, or a 2 MB payload per activity call by the end. Payloads over the default 256 KiB threshold go to external storage, so every late turn uploads a new copy of the conversation as an activity argument.

To measure this, we ran 20-turn conversations that grow to 500k tokens, with many conversations running at once:

Workload	Uploaded	Downloaded before	Downloaded after
200 conversations, normal caching	3,800 objects, 4.4 GiB	7,400 objects, 8.3 GiB	3,800 objects, 4.4 GiB
100 conversations, a cache miss on almost every workflow task	1,900 objects, 2.2 GiB	20,617 objects, 18.2 GiB	1,900 objects, 2.2 GiB

• With normal caching, downloads drop by half. Every stored object is now downloaded exactly once.
• With frequent cache misses, downloads drop by about 8×. In this workload the only large payloads in history were activity arguments, so replays no longer download anything extra. Workflows with large activity results still download those again on a cache miss, since the workflow needs them.
• Uploads don't change.

Other SDKs

Python doesn't have this problem

The Python SDK is built on the Rust core, and its workflow worker doesn't read history events itself. The core matches the workflow's commands against history and gives Python an activation: a list of things the workflow has to react to, such as its arguments, signals, queries, updates, and activity, child workflow and Nexus results. Python only downloads external storage payloads in that activation (decode_activation in temporalio/bridge/worker.py). No activation job carries the inputs the workflow sent, so Python already downloads each activity argument once, which is what this change does for Java.

Go has the same problem

The Go SDK's workflow worker downloads external storage payloads in every history event it receives, both in the polled workflow task (internal/internal_task_pollers.go) and in later history pages. The only payloads it skips are search attributes (visitProtoPayloads in internal/payload_visitor.go). So, like Java before this change, it downloads each large activity, child workflow, signal and Nexus input it sent, and downloads them all again on a cache miss. A similar fix would work there.

Breaking changes?

None.

Server PR

Not required.

Testing

• New unit test for which payloads are skipped.
• New end-to-end tests: large activity arguments, child workflow inputs and signal inputs are each downloaded once, by the worker that uses them, including when history is replayed. Before this change each was downloaded three times. Large activity results still reach the workflow.
• External storage, worker, replay, activity, child workflow and signal tests pass.

[diwu-sf]

@jmaeagle99 this is related to the external storage changes and fixes a double-download issue that we saw when running on Google storage, the root cause is cloud agnostic.