Skip to content

Add a GCS external payload storage driver - #3151

Open
diwu-sf wants to merge 20 commits into
temporalio:mainfrom
diwu-sf:codex/gcs-payload-storage
Open

diwu-sf wants to merge 20 commits into
temporalio:mainfrom
diwu-sf:codex/gcs-payload-storage

Conversation

@diwu-sf

@diwu-sf diwu-sf commented Oct 16, 2026 •

Copy link
Copy Markdown

What changed?

Added an experimental Google Cloud Storage driver for external payloads and an adapter for the official Google Cloud Storage Java client.

Object names follow the existing GCS drivers' encoding and namespace layout, with a namespace-preserving fallback for long names. Claims use object_name. Uploads leave existing objects unchanged, and downloads verify SHA-256 before decoding the complete payload, including metadata.

The application owns the Google client and executor. Includes setup documentation and automatic integration tests against fsouza/fake-gcs-server. Gradle downloads and caches a checksum-verified native binary; JUnit starts the server on a free local port and cleans up afterward.

The quickstart uses StorageOptions.grpc() for automatic DirectPath selection and includes a short note on preserving Google's runtime dependencies, service registrations, and Conscrypt native libraries when packaging an application.

Why?

Store large Temporal payloads in GCS without writing a custom storage driver.

Related to temporalio/features#924.

Breaking changes?

No existing SDK API changes. The optional Google client adapter brings newer protobuf and gRPC dependencies.

Server PR

Not required.

Testing

  • All 54 GCS tests passed without skips, including seven tests using the real Google client against fake GCS.
  • Integration coverage includes canonical claims, Unicode object names, a 6 MiB payload, missing objects and buckets, concurrent conditional uploads, integrity failures, empty namespaces, and long-name fallback.
  • ./gradlew --offline spotlessApply passed.
  • Both GCS modules passed build, including tests and Javadocs, with Java 21 and Java 8 release compatibility checks.
  • Confirmed that the existing dev-server preparation task also prepares the cached GCS emulator for offline tests.
  • Manual validation with a temporary GCE VM and a co-located GCS bucket confirmed ALTS DirectPath for uploads and downloads.
  • The full repository test suite was not run.

diwu-sf and others added 8 commits October 21, 2026 00:21
The README example used a bounded ThreadPoolExecutor with an abort
policy. A store or retrieve submits one task per payload, so any batch
larger than the pool and queue combined was rejected and failed as a
whole, and failed again on every retry.

Recommend Executors.newVirtualThreadPerTaskExecutor() instead, which
runs each GCS call on its own virtual thread and never rejects work.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
store() built each claim before uploading and then read the object
name back out of the claim map by key. Keep the object name and hash on
PreparedPayload instead, and build the claim once the upload succeeds.

retrieve() validated each hash with String.matches, which compiles the
regex on every call. Use a precompiled Pattern.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@diwu-sf

diwu-sf commented Oct 28, 2026

Copy link
Copy Markdown
Author

@jmaeagle99 this java GCS support is ready for review now, thanks

The README only covered virtual threads, which require Java 21, while
the SDK supports Java 8. Testing against real GCS showed that a fixed
thread pool sized to the worker's slots performs the same, and that a
smaller pool queues storage calls and slows workflow tasks.

Explain that worker slots, not the executor, limit concurrent GCS calls
and payload memory, since the SDK waits on each storage operation from
the worker thread. Warn against rejecting queues and CallerRunsPolicy.

Recommend gRPC only for workers in Google Cloud, where DirectPath
outperformed HTTP, and HTTP elsewhere, where gRPC uploads were much
slower. Note that the gRPC client exports metrics to Cloud Monitoring by
default and how to avoid the resulting permission warnings.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@diwu-sf

diwu-sf commented Oct 30, 2026 •

Copy link
Copy Markdown
Author

Tested against real GCS

I ran this branch end to end against real Google Cloud Storage and Temporal, over both HTTP and gRPC. Tests ran from AWS and from a GCP VM in the same region as the bucket.

No bugs found. Over 150,000 storage calls, nothing failed and every payload came back exactly as stored. Large payloads, unusual names, tampered files, cancelled uploads and worker restarts all behaved correctly.

The main test was an AI agent that sends its whole conversation to each LLM call, 200 conversations at once. A 500k-token conversation is about 2 MB per call. On the GCP VM:

HTTP gRPC
Total time 53 s 33 s
Typical upload 333 ms 62 ms
Timeouts (retried automatically) 70 0

Running this on GCP

  1. Use gRPC (StorageOptions.grpc()) when workers run in GCP, in the same region as the bucket. It was about twice as fast as HTTP. Outside GCP, use HTTP; gRPC was 3–7× slower there.
  2. Give the worker's service account roles/monitoring.metricWriter, or call setEnableGrpcClientMetrics(false). Otherwise the gRPC client fills the logs with permission warnings.
  3. Limit load with the worker's slot settings, not the storage thread pool. 32 slots handled all 200 conversations at full speed in a 4 GB heap.
  4. On Java 8–20, use a fixed thread pool at least as large as the worker's slots. A 16-thread pool was about 20% slower.
  5. Make the workflow cache large enough for active workflows. Each cache miss downloads all of that workflow's stored payloads again.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants