feat(storage): add adaptive bounded range reads for open-ended downloads by rajeevpodar · Pull Request #16586 · googleapis/google-cloud-cpp

rajeevpodar · 2026-10-23T09:11:59Z

When an application reads a whole object, or from an offset to the end (ReadFromOffset(N)), the client sends one unbounded GET. Many readers (Parquet, ArrayRecord, Riegeli, ...) only need the start of the download and then close the stream. Closing early aborts the transfer: the connection is torn down, and every byte the service already sent is discarded.

This PR adds the experimental storage_experimental::AdaptiveReadRangeOption. With the option set, open-ended downloads are requested as a series of growing, bounded windows instead of one unbounded GET:

caller: ReadObject(bucket, object, ReadFromOffset(N))   // or no range
client: GET Range: bytes=N-(N+1MiB-1)                   // first window: 1 MiB
        caller keeps reading: next window, 2, 4, 8, ... up to 64 MiB
        caller closes early: at most one small window outstanding

Changes

AdaptiveObjectReadSource (internal/adaptive_object_read_source.{h,cc}): a new ObjectReadSource decorator. Each window is a regular, retried download of a ReadRange(), so a retry resumes inside the window. The next window opens on a background thread once a read reaches prefetch_threshold (default 50%) of the current window.
Consistency and checksums: every window pins the generation returned by the first window. ObjectReadStream::size() and the headers describe the full download, and full-object CRC32C/MD5 validation covers all the windows.
Ignored ranges: if the first window shows that the service ignored the range (decompressive transcoding, or a JSON API response without content-range), the download switches to a regular unbounded read, at the cost of one extra request.
Scope: ReadRange() and ReadLast() downloads are unchanged. Windows are not hedged.
Attribution: with the JSON API, window requests add gccl-feat/adaptive-read to x-goog-api-client.
Environment variable: GOOGLE_CLOUD_CPP_STORAGE_ENABLE_ADAPTIVE_READ_RANGE=true|false turns the feature on, or off as a kill switch, without code changes. This is for applications where a library creates the Client.

Trade-off: a full sequential read takes more requests, 22 for a 1 GiB object with the defaults.

Usage

namespace gcs = google::cloud::storage;
namespace gcs_ex = google::cloud::storage_experimental;

// Every read from this client (defaults: 1 MiB first window, up to 64 MiB).
auto client = gcs::Client(
    google::cloud::Options{}.set<gcs_ex::AdaptiveReadRangeOption>({}));

// A single call, on a client that does not set the option.
auto reader = other_client.ReadObject(
    "bucket", "object", gcs::ReadFromOffset(4096),
    google::cloud::Options{}.set<gcs_ex::AdaptiveReadRangeOption>({}));

Testing

New unit tests: internal/adaptive_object_read_source_test.cc (58) and internal/connection_impl_adaptive_read_test.cc (12), plus new cases in internal/rest/stub_test.cc, client_test.cc and internal/grpc/default_options_test.cc. All pass locally (CMake, GCC 15). The two new suites also passed 30 shuffled repeats.
New integration tests in tests/object_read_range_integration_test.cc and tests/decompressive_transcoding_integration_test.cc. They compile, but I haven't run them locally (no testbench), so I'm relying on CI.

Before this is ready for review

Cap the drain on Close() at 1 MiB, whatever initial_window is.
Drop IfMetagenerationMatch / IfMetagenerationNotMatch after the first window. The pinned generation already keeps the data consistent.
Propagate CallContext to prefetches, so their spans are children of the ReadObject span.
Revisit the per-client prefetch thread pool for applications that create a client per reader.
TSAN build, a libcurl-level check that windows end with 206 and reuse the connection, and a throughput benchmark.

Follow-ups in separate PRs: gRPC attribution (needs a generator change) and AsyncClient::ReadObject.

Add the experimental `storage_experimental::AdaptiveReadRangeOption`. When it is set, a `ReadObject()` download that does not bound its range (the full object, or `ReadFromOffset()`) is requested as a series of growing, bounded windows (1 MiB, doubling up to 64 MiB) instead of a single unbounded GET. Applications that read the beginning of an object and then close the stream leave at most one small window outstanding, so the service sends far fewer bytes that are then discarded, and the connection can be reused instead of torn down. - `AdaptiveObjectReadSource` is a new `ObjectReadSource` decorator. Each window is a regular, retried download of a `ReadRange()`, so a retry resumes inside the window. The next window is opened on a background thread once a read reaches `prefetch_threshold` (default 50%) of the current window. - Every window pins the generation returned by the first window. `ObjectReadStream::size()` and the headers describe the full download, and full-object CRC32C/MD5 validation covers all the windows. - If the first window shows that the service ignored the range (decompressive transcoding, or a JSON API response without `content-range`), the download switches to a regular unbounded read, at the cost of one extra request. - `ReadRange()` and `ReadLast()` downloads are unchanged. The windows are not hedged. - With the JSON API, window requests add `gccl-feat/adaptive-read` to the `x-goog-api-client` header. - `GOOGLE_CLOUD_CPP_STORAGE_ENABLE_ADAPTIVE_READ_RANGE` (`true` or `false`) turns the feature on or off without code changes, for applications where a library creates the `Client`.

gemini-code-assist

Code Review

This pull request implements adaptive bounded range reads for Google Cloud Storage, allowing open-ended downloads to be fetched in growing, bounded windows. This feature is configurable via AdaptiveReadRangeOption and the GOOGLE_CLOUD_CPP_STORAGE_ENABLE_ADAPTIVE_READ_RANGE environment variable. The feedback primarily addresses style guide violations, specifically recommending the use of std::string_view over absl::string_view and requesting explicit type declarations instead of auto where domain types, StatusOr, or function return types are obscured.

gemini-code-assist · 2026-10-23T09:15:06Z

+#include "google/cloud/internal/make_status.h"
+#include "absl/strings/numbers.h"
+#include "absl/strings/str_cat.h"
+#include "absl/strings/string_view.h"


The repository style guide prefers std::string_view over absl::string_view. Please include <string_view> instead of "absl/strings/string_view.h".

#include <string_view>

References

Use std::string_view instead of absl::string_view. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+  // The size is `*` when the service does not know it, `SimpleAtoi()` rejects
+  // that value.
+  std::int64_t size = 0;
+  if (!absl::SimpleAtoi(absl::string_view(f->second).substr(pos + 1), &size)) {


The repository style guide prefers std::string_view over absl::string_view. Please use std::string_view here.

if (!absl::SimpleAtoi(std::string_view(f->second).substr(pos + 1), &size)) {

References

Use std::string_view instead of absl::string_view. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+                                   std::int64_t limit) {
+  std::vector<char> buffer(static_cast<std::size_t>(
+      std::clamp<std::int64_t>(limit, 1, kMaxDiscardBuffer)));
+  auto metadata = EmptyResult();


The repository style guide disallows using auto when it hides domain objects or function return types. Please explicitly type metadata as ReadSourceResult.

ReadSourceResult metadata = EmptyResult();

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+    // All the data in this window has been delivered. Reading the end of the
+    // response completes the request, so the connection can be reused. Errors
+    // here cannot lose any data, and are ignored.
+    auto metadata = Discard(*source, 0);


The repository style guide disallows using auto when it hides StatusOr<T>, domain objects, or function return types. Please explicitly type metadata as StatusOr<ReadSourceResult>.

StatusOr<ReadSourceResult> metadata = Discard(*source, 0);

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+  auto const nominal = NextAdaptiveWindowSize(window_nominal_, settings_);
+  auto const window = MakeWindow(offset_, nominal);
+  StatusOr<std::unique_ptr<ObjectReadSource>> source = [&] {
+    auto prefetched = TakePrefetch(window);


The repository style guide disallows using auto when it hides StatusOr<T>, domain objects, or function return types. Please explicitly type prefetched as std::optional<StatusOr<std::unique_ptr<ObjectReadSource>>>.

std::optional<StatusOr<std::unique_ptr<ObjectReadSource>>> prefetched = TakePrefetch(window);

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+    AdaptiveObjectReadSource::Executor executor, AdaptiveReadSettings settings,
+    std::int64_t offset, std::optional<std::int64_t> generation) {
+  AdaptiveObjectReadSource::Window first{
+      offset,


The repository style guide disallows using auto when it hides StatusOr<T>, domain objects, or function return types. Please explicitly type source as StatusOr<std::unique_ptr<ObjectReadSource>>.

StatusOr<std::unique_ptr<ObjectReadSource>> source = (*factory)(first);

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

@@ -406,9 +409,20 @@ StatusOr<ObjectMetadata> StorageConnectionImpl::GetObjectMetadata(
 StatusOr<std::unique_ptr<ObjectReadSource>> StorageConnectionImpl::ReadObject(
    ReadObjectRangeRequest const& request) {
  auto current = google::cloud::internal::SaveCurrentOptions();


The repository style guide disallows using auto when it hides domain objects or function return types. Please explicitly type current as google::cloud::internal::ImmutableOptions.

google::cloud::internal::ImmutableOptions current = google::cloud::internal::SaveCurrentOptions();

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+  auto const settings = MakeAdaptiveReadSettings(
+      current->get<storage_experimental::AdaptiveReadRangeOption>());


The repository style guide disallows using auto when it hides domain objects or function return types. Please explicitly type settings as AdaptiveReadSettings const.

AdaptiveReadSettings const settings = MakeAdaptiveReadSettings(current->get<storage_experimental::AdaptiveReadRangeOption>());

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+  auto const adaptive_reads =
+      GetEnv("GOOGLE_CLOUD_CPP_STORAGE_ENABLE_ADAPTIVE_READ_RANGE");


The repository style guide disallows using auto when it hides std::optional<T>, domain objects, or function return types. Please explicitly type adaptive_reads as std::optional<std::string> const.

std::optional<std::string> const adaptive_reads = GetEnv("GOOGLE_CLOUD_CPP_STORAGE_ENABLE_ADAPTIVE_READ_RANGE");

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

gemini-code-assist · 2026-10-23T09:15:07Z

+                   request.bucket_name(), "/o/",
+                   UrlEncode(request.object_name())),
+      AdaptiveReadApiClientHeader(options));
  auto headers = AddHeaders(options, builder);


The repository style guide disallows using auto when it hides StatusOr<T>, domain objects, or function return types. Please explicitly type headers as Status.

Status headers = AddHeaders(options, builder);

References

Reject auto when it hides StatusOr, domain objects, or function return types. ^(link)

codecov · 2026-10-23T09:56:45Z

Codecov Report

❌ Patch coverage is 97.79857% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.44%. Comparing base (6831cf3) to head (54d9794).

Files with missing lines	Patch %	Lines
...orage/internal/adaptive_object_read_source_test.cc	98.02%	20 Missing ⚠️
...ud/storage/internal/adaptive_object_read_source.cc	95.44%	16 Missing ⚠️
...age/internal/connection_impl_adaptive_read_test.cc	99.08%	3 Missing ⚠️
google/cloud/storage/internal/connection_impl.cc	97.56%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16586      +/-   ##
==========================================
+ Coverage   92.38%   92.44%   +0.05%     
==========================================
  Files        2266     2270       +4     
  Lines      219732   221543    +1811     
==========================================
+ Hits       203009   204798    +1789     
- Misses      16723    16745      +22

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rajeevpodar temporarily deployed to internal October 23, 2026 09:12 — with GitHub Actions Inactive

product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Oct 23, 2026

gemini-code-assist Bot reviewed Oct 23, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(storage): add adaptive bounded range reads for open-ended downloads - #16586

feat(storage): add adaptive bounded range reads for open-ended downloads#16586
rajeevpodar wants to merge 1 commit into
mainfrom
adaptiveReadRange

rajeevpodar commented Oct 23, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Uh oh!

codecov Bot commented Oct 23, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		auto const settings = MakeAdaptiveReadSettings(
		current->get<storage_experimental::AdaptiveReadRangeOption>());

		auto const adaptive_reads =
		GetEnv("GOOGLE_CLOUD_CPP_STORAGE_ENABLE_ADAPTIVE_READ_RANGE");

Conversation

rajeevpodar commented Oct 23, 2026

Changes

Usage

Testing

Before this is ready for review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Oct 23, 2026

Codecov Report

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant