Repository navigation
feat(oauth2): support certificate-bound OAuth2 tokens and mTLS rotation for Workload Identity Federation - #14781
Conversation
…#13955) ## Summary Adds support for actor tokens in `IdentityPoolCredentials` during certificate-bound OAuth2 STS token exchange, and implements per-cycle mTLS certificate snapshotting and single-retry rotation. --- ## Architectural Changes & Review Feedback Resolutions ### 1. Actor Token Extraction & Atomic Reads - Added package-private `IdentityPoolActorTokenSupplier` interface. - Enhanced `FileIdentityPoolSubjectTokenSupplier` with `readTokens()` to read and parse JSON credential files in a single atomic I/O operation, returning a `TokenPair` for subject and actor tokens. - Removed unreliable file timestamp caching; the file is read at most once per refresh cycle. - Added early validation in `IdentityPoolCredentialSource`: `actorTokenFieldName` must be non-empty, distinct from `subjectTokenFieldName`, and is supported only on JSON file sources. ### 2. Per-Cycle Certificate Pinning & 401 Rotation Retry - `refreshAccessToken()` snapshots the `KeyStore` from `X509Provider` at the start of each refresh cycle, creating a cycle-isolated `MtlsHttpTransportFactory` passed to the STS exchange. - On 401 Unauthorized from STS, `refreshAccessToken()` re-reads a fresh `KeyStore` from `x509Provider` and retries the exchange once (preserving original error context via `addSuppressed` if cert reload fails). - Relaxed mTLS endpoint validation to check for `.mtls.` to support custom universes and PSC endpoints. ### 3. Serialization & Binary Backward Compatibility - **SUID Compatibility**: Preserved `FileIdentityPoolSubjectTokenSupplier` class name and declared static `serialVersionUID = 7152208690659890358L;`, matching the synthetic SUID generated before this PR to prevent `InvalidClassException` on existing serialized credentials. - **Binary ABI Preservation**: Kept `public NetHttpTransport create()` return type on `MtlsHttpTransportFactory` to preserve binary compatibility for compiled callers via covariant return typing. - **Post-Deserialization Usability**: Implemented custom `readObject()` in `IdentityPoolCredentials` to reconstruct transient `x509Provider` and `MtlsHttpTransportFactory` from the serialized `credentialSource`, ensuring `refreshAccessToken()` and `createScoped()` remain fully functional after deserialization. - **KeyStore Validation**: `MtlsHttpTransportFactory.hasKeyStore()` checks `mtlsKeyStore != null && mtlsKeyStore.size() > 0`, ensuring uninitialized or empty KeyStores are not treated as mTLS-configured. ### 4. API Visibility - Made `setActorTokenSupplier`, `setActorTokenType`, and `setX509Provider` package-private on `IdentityPoolCredentials.Builder` for internal/testing use. --- ## Test Coverage - **1009 unit tests passing** (`oauth2_http` module). - **Production Path Tests**: End-to-end tests parsing JSON configs via `ExternalAccountCredentials.fromStream()` and executing `refreshAccessToken()`. - **Serialization Tests**: Verified backward compatibility against pre-PR serialized byte fixtures and verified token refresh on deserialized instances. - **Cert Pinning & Retry Tests**: Verified single-snapshot cert consistency and 401 cert rotation retries. - **Google Java Format**: Verified 100% compliance with `com.spotify.fmt:fmt-maven-plugin:2.25`. ### Manual Testing - Verified end-to-end execution against live Google STS endpoints, token exchange with actor tokens over mTLS, scoping, serialization round-trip with `readObject()` restoration, and multi-cycle refreshes. - Sample Test App: https://paste.googleplex.com/6209850244726784 - Sanitized Verification Output: https://paste.googleplex.com/6325994750345216 ## Next Steps The next phase (which will follow in a separate PR to keep reviews scoped) will introduce dynamic mTLS transport rotation natively in GAX and a 401 connection-draining interceptor required for downstream service retry logic. Based on design: https://docs.google.com/document/d/1NIKeJX86ETNAjoQA-lZIE_G_8mHspisBaL8gwNCO3dA/edit?resourcekey=0-z7teBZZk0WFIJEHL2ODZxw&tab=t.0 See b/542238030 for tracking
Removes a duplicate `google-cloud-storage` dependency declaration in `java-bigquery/google-cloud-bigquery/pom.xml`. The duplicate entry caused Maven project model validation failures during CI builds. The other declaration is in line 34 - 37.
…lters for flatten-maven-plugin (#14598) ## Summary Fixes `java.lang.VerifyError` in downstream tests (such as `java-bigquery-jdbc`) on GitHub Actions runners upgraded to **Apache Maven `3.10.0`**. ## Cause Maven `3.10.0` (Maven Resolver `2.0.24`) enables `aether.remoteRepositoryFilter.prefixes` and `aether.remoteRepositoryFilter.groupId` by default. Because `flatten-maven-plugin:1.3.0` reads dependency POMs with `repositories = null` (`[]`), the active remote repository filter rejects cached POMs in `~/.m2/repository`, causing `flatten-maven-plugin` to drop `<exclusions>` (such as `protobuf-javalite` on `io.grpc:grpc-protobuf-lite`) when generating `.flattened-pom.xml`. ## Fix Disable the default remote repository filters in `.mvn/maven.config`: ```text -Daether.remoteRepositoryFilter.prefixes=false -Daether.remoteRepositoryFilter.groupId=false ```
…patibility (#14609) Support for '#' comments in .mvn/maven.config was introduced in Apache Maven 3.9.0 (MNG-7682). Prior versions of Maven (such as 3.8.x) and IDE Maven parsers split the entire file on whitespace and reject '#' comment tokens with: Unable to parse maven.config: Unrecognized maven.config entries: [#, TODO:, ...] Remove the comment line from .mvn/maven.config so that it parses cleanly across both Maven 3.8.x and 3.9+.
…401 recovery (#14212) ### 🥞 Stacked PRs - 👉 **This PR** (*feat: implement IAM impersonation mTLS transport pinning and 401 recovery*) - ⏳ Stacked PR #14220 (*test: add mTLS in-process socket tests and live GCP WIF integration tests*) --- ## Summary Propagates the cycle-pinned mTLS transport factory into `ImpersonatedCredentials` during Workload Identity Federation with Service Account Impersonation, ensuring the IAM `generateAccessToken` call uses the exact same pinned client certificate as the STS token exchange. Also extends 401 recovery in `IdentityPoolCredentials` to catch unauthorized responses across both STS and IAM calls, reloading fresh certificates and re-executing the full refresh cycle. ## Test Coverage - **1,030 unit tests passing** across `oauth2_http` (including transport pinning lifecycle, multi-cycle rotation, 401 retry on IAM, and suppressed error handling). - **Google Java Format**: 100% compliant (`com.spotify.fmt:fmt-maven-plugin:2.25`). ### Manual Testing Sample test "app": https://paste.googleplex.com/5495919190081536 Test results: https://paste.googleplex.com/5757145073713152 See b/542238030 for tracking.
…ration tests (#14220) ## Description This PR adds comprehensive integration tests for Certificate-Bound OAuth 2.0 (mTLS), Actor Tokens, and Service Account Impersonation transport pinning across both hermetic in-process socket tests and live GCP Workload Identity Federation suites. ### Summary of Changes: 1. **Hermetic In-Process Socket Suite (`MtlsPipelineLocalTest.java`)**: - Uses JDK `HttpsServer` on `localhost` requiring client authentication (`setNeedClientAuth(true)`). - `testMtlsPipeline_verifiesPeerCertAndPayload`: Verifies server extracts client peer certificate from `SSLSession` and parses token exchange body (`grant_type`, `subject_token`, `actor_token`, `actor_token_type`). - `testMtlsPipeline_withImpersonation_usesSameCertForStsAndIam`: Verifies both STS and IAM calls execute on the exact same pinned mTLS transport and verifies the intermediate `Authorization: Bearer <token>` header. - `testMtlsPipeline_withImpersonation_401OnIam_retriesWithFreshCert`: Verifies full-cycle recovery when IAM returns 401 (reloads fresh KeyStore from disk, re-exchanges at STS with new cert, and calls IAM with new token + cert). - `testMtlsPipeline_concurrentRefreshes`: Multi-threaded refresh verifying thread-isolated transport snapshots. - `testMtlsPipeline_atomicTokenRead`: Verifies single-pass disk I/O for subject + actor tokens. 2. **Live GCP Suite Extensions (`ITWorkloadIdentityFederationTest.java`)**: - `identityPoolCredentials_withCertificateBoundWorkloadAndActorToken`: E2E verification of JSON config with certificate config and actor tokens. - `identityPoolCredentials_withProgrammaticMtlsAndActorToken`: E2E verification of programmatic builder configuration. 3. **`OAuthException.java`**: - Safely handles null HTTP error response content without NPE. ### Verification: - **Unit & Socket Tests**: 1,040 / 1,040 tests passing (including hermetic socket tests). - **Google Java Format**: 100% compliant (`mvn com.spotify.fmt:fmt-maven-plugin:2.25:check`). Source-Link: b/542238030
… IdentityPoolCredentials (#14430) ## Summary Allows `actor_token` and `actor_token_type` to be used in `IdentityPoolCredentials` with standard (non-mTLS) STS and IAM impersonation endpoints and without requiring client certificate (`certificate_config`) configuration. ## Context & Rationale In #13955, client-side guardrails (`isMtlsConfigured()` and `validateMtlsEndpoint()`) were enforced in the `IdentityPoolCredentials` constructor because Google STS initially required actor tokens to be paired with certificate-bound tokens over mTLS. Per the original design discussion, this was intentionally designed as a one-way door that could be loosened in a non-breaking manner once backend support for non-mTLS actor token exchanges was ready. ## Changes 1. **Removed Constructor mTLS Restrictions:** - Removed `isMtlsConfigured()` check in `IdentityPoolCredentials(Builder)` so actor tokens can be configured with standard `HttpTransportFactory` instances and without a `certificate` block. - Removed `validateMtlsEndpoint()` checks on `tokenUrl` and `serviceAccountImpersonationUrl`, allowing standard public endpoints (e.g., `https://sts.googleapis.com/v1/token`) to be used with actor tokens. - Preserved strict pairing validation between `actorTokenSupplier` and `actorTokenType`, as well as JSON format checks for file-based actor token extraction. 2. **401 Handling & Cert Rotation:** - When `x509Provider == null` (non-mTLS credentials), `refreshAccessToken()` uses standard transport without snapshotting a `KeyStore`, and `401 Unauthorized` errors propagate immediately without retry. - When `x509Provider != null && transportFactory instanceof MtlsHttpTransportFactory` (mTLS credentials), per-cycle certificate pinning and single-retry cert reload on `401` remain unchanged. 3. **Javadoc & Test Coverage:** - Updated class-level and builder Javadocs on `IdentityPoolCredentials`. - Updated negative constructor tests to positive verification tests (`*_succeeds`). - Added `refreshAccessToken_401WithActorTokenAndNonMtlsTransport_bubblesUpWithoutRetry` and `fromStream_fileCredentialSource_withoutCertificateConfig_andActorToken_withNonMtlsUrl_refreshesSuccessfully` to verify end-to-end non-mTLS actor token exchanges. ## Verification - All **1,035** unit tests passing in `oauth2_http` module (including all **88** tests in `IdentityPoolCredentialsTest`). - 100% compliant with `fmt-maven-plugin:2.25:check`. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…ult (#14554) ### Summary Enables the 4 mTLS + `actor_token` Workload Identity Federation integration tests in `ITWorkloadIdentityFederationTest` to run by default in CI without requiring `GOOGLE_API_CERTIFICATE_CONFIG` or external Secret Manager secrets: 1. **Default mTLS Certificate Config Fallback:** Falls back to the checked-in `testresources/mtls/certificate_config.json` (`test_cert.pem` / `test_key.pem`) when `GOOGLE_API_CERTIFICATE_CONFIG` is unset, removing the `assumeTrue` skip. 2. **Signed OIDC JWTs with `cnf.x5t#S256` and `may_act`:** Targets `oidc-jwt-1` (`OIDC_JWT_AUDIENCE`) and uses the `GOOGLE_APPLICATION_CREDENTIALS` service account private key (whose public key is registered in `oidc-jwt-1`'s `jwksJson`) to sign OIDC JWTs containing RFC 8705 certificate thumbprint binding (`cnf.x5t#S256`) and delegation (`may_act`) claims.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for actor tokens in Workload Identity Federation, enabling certificate-bound workloads and delegation. It updates IdentityPoolCredentials and ImpersonatedCredentials to handle atomic subject and actor token reads, per-cycle mTLS transport pinning, and robust 401/400 retry mechanisms with certificate reloading. Additionally, MtlsHttpTransportFactory is made serializable, and extensive hermetic tests are added to validate the mTLS pipeline. Feedback on the changes suggests wrapping keyStore.getKey in a nested try-catch block within OAuth2Utils to prevent password mismatches from disabling token caching, and deferring InputStreamReader instantiation in FileIdentityPoolSubjectTokenSupplier to avoid consuming bytes before JSON parsing.
…e-bound tokens" This reverts commit a2778ef.
| */ | ||
| FileIdentityPoolSubjectTokenSupplier(IdentityPoolCredentialSource credentialSource) { | ||
| this.credentialSource = credentialSource; | ||
| this.credentialSource = checkNotNull(credentialSource, "credentialSource cannot be null"); |
There was a problem hiding this comment.
qq: From what I see FileIdentityPoolSubjectTokenSupplier is an existing class, is this guaranteed to not be a behavior change? Would null credentialSource worked before?
There was a problem hiding this comment.
FileIdentityPoolSubjectTokenSupplier is package-private (class FileIdentityPoolSubjectTokenSupplier) and is only instantiated inside IdentityPoolCredentials after verifying credentialSource != null. Previously, a null credentialSource would have thrown a NullPointerException on the first call to getSubjectToken() (at credentialSource.getCredentialLocation()); Preconditions.checkNotNull fails fast with the same NullPointerException at construction time.
| "Actor tokens are only supported for JSON-formatted credential files with distinct field" | ||
| + " names."); |
There was a problem hiding this comment.
qq: What does distinct field names mean in this case? This is checking for format type == text, but the error message suggests something else
There was a problem hiding this comment.
Good catch! Updated the error message in e05fa17 to "Actor tokens are only supported for JSON-formatted credential files."
| static String parseToken(InputStream inputStream, IdentityPoolCredentialSource credentialSource) | ||
| throws IOException { | ||
| if (credentialSource.credentialFormatType == CredentialFormatType.TEXT) { | ||
| private String getToken(@Nullable String targetFieldName) throws IOException { |
There was a problem hiding this comment.
qq, should targetFieldName allowed to be nullable? What cases would be supported for this? Would it just be easier to assert non-null here?
There was a problem hiding this comment.
targetFieldName is null when reading a TEXT-formatted subject token: getSubjectToken() passes credentialSource.subjectTokenFieldName, which is null for TEXT format (since the entire file content is read as the raw token string rather than extracting a JSON field). Added an explanatory comment above getToken in e05fa17 to clarify this.
| static boolean isDefaultOrMtlsTransportFactory(@Nullable HttpTransportFactory transportFactory) { | ||
| return transportFactory == null | ||
| || transportFactory instanceof OAuth2Utils.DefaultHttpTransportFactory | ||
| || transportFactory.getClass() == MtlsHttpTransportFactory.class | ||
| || (transportFactory instanceof MtlsHttpTransportFactory | ||
| && !((MtlsHttpTransportFactory) transportFactory).hasKeyStore()); | ||
| } |
There was a problem hiding this comment.
nit: I think I've seen this type of check added a few times now. I wonder if this can be put in a shared util?
There was a problem hiding this comment.
Done in e05fa17 — moved isDefaultOrMtlsTransportFactory, hasMtlsKeyStore, and isKeylessMtlsTransportFactory into OAuth2Utils and updated IdentityPoolCredentials and ImpersonatedCredentials to use the shared helpers.
| ImpersonatedCredentials impersonated = getImpersonatedCredentials(); | ||
| if (impersonated != null) { | ||
| return impersonated.refreshAccessToken(); | ||
| } |
There was a problem hiding this comment.
nit: I don't recall the logic for this from the old PRs, can you add a comment that explains this logic and why it's needed?
There was a problem hiding this comment.
Done in e05fa17 — added an inline comment in both PluggableAuthCredentials and AwsCredentials explaining why getImpersonatedCredentials() must be reconciled with the parent builder's transportFactory when service account impersonation is configured.
| if (content == null || content.trim().isEmpty()) { | ||
| OAuthException oauthException = | ||
| new OAuthException( | ||
| "http_error_" + e.getStatusCode(), e.getStatusMessage(), null, e.getStatusCode()); |
There was a problem hiding this comment.
qq, what is this format string exception http_error_{status}_{message}? I think it may just be clearer as a string
There was a problem hiding this comment.
Those are actually two separate arguments to the OAuthException(String errorCode, @Nullable String errorDescription, @Nullable String errorUri) constructor ("http_error_" + e.getStatusCode() is errorCode, and e.getStatusMessage() is errorDescription), rather than a single concatenated string. OAuthException enforces Preconditions.checkNotNull(errorCode) and formats the message as "Error code " + errorCode + ": " + errorDescription. Added an inline comment in e05fa17 to make this clearer.
| JsonParser parser = OAuth2Utils.JSON_FACTORY.createJsonParser(content); | ||
| GenericJson errorResponse = parser.parseAndClose(GenericJson.class); | ||
| if (errorResponse == null) { | ||
| OAuthException oauthException = | ||
| new OAuthException( | ||
| "http_error_" + e.getStatusCode(), e.getStatusMessage(), null, e.getStatusCode()); | ||
| oauthException.initCause(e); | ||
| return oauthException; | ||
| } | ||
|
|
||
| String errorCode = null; | ||
| String errorDescription = null; | ||
| Object rawError = errorResponse.get("error"); | ||
| if (rawError instanceof String) { | ||
| errorCode = (String) rawError; | ||
| } else if (rawError instanceof Map) { | ||
| Map<?, ?> errorMap = (Map<?, ?>) rawError; | ||
| if (errorMap.get("status") instanceof String) { | ||
| errorCode = (String) errorMap.get("status"); | ||
| } | ||
| if (errorMap.get("message") instanceof String) { | ||
| errorDescription = (String) errorMap.get("message"); | ||
| } | ||
| } |
There was a problem hiding this comment.
nit: Can you add comments to explain what this big block of code is doing? This looks to be parsing out status/ messages from the error details? What is the format of that
There was a problem hiding this comment.
Done in e05fa17 — added inline comments documenting the two JSON error formats supported here (standard OAuth 2.0 RFC 6749 flat JSON {"error": "...", "error_description": "...", "error_uri": "..."} from the STS token endpoint vs. nested Google API Standard Error google.rpc.Status JSON {"error": {"code": 401, "status": "UNAUTHENTICATED", "message": "..."}} from the IAM Credentials API) and the fallback order.
…nsport helpers
| // Token and IAM endpoints can return errors in two JSON schemas: | ||
| // 1. RFC 6749 Section 5.2 OAuth 2.0 format, where "error" is a top-level String: | ||
| // {"error": "invalid_grant", "error_description": "...", "error_uri": "..."} | ||
| // 2. Google API Standard Error format (google.rpc.Status / AIP-193), returned by GFE/STS on | ||
| // mTLS or HTTP 401/403 authentication errors, where "error" is a nested JSON object: | ||
| // {"error": {"code": 401, "status": "UNAUTHENTICATED", "message": "..."}} |
There was a problem hiding this comment.
Thanks for the comment. This is for the token/ IAM endpoints, does the above check come from a different endpoint response?
This method is a bit long, can the token/ IAM endpoint logic here be moved to a helper?
There was a problem hiding this comment.
The whole createFromHttpResponseException method handles error responses from the OAuth, STS, and IAM endpoints — the checks above (content == null || content.trim().isEmpty() and errorResponse == null) are just the guards for when an HTTP error response has an empty or null body before inspecting the parsed JSON fields.
Extracted the JSON field extraction into a createFromJsonErrorResponse helper in 65d592b and updated the comments to make that distinction clearer.
lqiu96
left a comment
lqiu96
left a comment
There was a problem hiding this comment.
This is merging already reviewed code into main. Reviewed the new changes and those LGTM
Summary
Merges the
oauth2-bound-tokensfeature branch intomain, adding end-to-end support for certificate-bound OAuth 2.0 tokens and automatic mTLS certificate rotation in Workload Identity Federation (IdentityPoolCredentialsandImpersonatedCredentials).Included Feature Branch PRs
feat(oauth2): Extract actor tokens for cert-bound OAuth2 STS exchangefeat(oauth2): implement IAM impersonation mTLS transport pinning and 401 recoverytest(oauth2): add mTLS in-process socket tests and live GCP WIF integration testsfeat(oauth2): support non-mTLS token URLs and unbound actor tokens in IdentityPoolCredentialstest(auth): enable mTLS and actor_token WIF integration tests by defaultKey Capabilities
Actor Token Support for Hybrid Certificate-Bound WIF (
IdentityPoolActorTokenSupplier,FileIdentityPoolSubjectTokenSupplier):actor_token(text or JSON viaactor_token_field_nameandactor_token_type) for STS token exchange.mtime, size, file key) and JWTexpclaims with a 5-minute expiration margin to avoid redundant disk reads.Per-Cycle mTLS Transport Pinning & Certificate Rotation (
IdentityPoolCredentials,ImpersonatedCredentials,MtlsHttpTransportFactory):KeyStoreonce per refresh cycle so leaf certificate extraction, the STS token exchange, and downstream IAM Service Account impersonation (generateAccessToken/generateIdToken) share the exact same certificate and private key snapshot.401 Unauthorized, OAuthinvalid_grant, and transient TLS/KeyStore load failures during non-atomic split cert/key writes by reloading theKeyStoreonce and retrying when a certificate change is detected (OAuth2Utils.hasCertificateChanged).ImpersonatedCredentialsconfigured withMtlsProvideras well as delegatedImpersonatedCredentialschains.Test Coverage:
IdentityPoolCredentialsTest,ImpersonatedCredentialsTest,FileIdentityPoolSubjectTokenSupplierTest,MtlsHttpTransportFactoryTest,OAuth2UtilsTest, andOAuthExceptionTest.MtlsPipelineLocalTest) and live GCP Workload Identity Federation integration tests (ITWorkloadIdentityFederationTest).Manual & Live Verification
End-to-end live GCP and real-TLS socket fault-injection testing was executed against
oauth2-bound-tokens(65d592bd783):sts.mtls.googleapis.com): File-based JSONsubject_token+actor_tokenwith RFC 8705cnf.x5t#S256certificate binding and atomic single-supplier file reads.readObject) +createScoped(): Deserialization ofIdentityPoolCredentialswith mTLS state reconstruction across multiple live STS refresh cycles.STS mTLS->IAM Impersonation mTLS->GCS mTLS): Cert-bound workload + actor token exchange atsts.mtls.googleapis.com, service account impersonation atiamcredentials.mtls.googleapis.com, and bucket access atstorage.mtls.googleapis.com(HTTP 200).Suppliers+MtlsHttpTransportFactory) Live 3-Hop mTLS: End-to-endSTS mTLS->IAM mTLS->GCS mTLS(HTTP 200) and verification ofOAuth2UtilsmTLS helpers (hasMtlsKeyStore,isKeylessMtlsTransportFactory,isDefaultOrMtlsTransportFactory).subject_token+actor_tokenexchange over publicsts.googleapis.com->iamcredentials.googleapis.com->storage.googleapis.com(HTTP 200).trust_chainhandshake againststs.mtls.googleapis.comviaCertificateIdentityPoolSubjectTokenSupplier.PluggableAuthCredentialsImpersonation Transport Reconciliation: RebuildingPluggableAuthCredentialswith a customHttpTransportFactoryand verifying both STS andImpersonatedCredentialsuse the reconciled transport against live STS, IAM, and GCS (HTTP 200).FileIdentityPoolSubjectTokenSupplierTEXT format subject/actor token handling andOAuthException.createFromJsonErrorResponseparsing across live STS 400 errors, nestedgoogle.rpc.StatusJSON, and empty-body HTTP error fallbacks.401 UnauthorizedIAM recovery (both fresh and cached STS token eviction),400 invalid_grantSTS cert rotation recovery, and 8-thread concurrent mTLS refresh.