[wallrj-cyberark]

The nightly GKE e2e test fails because the CI project now forbids the DNS-based control-plane endpoint that #845 relies on. This PR reaches the cluster through the Connect gateway instead.

What is broken

Since the 2026-10-21 nightly, gcloud container clusters create in hack/e2e/test.sh fails with:

Operation denied by org policy: ["customConstraints/custom.disableGkeDnsEndpoint": "Prevents the the use and enablement of Google-managed DNS names for the GKE Control Plane."]

The 2026-10-20 nightly passed on the same commit, so the policy is new. Failing run: https://github.com/jetstack/jetstack-secure/actions/runs/39709299106/job/119596540932

The project now forbids all three ways a GitHub-hosted runner could reach the API server directly: public nodes, a public control-plane IP and the DNS endpoint.

What this changes

• The cluster is created with --enable-fleet instead of --enable-dns-access. This registers it as a fleet membership in the same project.
• The script fetches credentials with gcloud container fleet memberships get-credentials, so kubectl and helm go through the Connect gateway. Access is authorised with IAM.
• Nodes and control plane stay private, as in Create the e2e GKE cluster with private nodes and no public control-plane IP #845.

The script uses no exec, port-forward or attach, which the gateway may not support. kubectl logs --follow and kubectl wait are plain streaming requests.

What might still need doing in the project

I could not test this before pushing: I do not have access to the CI project. If the test-e2e run fails, the likely causes are:

• The GKE Hub and Connect gateway APIs (gkehub.googleapis.com, connectgateway.googleapis.com) are not enabled.
• The gke-cluster-creation service account lacks roles/gkehub.admin (to register the cluster) or roles/gkehub.gatewayEditor and roles/gkehub.viewer (to use the gateway).

The run log will say which. Someone with project admin rights will then need to grant it.

Testing

The test-e2e label on this PR runs the GKE job. I will link the result here.

[with Claude]

[wallrj-cyberark]

The test-e2e run fails because the GKE Hub API is not enabled in machineidentitysecurity-jsci-e. Someone with project admin rights needs to run this once:

gcloud services enable gkehub.googleapis.com connectgateway.googleapis.com \
  --project machineidentitysecurity-jsci-e
gcloud projects add-iam-policy-binding machineidentitysecurity-jsci-e \
  --member serviceAccount:gke-cluster-creation@machineidentitysecurity-jsci-e.iam.gserviceaccount.com \
  --role roles/gkehub.admin
gcloud projects add-iam-policy-binding machineidentitysecurity-jsci-e \
  --member serviceAccount:gke-cluster-creation@machineidentitysecurity-jsci-e.iam.gserviceaccount.com \
  --role roles/gkehub.gatewayEditor

The role bindings may already be covered by the account's existing roles. The run has not got far enough to tell. After that, re-run the test-e2e job on this PR.

We will not know whether the org policy accepts this cluster until the API is enabled. GKE checks the fleet first.

Error from run 39775167645

https://github.com/jetstack/jetstack-secure/actions/runs/39775167645/job/119812147262

ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Failed to get fleet default config: : error when calling fleet service: generic::permission_denied: Write access to project 'projects/machineidentitysecurity-jsci-e/locations/europe-west1' was denied: producer API gkehub-pa.googleapis.com access denied because consumer service gkehub.googleapis.com is not activated on the project [...] please ensure the gkehub.googleapis.com API is enabled.

[wallrj-cyberark]

Thanks for enabling the API. The re-run got further, but cluster creation now fails on a third org policy, and only an org policy admin can fix it:

Constraint constraints/compute.restrictPrivateServiceConnectProducer violated for projects/machineidentitysecurity-jsci-e. Forwarding Rule projects/machineidentitysecurity-jsci-e/regions/europe-west1/forwardingRules/gk3-test-secretless-261022-160708-e3fc49ac-30e8335a-pe targeting a Service Attachment in projects/346361970074 is not allowed.

A GKE cluster with a private control plane reaches it through a Private Service Connect endpoint in our VPC. That endpoint points at a service attachment in a Google-owned GKE project (346361970074 here). The policy does not allow that producer, so GKE cannot create the endpoint.

This policy is also new. The nightlies created clusters with the same --enable-private-endpoint flag until 2026-10-20.

Please ask whoever owns the org policies to let this project create PSC endpoints to GKE's producer. That can be an exception for machineidentitysecurity-jsci-e, or an allowed value that covers Google's GKE service attachments. Once that is in place, re-run test-e2e here. The fleet registration and DNS-endpoint changes in this PR are still needed. The re-run showed the cluster now passes both of those checks.

What I tried that did not work

• Run 39775167645 attempt 2: this PR as it stands. Fleet registration and the DNS-endpoint policy pass; creation fails at the PSC endpoint.
• Run 39908174477: I added --master-ipv4-cidr, hoping GKE would use VPC Network Peering instead of PSC. It still created the gk3-…-pe forwarding rule and failed the same way. I have removed that commit.
• No control-plane access mode is left that avoids all three policies. Public IP, DNS endpoint and PSC private endpoint are each forbidden.