ASM with remote fleet

[vedantthapa]

Hello, I'm trying to provision Anthos Service Mesh on a GKE cluster in a different google cloud project using the fleet API. The service mesh status shows healthy, however, while provisioning an ingress gateway, the istio-proxy container fails to get into a ready state with the following logs:

2024-01-31T16:48:52.804710Z	warning	envoy config external/envoy/source/common/config/grpc_stream.h:163	StreamAggregatedResources gRPC config stream to xds-grpc closed: 16, transport: per-RPC creds failed due to error: token manager failed to generate access token: federated token response does not have access token. {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}	thread=16
2024-01-31T16:49:06.720599Z	info	token	Prepared federated token request for aud "identitynamespace:<FLEET-HOST-PROJECT-ID>.svc.id.goog:https://gkehub.googleapis.com/projects/<FLEET-HOST-PROJECT-ID>/locations/northamerica-northeast1/memberships/<MEMBERSHIP-NAME>"
2024-01-31T16:49:06.744080Z	error	token	federated token response does not have access token{"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}
2024-01-31T16:49:06.745722Z	warning	envoy config external/envoy/source/common/config/grpc_stream.h:163	StreamAggregatedResources gRPC config stream to xds-grpc closed: 16, transport: per-RPC creds failed due to error: token manager failed to generate access token: federated token response does not have access token. {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}	thread=16
2024-01-31T16:49:10.936590Z	error	googleca	Failed to create certificate: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: token exchange failed: exchange failed all retries, last error: token exchange request failed: status code 400 body {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}, (aud: identitynamespace:<FLEET-HOST-PROJECT-ID>.svc.id.goog:https://gkehub.googleapis.com/projects/<FLEET-HOST-PROJECT-ID>/locations/northamerica-northeast1/memberships/<MEMBERSHIP-NAME>, STS endpoint: https://sts.googleapis.com/v1/token)

The <FLEET-HOST-PROJECT-ID>.svc.id.goog doesn't exist yet. Do I need to create it manually or by creating an empty cluster in the <FLEET-HOST-PROJECT-ID> project? In any case, this doesn't seem to be documented (might be totally wrong here).

I'm following the docs from here: https://cloud.google.com/anthos/fleet-management/docs/register/gke#gcloud (Switch to gcloud and checkout the second bullet point under Register an existing cluster sub-heading) and https://cloud.google.com/service-mesh/docs/managed/provision-managed-anthos-service-mesh#requirement.... I can also confirm that the required permissions for cross-project fleet access i.e, service mesh agent and anthos agent are set up.

Incase it helps, here's my mesh status:

> gcloud container fleet mesh describe --project <FLEET-HOST-PROJECT-ID>
createTime: '2023-12-22T14:22:04.472550147Z'
fleetDefaultMemberConfig:
  mesh:
    management: MANAGEMENT_AUTOMATIC
membershipSpecs:
  projects/<FLEET-HOST-PROJECT-NUMBER>/locations/northamerica-northeast1/memberships/<MEMBERSHIP-NAME>:
    mesh:
      management: MANAGEMENT_AUTOMATIC
    origin:
      type: USER
membershipStates:
  projects/<FLEET-HOST-PROJECT-NUMBER>/locations/northamerica-northeast1/memberships/<MEMBERSHIP-NAME>:
    servicemesh:
      controlPlaneManagement:
        details:
        - code: REVISION_READY
          details: 'Ready: asm-managed'
        state: ACTIVE
      dataPlaneManagement:
        details:
        - code: OK
          details: Service is running.
        state: ACTIVE
    state:
      code: OK
      description: |-
        Revision(s) ready for use: asm-managed.
        All Canonical Services have been reconciled successfully.
      updateTime: '2024-01-31T12:37:00.976267024Z'
name: projects/<FLEET-HOST-PROJECT-ID>/locations/global/features/servicemesh
resourceState:
  state: ACTIVE
spec: {}
updateTime: '2024-01-31T12:37:02.221523161Z'

Thanks!

[mahmoudrabie]

Hi, 

The error logs you provided indicate a problem with obtaining a federated access token due to an "invalid_target" error, which suggests an issue with the audience parameter configuration or the existence of the service account or resource targeted.

Here are some steps to troubleshoot:

(1) Ensure that GKE cluster is registered to the fleet in the host project. 

(2) Ensure that the service account used by the Istio-proxy (part of the ASM on your GKE cluster) has the necessary permissions across both the fleet host project and the GKE cluster's project.

(3) Verify that the service account has roles such as:

-  roles/gkehub.connect for the fleet registration and

-roles/servicemanagement.serviceControlland 

and 

roles/serviceusage.serviceUsageConsumer for service management.

Kind regards

Mahmoud