GCP

This enterprise-focused document provides comprehensive instructions for deploying the Portkey software in a hybrid mode on Google Kubernetes Engine clusters, designed to meet the needs of large-scale, mission-critical applications. It includes specific recommendations for component sizing, high availability, and integration with monitoring systems.

Components and Sizing Recommendations

Component	Options	Sizing Recommendations
AI Gateway	Deploy in your GKE cluster using Helm charts.	Use GKE worker nodes, each providing at least 2 vCPUs and 4 GiB of memory. For high availability, deploy them across multiple Availability Zones.
Logs Store (optional)	Google Cloud Storage or S3-compatible Storage	Each log document is ~10kb in size (uncompressed)
Cache (Prompts, Configs & Providers)	Built-in Redis, Google Memorystore Redis or Valkey	Deployed within the same VPC as the Portkey Gateway.

Prerequisites

Ensure that following tools and resources are installed and available:

• A running GKE cluster with at least 2 worker nodes. ( Best Practice: Use 2 nodes, with 1 node in each Availability Zone, to ensure high availability.)
• VPC which host GKE cluster must have ACTIVE subnet with purpose REGIONAL_MANAGED_PROXY.
• gcloud CLI
• Kubectl
• Helm (v3 or above)

Create a Portkey Account

• Go to the Portkey website.
• Sign up for a Portkey account.
• Once logged in, locate and save your Organisation ID for future reference. You can find it in the browser URL: https://app.portkey.ai/organisation/<organisation_id>/
• Contact the Portkey AI team and provide your Organisation ID and the email address used during signup.
• The Portkey team will share the following information with you:
  • Docker credentials for the Gateway images (username and password).
  • License: Client Auth Key.

Setup Project Environment

CLUSTER_NAME=<GKE_CLUSTER_NAME>       # Specify the name of the GKE cluster where the gateway will be deployed.
NAMESPACE=<NAMESPACE>                 # Specify the namespace where the gateway should be deployed (for example, portkeyai).
KSA=<KSA>                             # Provide a name for the Service Account to be associated with Gateway Pod (for example, gateway-sa)

mkdir portkey-gateway
cd portkey-gateway
touch values.yaml

Image Credentials Configuration

# Update the values.yaml file
imageCredentials:
  - name: portkey-enterprise-registry-credentials
    create: true
    registry: https://index.docker.io/v1/
    username: <PROVIDED BY PORTKEY>
    password: <PROVIDED BY PORTKEY>

  gatewayImage:
    repository: "docker.io/portkeyai/gateway_enterprise"
    pullPolicy: Always
    tag: "latest"
  dataserviceImage:
    repository: "docker.io/portkeyai/data-service"
    pullPolicy: Always
    tag: "latest"
  redisImage:
    repository: "docker.io/redis"
    pullPolicy: IfNotPresent
    tag: "7.2-alpine"
environment:
  create: true
  secret: true
  data:
    ANALYTICS_STORE: control_plane
    SERVICE_NAME: <SERVICE_NAME>                       # Specify a name for the service
    PORTKEY_CLIENT_AUTH: <PROVIDED BY PORTKEY>
    ORGANISATIONS_TO_SYNC: <ORGANISATION_ID>           # This is obtained after signing up for a Portkey account.  

Configure Components

Based on the choice of components and their configuration update the values.yaml.

MCP Gateway (Optional)

By default, only the AI Gateway is enabled in the deployment. To enable the MCP Gateway, add the following configuration to values.yaml:

environment:
  data:
    SERVER_MODE: "mcp/all"
    MCP_PORT: "8788"
    MCP_GATEWAY_BASE_URL: "<This must be set to MCP LoadBalancer URL or Domain pointing to MCP Service>"

Note:

• MCP_GATEWAY_BASE_URL must include the protocol prefix — either http:// or https://.
• This value is not required for the initial deployment. After the first deployment, once the MCP Load Balancer is provisioned and a hostname is mapped to the MCP Service, set this value and redeploy.

Server Modes

1. "" (empty or not provided): Deploys only the AI Gateway. This is the default configuration.
2. "mcp": Deploys only the MCP Gateway.
3. "all": Deploys both the AI Gateway and MCP Gateway.

Cache Store

The Portkey Gateway deployment includes a Redis instance pre-installed by default. You can either use this built-in Redis or connect to an external cache like Google Memorystore Redis or Valkey.

Built-in Redis

No additional permissions or network configurations are required.

## To use the built-in Redis, add the following configuration to the values.yaml file.
environment:
  data:
    CACHE_STORE: redis
    REDIS_URL: "redis://redis:6379"
    REDIS_TLS_ENABLED: "false"

Google Memorystore

To enable the gateway to work with a Memorystore cache, ensure that network access from GKE cluster on required port.

No Auth

## To use Google Memorystore Redis or Valkey, add the following configuration in the values.yaml file.
environment:
  data:
    CACHE_STORE: memory-store
    REDIS_URL: "redis://<GCP_MEMORY_STORE_IP>:<Port>" 
    REDIS_TLS_ENABLED: "false"                            ## "true"/"false"
    REDIS_MODE: cluster                                   ## Add this parameter only if cluster mode is enabled on Memorystore

AUTH String

To set up IAM-based authentication for Portkey Gateway to GCP Memorystore, follow the steps and add following configuration in values.yaml.

## To use IAM based authentication with Google Memorystore Redis or Valkey, add the following configuration in the values.yaml file.
environment:
  data:
    CACHE_STORE: memory-store
    GCP_REDIS_AUTH_MODE: workload
    REDIS_URL: "redis://<MEMORY_STORE_IP>:<Port>" 
    REDIS_TLS_ENABLED: "false"                             ## "true"/"false"
    REDIS_MODE: cluster                                   ## Add this parameter only if cluster mode is enabled on Memorystore
    REDIS_PASSWORD: <MEMORY_STORE_AUTH_STRING>

TLS (Optional)

If TLS is enabled on your GCP Memorystore Redis instance, you must provide the self-signed certificate to the Gateway to enable SSL/TLS connections.

1. Download the certificate file server-ca.pem from your GCP Memorystore Redis cluster.
2. Create a Kubernetes secret to store the Memorystore certificate:
   kubectl create secret generic memorystore-tls-certs  --from-file=server-ca.pem -n $NAMESPACE
   
3. Add the following configuration to values.yaml:
   environment:
     data:
       REDIS_TLS_CERTS: /etc/ssl/certs/server-ca.pem
       REDIS_TLS_ENABLED: "true"
   volumes:
     - name: memorystore-tls-certs
       secret:
         secretName: memorystore-tls-certs
   
   volumeMounts:
     - name: memorystore-tls-certs
       mountPath: /etc/ssl/certs/server-ca.pem
       subPath: server-ca.pem
   

Log Store

Google Cloud Storage

1. Create a GCS bucket for storing LLM access logs.

2. Set up access to the log store. The Gateway supports the following methods for connecting to GCS bucket for log storage:

   • Workload Identity Federation
   • HMAC

   Depending on the chosen GCS access method, update values.yaml with the following configuration.

   Workload Identity Federation

   To set up IAM-based authentication for Portkey Gateway to GCP bucket, follow the steps and add following configuration in values.yaml.

   ## To enable `Workload Identity Federation` update values.yaml with the following details:-
   serviceAccount:
     create: true
     automount: true
     # Provide the name of service account. Must be same as the name you provide while binding GSA and KSA during workload identity permission setup.
     name: <KSA>             
     annotations:
     # Replace <GSA> and <PROJECT_ID_A> with Google Service Account name and Project ID of service account respectively.
       iam.gke.io/gcp-service-account: <GSA>@<PROJECT_ID_A>.iam.gserviceaccount.com      
   
   environment:
     data:
       LOG_STORE: gcs_assume
       GCP_AUTH_MODE: workload
       LOG_STORE_REGION: <GCS_BUCKET_REGION>                     # Specify the GCP region where the GCS log bucket resides (e.g., us-east1).
       LOG_STORE_GENERATIONS_BUCKET: <GCS_BUCKET_NAME>           # Specify the name of GCS log bucket.
   

   HMAC Keys

   ## To enable HMAC based access update values.yaml with following details:-
   serviceAccount:
     create: true
     automount: true
     name: <KSA>                                  
   
   environment:
     data:
       LOG_STORE: gcs
       LOG_STORE_REGION: <GCS_BUCKET_REGION>                     # Specify the GCP region where the GCS log bucket resides (e.g., us-east1).
       LOG_STORE_GENERATIONS_BUCKET: <GCS_BUCKET_NAME>           # Specify the name of GCS log bucket.
       LOG_STORE_ACCESS_KEY: <HMAC_ACCESS_KEY>                   # Specify the HMAC access key of service account.
       LOG_STORE_SECRET_KEY: <HMAC_SECRET_KEY>                   # Specify the HMAC secret key of service account.
   

3. (Optional) Configure log path format using LOG_STORE_FILE_PATH_FORMAT. See Log Object Path Format for details.

Data Service (Optional)

The Data Service is a component of the Portkey deployment responsible for batch processing, fine-tuning, and log exports.

To enable Data Service, add the following configuration to the values.yaml file.

dataservice:
  name: "dataservice"
  enabled: true
  env:
    DEBUG_ENABLED: false
    SERVICE_NAME: "portkeyenterprise-dataservice"
  serviceAccount:
    create: true
    name: <KSA>

Network Configuration

Set Up External Access

To make the Gateway service accessible externally, you can set up either of the following:

• GCS Application Load Balancer with Kubernetes Ingress
• GCS Network Load Balancer with Kubernetes Service

Prerequisites

• GKE cluster must have HTTP Load Balancing add-on enabled.
• Load Balancers require an active subnet with purpose REGIONAL_MANAGED_PROXY. If you don’t have one, create it:

  # Replace <GKE_CLUSTER_REGION> with the region in which GKE cluster is created.
  # Replace <VPC_NAME> with name of vpc in which your GKE cluster is created.
  # Replace <SUBNET_CIDR> with CIDR to associate with subnet. CIDR MUST be part of VPC CIDR e.g., 10.0.2.0/23 etc

  gcloud compute networks subnets create lb-subnet \
    --purpose=REGIONAL_MANAGED_PROXY \
    --role=ACTIVE \
    --region=<GKE_CLUSTER_REGION> \
    --network=<VPC_NAME> \
    --range=<SUBNET_CIDR>

GCP Load Balancer Ingress

To create Application Load Balancer Ingress update the values.yaml file with following configuration:

ingress:
  enabled: true
  ingressClassName: gce                                           # 'gce-internal' for creating internal ALB
  # hostname: "<AI Gateway Hostname>"
  # hostBased: true
  # mcpHostname: "<MCP Gateway Hostname>"
  annotations: 
    kubernetes.io/ingress.class : gce                             # 'gce-internal' for creating internal ALB
    ingress.gcp.kubernetes.io/healthcheck-path: /v1/health

Note: If SERVER_MODE is set to all (i.e., both AI Gateway and MCP Gateway are enabled), you must enable host-based routing by setting hostBased to true and provide the hostname on which the AI Gateway and MCP Gateway will be accessible.

GCP Load Balancer Controller provides additional annotations (like TLS, custom health checks etc ) for managing Ingress Load Balancer. For a comprehensive list of available annotations, refer to the GCP Ingress Load Balancer.

GCP Load Balancer Service

To create Load Balancer update the values.yaml with following configuration:

service:
  type: LoadBalancer
  port: 80                                                                                          # NLB listener port                                                 
  containerPort: 8787                                             
  annotations:
    cloud.google.com/l4-rbs: "enabled"                              # Use this annotation for creating external Load Balancer. 
    # networking.gke.io/load-balancer-type: "Internal"              # Use this annotation for creating internal Load Balancer.                      
    spec.loadBalancerSourceRanges: "0.0.0.0/0"         

GCP Load Balancer Controller provides additional annotations (like TLS, custom health checks etc ) for managing Service Load Balancer. For a comprehensive list of available annotations, refer to the GCP Service Load Balancer.

Deploying Portkey Gateway

# Add the Portkey AI Gateway helm repository
helm repo add portkey-ai https://portkey-ai.github.io/helm
helm repo update

# Install the chart
helm upgrade --install portkey-ai portkey-ai/gateway -f ./values.yaml -n ${NAMESPACE} --create-namespace

Verify the deployment

To confirm that the deployment was successful, follow these steps:

• Verify that all pods are running correctly.

# 
kubectl get pods -n ${NAMESPACE}
# You should see all pods with a 'STATUS' of 'Running'.

Note: If pods are in a Pending, CrashLoopBackOff, or other error state, inspect the pod logs and events to diagnose potential issues.

• Test Gateway by sending a cURL request.

  1. Port-forward the Gateway pod
    kubectl port-forward  <POD_NAME> -n ${NAMESPACE} 9000:8787       # Replace <POD_NAME> with your Gateway pod's actual name.
  
  2. Once port forwarding is active, open a new terminal window or tab and send a test request by running:
  # Specify LLM provider and Portkey API keys
  OPENAI_API_KEY=<OPENAI_API_KEY>                           # Replace <OPENAI_API_KEY> with an actual API key
  PORTKEY_API_KEY=<PORTKEY_API_KEY>                         # Replace <PORTKEY_API_KEY> with Portkey API key which can be created from Portkey website(https://app.portkey.ai/api-keys).
  
  # Configure and send the curl request
  curl 'http://localhost:9000/v1/chat/completions'`\
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY"  \
  -H "x-portkey-provider: openai" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY"  \
  -d '{ 
      "model": "gpt-4o-mini", 
      "messages": [{"role": "user","content": "What is a fractal?"}]  
  }'
  
  3. Test gateway service integration with Load Balancer.
  # Replace <LOAD_BALANCER_IP> and <LB_LISTENER_PORT_NUMBER> with the DNS name and listener port of the created load balancer, respectively.
  curl 'http://<LOAD_BALANCER_IP>:<LB_LISTENER_PORT_NUMBER>/v1/chat/completions' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY"  \
  -H "x-portkey-provider: openai" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY"  \
  -d '{
      "model": "gpt-4o-mini",
      "messages": [{"role": "user","content": "What is a fractal?"}]
  }'
  

Integrating Gateway with Control Plane

Outbound Connectivity (Data Plane to Control Plane)

Portkey supports the following methods for integrating the Data Plane with the Control Plane for outbound connectivity:

• GCP Private Service Connect
• Over the Internet

Ensure Outbound Network Access

By default, Kubernetes allows full outbound access, but if your cluster has NetworkPolicies that restrict egress, configure them to allow outbound traffic.

Example NetworkPolicy for Outbound Access:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress
  namespace: portkeyai
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0

This allows the gateway to access LLMs hosted both within your VPC and externally. This also enables connection for the sync service to the Portkey Control Plane.

GCP Private Service Connect

Establishes a secure, private connection between the Control Plane and Data Plane within the GCP network.

Steps to establish GCP Private Service Connect connectivity:

1. (Optional) Create a subnet in region us-east4 if you don’t already have one.

# Replace <SUBNET_CIDR> with CIDR to associate with subnet. CIDR MUST be part of VPC CIDR e.g., 10.0.2.0/28 etc
# Replace <VPC_NAME> with name of vpc in which your GKE cluster is created.
gcloud compute networks subnets create psc-endpoint-subnet \
  --network=<VPC_NAME> \
  --range=<SUBNET_CIDR> \
  --region=us-east4

2. Create an IP address for the PSC endpoint.

# If you already had a subnet in us-east4 and you skipped step 1,
# replace psc-endpoint-subnet with the name of your existing subnet.
gcloud compute addresses create psc-endpoint-ip \
  --region=us-east4 \
  --subnet=psc-endpoint-subnet 

3. Contact the Portkey team, share your GCP Project ID, and request private connectivity.
4. Once your project has been whitelisted and Portkey has shared the SERVICE_ATTACHMENT_URI, create the forwarding rule for the Control Plane’s private endpoint.

# Replace <VPC_NAME> with name of vpc in which your GKE cluster is created. 
# Replace <SERVICE_ATTACHMENT_URI> with URI you received from Portkey.
gcloud compute forwarding-rules create portkey-cp-pvt-endpoint-rule \
  --region=us-east4 \
  --network=<VPC_NAME> \
  --address=psc-endpoint-ip \
  --target-service-attachment=<SERVICE_ATTACHMENT_URI> 

5. (Optional) If your GKE cluster is in a different region from us-east4, enable global access on the PSC endpoint.

gcloud compute forwarding-rules update portkey-cp-pvt-endpoint-rule \
  --region=us-east4 \
  --allow-psc-global-access

6. Fetch Connection ID of PSC endpoint and share it with the Portkey for connection approval.

gcloud compute forwarding-rules describe portkey-cp-pvt-endpoint-rule \
    --region=us-east4 \
    --format="value(pscConnectionId)"

7. Once Portkey confirms approval, verify the connection status:

gcloud compute forwarding-rules describe portkey-cp-pvt-endpoint-rule \
  --region=us-east4 \
  --format="get(pscConnectionStatus)"

# Should return "ACCEPTED"

8. Fetch Private IP of PSC endpoint.

PSC_IP=$(gcloud compute addresses describe psc-endpoint-ip \
  --region=us-east4 \
  --format="value(address)")
echo ${PSC_IP}

9. Create a Private DNS Zone for DNS resolution of Control Plane PSC endpoint.

# Replace <VPC_NAME> with VPC name of GKE cluster.
gcloud dns managed-zones create portkey-control-plane-pdz \
  --dns-name="privatelink-gcp.portkey.ai." \
  --description="Hosted Zone for DNS resolution of Portkey control plane endpoint" \
  --visibility="private" \
  --networks="<VPC_NAME>"

10. Create a DNS record to point to Portkey PSC endpoint IP.

gcloud dns record-sets create "us-east4-gcp-cp.privatelink-gcp.portkey.ai." \                                                         130 ↵
  --rrdatas="${PSC_IP}" \
  --type="A" \
  --ttl=300 \
  --zone="portkey-control-plane-pdz"

11. If connection status changes to ACCEPTED, update the values.yaml file with the following environment variables.

environment:
  create: true
  secret: true
  data:
      ALBUS_BASEPATH: "https://us-east4-gcp-cp.privatelink-gcp.portkey.ai/albus"
      CONTROL_PLANE_BASEPATH: "https://us-east4-gcp-cp.privatelink-gcp.portkey.ai/api/v1"
      SOURCE_SYNC_API_BASEPATH: "https://us-east4-gcp-cp.privatelink-gcp.portkey.ai/api/v1/sync"
      CONFIG_READER_PATH: "https://us-east4-gcp-cp.privatelink-gcp.portkey.ai/api/model-configs" 

12. Re-deploy the gateway.
    helm upgrade --install portkey-ai portkey-ai/gateway -f ./values.yaml  -n ${NAMESPACE}  --create-namespace
    
13. Check the Gateway pod logs to verify that no errors related to connection timeout or DNS resolution appear.

Over the Internet

Ensure Gateway has access to following endpoints over the internet.

• https://api.portkey.ai
• https://albus.portkey.ai

Inbound Connectivity (Control Plane to Data Plane)

• GCP Private Service Connect
• IP Whitelisting

GCP Private Service Connect

Establishes a secure, private connection between the Control Plane and Data Plane within the GCP network.

Prerequisites:

• Portkey Gateway must be exposed via either:

  • Regional internal Application Load Balancer, or
  • Regional internal proxy Network Load Balancer

• A Private DNS Zone assocated to GKE VPC network and A record pointing to internal Load Balancer’s Private IP.

  domain                Type    IP
  gateway.example.com     A   x.x.x.x
  mcp.example.com         A   x.x.x.x
  

• PSC requires an active subnet with purpose PRIVATE_SERVICE_CONNECT. If you don’t have one, create it:

  # Replace <GKE_CLUSTER_REGION> with the region in which GKE cluster is created.
  # Replace <VPC_NAME> with name of vpc in which your GKE cluster is created.
  # Replace <SUBNET_CIDR> with CIDR to associate with subnet. CIDR MUST be part of VPC CIDR e.g., 10.0.2.0/28 etc

  gcloud compute networks subnets create psc-nat-subnet \
    --purpose=PRIVATE_SERVICE_CONNECT \
    --role=ACTIVE \
    --region=<GKE_CLUSTER_REGION> \
    --network=<VPC_NAME> \
    --range=<SUBNET_CIDR>

Steps to establish GCP Private Service Connect connectivity:

1. Go to Private Service Connect > Published services and click Publish service.

2. Under Target details, select the Load Balancer option:

   • If you exposed the Gateway using Ingress in values.yaml, select Regional internal Application Load Balancer.
   • Otherwise, select Regional internal proxy Network Load Balancer.

3. Select the Portkey Gateway’s Load Balancer and the forwarding rule associated with it.

4. Under Service details:

   • Provide a name for the service (e.g., <org_name>-gateway-psc).
   • Select the subnet that was created for PSC.

5. Under Connections preference, select Accept connections for selected projects and add project pk-production-project to the accepted projects list.

6. Once the PSC service is created, copy the Service attachment and share it with the Portkey team so they can initiate a connection request. In addition to that also share AI Gateway URL (e.g., https://gateway.example.com).

7. Once the connection is initiated from the Control Plane, go to your PSC Published Service and approve the connection request.

8. To verify connectivity from the Control Plane to the Data Plane, send a test request to the Gateway and check if you can view the full log details on the Portkey app after clicking a log entry.

IP Whitelisting

Allows control plane to access the Data Plane over the internet by restricting inbound traffic to specific IP address of Control Plane. This method requires the Data Plane to have a publicly accessible endpoint. To whitelist, add an inbound rule to the VPC Firewall allowing connections from the Portkey Control Plane’s IPs (54.81.226.149, 34.200.113.35, 44.221.117.129) on Load Balancer listner port.

To integrate the Control Plane with the Data Plane, contact the Portkey team and provide the Public Endpoint of the Data Plane.

Verifying Gateway Integration with the Control Plane

• Send a test request to Gateway using curl.
• Go to Portkey website -> Logs.
• Verify that the test request appears in the logs and that you can view its full details by selecting the log entry.

Uninstalling Portkey Gateway

helm uninstall portkey-ai -n ${NAMESPACE}

Setting up IAM Permission

To enable the Portkey Gateway to access GCS bucket for log storage and, optionally, Vertex AI for model invocation, specific permissions are required.

Follow the steps below to configure permissions based on your chosen access method.

Workload Identity Federation

1. Create a Google Service Account.
   PROJECT_ID_A=<SERVICE_ACCOUNT_PROJECT_ID>     # Specify id of project in which service account is to be created.
   GSA=<GSA_NAME>                                # Specify name of Google Service Account to be created.
   bucket_name=<GCS_BUCKET_NAME>                 # Specify the name of GCS bucket which will store logs. Bucket must already be created.
   
   
   gcloud iam service-accounts create ${GSA} \
     --display-name="Portkey Gateway Service Account"
   
2. Create an IAM Policy binding to bind GSA to Gateway’s KSA.
   gcloud iam service-accounts \
     add-iam-policy-binding ${GSA}@${PROJECT_ID_A}.iam.gserviceaccount.com \
     --role roles/iam.workloadIdentityUser  \
     --member "serviceAccount:${PROJECT_ID_A}.svc.id.goog[${NAMESPACE}/${KSA}]"
   
3. Grant the required permissions to the GSA for accessing the GCS bucket. You can either assign the roles/storage.objectAdmin role to the GSA, or create a custom role with only the necessary permissions, such as storage.objects.create and storage.objects.get, and attach that role instead.

   GCS Bucket

   Same Project Access
   gcloud projects add-iam-policy-binding ${PROJECT_ID_A} \
     --member="serviceAccount:${GSA}@${PROJECT_ID_A}.iam.gserviceaccount.com" \
     --role="roles/storage.objectAdmin"
   
   Cross Project Access
   # Replace <PROJECT_B_ACCOUNT_ID> with Project Id 
   # in which GCS bucket is created.
   PROJECT_ID_B=<PROJECT_B_ACCOUNT_ID>
   gcloud projects add-iam-policy-binding ${PROJECT_ID_B} \
     --member='serviceAccount:${GSA}@${PROJECT_ID_A}.iam.gserviceaccount.com' 
     --role='roles/storage.objectAdmin'
   
4. (Optional) To allow the Gateway to access Vertex AI models, grant the roles/aiplatform.user role to the GSA.

   Vertex AI

   Same Project Access
   gcloud projects add-iam-policy-binding ${PROJECT_ID_A} \
   --member="serviceAccount:${GSA}@${PROJECT_ID_A}.iam.gserviceaccount.com" \
   --role="roles/aiplatform.user"
   
   Cross Project Access
   # Replace <PROJECT_B_ACCOUNT_ID> with Project Id 
   # in which Vertex AI is to be called.
   PROJECT_ID_B=<PROJECT_B_ACCOUNT_ID>
   gcloud projects add-iam-policy-binding ${PROJECT_ID_B} \
     --member='serviceAccount:${GSA}@${PROJECT_ID_A}.iam.gserviceaccount.com' 
     --role='roles/aiplatform.user'