Customer-managed encryption keys

This page describes how to use your own encryption key to protect your data stores in the US and EU multi-regions.

By default, Vertex AI Agent Builder encrypts your content stored at rest. Vertex AI Agent Builder handles and manages this default encryption for you without any additional actions on your part.

However, if you have specific compliance or regulatory requirements related to the keys that protect your data, you can use customer-managed encryption keys (CMEK) to protect your resources. In this case, you'll use Cloud KMS keys and follow the procedure on this page. The key is associated with a specific location: the US multi-region or the EU multi-region.

Cloud KMS keys are used to encrypt and decrypt the data in your data stores and apps. For general information about Cloud KMS, see the Cloud Key Management Service documentation.

Limitations of Cloud KMS in Vertex AI Agent Builder

The following limitations apply to CMEK (Cloud KMS) keys in Vertex AI Agent Builder:

Keys that are already applied to a data store can't be changed.
If you have CMEK organization policies, you must create new data stores using the API, not the Google Cloud console. Creating new data stores using the Google Cloud console fails if you have CMEK organization policies enabled.
After a key has been registered, it can't be deregistered or removed from a data store.
You must use US or EU multi-region data stores and apps (not global ones). For more information about multi-regions and data residency, including limits associated with using non-global locations, see Vertex AI Search locations or Vertex AI Agents locations.
If you need to register more than one key for a project, contact your Google account team to request a quota increase for CMEK configurations, providing a justification for why you need more than one key.
Using external key manager (EKM) or hardware security module (HSM) with CMEK is in GA with allowlist. To use EKM or HSM with CMEK, contact your Google account team.

The following limitations apply to EKM or HSM with CMEK:
- Your EKM and HSM quota for encrypt and decrypt calls should have at least 1,000 QPM of headroom. For how to check your quotas, see Check your Cloud KMS quotas.
- If using EKM, the key must be reachable for more than 90% of any time window of longer than 30 seconds. If the key isn't reachable for this amount of time, it can negatively impact indexing and search freshness.
- If there are billing issues, persistent out-of-quota issues, or persistent unreachability issues for more than 12 hours, the service automatically turns down the CmekConfig associated with the EKM or HSM key.
Data stores created before a key is registered to the project can't be protected by the key.
For Vertex AI Search, Enterprise edition is required. For information about Enterprise edition, see About advanced features.
You can't tune search models for data stores that are protected by CMEK keys.
Healthcare search data stores are CMEK-compliant. However, other third-party connector data stores and the BigQuery periodic connector are not CMEK-compliant. For general information about healthcare data stores, see Create a healthcare search data store. For general information about third-party connectors, see Connect a third-party data source.
Key rotation is not supported for recommendations apps. If you disable or destroy a key version that protects a data store associated with a recommendations app, the recommendations app stops working.
Key rotation is not compatible with analytics. If you rotate keys for a data store, apps that use that data store no longer display analytics.
CMEK keys don't apply to the following RAG APIs: check grounding, ranking, and grounded generation.

Before you begin

Make sure you satisfy the following prerequisites:

A symmetric Cloud KMS key with the rotation period set to Never (Manual rotation). See Create a key ring and Create a key in the Cloud KMS documentation.
The CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) on the key has been granted to the Discovery Engine service agent. For general instructions on how to add a role to a service agent, see Grant or revoke a single role.
The CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) on the key has been granted to the Cloud Storage service agent. If this role is not granted, data import for CMEK-protected data stores will fail because Discovery Engine is not able to make the CMEK-protected, temporary bucket and directory that is required for importing.
Don't create any data stores or apps that you want managed by your key until after you have completed the key registration instructions on this page.
Enterprise edition features are turned on for the app. See Turn Enterprise edition on or off.

Register your Cloud KMS key

To register your own managed key for Vertex AI Agent Builder, follow these steps:

Call the UpdateCmekConfig method with the Cloud KMS key that you want to register.

curl -X PATCH \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{"kms_key":"projects/KMS_PROJECT_ID/locations/KMS_LOCATION/keyRings/KEY_RING/cryptoKeys/KEY_NAME"}' \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/cmekConfigs/CMEK_CONFIG_ID?set_default=SET_DEFAULT"

KMS_PROJECT_ID: The ID of your project that contains the key. The project number won't work.
KMS_LOCATION: The multi-region of your KMS key: us or europe.
KEY_RING: The name of the key ring that holds the key.
KEY_NAME: The name of the key.
PROJECT_ID: The ID of your project that contains the data store.
LOCATION: The multi-region of your data store: us or eu.
CMEK_CONFIG_ID: The ID of the CmekConfig resource.
SET_DEFAULT: Set to true to use the key as the default key for subsequent data stores created in the multi-region.

An example curl call and response looks like this:

$ curl -X PATCH
-H "Authorization: Bearer $(gcloud auth print-access-token)"
-H "Content-Type: application/json
-d '{"kms_key":"projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"}'
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1?set_default=true"
 
{
 "name": "projects/my-ai-app-project-123/locations/us/operations/update-cmek-config-56789",
 "metadata": {
  "@type": "type.googleapis.com/google.cloud.discoveryengine.v1alpha.UpdateCmekConfigMetadata"
 }
}

Optional: Record the name value returned by the method and follow the instructions in Get details about a long-running operation to see when the operation is complete.

It typically takes a few minutes to register a key.

After the operation completes, new data stores in that multi-region are protected by the key. For general information about creating data stores, see Create a search data store.

View Cloud KMS keys

To view a registered key for Vertex AI Agent Builder, do one of the following:

If you have the CmekConfig resource name, call the GetCmekConfig method:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/cmekConfigs/CMEK_CONFIG_ID"

LOCATION: The multi-region of your data store: us or eu.
PROJECT_ID: The ID of your project that contains the data
CMEK_CONFIG_ID: The ID of the CmekConfig resource.

An example curl call and response looks like this:

$ curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1"
 
{
  "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1",
  "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
  "state": "ACTIVE"
  "is_default": true
}

If you don't have the CmekConfig resource name, call the ListCmekConfigs method:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/cmekConfigs"

LOCATION: The multi-region of your data store: us or eu.
PROJECT_ID: The ID of your project that contains the data

An example curl call and response looks like this:

$ curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/cmekConfigs"
 
{
  "cmek_configs": [
    {
      "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1",
      "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
      "state": "ACTIVE"
      "is_default": true
    }
    {
      "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-2",
      "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key-2"
      "state": "ACTIVE"
    }
  ]
}

Optional: Verify that a data store is protected by a key

Data stores that were created before your key was registered are not protected by the key. If you want to confirm that a particular data store is protected by your key, follow these steps:

Run the following curl command on the data store:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-H "x-goog-user-project: PROJECT_ID" \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/collections/default_collection/dataStores/DATA_STORE_ID"

LOCATION: The multi-region of your data store: us or eu.
PROJECT_ID: The ID of your project that contains the data store.
DATA_STORE_ID: The ID of the data store.

An example curl call looks like this:

curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
-H "Content-Type: application/json"
-H "x-goog-user-project: my-ai-app-project-123"
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/collections/default_collection/dataStores/my-data-store-1"

Review the output from the command: If the cmekConfig field is in the output and the kmsKey field shows the key that you registered, then the data store is protected by the key.

An example response looks like this:

{
 "name": "projects/969795412903/locations/us/collections/default_collection/dataStores/my-data-store-1",
 "displayName": "my-data-store-1",
 "industryVertical": "GENERIC",
 "createTime": "2023-09-05T21:20:21.520552Z",
 "solutionTypes": [
   "SOLUTION_TYPE_SEARCH"
 ],
 "defaultSchemaId": "default_schema",
 "cmekConfig": {
   "name": "projects/969795412903/locations/us/collections/default_collection/dataStores/my-data-store-1/cmekConfigs/cmek-config-1",
   "kmsKey": "projects/my-ai-app-project-123/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
 }
}

Rotate keys

When you rotate keys, you are creating a new version of the key and setting the new version as the primary version. Leave the original version of the key enabled for a while before disabling it. This gives any long-running operations that might be using the older key time to complete.

The following procedure outlines the steps to rotate keys for a Vertex AI Agent Builder data store. For general information about rotating keys, see Key rotation in the Cloud KMS guide.

Important: Do not rotate keys on data stores associated with recommendations apps or with any apps that need analytics. See Limitations of Cloud KMS in Vertex AI Agent Builder.

Reregister your key. Do this by repeating step 1 of Register your Cloud KMS key.

Note: Reregistering is much faster than registering. You won't need to monitor reregistering as a long-running operation.
See the instructions in the Manage keys section of the Cloud KMS guide to do the following:
1. Create a new key version, enable it, and make it primary.
2. Leave the older key version enabled.
3. After a week or so, disable the older key version and make sure that everything is working as before.
4. At some later date, when you are certain that no problems were caused by disabling the older key version, you can destroy the older key version.

If a key is disabled or revoked

If a key is disabled or permissions for the key are revoked, the data store stops ingesting data and stops serving data within 15 minutes. However, re-enabling a key or restoring permissions takes a long time. It can take up to 24 hours before the data store can resume serving data.

Therefore, don't disable a key unless necessary. Disabling and enabling a key on a data store is a time-consuming operation. For example, repeatedly switching a key between disabled and enabled means it will take a long time for the data store to reach a protected state. Disabling a key and re-enabling it immediately afterward could result in days of downtime because the key is first disabled from the data store and subsequently re-enabled.