Use customer-managed encryption keys (CMEK)

This page describes how customer-managed encryption keys (CMEK) work with Datastream.

Is CMEK right for you?

CMEK is intended for organizations that have sensitive or regulated data that requires them to manage their encryption keys.

Google-managed encryption versus customer-managed encryption

The CMEK feature lets you use your own cryptographic keys for data at rest in Datastream. After adding CMEK, whenever an API call is made, Datastream uses your key to access data.

Datastream uses Google-managed data encryption keys (DEK) and key encryption keys (KEK) to encrypt Datastream. There are two levels of encryption:

  1. The DEK encrypts data.
  2. The KEK encrypts the DEK.

Datastream stores the encrypted DEK alongside the encrypted data, and Google manages the Google KEK. With CMEK, you create a key that wraps the Google KEK. CMEK lets you create, revoke, and delete the KEK.

CMEK, including software, hardware, and external keys, are all managed through the Cloud Key Management Service (KMS) API.

Which locations support CMEK-enabled Datastream streams?

CMEK is available in all Datastream locations.

Understand service accounts

When your Datastream streams have CMEK enabled, you need to use a service account to request key access from Cloud Key Management Service.

To use a CMEK on a project, you must have a service account and you must grant the key access to the service account. The service account must exist inside of the project. The service account is visible in all regions.

If you use the Console to create a stream, Datastream creates the service account automatically when you first choose the Customer-managed key option (if a service account doesn't exist already). You don't need to have special permissions on your user account when Datastream creates the service account automatically.

Understand keys

In Cloud Key Management Service, you need to create a keyring with a cryptographic key, set with a location. When you create a new stream in Datastream, you select this key to encrypt the stream.

You need to know the key ID and key region when you create new streams that use CMEK. You must put new streams in the same region as the CMEK associated with the streams. You can create one project for both keys and streams, or different projects for each.

CMEK uses the following format:

projects/[CMEK_ENABLED_PROJECT]/locations/[REGION]/keyRings/[RING_NAME]/cryptoKeys/[KEYNAME]

If Datastream can't access the key (for example, you disable the key version), then Datastream changes the state of the stream to FAILED and an associated error message appears. After troubleshooting any issues associated with the errors message so that the key becomes accessible again, Datastream resumes the stream automatically.

External key managers

You can use keys stored in external key managers, such as Fortanix, Ionic, or Thales, as your CMEK. To learn how to use external keys with Cloud Key Management Service, see Cloud External Key Manager.

How do you make CMEK-encrypted data inaccessible permanently?

You might have situations where you want to permanently destroy data encrypted with CMEK. To do this, you destroy the CMEK version. You can't destroy the keyring or key, but you can destroy key versions of the key.

Restrictions

The following restrictions apply when using CMEK:

  • You can't update CMEK on a running stream.

  • Although you can use CMEK to encrypt rows from the source database, you can't use these keys to encrypt stream metadata, such as the stream ID, IP address of the source database, names of the source database tables, and so on.

Use CMEK

Now that you understand CMEK, you're ready to set up a service account and keys for CMEK. In addition, you'll learn how to set up Datastream to use CMEK.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  5. Make sure that billing is enabled for your Google Cloud project.

  6. Install and initialize the Cloud SDK.
  7. Make sure that you have the Datastream Admin role assigned to your user account.

    Go to the IAM page

  8. Enable the Cloud Key Management Service API.

    Enable the API

  9. Enable the Datastream API.

    Enable the API

Workflow to create a stream in Datastream with CMEK

  1. gcloud and API users only: Make sure that you have a service account for each project that requires CMEK. If you don't, then here's how to create a service account.
  2. Create a keyring and key, and set the location for each key. The location is the Google Cloud region.
  3. gcloud and API users only: Grant the key access to the service account.
  4. Copy or write down the key ID (KMS_KEY_ID) and location for the key, and the ID (KMS_KEYRING_ID) for the keyring. You need this information when granting the key access to the service account.
  5. Go to a project and create a stream in Datastream with the following options:
    1. The same location as the CMEK
    2. The CMEK configuration
    3. The CMEK ID

Your stream in Datastream is now enabled with CMEK.

Create a service account

You need to create a service account for each project that requires CMEK.

To allow a user to manage service accounts, grant one of the following roles:

  • Service Account User (roles/iam.serviceAccountUser): Includes permissions to list service accounts, get details about a service account, and impersonate a service account.
  • Service Account Admin (roles/iam.serviceAccountAdmin): Includes permissions to list service accounts and get details about a service account. Also includes permissions to create, update, and delete service accounts, and to view or change the Datastream policy on a service account.

Currently, you can use only gcloud commands to create the type of service account you need for CMEK. If you're using the Console, then Datastream creates this service account for you automatically.

To create a service account with gcloud, run the following command:

gcloud beta services identity create \
--service=datastream.googleapis.com \
--project=PROJECT_ID

The previous command returns a service account name. You use this service account name during the procedure in Grant the key access to the service account.

Create a key

You can create the key in the same Google Cloud project as the stream in Datastream, or in a separate user project. The Cloud KMS key ring location must match the region where you want to create the stream. A multi-region or global region key won't work. If the regions don't match, then you can't create the stream.

To create a Cloud KMS key:

Console

  1. In the Google Cloud console, go to the Cryptographic keys page.
  2. Click Create key ring.
  3. Add a Key ring name. Note this name because you need it when granting the key access to the service account.
  4. Add a Key ring location.
  5. Click Create. The Create key page opens.
  6. Add a Key name.
  7. Select a Purpose (symmetric or asymmetric).
  8. Select a Rotation period and Starting on date.
  9. Click Create.
  10. On the Keys table, click the three dots in the last column, and select Copy Resource ID or write it down. This is the KMS_KEY_ID. You need the KMS_KEY_ID when granting the key access to the service account.

gcloud

  1. Create a new key ring. 
    gcloud kms keyrings create KMS_KEYRING_ID \
--location=GCP_REGION
    Write down this name because you need it when granting the key access to the service account.
  2. Create a key on the key ring. 
    gcloud kms keys create KMS_KEY_ID \
--location=GCP_REGION \
--keyring=KMS_KEYRING_ID \
--purpose=encryption
    Write down this name because you need it when granting the key access to the service account.

Grant the key access to the service account

You only need to perform this procedure if you're using gcloud or the API.

To grant access to the service account, use the following code:

gcloud kms keys add-iam-policy-binding KMS_KEY_ID \
--location=GCP_REGION \
--keyring=KMS_KEYRING_ID \
--member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-datastream.iam.gserviceaccount.com \
--role=roles/cloudkms.cryptoKeyEncrypterDecrypter

Create a stream in Datastream with CMEK

As part of creating a stream in Datastream, you can use your CMEK to manage the encryption of your data.

What's next