Connecting to Cloud Storage Buckets

This page introduces Cloud Storage as an option for storing your machine learning data and training output, and describes how to give your Cloud TPU access to the data objects on Cloud Storage.

Before you begin

You need a Cloud TPU Service Account in order to access a Cloud Storage bucket.

  1. Create a Cloud TPU Service Account for your project.

    gcloud beta services identity create --service tpu.googleapis.com --project $PROJECT_ID

    The command returns a Cloud TPU Service Account with following format:

    service-PROJECT_NUMBER@cloud-tpu.iam.gserviceaccount.com

  2. Follow the instructions found in either the Cloud TPU quickstart guide or the Creating and deleting TPUs document to configure your Google Cloud project and create your Cloud TPU VM and TPU resources.

Writing data to Cloud Storage

Console

  1. Go to the Cloud Storage page on the Google Cloud console.

    Go to the Cloud Storage page

  2. Create a new bucket, specifying the following options:

    • A unique name of your choosing.
    • Default storage class: Standard
    • Location: us-central1

CLI

  1. Use the gcloud storage buckets create command to create a Cloud Storage bucket:

    gcloud storage buckets create gs://bucket-name --location region

    where:

    • region is the region where you created the Cloud TPU. For more information about regions and TPU availability, see TPU regions and zones.

    • bucket-name is the name of the bucket you want to create.

  2. Use the gcloud storage cp command to write files to the Cloud Storage bucket:

    gcloud storage cp -r local-data-dir gs://bucket-name

    where local-data-dir is a local path to your data. For example:$HOME/your-data

Giving your Cloud TPU access to Cloud Storage

You need to give your Cloud TPU read and write access to your Cloud Storage objects. To do that, you must grant the required access to the Cloud TPU Service Account used by the Cloud TPU. Follow these steps to find the Cloud TPU Service Account and grant the necessary access:

Authorize the Cloud TPU Service Account

Using fine-grained ACLs for Cloud TPU (Recommended)

If you store training data on Cloud Storage, the Cloud TPU Service Account needs read and write permission on the bucket.

Console

  1. Go to the Cloud Storage browser page to view the buckets you own.

    Go to the Cloud Storage browser

  2. Select the bucket whose ACL you want to modify.

  3. Select the Permissions tab.

  4. Select Add to add a new permission and type the complete Service Account name in the New Principals edit box.

  5. If you are reading from this bucket, you must authorize the TPU Service Account to read from the resource. Do this by granting the Service Account the Storage Legacy > Storage Legacy Bucket Reader role.

  6. If you are writing to this bucket, you must authorize the TPU Service Account to write to the resource. Do this by granting the Service Account the Storage Legacy > Storage Legacy Bucket Writer role.

CLI

  1. If you are reading from this bucket, grant read permission for the Cloud TPU Service Account:

     gcloud storage buckets add-iam-policy-binding gs://bucket-name --member=serviceAccount:service-account --role=roles/storage.objectViewer

  2. If you are writing to this bucket, grant write permission for the Cloud TPU Service Account:

     gcloud storage buckets add-iam-policy-binding gs://bucket-name --member=serviceAccount:service-account --role=roles/storage.objectCreator

Using IAM permissions for Cloud TPU (Alternative)

If you want to grant broader permissions instead of granting access to each bucket explicitly, you can grant the Identity Access Management (IAM) Storage Admin role to the Cloud TPU Service Account.

  1. Go to IAM

  2. Click the +Add button to add principals to the project.

  3. Enter the names of the Cloud TPU Service Account in the Principals text box.

  4. Click the Roles dropdown list.

  5. Enable the following roles:

    • Project > Viewer

    • Storage > Storage Admin

Cloud Storage FUSE

Cloud Storage FUSE allows you to mount and access Cloud Storage buckets as local file systems. This allows applications to read and write objects in your bucket using standard file system semantics.

See the Cloud Storage FUSE documentation for details about how Cloud Storage FUSE works and a description of how Cloud Storage FUSE operations map to Cloud Storage operations. You can find additional information about how to use Cloud Storage FUSE, such as how to install the gcsfuse CLI and mounting buckets on GitHub.

What's next