Control access with IAM

This document shows you how to do the following in Dataform:

• Grant Dataform required access.
• Control access to Dataform with IAM.
• Control access to individual tables with IAM.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the BigQuery and Dataform APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the BigQuery and Dataform APIs.

   Enable the APIs

Grant Dataform required access

This section shows you how to grant the Identity and Access Management (IAM) roles that Dataform service accounts require to execute workflows in BigQuery.

About service accounts in Dataform

When you create your first Dataform repository, Dataform automatically generates a default service account. Dataform uses the default service account to interact with BigQuery on your behalf. The default Dataform service account is not granted any BigQuery roles or permissions by default. You must grant the required access to the default Dataform service account.

Your default Dataform service account ID is in the following format:

service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com

Replace PROJECT_NUMBER with the numeral ID of your Google Cloud project. You can find your Google Cloud project ID in the Google Cloud console dashboard. For more information, see Identifying projects.

In addition to the default Dataform service account, you can use other service accounts to execute workflows on your behalf. You can configure custom service accounts in the following ways:

• At the repository level, to run all workflows in a given repository.

• Individually for each workflow configuration.

When you create a Dataform repository or workflow configuration, you can select any service account associated with your Google Cloud project that you have access to. You must configure the required permissions for all the service accounts associated with your Dataform resources.

A custom service account associated with a Dataform repository is only used to execute workflows from that repository. All other repository operations are still performed by the default Dataform service account.

Required roles for Dataform service accounts

Default and custom service accounts used in Dataform require the following BigQuery IAM roles to be able to execute workflows in BigQuery:

• BigQuery Data Editor on projects to which Dataform needs both read and write access. These usually include the project hosting your Dataform repository.
• BigQuery Data Viewer on projects to which Dataform needs read-only access.
• BigQuery Job User on the project hosting your Dataform repository.
• BigQuery Data Owner if you want to query BigQuery datasets.
• BigQuery roles for column-level access control if you want to use BigQuery policy tags.

Additionally, you need to grant the default Dataform service account the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) so that it can access any custom service accounts that you want to use in Dataform.

Security considerations for Dataform service accounts

Granting the roles required by Dataform to a service account comes with the following security considerations:

• Any service account granted the required roles might gain access to BigQuery or Secret Manager in the project that the service account belongs to, regardless of VPC Service Controls.

  For more information, see Configure VPC Service Controls for Dataform.

• Any user who has the dataform.repositories.create IAM permission can execute code using the default Dataform service account and all the permissions granted to that service account.

  For more information, see Security considerations for Dataform permissions.

To restrict the data that a user or service account can read or write in BigQuery, you can grant granular BigQuery IAM permissions to selected BigQuery datasets or tables. For more information, see Controlling access to datasets and Controlling access to tables and views.

Grant required BigQuery roles to a service account used in Dataform

To grant the required BigQuery IAM roles to your default Dataform service account, or a custom service account that you want to use in Dataform, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select or create a repository.

3. In the Google Cloud console, go to the IAM page.

   Go to IAM

4. Click Grant Access.

5. In the New principals field, enter the service account ID.

6. In the Select a role list, select the BigQuery Job User role.

7. Click Add another role, and then in the Select a role list, select the BigQuery Data Editor role.

8. Click Add another role, and then in the Select a role list, select the BigQuery Data Viewer role.

9. Click Save.

Grant token creation access to a custom service account

To use a custom service account in Dataform, the default Dataform service account must be able to access the custom service account. To grant this access, you need to add the default Dataform service account as a principal to the custom service account with the Service Account Token Creator role.

To grant the default Dataform service account access to a custom service account, follow these steps:

1. In the Google Cloud console, go to IAM > Service accounts.

   Go to Service accounts

2. Select a project.

3. On the Service accounts for project "PROJECT_NAME" page, select your custom Dataform service account.

4. Go to Permissions, and then click Grant Access.

5. In the New principals field, enter your default Dataform service account ID.

   Your default Dataform service account ID is in the following format:

   service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com
   

6. In the Select a role list, select the Service Account Token Creator role.

7. Click Save.

Control access to Dataform with IAM

This document describes the access control options for Dataform and shows you how to view and grant Dataform roles. Dataform uses Identity and Access Management (IAM) for access control. For more information about roles and permissions in IAM, see Understanding roles and permissions.

Predefined Dataform roles

The following table lists the predefined roles that give you access to Dataform resources:

Custom Dataform roles

Custom roles can include any permissions that you specify. You can create custom roles that include permissions to perform specific administrative operations, like creating development workspaces or creating files and directories within a development workspace. To create custom roles, see Creating and managing custom roles.

Security considerations for Dataform permissions

Any user who has the dataform.repositories.create permission can execute code in BigQuery using the default Dataform service account and all permissions granted to that service account. This includes execution of Dataform workflows.

The dataform.repositories.create permissions is included in the following IAM roles:

• BigQuery Admin (roles/bigquery.admin)
• BigQuery Job User (roles/bigquery.jobUser)
• BigQuery Studio User (roles/bigquery.studioUser)
• BigQuery User (roles/bigquery.user)
• Code Creator (roles/dataform.codeCreator)
• Code Editor (roles/dataform.codeEditor)
• Code Owner (roles/dataform.codeOwner)
• Colab Enterprise User (roles/aiplatform.colabEnterpriseUser)
• Dataform Admin (roles/dataform.admin)

To restrict the data that a user or service account can read or write in BigQuery, you can grant granular BigQuery IAM permissions to selected BigQuery datasets or tables. For more information, see Controlling access to datasets and Controlling access to tables and views.

For more information about the default Dataform service account and the roles and permissions it requires, see Grant Dataform required access.

View Dataform roles

Within the Google Cloud console, perform the following steps:

1. Go to the IAM & Admin > Roles page.

   Go to Roles

2. In the Filter field, select Used in, type Dataform, and then press Enter.

3. Click one of the listed roles to view the permissions of the role in the right pane.

   For example, the Dataform Admin role has full access to all Dataform resources.

For more information about granting a role on a project, see Grant a role. You can grant predefined or custom roles in this way.

Control access to an individual repository

To control access to Dataform with granular permissions, you can set Dataform IAM roles on individual repositories by using the Dataform API repositories.setIamPolicy request.

To set Dataform IAM roles on an individual Dataform repository, follow these steps:

1. In the terminal, pass the Dataform API repositories.setIamPolicy request with an access policy.

2. In the policy, bind a user, group, domain, or service account to a selected role in the following format:

   {
   "policy":
      {
         "bindings": [
         {
            "role": "roles/ROLE",
            "members": [
               "TYPE:IDENTIFIER",
            ]
         },
         ],
      }
   }
   

   Replace the following:

   • ROLE: the Dataform IAM role that you want to grant on the repository.
   • TYPE: user, group, domain, or serviceAccount.
   • IDENTIFIER: the user, group, domain, or service account that you want to grant the role to.

3. In the IAM page, ensure that all users can view the full list of Dataform repositories through a Dataform role with the dataform.repositories.list permission.

4. In IAM, ensure that only users who require full access to all Dataform repositories are granted the Dataform Admin role on all repositories.

The following command passes the repositories.setIamPolicy Dataform API request that grants the Dataform Editor role on the sales repository to a single user:

curl -H "Content-Type: application/json" -X POST -d '{ "policy": { "bindings": [{ "role": "roles/dataform.editor", "members": ["user:[email protected]"]}] }}' "https://dataform.googleapis.com/v1beta1/projects/examplepetstore/locations/us-central1/repositories/sales:setIamPolicy"

Grant public access to a repository

You can grant public access to a Dataform repository by granting IAM roles on the repository to the allAuthenticatedUsers principal.

When you assign an IAM role to the allAuthenticatedUsers principal, service accounts and all users on the internet who have authenticated with a Google Account are granted that role. This includes accounts that aren't connected to a Google Workspace account or Cloud Identity domain, such as personal Gmail accounts. Users who aren't authenticated, such as anonymous visitors, aren't included. For more information, see All authenticated users.

For example, when you grant the Dataform Viewer role to allAuthenticatedUsers on the sales repository, all service accounts and users on the internet who have authenticated with a Google Account have read-only access to all sales code resources.

To grant public access to a Dataform repository, follow these steps:

1. In the terminal, pass the Dataform API repositories.setIamPolicy request with an access policy.

2. In the policy, bind the allAuthenticatedUsers principal to a selected role in the following format:

   {
   "policy":
      {
         "bindings": [
         {
            "role": "roles/ROLE",
            "members": [
               "allAuthenticatedUsers",
            ]
         },
         ],
      }
   }
   

   Replace ROLE with a Dataform IAM role that you want to grant to all authenticated users.

The following command passes the repositories.setIamPolicy Dataform API request that grants the Dataform Viewer role on the sales repository to allAuthenticatedUsers:

curl -H "Content-Type: application/json" -X POST -d '{ "policy": { "bindings": [{ "role": "roles/dataform.viewer", "members": ["allAuthenticatedUsers"]}] }}' "https://dataform.googleapis.com/v1beta1/projects/examplepetstore/locations/us-central1/repositories/sales:setIamPolicy"

Prevent public access to repositories

To ensure that no access is granted to the public on any Dataform repository, you can restrict the allAuthenticatedUsers principal in your project.

To restrict allAuthenticatedUsers in your project, you can set the iam.allowedPolicyMemberDomains policy, and remove allAuthenticatedUsers from the list of allowed_values.

When you restrict allAuthenticatedUsers in the iam.allowedPolicyMemberDomains policy, the allAuthenticatedUsers principal cannot be used in any IAM policy in your project, which prevents granting public access to all resources, including Dataform repositories.

For more information about the iam.allowedPolicyMemberDomains policy and also instructions to set it, see Restricting identities by domain.

Workforce identity federation in Dataform

Workforce identity federation lets you use an external identity provider (IdP) to authenticate and authorize users to Google Cloud services with IAM.

Dataform supports workforce identity federation with no known limitations.

Control access to individual tables with IAM

This section shows you how to grant and revoke BigQuery IAM roles for individual Dataform tables and views.

When Dataform executes a table or view, it creates the resource in BigQuery. During development in Dataform, you can grant BigQuery roles to individual tables and views to control their access in BigQuery after execution.

For more information about granting and revoking access to resources, see Grant access to a resource.

Grant BigQuery roles to a table or view

You can grant BigQuery roles to a table or view in Dataform by adding a post_operations block with the GRANT DCL statement to the .sqlx definition file of the selected table or view.

To grant BigQuery roles to a selected table or view, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select a repository, and then a select a workspace.

3. In the Files pane, expand the definitions/ directory.

4. Select the .sqlx definition file of the table or view that you want to grant access to.

5. In the file, enter the following code snippet:

   post_operations {
       GRANT "ROLE_LIST"
       ON "RESOURCE_TYPE" ${self()}
       TO "USER_LIST"
   }
   

   Replace the following:

   • ROLE_LIST: the BigQuery role or list of comma-separated BigQuery roles that you want to grant.

   • RESOURCE_TYPE: TABLE or VIEW.

   • USER_LIST: the comma-separated list of users that the role is granted to.

     For a list of valid formats, see user_list.

6. Optional: Click Format.

7. Execute the table or view.

8. If you granted access on an incremental table, remove the GRANT statement from the table definition file after the first execution.

The following code sample shows the BigQuery Viewer role granted on a table to a user:

config { type: "table" }

SELECT ...

post_operations {
  GRANT `roles/bigquery.dataViewer`
  ON TABLE ${self()}
  TO "user:[email protected]"
}

Revoke BigQuery roles from a table or view

You can revoke BigQuery roles from a table or view by adding a post_operations block with the REVOKE DCL statement to the .sqlx definition file of the selected table or view.

To revoke BigQuery roles from a selected table or view, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select a repository, and then a select a workspace.

3. In the Files pane, expand the definitions/ directory.

4. Select the .sqlx definition file of the table or view that you want to revoke access to.

5. In the post_operations block, enter the following REVOKE statement:

       REVOKE "ROLE_LIST"
       ON "RESOURCE_TYPE" ${self()}
       FROM "USER_LIST"
   

   Replace the following:

   • ROLE_LIST: the BigQuery role or list of comma-separated BigQuery roles that you want to revoke.
   • RESOURCE_TYPE: TABLE or VIEW.
   • USER_LIST: the comma-separated list of users that the role is revoked from. For a list of valid formats, see user_list.

6. To revoke the access granted in a GRANT statement in the file, replace the GRANT statement with a REVOKE statement.

7. Optional: Click Format.

8. Execute the table or view.

9. If you revoked access to an incremental table, remove the REVOKE statement from the table definition file after the first execution.

The following code sample shows the BigQuery Viewer role revoked from a user on a table:

config { type: "table" }

SELECT ...

post_operations {
  REVOKE `roles/bigquery.dataViewer`
  ON TABLE ${self()}
  FROM "user:[email protected]"
}

Collectively manage BigQuery roles for tables and views

To control BigQuery access to individual tables and views in a single location, you can create a dedicated type: "operations" file with GRANT and REVOKE DCL statements.

To manage BigQuery table access in a single type: "operations" file, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select a repository, and then a select a workspace.

3. In the Files pane, next to definitions/, click the More menu.

4. Click Create file.

5. In the Add a file path field, enter the name of the file followed by .sqlx after definitions/. For example, definitions/table-access.sqlx.

   Filenames can only include numbers, letters, hyphens, and underscores.

6. Click Create file.

7. In the Files pane, expand the definitions/ directory, and select the newly created file.

8. In the file, enter the following code snippet:

     config { type: "operations" }
   
     GRANT "ROLE_LIST"
     ON RESOURCE_TYPE RESOURCE_NAME
     TO "USER_LIST"
   
     REVOKE "ROLE_LIST"
     ON { "<var>" }}RESOURCE_TYPE RESOURCE_NAME
     TO "USER_LIST"
   

   Replace the following:

   • ROLE_LIST: the BigQuery role or list of comma-separated BigQuery roles that you want to grant or revoke.
   • RESOURCE_TYPE: TABLE or VIEW.
   • RESOURCE_NAME: the name of the table or view.
   • USER_LIST: the comma-separated list of users that the role is granted to or revoked from. For a list of valid formats, see user_list.

9. Add GRANT and REVOKE statements as needed.

   1. To revoke access granted in a GRANT statement in the file, replace the GRANT statement with a REVOKE statement.

      Removing the GRANT statement without adding the REVOKE statement does not revoke access.

10. Optional: Click Format.

11. Execute the file after each update.

    1. If you granted or revoked access on an incremental table, remove the GRANT or REVOKE statement from the file after the first execution of the statement.

What's next

• To learn more about IAM, see IAM overview.
• To learn more about managing access to resources, see Manage access to projects, folders, and organizations
• To learn more about the key concepts of workforce identity federation, see Workforce identity federation.
• To learn more about BigQuery IAM roles and permissions, see Access control with IAM.
• To learn more about granting granular permissions to BigQuery datasets, see Controlling access to datasets.