Orchestrate notebooks

To provide feedback for notebook orchestration, email [email protected].

This document describes how to orchestrate Colab Enterprise notebooks in BigQuery, including how to schedule notebooks and inspect scheduled notebook runs.

Notebooks are code assets powered by Dataform.

You can schedule a notebook to automatically run at a specified time and frequency—for example, to train an ML model, call external APIs, or run BigQuery DataFrames code.

Changes that you make to a notebook are automatically saved, but are available only to you and to users who have access to the notebook. To update the schedule with a new version of the notebook, you need to deploy the notebook. By deploying a notebook, you update its schedule with your current version of the notebook. Schedules run the latest deployed version of the notebook.

Each notebook schedule is run by a custom Dataform service account, which you select during schedule creation. For more information about the types of service accounts in Dataform, see About service accounts in Dataform.

Dataform writes the output of scheduled notebook runs to the Cloud Storage bucket selected during schedule creation.

Notebook schedules use a standard E2 runtime. Colab Enterprise charges for runtimes apply. You are charged for runtime processing based on the E2 machine type. For information about pricing of standard E2 runtimes, see Colab Enterprise pricing.

Before you begin

Before you begin, create a notebook.

Enable notebook scheduling

To schedule notebooks, you must grant the following roles to the custom Dataform service account which you plan to use for notebook schedules:

Notebook Executor User (roles/aiplatform.notebookExecutorUser)

Follow Grant a single role on a project to grant the Notebook Executor User role to your custom Dataform service account on the selected project.

Storage Admin (roles/storage.admin)

Follow Add a principal to a bucket-level policy to add your custom Dataform service account as a principal to the Cloud Storage bucket which you plan to use for storing output of scheduled notebook runs, and grant the Storage Admin role to this principal.

Service Account User (roles/iam.serviceAccountUser)

Follow Grant a single role on a service account to add your custom Dataform service account as a principal to itself. In other words, add the custom Dataform service account as a principal to the same custom Dataform service account. Then, grant the Service Account User role to this principal.

Additionally, you must grant the following role to the default Dataform service account:

Service Account Token Creator(roles/iam.serviceAccountTokenCreator)

Follow Grant token creation access to a custom Dataform service account to add the default Dataform service account as a principal to your custom Dataform service account, and grant the Service Account Token Creator role to this principal.

To learn more about service accounts in Dataform, see About service accounts in Dataform.

Required permissions

To create notebook schedules, you need the following roles:

• Dataform Admin (roles/dataform.admin)
• BigQuery Read Session User (roles/bigquery.readSessionUser) or BigQuery Studio User (roles/bigquery.studioUser)

To edit and delete notebook schedules (Preview), you need the Dataform Editor (roles/dataform.editor) role.

To view notebook schedules (Preview), you need the Dataform Viewer (roles/dataform.viewer) role.

For more information about BigQuery IAM, see Access control with IAM.

Create a schedule

To schedule a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the notebook that you want to schedule.

3. Click Schedule.

4. In the Schedule Notebook pane, in the Schedule name field, enter a name for the schedule.

5. In the Runtime template field, select a runtime.

6. In the Service account field, select a custom Dataform service account.

7. In the Cloud Storage bucket field, click Browse and select or create a Cloud Storage bucket.

   The selected custom Dataform service account must be granted the Storage Admin IAM role on the selected bucket. For more information, see Enable notebook scheduling.

8. In the Schedule frequency section, do the following:

   1. In the Repeats drop-down, select the frequency of scheduled notebook runs.
   2. In the At time field, enter the time for scheduled notebook runs.
   3. In the Timezone drop-down, select the timezone for the schedule.

9. Click Create schedule.

When you create the schedule, the current version of the notebook is automatically deployed. To update the schedule with a new version of the notebook, deploy the notebook.

The latest deployed version of the notebook runs at the selected time and frequency.

Deploy a notebook

Deploying a notebook updates its schedule with the current version of the notebook. Schedules run the latest deployed version of the notebook.

To deploy a notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click Deploy.

The corresponding schedule is updated with the current version of the notebook. The latest deployed version of the notebook runs at the scheduled time.

Manually run a deployed notebook

When you manually run a notebook deployed in a selected schedule, BigQuery executes the deployed notebook once, independently from the schedule.

To manually run a deployed notebook, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Click the name of the selected notebook schedule.

3. On the Schedule details page, click Run.

View all schedules

To view all notebook schedules in your project, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Optional: To display additional columns with notebook schedule details, click view_column Column display options, and then select columns and click OK.

View schedule details

You can view details of a selected schedule in the Explorer pane or on the Orchestration page.

To view schedule details for a selected notebook, follow these steps:

View past executions

You can view past executions of a selected notebook schedule in the Explorer pane or on the Orchestration page.

To view past executions of a selected notebook schedule, follow these steps:

Disable a schedule

To pause scheduled runs of a selected notebook without deleting the schedule, you can disable the schedule.

To disable a schedule for a selected notebook, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click View schedule.

4. In the schedule details table, in the Schedule state row, click the Schedule is enabled toggle.

Enable a schedule

To resume scheduled runs of a disabled schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

3. Click View schedule.

4. In the schedule details table, in the Schedule state row, click the Schedule is disabled toggle.

Edit a schedule

You can edit a schedule in the Explorer pane or on the Orchestration page.

To edit a schedule, follow these steps:

Delete a schedule

To permanently delete a schedule for a selected notebook, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Do either of the following:

   • Click the name of the selected schedule, and then on the Schedule details page, click Delete.

   • In the row that contains the selected schedule, click more_vert View actions in the Actions column, and then click Delete.

3. In the dialog that appears, click Delete.

What's next

• Learn more about Colab Enterprise notebooks in BigQuery.
• Learn how to create notebooks.