YouTube Channel transfers

The BigQuery Data Transfer Service for YouTube connector lets you automatically schedule and manage recurring load jobs for YouTube Channel reports.

Supported Reports

• Channel Reports (automatically loaded into BigQuery)

The BigQuery Data Transfer Service for YouTube Channel reports supports the following reporting options:

For information on how YouTube Channel reports are transformed into BigQuery tables and views, see YouTube Channel report transformations.

Data ingestion from YouTube Channel transfers

Refresh windows

A refresh window is the number of days that a data transfer retrieves data when a data transfer occurs. For example, if the refresh window is three days and a daily transfer occurs, the BigQuery Data Transfer Service retrieves all data from your source table from the past three days. In this example, when a daily transfer occurs, the BigQuery Data Transfer Service creates a new BigQuery destination table partition with a copy of your source table data from the current day, then automatically triggers backfill runs to update the BigQuery destination table partitions with your source table data from the past two days. The automatically triggered backfill runs will either overwrite or incrementally update your BigQuery destination table, depending on whether or not incremental updates are supported in the BigQuery Data Transfer Service connector.

When you run a data transfer for the first time, the data transfer retrieves all source data available within the refresh window. For example, if the refresh window is three days and you run the data transfer for the first time, the BigQuery Data Transfer Service retrieves all source data within three days.

Refresh windows are mapped to the TransferConfig.data_refresh_window_days API field.

To retrieve data outside the refresh window, such as historical data, or to recover data from any transfer outages or gaps, you can initiate or schedule a backfill run.

Limitations

• The maximum supported file size for each report is 1710 GB.
• The minimum frequency that you can schedule a data transfer for is once every 24 hours. By default, a data transfer starts at the time that you create the transfer. However, you can configure the data transfer start time when you set up your transfer.
• The BigQuery Data Transfer Service does not support incremental data transfers during a YouTube Content Owner transfer. When you specify a date for a data transfer, all of the data that is available for that date is transferred.
• You cannot create a YouTube channel data transfer if you are signed in as a federated identity. You can only create a YouTube channel transfer while signed in using a Google Account.

Before you begin

Before you create a YouTube Channel data transfer:

• Verify that you have completed all actions required to enable the BigQuery Data Transfer Service.
• Create a BigQuery dataset to store the YouTube data.

Required permissions

Creating a YouTube Channel data transfer requires the following:

• YouTube: Ownership of the YouTube channel

• BigQuery: The following Identity and Access Management (IAM) permissions in BigQuery:

  • bigquery.transfers.update to create the transfer.
  • bigquery.datasets.get and bigquery.datasets.update on the target dataset.
  • If you intend to set up transfer run notifications for Pub/Sub, you must have pubsub.topics.setIamPolicy permissions. Pub/Sub permissions are not required if you just set up email notifications. For more information, see BigQuery Data Transfer Service run notifications.

The bigquery.admin predefined IAM role includes all of the BigQuery permissions that you need to create a YouTube Channel data transfer. For more information about IAM roles in BigQuery, see Predefined roles and permissions.

Set up a YouTube Channel transfer

Setting up a YouTube Channel data transfer requires a:

• Table Suffix: A user-friendly name for the channel provided by you when you set up the data transfer. The suffix is appended to the job ID to create the table name, for example reportTypeId_suffix. The suffix is used to prevent separate transfers from writing to the same tables. The table suffix must be unique across all transfers that load data into the same dataset, and the suffix should be short to minimize the length of the resulting table name.

If you use the YouTube Reporting API and have existing reporting jobs, the BigQuery Data Transfer Service loads your report data. If you don't have existing reporting jobs, setting up the transfer automatically enables YouTube reporting jobs.

To create a YouTube Channel data transfer:

bq mk \
--transfer_config \
--project_id=project_id \
--target_dataset=dataset \
--display_name=name \
--params='parameters' \
--data_source=data_source

bq mk \
--transfer_config \
--target_dataset=mydataset \
--display_name='My Transfer' \
--params='{"table_suffix":"MT","configure_jobs":"true"}' \
--data_source=youtube_channel

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.ProjectName;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.protobuf.Struct;
import com.google.protobuf.Value;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

// Sample to create youtube channel transfer config.
public class CreateYoutubeChannelTransfer {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    final String projectId = "MY_PROJECT_ID";
    String datasetId = "MY_DATASET_ID";
    String tableSuffix = "_test";
    Map<String, Value> params = new HashMap<>();
    params.put("table_suffix", Value.newBuilder().setStringValue(tableSuffix).build());
    TransferConfig transferConfig =
        TransferConfig.newBuilder()
            .setDestinationDatasetId(datasetId)
            .setDisplayName("Your Youtube Channel Config Name")
            .setDataSourceId("youtube_channel")
            .setParams(Struct.newBuilder().putAllFields(params).build())
            .build();
    createYoutubeChannelTransfer(projectId, transferConfig);
  }

  public static void createYoutubeChannelTransfer(String projectId, TransferConfig transferConfig)
      throws IOException {
    try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
      ProjectName parent = ProjectName.of(projectId);
      CreateTransferConfigRequest request =
          CreateTransferConfigRequest.newBuilder()
              .setParent(parent.toString())
              .setTransferConfig(transferConfig)
              .build();
      TransferConfig config = client.createTransferConfig(request);
      System.out.println("Youtube channel transfer created successfully :" + config.getName());
    } catch (ApiException ex) {
      System.out.print("Youtube channel transfer was not created." + ex.toString());
    }
  }
}

Query your data

When your data is transferred to BigQuery, the data is written to ingestion-time partitioned tables. For more information, see Introduction to partitioned tables.

If you query your tables directly instead of using the auto-generated views, you must use the _PARTITIONTIME pseudocolumn in your query. For more information, see Querying partitioned tables.

Troubleshoot YouTube Channel transfer setup

If you are having issues setting up your data transfer, see YouTube transfer issues in Troubleshooting transfer configurations.