YouTube Content Owner transfers

The BigQuery Data Transfer Service for YouTube Content Owner connector lets you automatically schedule and manage recurring load jobs for YouTube Content Owner reports.

Supported Reports

• Content Owner Reports
• System-Managed Reports
  • Overview
  • Reports
  • Fields

The BigQuery Data Transfer Service for YouTube Content Owner reports supports the following reporting options:

For information on how YouTube Content Owner reports are transformed into BigQuery tables and views, see YouTube Content Owner report transformations.

Data ingestion from YouTube Content Owner transfers

Refresh windows

A refresh window is the number of days that a data transfer retrieves data when a data transfer occurs. For example, if the refresh window is three days and a daily transfer occurs, the BigQuery Data Transfer Service retrieves all data from your source table from the past three days. In this example, when a daily transfer occurs, the BigQuery Data Transfer Service creates a new BigQuery destination table partition with a copy of your source table data from the current day, then automatically triggers backfill runs to update the BigQuery destination table partitions with your source table data from the past two days. The automatically triggered backfill runs will either overwrite or incrementally update your BigQuery destination table, depending on whether or not incremental updates are supported in the BigQuery Data Transfer Service connector.

When you run a data transfer for the first time, the data transfer retrieves all source data available within the refresh window. For example, if the refresh window is three days and you run the data transfer for the first time, the BigQuery Data Transfer Service retrieves all source data within three days.

Refresh windows are mapped to the TransferConfig.data_refresh_window_days API field.

To retrieve data outside the refresh window, such as historical data, or to recover data from any transfer outages or gaps, you can initiate or schedule a backfill run.

Limitations

• The maximum supported file size for each report is 1710 GB.
• The minimum frequency that you can schedule a data transfer for is once every 24 hours. By default, a data transfer starts at the time that you create the data transfer. However, you can configure the transfer start time when you set up your transfer.
• The BigQuery Data Transfer Service does not support incremental data transfers during a YouTube Content Owner transfer. When you specify a date for a data transfer, all of the data that is available for that date is transferred.

Before you begin

Before you create a YouTube Content Owner data transfer:

• Verify that you have completed all actions required to enable the BigQuery Data Transfer Service.
• Create a BigQuery dataset to store the YouTube data.
• Verify that you have a YouTube Content Owner account. A YouTube Content Owner is not the same as a YouTube channel. Typically, you only have a YouTube Content Owner account if you manage many different channels.
• If you intend to setup transfer run notifications for Pub/Sub, you must have pubsub.topics.setIamPolicy permissions. Pub/Sub permissions are not required if you just set up email notifications. For more information, see BigQuery Data Transfer Service run notifications.

Required permissions

Ensure that the person creating the data transfer has the following required permissions:

• BigQuery:

  • bigquery.transfers.update permissions to create the data transfer
  • Both bigquery.datasets.get and bigquery.datasets.update permissions on the target dataset

  The bigquery.admin predefined IAM role includes bigquery.transfers.update, bigquery.datasets.update and bigquery.datasets.get permissions. For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

• YouTube:

  • YouTube Content Manager or YouTube Content Owner.

  A Content Manager is granted rights to administer YouTube content for a Content Owner. A Content Owner is an umbrella account that owns one or more YouTube channels and the videos on those channels.

  • Hide revenue data is unchecked in YouTube Content Owner report settings.

  For revenue-related reports to transfer, the YouTube reports permission setting Hide revenue data should be unchecked for the user creating the transfer.

  

Set up a YouTube Content Owner transfer

Setting up a YouTube Content Owner data transfer requires a:

• Content Owner ID: Provided by YouTube. When you log in to YouTube as a Content Owner or Manager, your ID appears in the URL after o=. For example, if the URL is https://studio.youtube.com/owner/AbCDE_8FghIjK?o=AbCDE_8FghIjK, the Content Owner ID is AbCDE_8FghIjK. To select a different Content Manager account, see Sign in to a Content Manager account or YouTube Channel Switcher. For more information on creating and managing your Content Manager account, see Configure Content Manager account settings.
• Table Suffix: A user-friendly name for the channel provided by you when you set up the transfer. The suffix is appended to the job ID to create the table name, for example reportTypeId_suffix. The suffix is used to prevent separate data transfers from writing to the same tables. The table suffix must be unique across all transfers that load data into the same dataset, and the suffix should be short to minimize the length of the resulting table name.

If you use the YouTube Reporting API and have existing reporting jobs, the BigQuery Data Transfer Service loads your report data. If you don't have existing reporting jobs, setting up the data transfer automatically enables YouTube reporting jobs.

To set up a YouTube Content Owner data transfer:

bq mk \
--transfer_config \
--project_id=project_id \
--target_dataset=dataset \
--display_name=name \
--params='parameters' \
--data_source=data_source \
--service_account_name=service_account_name

bq mk \
--transfer_config \
--target_dataset=mydataset \
--display_name='My Transfer' \
--params='{"content_owner_id":"abCDE_8FghIjK","table_suffix":"MT","configure_jobs":"true"}' \
--data_source=youtube_content_owner

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.ProjectName;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.protobuf.Struct;
import com.google.protobuf.Value;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

// Sample to create youtube content owner channel transfer config
public class CreateYoutubeContentOwnerTransfer {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    final String projectId = "MY_PROJECT_ID";
    String datasetId = "MY_DATASET_ID";
    String contentOwnerId = "MY_CONTENT_OWNER_ID";
    String tableSuffix = "_test";
    Map<String, Value> params = new HashMap<>();
    params.put("content_owner_id", Value.newBuilder().setStringValue(contentOwnerId).build());
    params.put("table_suffix", Value.newBuilder().setStringValue(tableSuffix).build());
    TransferConfig transferConfig =
        TransferConfig.newBuilder()
            .setDestinationDatasetId(datasetId)
            .setDisplayName("Your Youtube Owner Channel Config Name")
            .setDataSourceId("youtube_content_owner")
            .setParams(Struct.newBuilder().putAllFields(params).build())
            .build();
    createYoutubeContentOwnerTransfer(projectId, transferConfig);
  }

  public static void createYoutubeContentOwnerTransfer(
      String projectId, TransferConfig transferConfig) throws IOException {
    try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
      ProjectName parent = ProjectName.of(projectId);
      CreateTransferConfigRequest request =
          CreateTransferConfigRequest.newBuilder()
              .setParent(parent.toString())
              .setTransferConfig(transferConfig)
              .build();
      TransferConfig config = client.createTransferConfig(request);
      System.out.println(
          "Youtube content owner channel transfer created successfully :" + config.getName());
    } catch (ApiException ex) {
      System.out.print("Youtube content owner channel transfer was not created." + ex.toString());
    }
  }
}

Query your data

When your data is transferred to BigQuery, the data is written to ingestion-time partitioned tables. For more information, see Partitioned tables.

If you query your tables directly instead of using the auto-generated views, you must use the _PARTITIONTIME pseudocolumn in your query. For more information, see Querying partitioned tables.

Troubleshoot YouTube Content Owner transfer setup

If you are having issues setting up your data transfer, see YouTube transfer issues in Troubleshooting transfer configurations.