Configure Cloud SQL and the external server for replication

This page describes how to configure your external server for replication to Cloud SQL, create a source representation instance on Cloud SQL, and replicate the data to Cloud SQL. You need to go through all the steps on this page before proceeding to the replication steps.

An alternative to the steps described on this page is the Database Migration Service, which offers continuous replication or one-time database migration from an external server to Cloud SQL.

Before you begin

Terminology

• External server. The MySQL server external to Cloud SQL that you want to replicate data from. It's also referred to as the source database or the external database server. It can be another Cloud SQL instance or any other database server, such as on-premises, Amazon Relational Database Service (RDS), and so on.

• Source representation instance. A mock of a Cloud SQL instance that represents the external server to the Cloud SQL replica. It's visible in the Google Cloud console and appears like a regular Cloud SQL instance, but it doesn't contain data, require configuration or maintenance, or affect billing.

• Cloud SQL replica. The Cloud SQL instance that replicates from the external server. Also known as the external primary read replica.

• Replication user account. The MySQL user account on the external server with sufficient permissions to allow replication between the external server and the Cloud SQL replica.

• Managed import. The process of importing data directly from the external server to the Cloud SQL replica. In this situation, Cloud SQL connects to the external server using the replication user account and runs the data dump directly on the external server to import data to the Cloud SQL replica.

Set up a Google Cloud project

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Cloud SQL Admin API.

   Enable the API

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the Cloud SQL Admin API.

   Enable the API

8. Make sure you have the Cloud SQL Admin, Storage Admin, and Compute Viewer roles on your user account.

   Go to the IAM page

Install the Google Cloud SDK

To configure replication, install Google Cloud SDK for your external server. You might want to install the SDK on your external server unless it's already installed elsewhere.

Set up the external server for replication

External server checklist

• Ensure that MySQL 5.5, 5.6, 5.7, 8.0, or 8.4 is installed on your external server. MySQL Community Edition, Cloud SQL for MySQL, Amazon Aurora, and MySQL on Amazon RDS are supported.

  • You must use a major version of MySQL on your Cloud SQL replica that is the same as or greater than the version of MySQL running on your external server.
  • If MySQL 5.5 is installed on the external server, GTID isn't supported.

• Ensure that binary logs are enabled and that:

  • Row-based binary logging is used.
  • Binary logs are retained for a long enough period to support the database migration. Generally a week is sufficient.

• Take note of whether your external server has GTID enabled. This affects the steps you need to take to set up replication.

• Take note of whether your external server has global read-lock permissions. This affects the steps you need to take to set up replication.

• If the Cloud SQL replica is enabled with a private IP address because the outgoing private IP address isn't static, configure the external server's firewall to allow the internal IP range allocated for the private services access of the VPC network that the Cloud SQL replica uses as its private network.

  The source database server's firewall must be configured to allow the entire internal IP range allocated for the private service connection of the VPC network that the Cloud SQL destination instance uses as the privateNetwork field of its ipConfiguration settings.

  To find the internal IP range:

  1. In the Google Cloud console, go to the VPC networks page.

     Go to the VPC networks page

  2. Select the VPC network that you want to use.

• Click the Private service connection tab.

• If your external server contains DEFINER clauses (views, events, triggers, or stored procedures), depending on the ordering of when these statements are executed, replication might fail. Learn more about DEFINER usage and potential workarounds in Cloud SQL.

• InnoDB is the only supported storage engine in Cloud SQL. Migrating with MyISAM might cause data inconsistency and requires data validation. For more information, see Converting tables from MyISAM to InnoDB in MySQL documentation.

Configure binary log retention

You should configure your external server to retain binary logs for at least 24 hours.

call mysql.rds_set_configuration('binlog retention ', HOURS);

Create a replication user account

A MySQL user account dedicated to managing replication to the Cloud SQL replica must exist on your external server. This user account should only be used to manage replication. Depending on the migration approach you use, you might need to grant permissions to the user later.

To add a replication user account, open a terminal on the external server and enter these MySQL commands:

      CREATE USER 'USERNAME'@'%' IDENTIFIED BY 'PASSWORD';
      GRANT SELECT, SHOW VIEW ON *.* TO 'USERNAME'@'HOST';
      FLUSH PRIVILEGES;

CREATE USER 'replicationUser'@'%' IDENTIFIED BY '8*&fwwd';
GRANT SELECT, SHOW VIEW ON *.* TO 'replicationUser'@'%';
FLUSH PRIVILEGES;

Set up a source representation instance

The source representation instance references the external server. It contains only the request data from the external server. Create the request data and use it in a curl command that creates the source representation instance in Cloud SQL.

Create the request data

The request data contains basic information about your external server in JSON format. The request data can be configured for a Cloud SQL replica on a public or private network and should contain this information:

    {
      "name": "SOURCE_NAME",
      "region": "REGION",
      "databaseVersion": "DATABASE_VERSION",
      "onPremisesConfiguration": {
        "hostPort": "SOURCE_HOST",
        "username": "USERNAME",
        "password": "PASSWORD",
        "caCertificate": "SOURCE_CERT",
        "clientCertificate": "CLIENT_CERT",
        "clientKey": "CLIENT_KEY"
      }
    }

// example of source.json for external server that
// - initiates replication from a Cloud SQL managed import
// - doesn't use SSL/TSL

{
  "name": "cloudsql-source-instance",
  "region": "us-central1",
  "databaseVersion": "MYSQL_5_7",
  "onPremisesConfiguration": {
    "hostPort": "192.0.2.0:3306",
    "username": "replicationUser",
    "password": "486#@%*@"
  }
}

// example of source.json for external server that
// - initiates replication from a file
// - uses SSL/TSL

{
  "name": "cloudsql-source-instance",
  "region": "us-central1",
  "databaseVersion": "MYSQL_5_7",
  "onPremisesConfiguration": {
    "hostPort": "192.0.2.0:"3306",
    "username": "replicationUser",
    "password": "486#@%*@",
    "dumpFilePath": "gs://replica-bucket/source-database.sql.gz",
    "caCertificate": "content of your certificate",
    "clientCertificate": "content of your certificate",
    "clientKey": "content of your client key"
  }
}

// example of source.json for external server that
// - initiates replication from a Cloud SQL custom import
// - doesn't use SSL/TSL

{
  "name": "cloudsql-source-instance",
  "region": "us-central1",
  "databaseVersion": "MYSQL_5_7",
  "onPremisesConfiguration": {
    "hostPort": "192.0.2.0:3306",
    "username": "replicationUser",
    "password": "486#@%*@"
  }
}

Create a source representation instance

Before you start this step, create a JSON file that contains your source request data.

Then, to create the source representation instance in Cloud SQL, open a terminal and run the following commands:

    gcloud auth login
    ACCESS_TOKEN="$(gcloud auth print-access-token)"
    curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
         --header 'Content-Type: application/json' \
         --data @JSON_PATH \
         -X POST \
         https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT-ID/instances

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data @./source.json \
     -X POST \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances

Update a source representation instance

If you update the request data from the external server, you can update the existing source representation instance to use the modified values.

Modify the request data

Update the request data to include any fields that have changed. This includes the hostPort, username, password, caCertificate, clientCertificate, and clientKey fields. After updating the request data, use it in a curl command to update the instance in Cloud SQL.

The following example shows updating the username and password fields with a different username and password:

    {
      "name": "SOURCE_NAME",
      "region": "REGION",
      "databaseVersion": "DATABASE_VERSION",
      "onPremisesConfiguration": {
        "username": "NEW_USERNAME",
        "password": "NEW_PASSWORD"
      }
    }

// example of source.json for external server that
// - initiates replication from a Cloud SQL managed import
// - doesn't use SSL/TSL

{
  "name": "cloudsql-source-instance",
  "region": "us-central1",
  "databaseVersion": "MYSQL_5_7",
  "onPremisesConfiguration": {
    "username": "newReplicationUser",
    "password": "525#@%*@"
  }
}

// example of source.json for external server that
// - initiates replication from a file
// - uses SSL/TSL

{
  "name": "cloudsql-source-instance",
  "region": "us-central1",
  "databaseVersion": "MYSQL_5_7",
  "onPremisesConfiguration": {
    "username": "newReplicationUser",
    "password": "486#@%*@"
  }
}

// example of source.json for external server that
// - initiates replication from a Cloud SQL custom import
// - doesn't use SSL/TSL

{
  "name": "cloudsql-source-instance",
  "region": "us-central1",
  "databaseVersion": "MYSQL_5_7",
  "onPremisesConfiguration": {
    "username": "newReplicationUser",
    "password": "486#@%*@"
  }
}

Modify a source representation instance

Before you start this step, create a JSON file that contains your modified request data.

Then, to modify the source representation instance in Cloud SQL, open a terminal and run the following commands:

    gcloud auth login
    ACCESS_TOKEN="$(gcloud auth print-access-token)"
    curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
         --header 'Content-Type: application/json' \
         --data @JSON_PATH \
         -X PATCH \
         https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT-ID/instances/SOURCE_NAME

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data @./source.json \
     -X PATCH \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances/cloudsql-source-instance

Set up a Cloud SQL replica

The Cloud SQL replica eventually contains the data from the external server. In this step, you create the request data and use it in a curl command that creates the Cloud SQL replica in Cloud SQL.

Create the request data

The request data contains basic information about your external server and Cloud SQL replica in JSON format. The request data can be configured for a Cloud SQL replica on a public or private network and should contain this information:

    {
        "settings": {
            "tier": "TIER",
            "dataDiskSizeGb": "DISK_SIZE",
            "ipConfiguration": {
                "ipv4Enabled": "PUBLIC_IP_STATUS",
                "privateNetwork": "projects/PROJECT_ID/global/networks/NETWORK_NAME"
             },
             "availabilityType": "AVAILABILITY_TYPE"
        },
        "masterInstanceName": "SOURCE_REPRESENTATION_INSTANCE_NAME",
        "region": "SOURCE_REGION",
        "databaseVersion": "DATABASE_VERSION",
        "name": "REPLICA_NAME"
    }

    {
        "settings": {
            "tier": "db-custom-4-15360",
            "dataDiskSizeGb": "100"
        },
        "masterInstanceName": "source-instance",
        "region": "us-central1",
        "databaseVersion": "MYSQL_8_0",
        "name": "replica-instance"
    }

Create the Cloud SQL replica

Before you start this step, create a JSON file that contains your replica request data. Then, to create a Cloud SQL replica, open a Cloud Shell terminal and run these commands:

    gcloud auth login
    ACCESS_TOKEN="$(gcloud auth print-access-token)"
    curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
         --header 'Content-Type: application/json' \
         --data @JSON_PATH \
         -X POST \
         https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT-ID/instances

    gcloud auth login
    ACCESS_TOKEN="$(gcloud auth print-access-token)"
    curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
         --header 'Content-Type: application/json' \
         --data @./replica.json \
         -X POST \
         https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances

Verify your setup

To ensure your instances were set up correctly, go to the Cloud SQL Instances page.

You should see your source representation instance and the Cloud SQL replica, in a listing similar to the following:

Also make sure that you have the cloudsql.instances.migrate permission on the Cloud SQL replica. This permission is included in the cloudsql.admin or cloudsql.editor IAM roles.

Add users to the Cloud SQL replica

You cannot import database user accounts from the external server, but you can create them on a Cloud SQL replica. Do this before you replicate from the external server.

Get the Cloud SQL replica's outgoing IP address

You can use the outgoing IP address of the Cloud SQL replica to create a secure connection between the external server and the Cloud SQL replica. You won't be charged for this IP address.

gcloud sql instances describe REPLICA_NAME --format="default(ipAddresses)"

Allow incoming connections on the external server

The Cloud SQL replica needs to connect to the external server for replication to succeed. You must configure the network firewall for your external server to accept connections from the Cloud SQL replica's outgoing IP address if the following conditions apply:

• The external server is behind a firewall or some other network restriction.
• Your Cloud SQL replica is using a public IP.

To connect to the Cloud SQL replica, you use the replica's primary IP address. This IP address is displayed in the Google Cloud console.

Update the source representation instance to allow replication to the Cloud SQL replica

After you set up the source representation instance for the Cloud SQL replica, you might need to update the source representation instance. For example, these scenarios require an update to your configurations:

• The host, port, or IP of the external server changes.
• You want to use a different MySQL replication user.
• The password of the MySQL replication user changes.
• The SSL certificates used to securely connect to the external server change.

Seed the Cloud SQL replica

There are three options for performing the initial load of data from the external server into the Cloud SQL replica:

• A managed import uses a service that extracts data from the external server and imports it into the Cloud SQL instance directly. Cloud SQL recommends this option. You must use this option if your external server doesn't use global transaction identifier (GTID) replication. For more information, see Using a managed import to set up replication from external databases.
• A dump file requires you to create a dump file of your external server, move it to a Cloud Storage bucket, and import it into Cloud SQL. You can only use this option if your external server uses GTID replication. For more information, see Using a dump file to set up replication from external databases.
• A custom import requires you to make a Cloud SQL primary instance, perform a custom data import using your preferred tools, demote it to a Cloud SQL replica, and then set up external server replication. This works best for multi-TB database migrations; however, you must use GTID replication. For more information, see Using a custom import to set up replication from large external databases.

Which option is right for you?

Monitor replication

When the Cloud SQL replica finishes the initial data load, it connects to the external server and applies all updates that were made after the export operation. Confirm your replication status.

It's important to check the replication status before promoting the replica to a standalone instance. If the replication process isn't successfully completed, a promoted replica doesn't have all the changes from your external server.

If replication delay is not trending toward 0, take steps to address it. You might want to check these metrics: /postgresql/external_sync/initial_sync_complete, postgresql/external_sync/max_replica_byte_lag, and database/replication/state. View the list of Cloud SQL metrics.

Once the Cloud SQL replica has caught up with the external server and there's no replication delay on the Cloud SQL replica, connect to your database. Run the appropriate database commands to make sure that the contents are as expected when compared with the external server. Retain your external server until the necessary validations are done.

Set up a cascading replica

After migration, you can create cascading read replicas under your Cloud SQL replica before promoting the Cloud SQL replica.

To create a cascading replica, run the following commands:

Promote the replica

Promote your replica by following these steps:

1. Promote the replica to a primary instance. Note that GTID and binary logs are turned on and the scheduled backups are enabled on the promoted replica.
2. Add read replicas to your instance.
3. Optional: Configure your instance for high availability (HA). To prevent additional downtime, you can enable HA while setting up a replica by setting AVAILABILITY_TYPE to REGIONAL.

What's next

• Learn how to use a custom import to set up replication from large external databases.
• Learn how to use a dump file to set up replication from external databases.

• Learn how to use a managed import to set up replication from external databases.