Connect from Cloud Build

This page contains information and examples for connecting to a Cloud SQL instance from a service running in Cloud Build.

Cloud SQL is a fully-managed database service that helps you set up, maintain, manage, and administer your relational databases in the cloud.

Cloud Build is a service that executes your builds on Google Cloud infrastructure.

Set up a Cloud SQL instance

  1. Enable the Cloud SQL Admin API in the Google Cloud project that you are connecting from, if you haven't already done so:

    Enable the API

  2. Create a Cloud SQL for SQL Server instance. We recommend that you choose a Cloud SQL instance location in the same region as your Cloud Run service for better latency, to avoid some networking costs, and to reduce cross region failure risks.

    By default, Cloud SQL assigns a public IP address to a new instance. You also have the option to assign a private IP address. For more information about the connectivity options for both, see the Connecting Overview page.

Set up an Artifact Registry Repository

  1. Enable the Artifact Registry API in the Google Cloud project that you are connecting from, if you haven't already done so:

    Enable the API

  2. Create a Docker Artifact Registry. To improve latency, reduce the risk of cross-region failure, and avoid additional networking costs, we recommend that you choose an Artifact Registry location in the same region as your Cloud Run service.

Configure Cloud Build

The steps to configure Cloud Build depend on the type of IP address that you assigned to your Cloud SQL instance.

Public IP (default)

Make sure your Cloud Build service account has the IAM roles and permissions required to connect to the Cloud SQL instance. The Cloud Build service account is listed on the Google Cloud console IAM page as the Principal [YOUR-PROJECT-NUMBER]@cloudbuild.gserviceaccount.com.

To view this service account in the Google Cloud console, select the Include Google-provided role grants checkbox.

Your Cloud Build service account needs one of the following IAM roles:

  • Cloud SQL Client (preferred)
  • Cloud SQL Admin
Or, you can manually assign the following IAM permissions:
  • cloudsql.instances.connect
  • cloudsql.instances.get

If the Cloud Build service account belongs to a different project than the Cloud SQL instance, then the Cloud SQL Admin API and IAM permissions need to be added for both projects.

Private IP

To connect to your Cloud SQL instance over private IP, Cloud Build must be in the same VPC network as your Cloud SQL instance. To configure this:

  1. Set up a private connection between the VPC network of your Cloud SQL instance and the service producer network.
  2. Create a Cloud Build private pool.

Once configured, your application will be able to connect directly using your instance's private IP address and port 1433 when your build is run in the pool.

Connect to Cloud SQL

After you configure Cloud Build, you can connect to your Cloud SQL instance.

Public IP (default)

For public IP paths, Cloud Build supports TCP sockets.

You can use the Cloud SQL Auth Proxy in a Cloud Build step to allow connections to your database. This configuration:

  1. Builds your container and pushes it to Artifact Registry.
  2. Builds a second container, copying in the Cloud SQL Auth Proxy binary.
  • Using the second container, starts the Cloud SQL Auth Proxy and runs any migration commands.
  • steps:
      - id: "docker-build"
        name: "gcr.io/cloud-builders/docker"
        args: ["build", "-t", "${_IMAGE_NAME}", "sql-proxy/."]
    
      - id: "docker-push"
        name: "gcr.io/cloud-builders/docker"
        args: ["push", "${_IMAGE_NAME}"]
    
      - id: "docker-layer"
        name: "gcr.io/cloud-builders/docker"
        entrypoint: /bin/bash
        args:
          - '-c'
          - |
            echo "FROM $_IMAGE_NAME
            COPY --from=gcr.io/cloud-sql-connectors/cloud-sql-proxy /cloud-sql-proxy /cloudsql/cloud-sql-proxy" > Dockerfile-proxy;
    
            docker build -f Dockerfile-proxy -t ${_IMAGE_NAME}-proxy .
    
      # For TCP connections
      - id: "migrate-tcp"
        name: "${_IMAGE_NAME}-proxy"
        dir: sql-proxy
        env:
          - "DATABASE_NAME=${_DATABASE_NAME}"
          - "DATABASE_USER=${_DATABASE_USER}"
          - "DATABASE_HOST=127.0.0.1"
          - "DATABASE_PORT=${_DATABASE_PORT}"
          - "DATABASE_TYPE=${_DATABASE_TYPE}"
        secretEnv:
          - DATABASE_PASS
        entrypoint: /bin/bash
        args:
          - '-c'
          - |
            /cloudsql/cloud-sql-proxy --port ${_DATABASE_PORT} ${_INSTANCE_CONNECTION_NAME} & sleep 2;
            python migrate.py # for example
    
      # For Unix Socket connections
      - id: "migrate-socket"
        name: "${_IMAGE_NAME}-proxy"
        dir: sql-proxy
        env:
          - "DATABASE_NAME=${_DATABASE_NAME}"
          - "DATABASE_USER=${_DATABASE_USER}"
          - "INSTANCE_CONNECTION_NAME=${_INSTANCE_CONNECTION_NAME}"
          - "DATABASE_TYPE=${_DATABASE_TYPE}"
        secretEnv:
          - DATABASE_PASS
        entrypoint: /bin/bash
        args:
          - '-c'
          - |
            /cloudsql/cloud-sql-proxy --unix-socket /cloudsql ${_INSTANCE_CONNECTION_NAME} & sleep 2;
            if [ $_DATABASE_TYPE = 'mssql' ]; then echo "MSSQL doesn't support Unix Sockets. Skippng."; exit 0; fi;
            python migrate.py # for example.
    
    options:
      dynamicSubstitutions: true
    
    substitutions:
      _DATABASE_USER: myuser
      _DATABASE_NAME: mydatabase
      _INSTANCE_CONNECTION_NAME: ${PROJECT_ID}:us-central1:myinstance
      _DATABASE_PORT: '5432'
      _DATABASE_TYPE: postgres
      _DATABASE_PASSWORD_KEY: database_password
      _AR_REPO_REGION: us-central1
      _AR_REPO_NAME: my-docker-repo
      _IMAGE_NAME: ${_AR_REPO_REGION}-docker.pkg.dev/${PROJECT_ID}/${_AR_REPO_NAME}/sample-sql-proxy
    
    availableSecrets:
      secretManager:
        - versionName: projects/$PROJECT_ID/secrets/${_DATABASE_PASSWORD_KEY}/versions/latest
          env: DATABASE_PASS

    The Cloud Build code sample above shows how you might run a hypothetical migrate script after deploying the sample app above to update its Cloud SQL database using the Cloud SQL Auth Proxy and Cloud Build. To run this Cloud Build code sample the setup steps required are:

    1. Create a folder name sql-proxy
    2. Create a Dockerfile file in the sql-proxy folder with the following single line of code for its file contents:

      FROM gcr.io/gcp-runtimes/ubuntu_20_0_4
    3. Create a cloudbuild.yaml file in the sql-proxy folder.
    4. Update the cloudbuild.yaml file:
      1. Copy the sample Cloud Build code above and paste it into the cloudbuild.yaml file.
      2. Update the example code _DATABASE_TYPE within the substitutions: block to be mssql.
      3. Update the example code _DATABASE_PORT within the substitutions: block to be 1433, which is the port used by SQL Server.
      4. Replace the following placeholder values with the values used in your project:
        • mydatabase
        • myuser
        • myinstance
    5. Create a secret named database_password in Secret Manager.
    6. Create a migrate.py script file in the sql-proxy folder.
      • The script can reference the following environment variables and the secret created in the cloudbuild.yaml file using the following examples:
        • os.getenv('DATABASE_NAME')
        • os.getenv('DATABASE_USER')
        • os.getenv('DATABASE_PASS')
        • os.getenv('INSTANCE_CONNECTION_NAME')
      • To reference the same variables from a Bash script (for example: migrate.sh) use the following examples:
        • $DATABASE_NAME
        • $DATABASE_USER
        • $DATABASE_PASS
        • $INSTANCE_CONNECTION_NAME
    7. Run the following gcloud builds submit command to build a container with the Cloud SQL Auth Proxy, start the Cloud SQL Auth Proxy, and run the migrate.py script:

      gcloud builds submit --config cloudbuild.yaml

    Private IP

    For private IP paths, your application connects directly to your instance through private pools. This method uses TCP to connect directly to the Cloud SQL instance without using the Cloud SQL Auth Proxy.

    Connect with TCP

    Connect using the private IP address of your Cloud SQL instance as the host and port 1433.

    Python

    To see this snippet in the context of a web application, view the README on GitHub.

    import os
    
    import sqlalchemy
    
    
    def connect_tcp_socket() -> sqlalchemy.engine.base.Engine:
        """Initializes a TCP connection pool for a Cloud SQL instance of SQL Server."""
        # Note: Saving credentials in environment variables is convenient, but not
        # secure - consider a more secure solution such as
        # Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
        # keep secrets safe.
        db_host = os.environ[
            "INSTANCE_HOST"
        ]  # e.g. '127.0.0.1' ('172.17.0.1' if deployed to GAE Flex)
        db_user = os.environ["DB_USER"]  # e.g. 'my-db-user'
        db_pass = os.environ["DB_PASS"]  # e.g. 'my-db-password'
        db_name = os.environ["DB_NAME"]  # e.g. 'my-database'
        db_port = os.environ["DB_PORT"]  # e.g. 1433
    
        pool = sqlalchemy.create_engine(
            # Equivalent URL:
            # mssql+pytds://<db_user>:<db_pass>@<db_host>:<db_port>/<db_name>
            sqlalchemy.engine.url.URL.create(
                drivername="mssql+pytds",
                username=db_user,
                password=db_pass,
                database=db_name,
                host=db_host,
                port=db_port,
            ),
            # ...
        )
    
        return pool
    
    

    Java

    To see this snippet in the context of a web application, view the README on GitHub.

    Note:

    • CLOUD_SQL_CONNECTION_NAME should be represented as <MY-PROJECT>:<INSTANCE-REGION>:<INSTANCE-NAME>
    • Using the argument ipTypes=PRIVATE will force the SocketFactory to connect with an instance's associated private IP
    • See the JDBC socket factory version requirements for the pom.xml file here .

    
    import com.zaxxer.hikari.HikariConfig;
    import com.zaxxer.hikari.HikariDataSource;
    import javax.sql.DataSource;
    
    public class TcpConnectionPoolFactory extends ConnectionPoolFactory {
    
      // Note: Saving credentials in environment variables is convenient, but not
      // secure - consider a more secure solution such as
      // Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
      // keep secrets safe.
      private static final String DB_USER = System.getenv("DB_USER");
      private static final String DB_PASS = System.getenv("DB_PASS");
      private static final String DB_NAME = System.getenv("DB_NAME");
    
      private static final String INSTANCE_HOST = System.getenv("INSTANCE_HOST");
      private static final String DB_PORT = System.getenv("DB_PORT");
    
    
      public static DataSource createConnectionPool() {
        // The configuration object specifies behaviors for the connection pool.
        HikariConfig config = new HikariConfig();
    
        // Configure which instance and what database user to connect with.
        config.setJdbcUrl(
            String.format("jdbc:sqlserver://%s:%s;databaseName=%s", INSTANCE_HOST, DB_PORT, DB_NAME));
        config.setUsername(DB_USER); // e.g. "root", "sqlserver"
        config.setPassword(DB_PASS); // e.g. "my-password"
    
    
        // ... Specify additional connection properties here.
        // ...
    
        // Initialize the connection pool using the configuration object.
        return new HikariDataSource(config);
      }
    }

    Node.js

    To see this snippet in the context of a web application, view the README on GitHub.

    const mssql = require('mssql');
    
    // createTcpPool initializes a TCP connection pool for a Cloud SQL
    // instance of SQL Server.
    const createTcpPool = async config => {
      // Note: Saving credentials in environment variables is convenient, but not
      // secure - consider a more secure solution such as
      // Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
      // keep secrets safe.
      const dbConfig = {
        server: process.env.INSTANCE_HOST, // e.g. '127.0.0.1'
        port: parseInt(process.env.DB_PORT), // e.g. 1433
        user: process.env.DB_USER, // e.g. 'my-db-user'
        password: process.env.DB_PASS, // e.g. 'my-db-password'
        database: process.env.DB_NAME, // e.g. 'my-database'
        options: {
          trustServerCertificate: true,
        },
        // ... Specify additional properties here.
        ...config,
      };
      // Establish a connection to the database.
      return mssql.connect(dbConfig);
    };

    Go

    To see this snippet in the context of a web application, view the README on GitHub.

    package cloudsql
    
    import (
    	"database/sql"
    	"fmt"
    	"log"
    	"os"
    	"strings"
    
    	_ "github.com/denisenkom/go-mssqldb"
    )
    
    // connectTCPSocket initializes a TCP connection pool for a Cloud SQL
    // instance of SQL Server.
    func connectTCPSocket() (*sql.DB, error) {
    	mustGetenv := func(k string) string {
    		v := os.Getenv(k)
    		if v == "" {
    			log.Fatalf("Fatal Error in connect_tcp.go: %s environment variable not set.\n", k)
    		}
    		return v
    	}
    	// Note: Saving credentials in environment variables is convenient, but not
    	// secure - consider a more secure solution such as
    	// Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
    	// keep secrets safe.
    	var (
    		dbUser    = mustGetenv("DB_USER")       // e.g. 'my-db-user'
    		dbPwd     = mustGetenv("DB_PASS")       // e.g. 'my-db-password'
    		dbTCPHost = mustGetenv("INSTANCE_HOST") // e.g. '127.0.0.1' ('172.17.0.1' if deployed to GAE Flex)
    		dbPort    = mustGetenv("DB_PORT")       // e.g. '1433'
    		dbName    = mustGetenv("DB_NAME")       // e.g. 'my-database'
    	)
    
    	dbURI := fmt.Sprintf("server=%s;user id=%s;password=%s;port=%s;database=%s;",
    		dbTCPHost, dbUser, dbPwd, dbPort, dbName)
    
    
    	// dbPool is the pool of database connections.
    	dbPool, err := sql.Open("sqlserver", dbURI)
    	if err != nil {
    		return nil, fmt.Errorf("sql.Open: %w", err)
    	}
    
    	// ...
    
    	return dbPool, nil
    }
    

    C#

    To see this snippet in the context of a web application, view the README on GitHub.

    using Microsoft.Data.SqlClient;
    using System;
    
    namespace CloudSql
    {
        public class SqlServerTcp
        {
            public static SqlConnectionStringBuilder NewSqlServerTCPConnectionString()
            {
                // Equivalent connection string:
                // "User Id=<DB_USER>;Password=<DB_PASS>;Server=<INSTANCE_HOST>;Database=<DB_NAME>;"
                var connectionString = new SqlConnectionStringBuilder()
                {
                    // Note: Saving credentials in environment variables is convenient, but not
                    // secure - consider a more secure solution such as
                    // Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
                    // keep secrets safe.
                    DataSource = Environment.GetEnvironmentVariable("INSTANCE_HOST"), // e.g. '127.0.0.1'
                    // Set Host to 'cloudsql' when deploying to App Engine Flexible environment
                    UserID = Environment.GetEnvironmentVariable("DB_USER"),         // e.g. 'my-db-user'
                    Password = Environment.GetEnvironmentVariable("DB_PASS"),       // e.g. 'my-db-password'
                    InitialCatalog = Environment.GetEnvironmentVariable("DB_NAME"), // e.g. 'my-database'
    
                    // The Cloud SQL proxy provides encryption between the proxy and instance
                    Encrypt = false,
                };
                connectionString.Pooling = true;
                // Specify additional properties here.
                return connectionString;
            }
        }
    }

    Ruby

    To see this snippet in the context of a web application, view the README on GitHub.

    tcp: &tcp
      adapter: sqlserver
      # Configure additional properties here.
      # Note: Saving credentials in environment variables is convenient, but not
      # secure - consider a more secure solution such as
      # Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
      # keep secrets safe.
      username: <%= ENV["DB_USER"] %>  # e.g. "my-database-user"
      password: <%= ENV["DB_PASS"] %> # e.g. "my-database-password"
      database: <%= ENV.fetch("DB_NAME") { "vote_development" } %>
      host: <%= ENV.fetch("INSTANCE_HOST") { "127.0.0.1" }%> # '172.17.0.1' if deployed to GAE Flex
      port: <%= ENV.fetch("DB_PORT") { 1433 }%> 

    PHP

    To see this snippet in the context of a web application, view the README on GitHub.

    namespace Google\Cloud\Samples\CloudSQL\SQLServer;
    
    use PDO;
    use PDOException;
    use RuntimeException;
    use TypeError;
    
    class DatabaseTcp
    {
        public static function initTcpDatabaseConnection(): PDO
        {
            try {
                // Note: Saving credentials in environment variables is convenient, but not
                // secure - consider a more secure solution such as
                // Cloud Secret Manager (https://cloud.google.com/secret-manager) to help
                // keep secrets safe.
                $username = getenv('DB_USER'); // e.g. 'your_db_user'
                $password = getenv('DB_PASS'); // e.g. 'your_db_password'
                $dbName = getenv('DB_NAME'); // e.g. 'your_db_name'
                $instanceHost = getenv('INSTANCE_HOST'); // e.g. '127.0.0.1' ('172.17.0.1' for GAE Flex)
    
                // Connect using TCP
                $dsn = sprintf(
                    'sqlsrv:server=%s;Database=%s',
                    $instanceHost,
                    $dbName
                );
    
                // Connect to the database
                $conn = new PDO(
                    $dsn,
                    $username,
                    $password,
                    # ...
                );
            } catch (TypeError $e) {
                throw new RuntimeException(
                    sprintf(
                        'Invalid or missing configuration! Make sure you have set ' .
                            '$username, $password, $dbName, and $instanceHost (for TCP mode). ' .
                            'The PHP error was %s',
                        $e->getMessage()
                    ),
                    $e->getCode(),
                    $e
                );
            } catch (PDOException $e) {
                throw new RuntimeException(
                    sprintf(
                        'Could not connect to the Cloud SQL Database. Check that ' .
                            'your username and password are correct, that the Cloud SQL ' .
                            'proxy is running, and that the database exists and is ready ' .
                            'for use. For more assistance, refer to %s. The PDO error was %s',
                        'https://cloud.google.com/sql/docs/sqlserver/connect-external-app',
                        $e->getMessage()
                    ),
                    (int) $e->getCode(),
                    $e
                );
            }
    
            return $conn;
        }
    }

    You can then create a Cloud Build step to run your code directly.

    steps:
      - id: "docker-build"
        name: "gcr.io/cloud-builders/docker"
        args: ["build", "-t", "${_IMAGE_NAME}", "sql-private-pool/."]
    
      - id: "docker-push"
        name: "gcr.io/cloud-builders/docker"
        args: ["push", "${_IMAGE_NAME}"]
    
      - id: "migration"
        name: "${_IMAGE_NAME}"
        dir: sql-private-pool
        env:
          - "DATABASE_NAME=mydatabase"
          - "DATABASE_USER=myuser"
          - "DATABASE_HOST=${_DATABASE_HOST}"
          - "DATABASE_TYPE=${_DATABASE_TYPE}"
        secretEnv:
          - DATABASE_PASS
        entrypoint: python   # for example
        args: ["migrate.py"] # for example
    
    options:
      pool:
        name: projects/$PROJECT_ID/locations/us-central1/workerPools/private-pool
      dynamicSubstitutions: true
    
    substitutions:
      _DATABASE_PASSWORD_KEY: database_password
      _DATABASE_TYPE: postgres
      _AR_REPO_REGION: us-central1
      _AR_REPO_NAME: my-docker-repo
      _IMAGE_NAME: ${_AR_REPO_REGION}-docker.pkg.dev/${PROJECT_ID}/${_AR_REPO_NAME}/sample-private-pool
    
    availableSecrets:
      secretManager:
        - versionName: projects/$PROJECT_ID/secrets/${_DATABASE_PASSWORD_KEY}/versions/latest
          env: DATABASE_PASS

    The Cloud Build code sample above shows how you might run a hypothetical migrate script after deploying the sample app above to update its Cloud SQL database using Cloud Build. To run this Cloud Build code sample the setup steps required are:

    1. Create a folder name sql-private-pool
    2. Create a Dockerfile file in the sql-private-pool folder with the following single line of code for its file contents:

      FROM gcr.io/gcp-runtimes/ubuntu_20_0_4
    3. Create a cloudbuild.yaml file in the sql-private-pool folder.
    4. Update the cloudbuild.yaml file:
      1. Copy the sample Cloud Build code above and paste it into the cloudbuild.yaml file.
      2. Replace the following placeholder values with the values used in your project:
        • mydatabase
        • myuser
        • databasehost, in the form host:port.
    5. Create a secret named database_password in Secret Manager.
    6. Create a migrate.py script file in the sql-proxy folder.
      • The script can reference the following environment variables and the secret created in the cloudbuild.yaml file using the following examples:
        • os.getenv('DATABASE_NAME')
        • os.getenv('DATABASE_USER')
        • os.getenv('DATABASE_PASS')
        • os.getenv('DATABASE_HOST')
      • To reference the same variables from a Bash script (for example: migrate.sh) use the following examples:
        • $DATABASE_NAME
        • $DATABASE_USER
        • $DATABASE_PASS
        • $DATABASE_HOST
    7. Run the following gcloud builds submit command to build a container with the Cloud SQL Auth Proxy, start the Cloud SQL Auth Proxy, and run the migrate.py script:

      gcloud builds submit --config cloudbuild.yaml

    Best practices and other information

    You can use the Cloud SQL Auth Proxy when testing your application locally. See the quickstart for using the Cloud SQL Auth Proxy for detailed instructions.

    You can also test using the Cloud SQL Proxy via a docker container.

    Database schema migrations

    By configuring Cloud Build to connect to Cloud SQL, you can run database schema migration tasks in Cloud Build using the same code you would deploy to any other serverless platform.

    Using Secret Manager

    You can use Secret Manager to include sensitive information in your builds.