Using the GKE Enterprise toolchain with Active Assist

Setting up your environment

In this tutorial, you run all of the commands in Cloud Shell.

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

2. Set variables for the project ID and project number of the current build Google Cloud project:

   export RECO_MGR_PROJECT=PROJECT_ID
   gcloud config set project $RECO_MGR_PROJECT
   export RECO_MGR_PROJECT_NUMBER=$(gcloud projects describe $RECO_MGR_PROJECT --format='value(projectNumber)')
   

   Replace PROJECT_ID with the project ID that you noted in the previous section.

3. Set variables for the deployment region:

   export REGION=us-central1
   export ZONE=us-central1-a
   

4. Clone the repository that contains the code for the sample app used in this tutorial:

   git clone https://github.com/GoogleCloudPlatform/activeassist-anthos-toolchain.git
   

5. Go to the project directory:

   cd activeassist-anthos-toolchain
   

Build the pipeline

In this section, you create the components to build the pipeline. Active Assist recommendations are generated based on usage patterns and system metrics. Each recommendation category can use a different default window of time to analyze usage data and metrics based on which recommendations are generated. To test the end-to-end pipeline, the repository that you cloned in an earlier step provides sample recommendations (stubs) that you use to run the end-to-end pipeline.

Alternatively, if you are running the pipeline in a sample project that has existing resources and recommendations, you can make appropriate changes to the sample code and then run the pipeline.

Set up sample private GitHub repositories

In the following sections, you set up the sample GitHub repositories for this tutorial.

Set up a private DRY GitHub repository

1. Create a private GitHub repository for the DRY repository. Make a note of the name you give the repository.

2. In Cloud Shell, create an environment variable for your GitHub username and the name of the DRY repository:

   export REPO_OWNER=YOUR_GITHUB_USERNAME
   export DRY_REPO_NAME=YOUR_PRIVATE_DRY_REPO
   

   Replace the following:

   • YOUR_GITHUB_USERNAME: your GitHub username.
   • YOUR_PRIVATE_DRY_REPO: the name of your DRY repository.

3. Create a personal access token (PAT) to create issues in this repository. The pipeline creates GitHub issues if there are Active Assist recommendations that need to be reviewed. For more information about creating PATs in GitHub, see the GitHub documentation.

   When you set a scope for this token, select Full control of private repositories.

4. In Cloud Shell, create an environment variable for the PAT that you generated:

   export GITHUB_TOKEN=YOUR_PERSONAL_ACCESS_TOKEN
   

   Replace YOUR_PERSONAL_ACCESS_TOKEN with your own token.

Set up a private hydrated GitHub repository

1. Create a private GitHub repository for the hydrated repository. Make a note of the name you give the repository.

2. In Cloud Shell, set an environment variable for the hydrated repository:

   export HYDRATED_REPO_NAME=YOUR_PRIVATE_HYDRATED_REPO
   export HYDRATED_REPO='[email protected]:$REPO_OWNER/$HYDRATED_REPO_NAME.git'
   

   Replace YOUR_PRIVATE_HYDRATED_REPO with the name of your hydrated repository.

3. Create a deploy key pair:

   ssh-keygen -t rsa -b 4096 \
   -C 'active-assist-robot' \
   -N '' \
   -f $(pwd)/active-assist-robot
   

   A deploy key lets you deploy to your private GitHub repository when you run a Cloud Build job to hydrate config files.

4. Print the generated key:

   cat $(pwd)/active-assist-robot.pub
   

5. Add the deploy key to the private GitHub repository. Make sure to select Allow write access when you add the deploy key. To learn how to add deploy keys to GitHub repositories, see the GitHub documentation for Managing deploy keys.

Upload GitHub keys to Secret Manager

1. In Cloud Shell, create a secret to store the private key from the deploy key pair:

   gcloud secrets create github-ssh-key \
     --data-file=$(pwd)/active-assist-robot
   

2. Create a secret to store the PAT:

   echo $GITHUB_TOKEN | gcloud secrets create github-pat --data-file=-
   

Create an GKE cluster

In this section, you create a GKE cluster with the Config Connector add-on, create an identity, and configure Config Connector. You also configure Config Sync. You can use Config Sync to create a common configuration across all your infrastructure, including custom policies, and apply it both on-premises and in the cloud. Config Sync evaluates changes and rolls them out to all Kubernetes clusters so that your desired state is always reflected in your clusters.

1. In Cloud Shell, create a new GKE cluster with the Config Connector add-on enabled:

   gcloud container clusters create sample-ops \
     --machine-type n1-standard-4 \
     --zone $ZONE \
     --release-channel regular \
     --addons ConfigConnector \
     --workload-pool=$RECO_MGR_PROJECT.svc.id.goog \
     --enable-stackdriver-kubernetes \
     --enable-ip-alias
   

2. Complete the following sections in the Installing with the GKE add-on guide, to create an identity and configure Config Connector.

   1. Creating an identity
   2. Configuring Config Connector

3. Install Config Sync in the GKE cluster that you created. When you configure Config Sync, you must do the following:

   1. Use a token to grant Config Sync read-only access to Git. Use the GitHub token that you created when you set up a private DRY GitHub repository. The token is available through the $GITHUB_TOKEN environment variable.
   2. Configure Config Sync using gcloud. Set the following settings:
      1. sourceFormat: hierarchy
      2. syncRepo: https://github.com/YOUR_GITHUB_USERNAME/YOUR_PRIVATE_HYDRATED_REPO
      3. syncBranch: main
      4. secretType: token
      5. policyDir: Don't fill in this option

Create a Cloud Build trigger to push to the hydrated repository

In the following sections, you create a Cloud Build trigger that is triggered when templates are pushed to the main branch of your YOUR_PRIVATE_DRY_REPO repository. This trigger runs the steps that hydrate the config-as-data KRM templates in the YOUR_PRIVATE_DRY_REPO repository and pushes the hydrated configuration files to your YOUR_PRIVATE_HYDRATED_REPO repository.

Connect Cloud Build to your GitHub repositories

In this section, you connect the YOUR_PRIVATE_DRY_REPO and YOUR_PRIVATE_HYDRATED_REPO GitHub repositories to Cloud Build.

1. Go to the GitHub marketplace page for the Cloud Build app.

   Go to the Cloud Build app page

2. Click Setup with Google Cloud Build.

3. If prompted, sign in to GitHub.

4. Select Only select repositories.

   Use the Select repositories drop-down to enable access to your YOUR_PRIVATE_DRY_REPO and YOUR_PRIVATE_HYDRATED_REPO repositories through the Cloud Build app.

5. Click Install.

6. Sign in to Google Cloud. The Authorization page is displayed and you are prompted to authorize the Google Cloud Build app to connect to Google Cloud.

7. Click Authorize Google Cloud Build by GoogleCloudBuild. You are redirected to the Google Cloud console.

8. Select your Google Cloud project.

9. Select the consent checkbox and click Next.

10. Click Install.

11. Sign in to Google Cloud. The Authorization page is displayed and you are prompted to authorize the Google Cloud Build app to connect to Google Cloud.

12. Click Authorize Google Cloud Build by GoogleCloudBuild. You are redirected to the Google Cloud console.

13. Select your Google Cloud project.

14. Select the consent checkbox and click Next.

15. In the Select repository page that appears, select the following GitHub repositories:

    • YOUR_PRIVATE_DRY_REPO
    • YOUR_PRIVATE_HYDRATED_REPO

16. Click Connect and then click Done.

Create a Cloud Build trigger for the DRY repository

1. In Cloud Shell, run the following command:

   envsubst < cloudbuild.template.yaml > cloudbuild.yaml
   

   The command generates a cloudbuild.yaml file.

2. Create the trigger:

   gcloud beta builds triggers create github \
     --name ActiveAssistDemo \
     --repo-name=$DRY_REPO_NAME \
     --repo-owner=$REPO_OWNER \
     --branch-pattern="main" \
     --build-config=cloudbuild.yaml
   

3. Give the Cloud Build service account permission to access Secret Manager:

   gcloud secrets add-iam-policy-binding github-ssh-key  \
     --member="serviceAccount:${RECO_MGR_PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
     --role="roles/secretmanager.secretAccessor"
   
   gcloud secrets add-iam-policy-binding github-pat  \
     --member="serviceAccount:${RECO_MGR_PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
     --role="roles/secretmanager.secretAccessor"
   

Create a scheduled Cloud Build trigger for Active Assist recommendations

In the following sections, you create a scheduled Cloud Build trigger that runs periodically. This trigger fetches Active Assist recommendations using a kpt function and determines if there are any active recommendations for the resources in your YOUR_PRIVATE_HYDRATED_REPO repository. The kpt function also creates a GitHub issue in your YOUR_PRIVATE_HYDRATED_REPO repository if there are active recommendations for resource configuration that need to be reviewed and actuated.

Generate a Cloud Build image

In this section, you generate a Cloud Build image that has kpt, gh, and Node components.

1. In Cloud Shell, build and push a Docker image to Container Registry:

   gcloud auth configure-docker
   
   docker build -t gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1 ./recommender-kpt-function
   
   docker push gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1
   

Create a Cloud Build trigger for your hydrated repo

1. In Cloud Shell, create the configuration file needed to set up the scheduled Cloud Build trigger:

    envsubst < cloudbuild-scheduled-recommendations.template.yaml > cloudbuild-scheduled-recommendations.yaml
   

2. Create the Cloud Build trigger:

   gcloud beta builds triggers create github \
     --name ActiveAssistScheduledRecommendations \
     --repo-name=YOUR_PRIVATE_HYDRATED_REPO \
     --repo-owner=$REPO_OWNER \
     --branch-pattern="main" \
     --build-config=cloudbuild-scheduled-recommendations.yaml
   

3. Get the ID of this trigger:

   export TRIGGER_ID=`gcloud beta builds triggers describe \
     ActiveAssistScheduledRecommendations \
     --format="value(id)"`
   

Create a Cloud Scheduler job to invoke your trigger

1. In Cloud Shell, create a service account:

   gcloud iam service-accounts create build-invoker \
      --description "Service Account used by Cloud Scheduler to invoke the sample scheduled Cloud Build job" \
      --display-name "recommender-scheduler-sa" \
      --project $RECO_MGR_PROJECT
   

   Cloud Scheduler jobs use this service account to run the recommender-parser service.

2. Give the service account the permissions to invoke a Cloud Build job:

   gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \
     --member serviceAccount:build-invoker@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
     --role roles/cloudbuild.builds.editor \
     --project $RECO_MGR_PROJECT
   
    gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \
      --member serviceAccount:build-invoker@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
      --role roles/serviceusage.serviceUsageConsumer \
      --project $RECO_MGR_PROJECT
   

3. Create a Cloud Scheduler job to invoke the trigger that you created in the previous step:

   gcloud scheduler jobs create http scheduled-build \
      --project $RECO_MGR_PROJECT \
      --time-zone "America/Los_Angeles" \
      --schedule="0 */3 * * *" \
      --uri="https://cloudbuild.googleapis.com/v1/projects/${RECO_MGR_PROJECT}/triggers/${TRIGGER_ID}:run" \
      --description="Scheduler job to invoke Cloud Build" \
      --oauth-service-account-email="build-invoker@$RECO_MGR_PROJECT.iam.gserviceaccount.com" \
      --headers="Content-Type=application/json" \
      --http-method="POST" \
   

   Select Y if your see the following message:

   There is no App Engine app in the project.

   If you are prompted to choose the region where you want your App Engine application located, select the us-central region.

Commit and push the Cloud Build configuration files to GitHub

Push the two Cloud Build configuration files that you created to your YOUR_PRIVATE_DRY_REPO repository:

git remote add dry https://github.com/$REPO_OWNER/$DRY_REPO_NAME.git

git add cloudbuild.yaml
git add cloudbuild-scheduled-recommendations.yaml
git commit -m "Added cloudbuild configuration YAMLs"
git push dry main

You might be prompted to enter your GitHub credentials when you push to your private repository.

Review the outcome of the Cloud Build job

When you commit and push changes to your YOUR_PRIVATE_DRY_REPO repository, the Cloud Build job is triggered. If the Cloud Build job runs successfully, several resources are created. In this section, you verify whether the resources are created after the Cloud Build job completes.

1. In Cloud Shell, in your sample-ops cluster, validate that you have a namespace called activeassist-kcc:

   kubectl get ns | grep activeassist-kcc
   

2. Config Connector deploys a running sample Compute Engine instance to your PROJECT_ID project.

   Validate that the Compute Engine instance is in the project:

    gcloud compute instances list | grep \
    computeinstance-sample-cloudmachine
   

   The MACHINE_TYPE type for this machine is n1-standard-1.

Run end-to-end tests

To let you test the end-to-end pipeline, the repository that you cloned for this tutorial provides sample recommendations (stubs). You use these stubs to run the end-to-end pipeline. The stub imitates an Active Assist recommendation payload and has a recommendation to change the machine type for the compute engine instance that was deployed from the n1-standard-1 instance type to the g1-small instance type.

In this section, you invoke the scheduled Cloud Build trigger manually to run the job that uses a kpt function to fetch Active Assist recommendations. You also verify that a GitHub issue is created in your YOUR_PRIVATE_DRY_REPO repository.

1. Open the Build Triggers page in the Google Cloud console.

   Open the Build Triggers page

2. Select the ActiveAssistScheduledRecommendations trigger.

3. To manually test the trigger, click Run on the entry for your trigger on the triggers list.

   The trigger creates a GitHub issue in your YOUR_PRIVATE_DRY_REPO repository. The issue is similar to the following:

   gcloud auth configure-docker
   
   docker build -t gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1 ./recommender-kpt-function
   
   docker push gcr.io/$RECO_MGR_PROJECT/kpt-dev-gh:1
   

   In the sample issue, the kpt function output shows that the current MACHINE_TYPE type for the Compute Engine instance is n1-standard-1 type. The Active Assist recommendation is to change it to a g1-small type.

   Version control reviewers in your enterprise can review automated GitHub issues and action them as appropriate for your enterprise.