Use the Prometheus sidecar for Cloud Run

Default configuration

Use the default RunMonitoring configuration

If you don't create a custom RunMonitoring configuration, then the sidecar collector synthesizes the following default configuration to scrape metrics from port 8080 at metrics path /metrics every 30 seconds:

apiVersion: monitoring.googleapis.com/v1beta
kind: RunMonitoring
metadata:
  name: run-gmp-sidecar
spec:
  endpoints:
  - port: 8080
    path: /metrics
    interval: 30s

Using the default configuration requires no additional setup beyond adding the sidecar to your Cloud Run service.

Add the sidecar to a Cloud Run service

To use the sidecar with the default RunMonitoring configuration, you need to modify your existing Cloud Run configuration to add the sidecar. To add the sidecar, do the following:

• Add a container-dependency annotation that specifies the startup and shutdown order of the containers. In the following example, the sidecar container, named "collector", starts after and shuts down before the application container, named "app" in the example.
• Create a container for the collector, named "collector" in the following example.

For example, add the lines preceded by the + (plus) character to your Cloud Run configuration, and then redeploy your service:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    run.googleapis.com/launch-stage: ALPHA
    run.googleapis.com/cpu-throttling: 'false'
  name: my-cloud-run-service
spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/execution-environment: gen2
+       run.googleapis.com/container-dependencies: '{"collector":["app"]}'
    spec:
      containers:
      - image: "REGION-docker.pkg.dev/PROJECT_ID/run-gmp/sample-app"
        name: app
        startupProbe:
          httpGet:
            path: /startup
            port: 8000
        livenessProbe:
          httpGet:
            path: /liveness
            port: 8000
        ports:
        - containerPort: 8000
+     - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.2.0"
+       name: collector

To redeploy the Cloud Run service with the configuration file run-service.yaml, run the following command:

gcloud run services replace run-service.yaml --region=REGION

Custom configuration

Create a custom RunMonitoring configuration

You can create a RunMonitoring configuration for a Cloud Run service if the default configuration is not sufficient. For example, you might create a configuration like the following:

apiVersion: monitoring.googleapis.com/v1beta
kind: RunMonitoring
metadata:
  name: my-custom-cloud-run-job
spec:
  endpoints:
  - port: 8080
    path: /metrics
    interval: 10s
    metricRelabeling:
    - action: replace
      sourceLabels:
      - __address__
      targetLabel: label_key
      replacement: label_value
  targetLabels:
    metadata:
    - service
    - revision

This configuration does the following:

• Scrapes metrics from the port 8080 and uses the default metrics path /metrics.
• Uses a 10-second scrape interval.
• Uses relabelling to add the label label_key with the value label_value to every metric scraped.
• Adds the service and revision metadata labels to every metric scraped.

For information on the available configuration options, see RunMonitoring spec: configuration options.

When you use a custom RunMonitoring configuration, you must perform the following additional configuration:

• Enable the Secret Manager API and grant your Cloud Run service account Secret Manager roles. For more information, see Before you begin.
• Store the custom configuration as a secret.
• Add the sidecar to a Cloud Run service .

Store the configuration by using Secret Manager

To provide a custom RunMonitoring configuration, do the following:

• Create a file that contains the custom configuration.
• Create a Secret Manager secret that contains the custom configuration.

The following command creates a secret named mysecret from the custom-config.yaml file:

gcloud secrets create mysecret --data-file=custom-config.yaml

To pick up a change to the configuration after you modify the custom-config.yaml file, you must delete and recreate the secret, and then redeploy the Cloud Run service.

Updating the configuration in Secret Manager

If you want to update the custom RunMonitoring configuration at any time, you can add a new version of the existing secret to Secret Manager.

For example, to update mysecret with a new config stored in file custom-config-updated.yaml, you can run:

gcloud secrets versions add mysecret --data-file=custom-config-updated.yaml

The sidecar automatically reloads and applies the changes to its configuration.

Add the sidecar to a Cloud Run service

After you have created a Secret Manager secret for your RunMonitoring configuration, you must modify your Cloud Run configuration to do the following:

• Add the Managed Service for Prometheus sidecar
  • Add a container-dependency annotation that specifies the startup and shutdown order of the containers. In the following example, the sidecar container, named "collector", starts after and shuts down before the application container, named "app" in the example.
  • Create a container for the collector, named "collector" in the following example.
• Add the secret by mounting the secret at the location /etc/rungmp/config.yaml:
  • Add a secrets annotation that points to the secret holding your custom RunMonitoring configuration.
  • Create a volume, named "config" in the following example, for the secret that points to the file config.yaml.
  • Mount the volume as part of the collector image at the mount path /etc/rungmp.

For example, add the lines preceded by the + (plus) character to your Cloud Run configuration, and then redeploy your service:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    run.googleapis.com/launch-stage: ALPHA
    run.googleapis.com/cpu-throttling: 'false'
  name: my-cloud-run-service
spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/execution-environment: gen2
+       run.googleapis.com/container-dependencies: '{"collector":["app"]}'
+       run.googleapis.com/secrets: 'mysecret:projects/PROJECT_ID/secrets/mysecret'
    spec:
      containers:
      - image: "REGION-docker.pkg.dev/PROJECT_ID/run-gmp/sample-app"
        name: app
        startupProbe:
          httpGet:
            path: /startup
            port: 8000
        livenessProbe:
          httpGet:
            path: /liveness
            port: 8000
        ports:
        - containerPort: 8000
+     - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.2.0"
+       name: collector
+       volumeMounts:
+       - mountPath: /etc/rungmp/
+         name: config
+     volumes:
+     - name: config
+       secret:
+         items:
+         - key: latest
+           path: config.yaml
+         secretName: 'mysecret'

To redeploy the Cloud Run service with the configuration file run-service.yaml, run the following command:

gcloud run services replace run-service.yaml --region=REGION

RunMonitoring spec: configuration options

This section describes RunMonitoring configuration spec for the Managed Service for Prometheus sidecar. The following shows an example custom configuration:

apiVersion: monitoring.googleapis.com/v1beta
kind: RunMonitoring
metadata:
  name: my-custom-cloud-run-job
spec:
  endpoints:
  - port: 8080
    path: /metrics
    interval: 10s
    metricRelabeling:
    - action: replace
      sourceLabels:
      - __address__
      targetLabel: label_key
      replacement: label_value
  targetLabels:
    metadata:
    - service
    - revision

RunMonitoring

The RunMonitoring configuration monitors a Cloud Run service.

RunMonitoringSpec

This spec describes how the RunMonitoring configuration monitors a Cloud Run service.

RunTargetLabels

The RunTargetLabels configuration lets you include Cloud Run labels from the discovered Prometheus targets.

Use Cloud Build

Use Cloud Build

The run-gmp-sidecar repository includes configuration files for Cloud Build. These files bundle together the steps needed to build and deploy the sample app.

You can also manually perform the steps handled by Cloud Build. For more information, see Build and run manually.

Set up for Cloud Build

These configuration files require a Cloud Build service account and an Artifact Registry repository. The run-gmp-sidecar repository also includes a script, create-sa-and-ar.sh that does the following:

• Creates a service account, run-gmp-sa@PROJECT_ID.iam.gserviceaccount.com.
• Grants the following roles to the service account:
  • roles/iam.serviceAccountUser
  • roles/storage.objectViewer
  • roles/logging.logWriter
  • roles/artifactregistry.createOnPushWriter
  • roles/secretmanager.admin
  • roles/secretmanager.secretAccessor
  • roles/run.admin
• Creates an Artifact Registry repository, run-gmp, for your container images.

To create the run-gmp-sa service account and run-gmp repository, run the following command:

./create-sa-and-ar.sh

It takes some time for the permissions to propagate, so we recommend waiting about 30 seconds before proceeding to the next step. Otherwise, you might see authorization errors.

Submit Cloud Build request

After you've set up the run-gmp-sa service account and run-gmp repository, you can start a Cloud Build job by submitting the Cloud Build configuration file. You can use the following gcloud builds submit command:

gcloud builds submit . --config=cloudbuild-simple.yaml --region=REGION

This command takes several minutes to run, but it almost immediately indicates you where you can find the Cloud Build build details. The message looks like the following:

Logs are available at [
https://console.cloud.google.com/cloud-build/builds/637860fb-7a14-46f2-861e-09c1dc4cea6b?project=PROJECT_NUMBER].

To watch the build progress, navigate to the returned URL in your browser. You can also view the sidecar logs in Cloud Logging.

Check the service URL

After the Cloud Build job ends, check the Cloud Run service endpoint URL by running the following command:

gcloud run services describe my-cloud-run-service --region=REGION --format="value(status.url)"

This command returns the service URL, which looks like the following:

https://my-cloud-run-service-abcdefghij-ue.a.run.app

Use the returned URL as the value of SERVICE_URL in the section Verify that your app is running.

Build and run manually

Build and run manually

You can manually perform the steps handled by Cloud Build, described in Use Cloud Build. The following sections describe how to manually perform the same tasks.

Set variables used by later steps

Several of the later steps use environment variables for common values. Set up these variables by using the following commands:

export GCP_PROJECT=PROJECT_ID
export REGION=REGION

Create a container image repository

Create an Artifact Registry repository for the container image by running the following command:

gcloud artifacts repositories create run-gmp \
    --repository-format=docker \
    --location=${REGION}

Build and push the sample app

This example uses Docker on Linux to build the sample app and push it to an Artifact Registry repository. You might need to adapt these commands if you are working in a Windows or macOS environment.

To build the sample app and push it to Artifact Registry, do the following:

1. Authenticate your Docker client with the Google Cloud CLI:

   gcloud auth configure-docker ${REGION}-docker.pkg.dev
   

2. Go to the simple-app directory in the run-gmp-sidecar repository:

   pushd sample-apps/simple-app
   

3. Build the simple-app app by running the following command:

   docker build -t ${REGION}-docker.pkg.dev/${GCP_PROJECT}/run-gmp/sample-app .
   

4. Push the image for the built app by running the following command:

   docker push ${REGION}-docker.pkg.dev/${GCP_PROJECT}/run-gmp/sample-app
   

5. Return to the run-gmp-sidecar directory:

   popd
   

Configure the Cloud Run service

The run-service-simple.yaml file in the run-gmp-sidecar repository defines a multicontainer Cloud Run service that uses the sample app and collector images that you built in previous steps. The run-service-simple.yaml file contains the following Service specification:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    run.googleapis.com/launch-stage: ALPHA
    run.googleapis.com/cpu-throttling: 'false'
  name: my-cloud-run-service
spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/execution-environment: gen2
        run.googleapis.com/container-dependencies: '{"collector":["app"]}'
    spec:
      containers:
      - image: "%SAMPLE_APP_IMAGE%"
        name: app
        startupProbe:
          httpGet:
            path: /startup
            port: 8000
        livenessProbe:
          httpGet:
            path: /liveness
            port: 8000
        ports:
        - containerPort: 8000
      - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/cloud-run-gmp-sidecar/cloud-run-gmp-sidecar:1.2.0"
        name: collector

This file contains a placeholder value for the images you've built in previous steps, so you must update the placeholders with the actual values for your Google Cloud project.

Replace the %SAMPLE_APP_IMAGE% placeholder by running the following command:

sed -i s@%SAMPLE_APP_IMAGE%@REGION-docker.pkg.dev/${GCP_PROJECT}/run-gmp/sample-app@g run-service-simple.yaml

Deploy the Cloud Run service

After you've updated the run-service-simple.yaml file with your values, you can create and deploy the Cloud Run service by running the following command:

gcloud run services replace run-service-simple.yaml --region=REGION
   

This command returns the service URL, which looks like the following:

https://my-cloud-run-service-abcdefghij-ue.a.run.app

Use the returned URL as the value of SERVICE_URL in the section Verify that your app is running.

Allow unauthenticated HTTP access

Before you can make a request to the service URL, you need to change the Cloud Run service policy to accept unauthenticated HTTP access. The policy.yaml file in the run-gmp-sidecar repository contains the necessary change.

To change the service policy, run the following command:

gcloud run services set-iam-policy my-cloud-run-service policy.yaml --region=REGION