Object change notification

Object change notification can be used to notify an application when an object is updated or added to a bucket.

Should you use Object change notification?

Generally, you should not use Object change notification. The recommended tool for generating notifications that track changes to objects in your Cloud Storage buckets is Pub/Sub notifications for Cloud Storage, because it's faster, more flexible, easier to set up, and more cost-effective. For a step-by-step guide to setting up Pub/Sub notifications for Cloud Storage, see Configuring Pub/Sub notifications for Cloud Storage.

How Object change notification Works

A client application can send a request to watch for changes to the objects in a particular bucket.

Completing a watch request creates a new notification channel. A notification channel is the means by which a notification message is sent to an application watching a bucket. Only one type of notification channel, a web hook, is currently supported.

After a notification channel is initiated, Cloud Storage notifies the application any time an object is added, updated, or removed from the bucket. For example, when you add a new picture to a bucket, an application could be notified to create a thumbnail.

Object change notification Details

Terminology

The following table contains a description of several terms used throughout the Object change notification documentation:

Term	Description
Application URL	The URL of your application. This is the address where notifications will be sent. Note that this must be an HTTPS URL; HTTP URLs are not permitted.
Channel Identifier	The identifier for a notification channel. Must be unique within a particular bucket, i.e., if there are multiple notification channels for a single bucket, each notification channel must have a distinct channel identifier. This identifier will be sent to your application along with each notification message.
Resource Identifier	An opaque identifier for the resource being watched. The resource identifier is required for stopping a notification channel. You can retrieve this identifier from the response to a watch request or from the `X-Goog-Resource-Id` header of notification event messages.
Client Token	(optional) Client tokens can be used to validate notifications events. To do this, set a custom client token with your watch request. Notification messages will contain this token so you can verify that they are authentic.

Watching a Bucket

To start watching a bucket for change notification events, you make a watchAll request. This creates a notification channel that sends notification events to the given address for the given bucket. The notification channel includes a custom client token and channel identifier if specified.

An example POST request for watching a bucket:

POST /storage/v1/b/BucketName/o/watch?alt=json HTTP/1.1
Host: storage.googleapis.com
Content-Length: 200
User-Agent: google-api-python-client/1.0
Content-Type: application/json
Authorization: Bearer oauth2_token

{
  "token": "ClientToken",
  "type": "web_hook",
  "id": "ChannelId",
  "address": "ApplicationUrl"
}

Notification Authorization

When watching a bucket, the notification channel being created will be associated with the Google Cloud console project of the application initiating the API request. This means, for example, that if a user gives access to an installed application or web application via an OAuth2 flow, a notification channel created by the application will be associated with the application's project, not the project containing the bucket being watched.

Since a service account is associated with a project, if you use a service account to watch a bucket, it will create a notification channel in the service account's project.

Removing a Notification Channel

To stop a notification channel, you make a stop request. This stops all notification events to the specified resource identifier (resourceId) and channel identifier (id) pair. Additional active channels for the same resource are unaffected. The resource and channel identifiers can be found in the response of a watch request or in the body of notification event messages.

An example POST request for stopping a channel:

POST /storage/v1/channels/stop HTTP/1.1
Host: storage.googleapis.com
Content-Length: 200
User-Agent: google-api-python-client/1.0
Content-Type: application/json
Authorization: Bearer oauth2_token

{
  "resourceId": "ResourceId",
  "id": "ChannelId"
}

Notification Event Message Types

Sync

A notification event is sent when a new notification channel is created after issuing a watch request. After receiving the sync event, all later changes to the bucket will be sent to the application URL configured for the channel.

The notification will be sent as a POST request to the configured application URL. There is no body in the request. The sync notification metadata is contained in the request's headers. The following is an example of the sync notification request:

POST /ApplicationUrlPath
Accept: */*
Content-Type: application/json; charset="utf-8"
Content_Length: 0
Host: ApplicationUrlHost
X-Goog-Channel-Id: ChannelId
X-Goog-Channel-Token: ClientToken
X-Goog-Resource-Id: ResourceId
X-Goog-Resource-State: sync
X-Goog-Resource-Uri: https://storage.googleapis.com/storage/v1/b/BucketName/o?alt=json

Object Addition, Update, or Deletion

A notification event is sent when a new object is added to a bucket, an existing object's content or metadata has been modified, or an object is deleted from a bucket.

The notification will be sent as a POST request to the configured application URL. The body of the request contains a JSON-encoded message as shown in the following notification request:

POST /ApplicationUrlPath
Accept: */*
Content-Length: 1097
Content-Type: application/json; charset="utf-8"
Host: ApplicationUrlHost
X-Goog-Channel-Id: ChannelId
X-Goog-Channel-Token: ClientToken
X-Goog-Resource-Id: ResourceId
X-Goog-Resource-State: ResourceState
X-Goog-Resource-Uri: https://storage.googleapis.com/storage/v1/b/BucketName/o?alt=json

{
 "kind": "storage#object",
 "id": "BucketName/ObjectName",
 "selfLink": "https://www.googleapis.com/storage/v1/b/BucketName/o/ObjectName",
 "name": "ObjectName",
 "bucket": "BucketName",
 "generation": "1367014943964000",
 "metageneration": "1",
 "contentType": "application/octet-stream",
 "updated": "2013-04-26T22:22:23.832Z",
 "size": "10",
 "md5Hash": "xHZY0QLVuYng2gnOQD90Yw==",
 "mediaLink": "https://content-storage.googleapis.com/storage/v1/b/BucketName/o/ObjectName?generation=1367014943964000&alt=media",
 "owner": {
  "entity": "[email protected]"
 },
 "crc32c": "C7+82w==",
 "etag": "COD2jMGv6bYCEAE="
}

where ResourceState is:

exists - for object additions and updates.
not_exists - for object deletions.

and where the contents of the JSON message contain the current representation of the object as described at Object Resource Description.

Reliable Delivery

Object change notification will attempt to deliver notifications to your application in a reliable way. However, be aware that notifications can be delayed indefinitely, and timeliness is not guaranteed. Since your application might not always be available, the following rules are followed when delivering notifications:

If a notification delivery attempt fails, additional attempts will be made. The interval between additional delivery attempts is determined by an exponential backoff algorithm that starts with a retry 30 seconds after the initial failure. Subsequent deliveries are attempted at increasing intervals, up to a maximum interval of 90 minutes. Note that the subsequent retry intervals are slightly randomized so they do not occur at exact exponential values. After the maximum retry interval of 90 minutes is reached, subsequent retries continue every 90 minutes for 7 days. If the notification cannot be delivered in that time, it is purged.
If your application cannot be reached after 20 seconds or if your application responds with one of the following HTTP response codes, the notification delivery attempt is treated as a failure and is retried:
- 500 Internal Server Error
- 502 Bad Gateway
- 503 Service Unavailable
- 504 Gateway Timeout
If your application responds with one of the following HTTP response codes, the notification is treated as successfully delivered:
- 102 Processing
- 200 OK
- 201 Created
- 202 Accepted
- 204 No Content
Any other HTTP response codes returned by your application are treated as permanent failures and are not retried.

Note: Because of the retry mechanism above, it is possible that notifications are delivered more than once. Ensure that your application is idempotent with respect to processing a unique notification. You can can do this by choosing a combination of notification headers to help uniquely identify notifications, including X-Goog-Channel-Id, X-Goog-Resource-Id, X-Goog-Resource-State, X-Goog-Generation, and X-Goog-Metageneration.

Client Application Example

This section explains how to create an App Engine client application which processes change notification events.

The example application contains a class called MainPage. When the user updates or adds an object to the bucket, the MainPage class processes the notification event. For simplicity, the post method that does the actual processing just logs a message with the time the notification was received. You can replace this code with your actual processing logic. If you are not yet comfortable developing App Engine applications, try deploying a sample app or following a tutorial before you continue on.

Configuring the application.
Create the configuration file app.yaml to specify the client application that handles the bucket's change notification events.

application: APPLICATION
version: 1
runtime: python38
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: change_notification_client.app

Creating the Application.
The following example implements a client application for handling a bucket's change notification events. Name it change_notification_client.py, then deploy your app:

"""Notification handling for Google Cloud Storage."""

import json
import logging

import webapp2


class MainPage(webapp2.RequestHandler):
  """Process notification events."""
  def get(self):
    logging.info("Get request to notification page.")
    self.response.write("Welcome to the notification app.")

  def post(self):  # pylint: disable-msg=C6409
    """Process the notification event.

    This method is invoked when the notification channel is first created with
    a sync event, and then subsequently every time an object is added to the
    bucket, updated (both content and metadata) or removed. It records the
    notification message in the log.
    """

    logging.debug(
        '%s\n\n%s',
        '\n'.join(['%s: %s' % x for x in self.request.headers.iteritems()]),
        self.request.body)

    # The following code is for demonstration. Replace
    # it with your own notification processing code.

    if 'X-Goog-Resource-State' in self.request.headers:
      resource_state = self.request.headers['X-Goog-Resource-State']
      if resource_state == 'sync':
        logging.info('Sync message received.')
      else:
        an_object = json.loads(self.request.body)
        bucket = an_object['bucket']
        object_name = an_object['name']
        logging.info('%s/%s %s', bucket, object_name, resource_state)
    else:
      logging.info("Other post.")

logging.getLogger().setLevel(logging.DEBUG)
app = webapp2.WSGIApplication([('/', MainPage)], debug=True)

Assigning the application's access permission to the bucket.
If your bucket is owned by a different service account than your App Engine app, grant the application OWNER access to the bucket.
Start watching the bucket for object changes.
Create a notification channel to your application by watching the bucket using a watchAll request with the address of the URL for your App Engine application, e.g., https://ApplicationId.appspot.com/.
Testing the Application.
To see if the application works as expected, perform the following steps:
1. To assure that the application has been deployed and works correctly, execute the following curl command:
```
curl -X Post https://APPLICATION_ID.appspot.com
```
  If you used your own domain name to deploy the application, use it instead of appspot.com in the previous command.
2. Go to the Logging page for your project. Refresh the list of the logged messages, if needed. Verify that the log message issued by the application is logged.
3. Add an object to the bucket. Cloud Storage notifies the application, which then logs a message.
4. Go to the Logging page for your project. Refresh the list of the logged messages and find the message for the object copy.
Remove the notification channel
Remove the notification channel by specifying the channel and resource identifiers returned when you issued the notification command to watch the bucket.