Feed management API

Supported in:

This reference guide explains how to use the Google Security Operations feed management API to programmatically create, run, and manage data feeds that send logs to your Google Security Operations instance. For details about how to use the Google Security Operations UI to create and manage feeds, see the Feed management user guide.

Prerequisites

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Google Security Operations. You can find the prerequisites as follows:

  • Prerequisites for each source type are listed in Configuration by source type .

  • Prerequisites for each log type ingested using the API feed source type are listed in Configuration by log type .

  • Prerequisites for all log types ingested using any source type are listed in the Google Security Operations UI. Go to Settings > Feeds > Add New, select a Source Type and Log Type, and review the required fields. For details, see Creating and editing feeds .

For example, if you set up a data feed from a Google Cloud Storage bucket, you might need to complete the following tasks:

  1. Use the feed management fetchFeedServiceAccount method to get a Google Security Operations service account that Google Security Operations uses to ingest data.
  2. Grant access to the Google Security Operations service account to the relevant Cloud Storage objects. For more information, see Grant access to the Google Security Operations service account.

How to authenticate with the Google Security Operations API

This Google Security Operations API uses the OAuth 2.0 protocol for authentication and authorization. Your application can complete these tasks using either of the following implementations:

  • Using the Google API Client Library for your computer language.

  • Directly interfacing with the OAuth 2.0 system using HTTP.

See the reference documentation for the Google Authentication library in Python.

Google Authentication libraries are a subset of the Google API client libraries. See other language implementations.

Getting API authentication credentials

Your Google Security Operations representative will provide you with a Google Developer Service Account Credential to enable the API client to communicate with the API.

You also must provide the Auth Scope when initializing your API client. OAuth 2.0 uses a scope to limit an application's access to an account. When an application requests a scope, the access token issued to the application is limited to the scope granted.

Use the following scope to initialize your Google API client:

https://www.googleapis.com/auth/chronicle-backstory

Python example

The following Python example demonstrates how to use the OAuth2 credentials and HTTP client using google.oauth2 and googleapiclient.

# Imports required for the sample - Google Auth and API Client Library Imports.
# Get these packages from https://pypi.org/project/google-api-python-client/ or run $ pip
# install google-api-python-client from your terminal
from google.auth.transport import requests
from google.oauth2 import service_account

SCOPES = ['https://www.googleapis.com/auth/chronicle-backstory']

# The apikeys-demo.json file contains the customer's OAuth 2 credentials.
# SERVICE_ACCOUNT_FILE is the full path to the apikeys-demo.json file
# ToDo: Replace this with the full path to your OAuth2 credentials
SERVICE_ACCOUNT_FILE = '/customer-keys/apikeys-demo.json'

# Create a credential using Google Developer Service Account Credential and Google Security Operations API
# Scope.
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)

# Build a requests Session Object to make authorized OAuth requests.
http_session = requests.AuthorizedSession(credentials)

# Your endpoint GET|POST|PATCH|etc. code will vary below

# Reference List example (for US region)
url = 'https://backstory.googleapis.com/v2/lists/COLDRIVER_SHA256'

# You might need another regional endpoint for your API call; see
# https://cloud.google.com/chronicle/docs/reference/ingestion-api#regional_endpoints

# requests GET example
response = http_session.request("GET", url)

# POST example uses json
body = {
  "foo": "bar"
}
response = http_session.request("POST", url, json=body)

# PATCH example uses params and json
params = {
  "foo": "bar"
}
response = http_session.request("PATCH", url, params=params, json=body)

# For more complete examples, see:
# https://github.com/chronicle/api-samples-python/

Chronicle API query limits

The Chronicle API enforces limits on the volume of requests that can be made by any one customer against the Google Security Operations platform. If you reach or exceed the query limit, the Chronicle API server returns HTTP 429 (RESOURCE_EXHAUSTED) to the caller. When developing applications for the Chronicle API, Google Security Operations recommends that you enforce rate limits within your system to avoid resource exhaustion. These limits apply to all of the Chronicle APIs, including the feed management API.

The feed management API enforces the following limits, which are measured in queries per second (QPS):

Chronicle API API Method Limit
Feed management Create Feed 1 QPS
Get Feed 1 QPS
List Feeds 1 QPS
Update Feed 1 QPS
Delete Feed 1 QPS

Control the rate of ingestion

When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations restricts the rate of ingestion for new data feeds to prevent a source with a high ingestion rate from affecting the ingestion rate of another data source. In this case, there is a delay but no data is lost. The ingestion volume and tenant's usage history determine the threshold.

You can request a rate limit increase by contacting Cloud Customer Care.

Limitations

Data feeds have a maximum log line size of 4 MB.

See the detailed list of the Chronicle API query limits.

Python example using OAuth2 credentials and HTTP client

The following Python example demonstrates how to use the OAuth2 credentials and the HTTP client using google.oauth2 and googleapiclient.

# Imports required for the sample - Google Auth and API Client Library Imports.
# Get these packages from https://pypi.org/project/google-api-python-client/ or
# run $ pip install google-api-python-client from your terminal

from google.auth.transport import requests
from google.oauth2 import service_account

SCOPES = ['https://www.googleapis.com/auth/chronicle-backstory']

# The apikeys-demo.json file contains the customer's OAuth 2 credentials.
# SERVICE_ACCOUNT_FILE is the full path to the apikeys-demo.json file
# ToDo: Replace this with the full path to your OAuth2 credentials

SERVICE_ACCOUNT_FILE = '/customer-keys/apikeys-demo.json'

# Create a credential using Google Developer Service Account Credential and 
Chronicle API Scope.

credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)

# Build an HTTP session to make authorized OAuth requests.

http_session = requests.AuthorizedSession(credentials)

# <your code continues here>

Regional endpoints

Google Security Operations provides regional endpoints for each API.

  • São Paulo—https://southamerica-east1-backstory.googleapis.com
  • Canada—https://northamerica-northeast2-backstory.googleapis.com
  • Dammam—https://me-central2-backstory.googleapis.com
  • Doha—https://me-central1-backstory.googleapis.com
  • Europe Multi-Region—https://europe-backstory.googleapis.com
  • Frankfurt—https://europe-west3-backstory.googleapis.com
  • London—https://europe-west2-backstory.googleapis.com
  • Mumbai—https://asia-south1-backstory.googleapis.com
  • Paris—https://europe-west9-backstory.googleapis.com
  • Singapore—https://asia-southeast1-backstory.googleapis.com
  • Sydney—https://australia-southeast1-backstory.googleapis.com
  • Tel Aviv—https://me-west1-backstory.googleapis.com
  • Tokyo—https://asia-northeast1-backstory.googleapis.com
  • Turin—https://europe-west12-backstory.googleapis.com
  • United States Multi-Region—https://backstory.googleapis.com
  • Zurich—https://europe-west6-backstory.googleapis.com

Feed Schema API reference

The Feed Schema API returns information that is useful for constructing valid feed management API requests. For example, you can get the data structure representing the entire feed schema. This structure defines the specific fields to specify for each valid combination of feed source type and log type. Alternatively, you can get a list of all log types compatible with a particular feed source type.

Specifically, the feed schema contains:

  • Information about each valid feed source type:
    • A human-readable name
    • A human-readable description
    • Whether feeds with a given feed source type can be modified using the API, or are read-only
  • Information about each log type:
    • A human-readable name
    • Whether feeds with a given log type can be modified using the API, or are read-only
  • Which log types are compatible with which feed source types
  • Information about the specific fields to specify for each valid combination of log type and feed source type:
    • A human-readable field name and description
    • Compatibility with other fields
    • Semantic type (e.g. URI, "secret", etc)
    • Whether the field is required
    • What a valid value for the field looks like

The schema can be accessed using a few different methods.

GetFeedSchema

This method returns a structure representing the entire feed schema. The schema is comprised of a list of "feed source type schemas" each of which describe the supported feed source types. Each feed source types' schema contains a list of "log type schemas" that correspond to the set of log types that are compatible with the feed source type and describe the log type. Each log type schema contains a list of "details field schemas" that describe those fields you would set when issuing a Create Feed request, for example, or expect to see as a result of a ListFeed or GetFeed response. The field schemas specified are unique to the particular combination of log type and feed source type.

Request

GET https://backstory.googleapis.com/v1/feedSchema

Response

{
  "feedSourceTypeSchemas": [{
      "name": "feedSourceTypeSchemas/AMAZON_S3",
      "displayName": "Amazon S3",
      "description": "Amazon Simple Storage Service, a service offered by Amazon Web Services that provides object storage through a web service interface",
      "feedSourceType": "AMAZON_S3",
      "logTypeSchemas": [{
          "name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL",
          "displayName": "AWS Cloudtrail",
          "logType": "AWS_CLOUDTRAIL",
          "detailsFieldSchemas": [{
              "fieldPath": "details.amazon_s3_settings.authentication.access_key_id",
              "displayName": "Access key ID",
              "description": "An account access key that is a 20-character alphanumeric string, for example AKIAIOSFODNN7EXAMPLE",
              "type": "STRING",
              "exampleInput": "AKIAIOSFODNN7EXAMPLE",
            },
            ...
            {
              "fieldPath": "details.amazon_s3_settings.s3_uri",
              "displayName": "S3 URI",
              "description": "The S3 bucket source URI",
              "type": "STRING_URI",
              "isRequired": true,
              "exampleInput": "s3://cs-prod-cannon-00afe0c847a8/data/",
            }],
        },
        ...
        {
          "name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/ABNORMAL_SECURITY",
          "displayName": "Abnormal Security",
          "logType": "ABNORMAL_SECURITY",
          ...
        }],
    },
    ...
    {
      "name": "feedSourceTypeSchemas/AMAZON_SQS",
      "displayName": "Amazon SQS",
      "description": "Amazon Simple Queue Service, a service offered by Amazon Web Services that provides fully managed message queuing service to transfer messages asynchronously",
      "feedSourceType": "AMAZON_SQS",
      ...
    }],
}

ListFeedSourceTypeSchemas

This method returns information about all feed source types.

Request
GET https://backstory.googleapis.com/v1/feedSourceTypeSchemas
Sample Response
{
  "feedSourceTypeSchemas": [{
      "name": "feedSourceTypeSchemas/AMAZON_S3",
      "displayName": "Amazon S3",
      "description": "Amazon Simple Storage Service, a service offered by Amazon Web Services that provides object storage through a web service interface",
      "feedSourceType": "AMAZON_S3",
    },
    ...
    {
      "name": "feedSourceTypeSchemas/AMAZON_SQS",
      "displayName": "Amazon SQS",
      "description": "Amazon Simple Queue Service, a service offered by Amazon Web Services that provides fully managed message queuing service to transfer messages asynchronously",
      "feedSourceType": "AMAZON_SQS",
    }],
}

ListLogTypeSchemas

This method returns information about all log types compatible with a particular feed source type.

Request
GET https://backstory.googleapis.com/v1/feedSourceTypeSchemas/{feed source type}/logTypeSchemas
Sample Request
https://backstory.googleapis.com/v1/feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas
Sample Response
{
  "logTypeSchemas": [{
      "name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL",
      "displayName": "AWS Cloudtrail",
      "logType": "AWS_CLOUDTRAIL",
    },
    ...
    {
      "name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/ABNORMAL_SECURITY",
      "displayName": "Abnormal Security",
      "logType": "ABNORMAL_SECURITY",
      ...
    }],
}

GetLogTypeSchema

This method returns detailed information about all the fields necessary to configure a feed for a particular source type and log type.

Request
GET https://backstory.googleapis.com/v1/feedSourceTypeSchemas/{feed source type}/logTypeSchemas/{log type}
Sample Request
https://backstory.googleapis.com/v1/feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL
Sample Response
{
  "name": "feedSourceTypeSchemas/AMAZON_S3/logTypeSchemas/AWS_CLOUDTRAIL",
  "displayName": "AWS Cloudtrail",
  "logType": "AWS_CLOUDTRAIL",
  "detailsFieldSchemas": [{
      "fieldPath": "details.amazon_s3_settings.authentication.access_key_id",
      "displayName": "Access key ID",
      "description": "An account access key that is a 20-character alphanumeric string, for example AKIAIOSFODNN7EXAMPLE",
      "type": "STRING",
      "exampleInput": "AKIAIOSFODNN7EXAMPLE",
    },
    ...
    {
      "fieldPath": "details.amazon_s3_settings.s3_uri",
      "displayName": "S3 URI",
      "description": "The S3 bucket source URI",
      "type": "STRING_URI",
      "isRequired": true,
      "exampleInput": "s3://cs-prod-cannon-01abc2d345e6/data/",
    }],
}

Feed management API reference

This section describes the endpoints for creating, enabling, and managing feeds.

When creating or editing a feed, you need to specify the feedSourceType and logType in the request body. For details about these fields, see Configuration by source type and Configuration by log type.

Create Feed

Creates a third party data feed in your Google Security Operations instance.

Request

POST https://backstory.googleapis.com/v1/feeds
Request body

This example shows how to collect authentication logs from Duo Security.

{
  "display_name": "some feed name",
  "details": {
    "feedSourceType": "API",
    "logType": "DUO_AUTH",
    "duoAuthSettings": {
      "authentication": {
        "user": "ABCUSERNAMEDEF",
        "secret": "aBcS3cReTdEf"
      },
      "hostname": "api-abc123.duosecurity.com"
    },
    "namespace": "my-asset-namespace",
    "labels": [{
      "key": "my-ingestion-label-key",
      "value": "my-ingestion-label-value"
    }]
  }
}
Sample request
https://backstory.googleapis.com/v1/feeds
{
  "display_name": "some feed name",
  "details": {
    "feedSourceType": "API",
    "logType": "DUO_AUTH",
    "duoAuthSettings": {
      "authentication": {
        "user": "ABCUSERNAMEDEF",
        "secret": "aBcS3cReTdEf"
      },
      "hostname": "api-abc123.duosecurity.com"
    }
  }
}
Sample successful response
{
 "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
 "display_name": "some feed name",
 "details": {
   "logType": "DUO_AUTH",
   "feedSourceType": "API",
   "duoAuthSettings": {
     "hostname": "api-abc123.duosecurity.com"
   }
 },
 "feedState": "ACTIVE"
}

If the response is unsuccessful, it returns an HTTP status code other than 200 (OK). Be sure to check the body of the response for details of the failure.

Asset namespace

To assign an asset namespace to all events that are ingested from a particular feed, set the "namespace" field within details. The namespace field is a string.

Ingestion label

Ingestion labels are part of Unified Data Model metadata. They are repeated key and value pairs. To assign ingestion labels to all events that are ingested from a particular feed, set the labels field within details. The labels field is an array of JSON objects with key and value fields.

DeleteFeed

Deletes a feed that was configured using the Google Security Operations feed management API.

Request

DELETE  https://backstory.googleapis.com/v1/feeds/{feedID}
Sample request
DELETE https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567
Sample response

If the operation is successful, Delete Feed returns an empty response with an HTTP status code 200 (OK).

{}

Enable Feed

Enables an INACTIVE feed, which allows it to be executed.

Request
POST https://backstory.googleapis.com/v1/feeds/{feedID}:enable
Sample request
POST https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567:enable
Sample response
{
 "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
 "display_name": "some feed name",
 "details": {
   "logType": "DUO_AUTH",
   "feedSourceType": "API",
   "duoAuthSettings": {
     "hostname": "api-abc123.duosecurity.com"
   }
 },
 "feedState": "ACTIVE"
}

Disable Feed

Disables a feed. A disabled feed has a status of INACTIVE. Disabled feeds will no longer fetch data.

Request
POST https://backstory.googleapis.com/v1/feeds/{feedID}:disable
Sample request
POST https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567:disable
Sample response
{
 "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
 "display_name": "some feed name",
 "details": {
   "logType": "DUO_AUTH",
   "feedSourceType": "API",
   "duoAuthSettings": {
     "hostname": "api-abc123.duosecurity.com"
   }
 },
 "feedState": "INACTIVE"
}

Get Feed

Gets the details of the feed that was configured.

Request

GET https://backstory.googleapis.com/v1/feeds/{feedID}
Sample request
https://backstory.googleapis.com/v1/feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567
Sample response
{
 "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
 "display_name": "some feed name",
 "details": {
   "logType": "DUO_AUTH",
   "feedSourceType": "API",
   "duoAuthSettings": {
     "hostname": "api-abc123.duosecurity.com"
   }
 },
 "feedState": "FAILED",
"last_feed_initiation_time": "2024-01-15T01:30:15.01Z",
"failure_details": {
  "error_code": "INVALID_ARGUMENT"
  "http_error_code": 400,
  "error_cause": "A connection to the source was established, but the feed failed because of invalid arguments",
  "error_action":"Check the feed configuration. Learn more about setting up the feeds.\nIf the problem continues, contact Chronicle Support"
 }
}

List Feeds

Retrieves all the feeds configured for a given Google Security Operations instance.

Request
GET https://backstory.googleapis.com/v1/feeds
Sample request
https://backstory.googleapis.com/v1/feeds
Sample response
{
 "feeds": [
   {
     "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
     "details": {
       "logType": "AZURE_AD_CONTEXT",
       "feedSourceType": "API",
       "azureAdContextSettings": {}
     },
     "feedState": "FAILED",
  "last_feed_initiation_time": "2024-01-15T01:30:15.01Z",
  "failure_details": {
    "error_code": "INVALID_ARGUMENT"
    "http_error_code": 400,
    "error_cause": "A connection to the source was established, but the feed failed because of invalid arguments",
    "error_action":"Check the feed configuration. Learn more about setting up the feeds.\nIf the problem continues, contact Chronicle Support"
   }
   },
   {
     "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
     "display_name": "some feed name",
     "details": {
       "logType": "PAN_PRISMA_CLOUD",
       "feedSourceType": "API",
       "panPrismaCloudSettings": {
         "hostname": "api2.prismacloud.io"
       }
     },
     "feedState": "ACTIVE",
  "last_feed_initiation_time": "2024-01-15T01:30:15.01Z",
   }
 ]
}

Read-only feeds

There may be feeds returned from a List Feeds request that have the field readOnly set to true. Read-only feeds cannot be created, updated, or deleted.

Feeds are read-only for a few reasons. For example:

  • Some feed source types are not fully supported by feed management at the moment, and were created before the release of feed management.
  • Some specialized log types are not available to every Google Security Operations user. If a feed exists with one of these types, it is considered read-only.

Update Feed

Updates the given feed with new details.

Request
PATCH https://backstory.googleapis.com/v1/feeds/{feedID}
Request body

The following examples shows how to update a Duo Auth feed.

Sample request
{
  "display_name": "my feed",
  "details": {
    "feedSourceType": "API",
    "logType": "DUO_AUTH",
    "duoAuthSettings": {
      "authentication": {
        "user": "ABCUSERNAMEDEF",
        "secret": "aBcS3cReTdEf"
      },
      "hostname": "api-abc123.duosecurity.com"
    }
  }
}
Sample response
{
 "display_name": "my feed",
 "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
 "details": {
   "logType": "DUO_AUTH",
   "feedSourceType": "API",
   "duoAuthSettings": {
     "hostname": "api-abc123.duosecurity.com"
   }
 },
 "feedState": "ACTIVE"
}
Sample Request that does not update displayName
{
 "name": "feeds/12a34567-bc8d-1111-e9f0-gh1ijk234567",
 "details": {
   "logType": "DUO_AUTH",
   "feedSourceType": "API",
   "duoAuthSettings": {
     "hostname": "api-abc123.duosecurity.com"
   }
 },
 "feedState": "ACTIVE"
}

Fetch service account

Gets a unique service account that Google Security Operations uses to ingest data. Use this method only if you're setting up a Cloud Storage feed.

Request
GET https://backstory.googleapis.com/v1/fetchFeedServiceAccount
Sample request
GET https://backstory.googleapis.com/v1/fetchFeedServiceAccount
Sample response
"serviceAccount": "[email protected]"

Response message fields

This section describes the following fields that are returned in response messages:

  • feedState
  • failureMsg

Feed state

The feedState field can be found in the response message of most operations. feedState gives some insight into the current state of a feed.

feedState Description
"ACTIVE" Feed successfully created and will begin fetching data.
"INACTIVE" Feed has been disabled.
"IN_PROGRESS" Feed is attempting to fetch data. A feed will only have this status if it has not previously failed.
"COMPLETED" Feed has recently fetched data successfully.
"FAILED" Feed has failed and has not successfully fetched data since it failed. Mis-configuration is the typical cause of feed failure. Please see the failureMsg field for more information.

Failure message

The failureMsg field can be found in the response message of most operations, but only for those feeds whose feedState is FAILED. It provides information regarding the error code, cause of the error, and how to troubleshoot the error. For information about error messages, see Troubleshooting. Refer to the following documentation for your particular feed type to understand how to correctly configure the feed.

Generate secret key and API key to authenticate the feed

You need to generate the secret key and API key to authenticate the feed when you set up a feed that has webhook or Amazon Data Firehose as the source type. You can reuse your existing API key to authenticate to Google Security Operations. You must generate a secret key for every new feed and can't reuse the secret key.

  1. To generate a secret key for a webhook or Amazon Data Firehose feed, run the following curl command that uses the generateSecret Chronicle API.

       curl --location --request POST -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID:generateSecret

    Replace the following:

    • REGIONAL_ENDPOINT: the Google Security Operations regional endpoint, such as us-chronicle.googleapis.com. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.
    • PROJECT_NUMBER: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.
    • REGION_ID: the code that Google assigns based on the region. The following are the supported region IDs: Asia-Southeast1, Australia-Southeast1, Europe, EU, Europe-West2, Europe-West3, Europe-West6, Govcloud-US, Me-West1, and US.
    • CUSTOMER_ID: the Google Security Operations customer ID.
    • FEED_ID: the Google Security Operations feed ID.

    A secret key is returned. Copy and store the secret key because you cannot view this secret again. You can use the generateSecret API again to generate a new secret key, but regeneration of the secret key makes the previous secret key obsolete.

  2. To generate the API key, do the following:

    1. Go to the Google Cloud console Credentials page.
    2. Click Create credentials, and then select API key.
    3. Restrict the API key access to the Chronicle API.

Configuration by source type

This section provides information about configuring feed source types. A feed source type defines where data is located and how it's accessed. Valid values for feedSourceType are as follows:

feedSourceType Description
API Ingest data from a third-party API.
HTTPS_PUSH_GOOGLE_CLOUD_PUBSUB Ingest data using a Pub/Sub push subscription.
GOOGLE_CLOUD_STORAGE Ingest data from a Cloud Storage bucket.
HTTPS_PUSH_AMAZON_KINESIS_FIREHOSE Ingest data from Amazon Data Firehose.
AMAZON_S3 Ingest data from an Amazon Simple Storage Service bucket.
AMAZON_SQS Ingest data from an Amazon Simple Queue Service queue whose entries point to files stored in S3
AZURE_BLOBSTORE Ingest data from Azure Blob Storage.
HTTP Ingest data from files accessible by an HTTP(S) request. Note that this *should not* be used to interact with third-party APIs. Use the API feed source type for third-party APIs supported by Google Security Operations.
HTTPS_PUSH_WEBHOOK Ingest data using an HTTPS webhook.

API

Use the API feed source type to ingest data from a third-party API. The configuration settings for the API feed source type are specific to the log type that you specify for the feed.

For example, when you use the Feeds API endpoints to configure settings for an API feed data source, you specify API as the value for the details.feedSourceType field. You also specify a supported log type as the value for the details.logType field. The log type determines the additional feed configuration settings that you can specify. Details about these settings are provided in the section for that specific log type in Configuration by log type.

Google Cloud Pub/Sub

Data source Ingest schedule details.feedSourceType
Pub/Sub Based on your implementation, it might take approximately a minute for a new feed to populate because Google Security Operations takes some time to process the data. HTTPS_PUSH_GOOGLE_CLOUD_PUBSUB

Prerequisites

Set up push ingestion using Pub/Sub

Data can be sent to Google Security Operations using Pub/Sub. You must first create a feed with the appropriate log type before configuring Pub/Sub to send data.

To set up HTTPS push ingestion using Pub/Sub, do the following:

  1. Create a Pub/Sub feed using the following create API request:

      {
    "displayName": "FEED_NAME",
    "details": {
      "feedSourceType": "HTTPS_PUSH_GOOGLE_CLOUD_PUBSUB",
      "logType": "projects/PROJECT_ID/locations/REGION_ID/instances/CUSTOMER_ID/logTypes/LOG_TYPE"
    }
  }

    Replace the following:

    • FEED_NAME: specify a name for the feed.
    • PROJECT_ID: the project ID of the project that is bound to Google Security Operations.
    • REGION_ID: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs: Asia-Southeast1, Australia-Southeast1, Europe, EU, Europe-West2, Europe-West3, Europe-West6, Govcloud-US, Me-West1, and US.
    • CUSTOMER_ID: the Google Security Operations customer ID.
    • LOG_TYPE: the type of data the feed ingests. Google Security Operations supports specific log types for the Pub/Sub feed.

    If the data to be ingested contains a delimiter that separate log lines, such as \\n, include the following in the details field of the feed request body:

      "httpsPushGoogleCLoudPubsubSettings": {
    "splitDelimiter": "LOG_DELIMITER"
  }

    Replace LOG_DELIMITER with the delimiter that separates the log lines, such as \\n.

  2. After you create a feed in Pub/Sub, create a push subscription, specify the HTTPS endpoint, and enable authentication. For more information about how to create a push subscription, see Create push subscriptions.

    • Specify the endpoint URL. The endpoint URL must have the following format: https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_ID/ locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID:importPushLogs

      • REGIONAL_ENDPOINT: the Google Security Operations regional endpoint, such as us-chronicle.googleapis.com. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.
      • PROJECT_ID: the project ID of the project that is bound to Google Security Operations.
      • REGION_ID: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs: Asia-Southeast1, Australia-Southeast1, Europe, EU, Europe-West2, Europe-West3, Europe-West6, Govcloud-US, Me-West1, and US.
      • CUSTOMER_ID: the Google Security Operations customer ID.
      • FEED_ID: the Google Security Operations feed ID.

    • Select Enable authentication and select a service account.

GOOGLE_CLOUD_STORAGE

Data source Ingest schedule details.feedSourceType details.logType
Google Cloud Storage Bucket Every 15 minutes GOOGLE_CLOUD_STORAGE See the Feed Schema API reference to get compatible log types.

Prerequisites

Before you set up a Cloud Storage feed, you must get a Google Security Operations service account and provide access to the account so that Google Security Operations can ingest data.

  1. Use the feed management fetchFeedServiceAccount method to get a Google Security Operations service account.
  2. Grant access to the Google Security Operations service account to the relevant Cloud Storage objects. For more information, see Grant access to the Google Security Operations service account.
  3. If VPC Service Controls is enabled, configure an ingress rule to provide access to the Cloud Storage bucket. In this document, see Configure VPC Service Controls.

Recommendation

  • If your Cloud Storage bucket contains many small files, it can increase the amount of time it takes to transfer the data. It is recommended to combine smaller files into one large file to accelerate the feed transfer process.

  • It is important to set a data retention policy to your Cloud Storage buckets so that the transferred files are deleted from the buckets and are not listed in any future transfer feeds. Alternatively, using the Google SecOps feed management UI, you can set the option to delete the source files from the storage buckets after they have been transferred.

  • As Google SecOps pulls files from Cloud Storage on a frequent basis, it is recommended to specify a storage class for the bucket that is the most cost-effective.

Configure VPC Service Controls

If VPC Service Controls is enabled, an ingress rule is required to provide access to the Cloud Storage bucket.

The following Cloud Storage methods must be allowed in the ingress rule:

  • google.storage.objects.list. Required for a single file feed.
  • google.storage.objects.get. Required for feeds that require directory or subdirectory access.
  • google.storage.objects.delete. Required for feeds that require deletion of the source file.

Sample ingress rule

- ingressFrom:
  identities:
    - serviceAccount:[email protected]
  sources:
  - accessLevel: "*"
  ingressTo:
  operations:
  - serviceName: storage.googleapis.com
    methodSelectors:
    - method: google.storage.objects.list
    - method: google.storage.objects.get
    - method: google.storage.objects.delete
  resources:
  - projects/PROJECT_ID

Type-specific request fields

Field Required Description
details.gcsSettings.bucketUri Yes The URI which corresponds to the Cloud Storage bucket. The format is the same format used by the gcloud storage command group to specify a resource.
details.gcsSettings.sourceType Yes The type of object indicated by bucketUri. Valid values are:
  • FILES: The URI points to a single file which will be ingested with each execution of the feed.
  • FOLDERS: The URI points to a directory. All files contained within the directory will be ingested with each execution of the feed.
  • FOLDERS_RECURSIVE: The URI points to a directory. All files and directories within the indicated directory will be ingested, including all files and directories within those directories.
details.gcsSettings.sourceDeletionOption Yes Whether to delete source files after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
  • SOURCE_DELETION_NEVER: Never delete files from the source.
  • SOURCE_DELETION_ON_SUCCESS:Delete files and empty directories from the source after successful ingestion.
  • SOURCE_DELETION_ON_SUCCESS_FILES_ONLY:Delete files from the source after successful ingestion.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "GOOGLE_CLOUD_STORAGE",
   "logType": "LOGTYPE_YOU_WANT_TO_BRING",
   "gcsSettings": {
     "bucketUri": "gs://bucket/file",
     "sourceType": "FOLDERS_RECURSIVE",
     "sourceDeletionOption": "SOURCE_DELETION_NEVER"
   }
 }
}

Amazon Data Firehose

Data source Ingest schedule details.feedSourceType
Amazon Data Firehose Based on your implementation, it might take approximately a minute for a new feed to populate because Google Security Operations takes some time to process the data. HTTPS_PUSH_AMAZON_KINESIS_FIREHOSE

Prerequisites

Set up push ingestion using Amazon Data Firehose

Data can be sent to Google Security Operations using Amazon Data Firehose. You must first create a feed with the appropriate log type before configuring Amazon Data Firehose to send data.

To set up HTTPS push ingestion using Amazon Data Firehose, do the following:

  1. Create an Amazon Data Firehose feed using the following create API request:

      {
    "displayName": "FEED_NAME",
    "details": {
      "feedSourceType": "HTTPS_PUSH_AMAZON_KINESIS_FIREHOSE",
      "logType": "projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/logTypes/LOG_TYPE"
    }
  }

    Replace the following:

    • FEED_NAME: specify a name for the feed.
    • PROJECT_NUMBER: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.
    • REGION_ID: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs: Asia-Southeast1, Australia-Southeast1, Europe, EU, Europe-West2, Europe-West3, Europe-West6, Govcloud-US, Me-West1, and US.
    • CUSTOMER_ID: the Google Security Operations customer ID.
    • LOG_TYPE: the type of data the feed ingests. Google Security Operations supports specific log types for the Amazon Data Firehose feed.

    If the data to be ingested contains a delimiter that separate log lines, such as \\n, include the following in the details field of the feed request body:

      "httpsPushAmazonKinesisFirehoseSettings": {
    "splitDelimiter": "LOG_DELIMITER"
  }

    Replace LOG_DELIMITER with the delimiter that separates the log lines, such as \\n.

  2. After you create a feed, generate a secret key for the feed and generate an API key to authenticate to Google Security Operations.

  3. In Amazon Data Firehose, specify the HTTPS endpoint and access key.

    • Specify the endpoint URL. Here is a sample Amazon Data Firehose endpoint: https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_NUMBER/ locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID%3AimportPushLogs?key=API_KEY

      The endpoint includes the following values:

      • REGIONAL_ENDPOINT: the Google Security Operations regional endpoint, such as us-chronicle.googleapis.com. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.
      • PROJECT_NUMBER: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.
      • REGION_ID: the region configured for your Google Security Operations instance. This was set when the tenant was provisioned. The following are the supported region IDs: Asia-Southeast1, Australia-Southeast1, Europe, EU, Europe-West2, Europe-West3, Europe-West6, Govcloud-US, Me-West1, and US.
      • CUSTOMER_ID: the Google Security Operations customer ID.
      • FEED_ID: the Google Security Operations feed ID.
      • API_KEY: the API key value.

    • In the Access key field, specify the secret key that you obtained using the generateSecret API.

AMAZON_S3

Data source Ingest schedule details.feedSourceType details.logType
Amazon Simple Storage Service Bucket Every 15 minutes AMAZON_S3 See the Feed Schema API reference to get compatible log types.

Prerequisites

  1. Create an S3 bucket.
  2. Create a security key for programmatic access.

To learn more about how to configure a feed to ingest data from an Amazon S3 bucket, see Ingest AWS logs.

Recommendation

  • If your S3 bucket contains many small files, it can increase the amount of time it takes to transfer the data. It is recommended to combine smaller files into one large file to accelerate the feed transfer process.

  • It is important to set a data retention policy to your S3 buckets so that the transferred files are deleted from the buckets and are not listed in any future transfer feeds. Alternatively, using the Google SecOps feed management UI, you can set the option to delete the source files from the storage buckets after they have been transferred.

  • As Google SecOps pulls files from S3 buckets on a frequent basis, it is recommended to specify a storage class for your buckets that is the most cost-effective.

Type-specific request fields

Field Required Description
details.amazonS3Settings.s3Uri Yes The S3 URI to ingest.
details.amazonS3Settings.sourceType Yes The type of file indicated by the URI. Valid values are:
  • FILES: The URI points to a single file which will be ingested with each execution of the feed.
  • FOLDERS: The URI points to a directory. All files within the directory will be ingested with each execution of the feed.
  • FOLDERS_RECURSIVE: The URI points to a directory. All files and directories within the indicated directory will be ingested, including all files and directories within those directories.
details.amazonS3Settings.sourceDeletionOption Yes Whether to delete source files after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
  • SOURCE_DELETION_NEVER: Never delete files from the source.
  • SOURCE_DELETION_ON_SUCCESS:Delete files and empty directories from the source after successful ingestion.
  • SOURCE_DELETION_ON_SUCCESS_FILES_ONLY:Delete files from the source after successful ingestion.
details.amazonS3Settings.authentication.region Yes The region where the S3 bucket resides. For a list of regions, see Amazon S3 regions.
details.amazonS3Settings.authentication.accessKeyId Yes This is the 20 character ID associated with your Amazon IAM account.
details.amazonS3Settings.authentication.secretAccessKey Yes This is the 40 character access key associated with your Amazon IAM account.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "AMAZON_S3",
   "logType": "LOGTYPE_YOU_WANT_TO_BRING",
   "amazonS3Settings": {
     "s3Uri": "s3://uri/to/file",
     "sourceType": "FILES",
     "sourceDeletionOption": "SOURCE_DELETION_NEVER",
     "authentication": {
       "region": "US_EAST_1",
       "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
       "secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
     },
   }
 }
}

Amazon S3 regions

AWS Region AWS Region code authentication.region
Asia Pacific (Mumbai) ap-south-1 AP_SOUTH_1
Asia Pacific (Seoul) ap-northeast-2 AP_NORTHEAST_2
Asia Pacific (Singapore) ap-southeast-1 AP_SOUTHEAST_1
Asia Pacific (Sydney) ap-southeast-2 AP_SOUTHEAST_2
Asia Pacific (Tokyo) ap-northeast-1 AP_NORTHEAST_1
AWS GovCloud (US-East) us-gov-east-1 US_GOV_EAST_1
AWS GovCloud (US-West) us-gov-west-1 US_GOV_CLOUD
Canada (Central) ca-central-1 CA_CENTRAL_1
China (Beijing) cn-north-1 CN_NORTH_1
China (Ningxia) cn-northwest-1 CN_NORTHWEST_1
Europe (Frankfurt) eu-central-1 EU_CENTRAL_1
Europe (Ireland) eu-west-1 EU_WEST_1
Europe (London) eu-west-2 EU_WEST_2
Europe (Paris) eu-west-3 EU_WEST_3
Europe (Stockholm) eu-north-1 EU_NORTH_1
South America (São Paulo) sa-east-1 SA_EAST_1
US East (N. Virginia) us-east-1 US_EAST_1
US East (Ohio) us-east-2 US_EAST_2
US West (N. California) us-west-1 US_WEST_1
US West (Oregon) us-west-2 US_WEST_2

AMAZON_SQS

Data source details.feedSourceType details.logType
Amazon Simple Storage Service Bucket (S3) that sends notifications to an Amazon Simple Queue Service (SQS) AMAZON_SQS See the Feed Schema API reference to get compatible log types.

You can ingest data from an Amazon SQS service whose entries point to files stored in an Amazon S3 bucket.

When you apply a policy, make sure that you have included sqs:DeleteMessage. Google Security Operations is not able to delete messages if the sqs:DeleteMessage permission is not attached to the SQS queue. All the messages are accumulated on the AWS side, which causes a delay as Google Security Operations repeatedly attempts to transfer the same files.

Amazon SQS regions

AWS Region AWS Region code authentication.region
Asia Pacific (Mumbai) ap-south-1 AP_SOUTH_1
Asia Pacific (Seoul) ap-northeast-2 AP_NORTHEAST_2
Asia Pacific (Singapore) ap-southeast-1 AP_SOUTHEAST_1
Asia Pacific (Sydney) ap-southeast-2 AP_SOUTHEAST_2
Asia Pacific (Tokyo) ap-northeast-1 AP_NORTHEAST_1
AWS GovCloud (US-East) us-gov-east-1 US_GOV_EAST_1
AWS GovCloud (US-West) us-gov-west-1 US_GOV_CLOUD
Canada (Central) ca-central-1 CA_CENTRAL_1
China (Beijing) cn-north-1 CN_NORTH_1
China (Ningxia) cn-northwest-1 CN_NORTHWEST_1
Europe (Frankfurt) eu-central-1 EU_CENTRAL_1
Europe (Ireland) eu-west-1 EU_WEST_1
Europe (London) eu-west-2 EU_WEST_2
Europe (Paris) eu-west-3 EU_WEST_3
Europe (Stockholm) eu-north-1 EU_NORTH_1
South America (São Paulo) sa-east-1 SA_EAST_1
US East (N. Virginia) us-east-1 US_EAST_1
US East (Ohio) us-east-2 US_EAST_2
US West (N. California) us-west-1 US_WEST_1
US West (Oregon) us-west-2 US_WEST_2

Prerequisites

Create the S3 bucket and the SQS queue in the same region.

  1. Create an S3 bucket.
  2. Create an SQS queue.
    • The queue must be a Standard queue, not a FIFO queue.
  3. Set up notifications on your S3 bucket to write to your SQS queue.
    • Be sure to attach an access policy.
  4. Create an access key which will be used to access both the SQS queue and the S3 bucket.

To learn more about how to configure a feed to ingest data from an Amazon SQS queue whose entries point to files stored in an Amazon S3 bucket, see Ingest AWS logs.

Recommendation

The use of Amazon SQS is preferred over Amazon S3 as the source type. When an Amazon SQS service is used, Google Security Operations reads the Amazon S3 notifications that are sent to the Amazon SQS service and pulls the corresponding files out of the Amazon S3 bucket. This is effectively a push-based version of an Amazon S3 feed and can be used to reduce the latency of ingestion.

Type-specific request fields

Field Required Description
details.amazonSqsSettings.queue Yes The SQS queue name.
details.amazonSqsSettings.region Yes The region where the SQS queue and S3 bucket reside. For a list of regions, see Amazon S3 regions.
details.amazonSqsSettings.accountNumber Yes The account number for the SQS queue and S3 bucket.
details.amazonSqsSettings.sourceDeletionOption Yes Whether to delete the source files in the S3 bucket after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
  • SOURCE_DELETION_NEVER: Never delete files from the source.
  • SOURCE_DELETION_ON_SUCCESS:Delete files and empty directories from the source after successful ingestion.
  • SOURCE_DELETION_ON_SUCCESS_FILES_ONLY:Delete files from the source after successful ingestion.
details.amazonSqsSettings.authentication.sqsAccessKeySecretAuth.accessKeyId Yes This is the 20 character ID associated with your Amazon IAM account.
details.amazonSqsSettings.authentication.sqsAccessKeySecretAuth.secretAccessKey Yes This is the 40 character access key associated with your Amazon IAM account.
details.amazonSqsSettings.authentication.additionalS3AccessKeySecretAuth.accessKeyId No This is the 20 character ID associated with your Amazon IAM account. Only specify if using a different access key for the S3 bucket.
details.amazonSqsSettings.authentication.additionalS3AccessKeySecretAuth.secretAccessKey No This is the 40 character access key associated with your Amazon IAM account. Only specify if using a different access key for the S3 bucket.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "AMAZON_SQS",
   "logType": "LOGTYPE_YOU_WANT_TO_BRING",
   "amazonSqsSettings": {
     "queue": "cs-prod-canon-queue-01234abc56de789f",
     "region": "US_EAST_1",
     "accountNumber": "123456789012",
     "sourceDeletionOption": "SOURCE_DELETION_NEVER"
     "authentication": {
       "sqsAccessKeySecretAuth": {
         "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
         "secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
       },
     },
   }
 }
}

AZURE_BLOBSTORE

Data source Ingest schedule details.feedSourceType details.logType
Microsoft Azure Blob Storage Container Every 15 minutes AZURE_BLOBSTORE See the Feed Schema API reference to get compatible log types.

Prerequisites

You will need either:

To learn more about how to configure a feed to ingest data from Azure Blob Storage, see Ingest Azure Activity logs.

Recommendation

  • If your Azure Blob Storage container contains many small files, it can increase the amount of time it takes to transfer the data. It is recommended to combine smaller files into one large file to accelerate the feed transfer process.

  • It is important to set a data retention policy to your Azure Blob Storage containers so that the transferred files are deleted from the containers and are not listed in any future transfer feeds.

  • As Google SecOps pulls files from containers on a frequent basis, it is recommended to specify a storage class for your containers that is the most cost-effective.

Type-specific request fields

Field Required Description
details.azureBlobStoreSettings.azureUri Yes The URI pointing to a Azure Blob Storage blob or container.
details.azureBlobStoreSettings.sourceType Yes The type of object indicated by the URI. Valid values are:
  • FILES: The URI points to a single blob that will be ingested with each execution of the feed.
  • FOLDERS_RECURSIVE: The URI points to a Blob Storage container.
details.azureBlobStoreSettings.sourceDeletionOption Yes Source file deletion is not supported in Azure. This field's value must be set to SOURCE_DELETION_NEVER.
details.azureBlobStoreSettings.authentication.sharedKey No A shared key, a 512-bit random string in base64 encoding, authorized to access Azure Blob Storage. Required if not specifying an SAS Token.
details.azureBlobStoreSettings.authentication.sasToken No A Shared Access Signature authorized to access the Azure Blob Storage container.

Azure URI source types

When specifying an Azure URI you must also indicate the type of object is indicated by the URI.

details.sourceType Source type
FILES The URI points to a single blob that will be ingested with each execution of the feed.
FOLDERS The URI points to a directory. All files contained within the directory will be ingested with each execution of the feed.
FOLDERS_RECURSIVE The URI points to a Blob Storage container.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "AZURE_BLOBSTORE",
   "logType": "LOGTYPE_YOU_WANT_TO_BRING",
   "azureBlobStoreSettings": {
     "azureUri": "https://myaccount.blob.core.windows.net/logging",
     "sourceType": "FOLDERS_RECURSIVE",
     "sourceDeletionOption": "SOURCE_DELETION_NEVER"
     "authentication": {
       "sharedKey": "Ab12CyDEFG3HI45JklMnopQrs00TU6xVw7xYZ8AbcdeFgHioJkL0MnoPqRsTUvWxYZaBCdEFg9hijKlm0N12pqR==",
     },
   }
 }
}

HTTP

Data source Ingest schedule details.feedSourceType details.logType
Files available over the open internet via an HTTP request. Every 15 minutes HTTP See the Feed Schema API reference to get compatible log types.

WARNING: The HTTP type should not be used to gather data from an API. Refer to the following supported API feed types.

Type-specific request fields

Field Required Description
details.httpSettings.uri Yes The URI pointing to a file or collection of files.
details.httpSettings.sourceType Yes The type of file indicated by the URI. Valid values are:
  • FILES: The URI points to a single file which will be ingested with each execution of the feed.
  • FOLDERS: The URI points to a directory. All files within the directory will be ingested with each execution of the feed.
  • FOLDERS_RECURSIVE: The URI points to a directory. All files and directories within the indicated directory will be ingested, including all files and directories within those directories.
details.httpSettings.sourceDeletionOption Yes Whether to delete source files after they have been transferred to Google Security Operations. This reduces storage costs. Valid values are:
  • SOURCE_DELETION_NEVER: Never delete files from the source.
  • SOURCE_DELETION_ON_SUCCESS:Delete files and empty directories from the source after successful ingestion.
  • SOURCE_DELETION_ON_SUCCESS_FILES_ONLY:Delete files from the source after successful ingestion.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "HTTP",
   "logType": "LOGTYPE_YOU_WANT_TO_BRING",
   "httpSettings": {
     "uri": "https://url.com/myfile",
     "sourceType": "FILES",
     "sourceDeletionOption": "SOURCE_DELETION_NEVER"
   }
 }
}

Webhook

Data source Ingest schedule details.feedSourceType
Data that is streamed to an HTTPS webhook. Based on your implementation, it might take approximately a minute for a new feed to populate because Google Security Operations takes some time to process the data. HTTPS_PUSH_WEBHOOK

Prerequisites

Set up push ingestion using an HTTPS webhook

Data can be sent to Google Security Operations using an HTTPS webhook. You must first create a feed with the appropriate log type before configuring an HTTPS webhook to send data.

To set up HTTPS push ingestion using an HTTPS webhook, do the following:

  1. Create an HTTPS webhook feed using the following create API request:

      {
    "displayName": "FEED_NAME",
    "details": {
      "feedSourceType": "HTTPS_PUSH_WEBHOOK",
      "logType": "projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/logTypes/LOG_TYPE"
    }
  }

    Replace the following:

    • FEED_NAME: specify a name for the feed.
    • PROJECT_NUMBER: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.
    • REGION_ID: the code that Google assigns based on the region. The following are the supported region IDs: Asia-Southeast1, Australia-Southeast1, Europe, EU, Europe-West2, Europe-West3, Europe-West6, Govcloud-US, Me-West1, and US.
    • CUSTOMER_ID: the Google Security Operations customer ID.
    • LOG_TYPE: the type of data the feed ingests. Google Security Operations supports specific log types for the HTTPS webhook feed.

    If the data to be ingested contains a delimiter that separate log lines, such as \\n, include the following in the details field of the feed request body:

      "httpsPushWebhookSettings": {
    "splitDelimiter": "LOG_DELIMITER"
  }

    Replace LOG_DELIMITER with the delimiter that separates the log lines, such as \\n.

  2. After you create a feed, generate a secret key for the feed and generate an API key to authenticate to Google Security Operations.

  3. In your client application, specify the HTTPS endpoint. Here is a sample HTTPS webhook endpoint:

    https://REGIONAL_ENDPOINT/v1alpha/projects/PROJECT_NUMBER/locations/REGION_ID/instances/CUSTOMER_ID/feeds/FEED_ID:importPushLogs

    The endpoint includes the following values:

    • REGIONAL_ENDPOINT: the Google Security Operations regional endpoint, such as us-chronicle.googleapis.com. For information about supported regional endpoints, see Google Security Operations regional endpoints section of this document.
    • PROJECT_NUMBER: an automatically generated unique identifier for your project. For information about project name, project ID, and project number, which are used to identify a project, see Creating and managing projects.
    • REGION_ID: the code that Google assigns based on the region. The following are the supported region IDs: Asia-Southeast1, Australia-Southeast1, Europe, EU, Europe-West2, Europe-West3, Europe-West6, Govcloud-US, Me-West1, and US.
    • CUSTOMER_ID: the Google Security Operations customer ID.
    • FEED_ID: the Google Security Operations feed ID.

  4. Enable authentication by specifying the API key and secret key as part of the custom header in the following format: X-goog-api-key = KEY_VALUE X-Webhook-Access-Key = SECRET_VALUE

    Replace the following:

    • SECRET_VALUE: the secret key value that you generated using the GenerateSecret API. You can also pass the secret as a query parameter in the endpoint URL (?secret=SECRET_VALUE). We recommend that you specify the secret as a header instead of specifying it in the URL.
    • KEY_VALUE: the API key value. You can also pass the API key as a query parameter in the endpoint URL (?key=KEY_VALUE). We recommend that you specify the API key as a header instead of specifying it in the URL.

Google Security Operations regional endpoints

When you construct the HTTPS endpoint URL to push feeds, use the following regional endpoints that Google Security Operations supports:

  • asia-northeast1-chronicle.googleapis.com
  • asia-south1-chronicle.googleapis.com
  • asia-southeast1-chronicle.googleapis.com
  • australia-southeast1-chronicle.googleapis.com
  • eu-chronicle.googleapis.com
  • europe-west2-chronicle.googleapis.com
  • europe-west3-chronicle.googleapis.com
  • europe-west6-chronicle.googleapis.com
  • europe-west12-chronicle.googleapis.com
  • govcloud-us-chronicle.googleapis.com
  • me-central1-chronicle.googleapis.com
  • me-central2-chronicle.googleapis.com
  • me-west1-chronicle.googleapis.com
  • northamerica-northeast2-chronicle.googleapis.com
  • us-chronicle.googleapis.com

Configuration by log type

The following table lists the log types that Google Security Operations supports for the API feed source type (that is, ingesting data from third-party APIs).

If a log type has Google Security Operations parser support, the ingested data is stored in Google Security Operations UDM format as well as raw log data.

Click a Data Source name for detailed reference information, prerequisites, and API examples for the log type.

To learn about prerequisites for other log types and feed source types, see Prerequisites.

Data Source Log Type Google Security Operations Parser Support
Anomali ThreatStream ANOMALI_IOC Yes
CrowdStrike Detection Monitoring CS_DETECTS Yes
Duo Authentication Logs DUO_AUTH Yes
Duo Users DUO_USER_CONTEXT Yes
Fidelis Cloud Passage Events CLOUD_PASSAGE Yes
Fox-IT FOX_IT_STIX No
Google Cloud Identity Devices GCP_CLOUDIDENTITY_DEVICES Yes
Google Cloud Identity Device Users GCP_CLOUDIDENTITY_DEVICEUSERS Yes
Google Workspace Activity WORKSPACE_ACTIVITY Yes
Google Workspace Alerts WORKSPACE_ALERTS Yes
Google Workspace Chrome WORKSPACE_CHROMEOS Yes
Google Workspace Groups WORKSPACE_GROUPS Yes
Google Workspace Mobile WORKSPACE_MOBILE Yes
Google Workspace Privileges WORKSPACE_PRIVILEGES Yes
Google Workspace Users WORKSPACE_USERS Yes
Imperva IMPERVA_WAF Yes
Microsoft Azure AD Directory Audit AZURE_AD_AUDIT Yes
Microsoft Azure AD Context AZURE_AD_CONTEXT Yes
Microsoft Azure AD Sign-Ins AZURE_AD Yes
Microsoft Azure MDM Intune Audit Events AZURE_MDM_INTUNE Yes
Microsoft Graph Security API MICROSOFT_GRAPH_ALERT Yes
Microsoft 365 Management Activity OFFICE_365 Yes
Mimecast Secure Email Gateway MIMECAST_MAIL Yes
Netskope Alerts NETSKOPE_ALERT Yes
Netskope Alerts V2 NETSKOPE_ALERT_V2 Yes
Okta System Log OKTA Yes
Okta Users OKTA_USER_CONTEXT Yes
Palo Alto Networks Autofocus PAN_IOC Yes
Palo Alto Networks Cortex XDR CORTEX_XDR Yes
Palo Alto Networks Prisma Cloud Audit Logs PAN_PRISMA_CLOUD Yes
Proofpoint on Demand PROOFPOINT_ON_DEMAND Yes
Proofpoint TAP PROOFPOINT_MAIL Yes
Qualys VM QUALYS_VM Yes
Qualys Scan QUALYS_SCAN No
Rapid7 InsightVM RAPID7_INSIGHT Yes
Recorded Future RECORDED_FUTURE_IOC Yes
RH-ISAC RH_ISAC_IOC Yes
Salesforce SALESFORCE Yes
SentinelOne Alert SENTINELONE_ALERT Yes
ServiceNow CMDB SERVICENOW_CMDB Yes
Symantec Event Export SYMANTEC_EVENT_EXPORT Yes
Thinkst Canary THINKST_CANARY Yes
ThreatConnect THREATCONNECT_IOC Yes
Workday WORKDAY Yes
Workday Audit Logs WORKDAY_AUDIT No
AWS EC2 Hosts AWS_EC2_HOSTS Yes
AWS EC2 Instances AWS_EC2_INSTANCES Yes
AWS EC2 VPCs AWS_EC2_VPCS Yes
AWS Identity and Access Management AWS_IAM Yes

Anomali ThreatStream

This section provides API reference details for the ANOMALI_IOC log type. For details about the data source, see the Anomali ThreatStream documentation.

Data source Ingest schedule details.feedSourceType details.logType
api.threatstream.com/api/v2/intelligence Every minute API ANOMALI_IOC

Prerequisites

  • Get the values for all required request fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.anomaliSettings.authentication.user Yes Username
details.anomaliSettings.authentication.secret Yes API key

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "ANOMALI_IOC",
   "anomaliSettings": {
     "authentication": {
       "user": "USERNAME",
       "secret": "APIKEY"
     },
   }
 }
}

AWS EC2 Hosts

This section provides API reference details for the AWS_EC2_HOSTS log type.

Prerequisites

  • Get the values for all required request fields.
  • Get the following required permissions:

Type-specific request fields

The following table lists the field values required when creating a feed to collect data for the AWS_EC2_HOSTS log type.

Field Required Description
details.awsEc2HostsSettings.authentication.user Yes This is the 20 character ID associated with your Amazon IAM account.
details.awsEc2HostsSettings.authentication.secret Yes This is the 40 character access key associated with your Amazon IAM account.

Sample Create Feed request

{
  "details": {
      "awsEc2HostsSettings": {
          "authentication": {
              "user": "AccessKeyID",
              "secret": "SecretAccessKey"
          }
      },
      "feedSourceType": "API",
      "logType": "AWS_EC2_HOSTS"
  }
}

AWS EC2 Instances

This section provides API reference details for the AWS_EC2_INSTANCES log type.

Prerequisites

  • Get the values for all required request fields.
  • Get the following required permissions:

Type-specific request fields

The following table lists the field values required when creating a feed to collect data for the AWS_EC2_INSTANCES log type.

Field Required Description
details.awsEc2InstancesSettings.authentication.user Yes This is the 20 character ID associated with your Amazon IAM account.
details.awsEc2InstancesSettings.authentication.secret Yes This is the 40 character access key associated with your Amazon IAM account.

Sample Create Feed request

{
  "details": {
      "awsEc2InstancesSettings": {
          "authentication": {
              "user": "AccessKeyID",
              "secret": "SecretAccessKey"
          }
      },
      "feedSourceType": "API",
      "logType": "AWS_EC2_INSTANCES"
  }
}

AWS EC2 VPCs

This section provides API reference details for the AWS_EC2_VPCS log type.

Prerequisites

  • Get the values for all required request fields.
  • Get the following required permissions:

Type-specific request fields

The following table lists the field values required when creating a feed to collect data for the AWS_EC2_VPCS log type.

Field Required Description
details.awsEc2VpcsSettings.authentication.user Yes This is the 20 character ID associated with your Amazon IAM account.
details.awsEc2VpcsSettings.authentication.secret Yes This is the 40 character access key associated with your Amazon IAM account.

Sample Create Feed request

{
  "details": {
      "awsEc2VpcsSettings": {
          "authentication": {
              "user": "AccessKeyID",
              "secret": "SecretAccessKey"
          }
      },
      "feedSourceType": "API",
      "logType": "AWS_EC2_VPCS"
  }
}

AWS Identity and Access Management

This section provides API reference details for the AWS_IAM log type.

Prerequisites

  • Get the values for all required request fields.
  • Get the following required permissions:

Type-specific request fields

The following table lists the field values required when creating a feed to collect data for the AWS_IAM log type.

Field Required Description
details.awsIamSettings.authentication.user Yes This is the 20 character ID associated with your Amazon IAM account.
details.awsIamSettings.authentication.secret Yes This is the 40 character access key associated with your Amazon IAM account.
details.awsIamSettings.apiType Yes API which needs to be called (Users/Roles/Groups).

Sample Create Feed request

{
  "details": {
      "awsIamSettings": {
          "authentication": {
              "user": "AccessKeyID",
              "secret": "SecretAccessKey"
          },
          "apiType": "USERS"
      },
      "feedSourceType": "API",
      "logType": "AWS_IAM"
  }
}

CrowdStrike Detection Monitoring

This section provides API reference details for the CS_DETECTS log type. For details about the data source, see the CrowdStrike Detection Monitoring documentation.

Data source Ingest schedule details.feedSourceType details.logType
  • api.crowdstrike.com/detects/queries/detects/v1
  • api.crowdstrike.com/detects/entities/summaries/GET/v1
 Every minute API CS_DETECTS

Prerequisites

  • Get the values for all required request fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.crowdstrikeDetectsSettings.authentication.clientId Yes Application ID
details.crowdstrikeDetectsSettings.authentication.clientSecret Yes Client Secret
details.crowdstrikeDetectsSettings.authentication.tokenEndpoint Yes Authentication URL
details.crowdstrikeDetectsSettings.hostname Yes API Endpoint URL

Sample Create Feed request

{
  "details": {
    "feedSourceType": "API",
    "logType": "CS_DETECTS",
    "crowdstrikeDetectsSettings": {
      "authentication": {
          "clientId": "CLIENT ID",
          "clientSecret": "CLIENT SECRET",
          "tokenEndpoint": "https://api.us-2.crowdstrike.com/oauth2/token"
      },
      "hostname": "api.crowdstrike.com"
    }
  }
}

Steps to enable Crowdstrike Feed

Create a CrowdStrike API Client

  1. Within the CrowdStrike application, create an API client by navigating to Support and resources > API clients and keys.

  2. Create a new API Client with API scopes granting permission to Read Detections.

  3. Record the values for: Base URL + Client ID + Client Secret. You will need these to set up the Feed in Google Security Operations.

Setup the Google Security Operations feed
  1. Launch Google Security Operations, then select the Settings menu.
  2. Select Feeds in the left pane and click Add New.
  3. Select the Third Party API Source Type and Crowdstrike Detection Monitoring log type.

  4. Fill in the requested parameters gathered earlier from CrowdStrike and click Submit.

The feed will begin to retrieve all detections from the CrowdStrike instance in chronological order. Detections older than 6 months will be dropped. After the backfill is complete, which can take some time depending on the number of detections in CrowdStrike, the feed checks for new detections every 5 minutes.

Duo Authentication Logs

This section provides API reference details for the DUO_AUTH log type. For details about the data source, see the Duo Authentication Logs documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/admin/v2/logs/authentication

Replace API_HOSTNAME with the fully qualified domain name of the API instance.

 Every 30 minutes API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.duoAuthSettings.authentication.user Yes The username to authenticate to Duo.
details.duoAuthSettings.authentication.secret Yes The secret to authenticate to Duo.
details.duoAuthSettings.hostname Yes The fully qualified domain name for your instance of the API, such as api-myinstance.duosecurity.com.

Test the endpoint

The Duo Admin API provides programmatic access to the administrative functionality of Duo Security's two-factor authentication platform.

To query your Duo account's authentication logs, you need to send a request to the /admin/v2/logs/authentication endpoint.

For details on how to use the API, see the Authentication Logs section in the Duo documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "DUO_AUTH",
     "duoAuthSettings": {
       "authentication": {
         "user": "USERNAME",
         "secret": "SECRET"
       },
       "hostname": "api-mytenant.duosecurity.com"
     }
   }
}

Duo Users

This section provides API reference details for the DUO_USER_CONTEXT log type. For details about the data source, see the Duo Users documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/admin/v1/users

Replace API_HOSTNAME with the fully qualified domain name of the API instance.

 Every 24 hours API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.duoUserContextSettings.authentication.user Yes The username to authenticate to Duo.
details.duoUserContextSettings.authentication.secret Yes The secret to authenticate to Duo.
details.duoUserContextSettings.hostname Yes The fully qualified domain name for your instance of the API, such as api-myinstance.duosecurity.com.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "DUO_USER_CONTEXT",
     "duoUserContextSettings": {
       "authentication": {
         "user": "USERNAME",
         "secret": "SECRET"
       },
       "hostname": "api-mytenant.duosecurity.com"
     }
   }
}

Fidelis Cloud Passage Events

This section provides API reference details for the CLOUD_PASSAGE log type. For details about the data source, see the Cloud Passage Events documentation.

Data source Ingest schedule details.feedSourceType
api.cloudpassage.com/events?event_types Every minute API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.cloudPassageSettings.authentication.user Yes The username that is used for authentication.
details.cloudPassageSettings.authentication.secret Yes The secret that is for authentication.
details.cloudPassageSettings.eventTypes No The type of events to include in the response. If you don't specify any event types, then the following event types are fetched: fim_target_integrity_changed, lids_rule_failed, and sca_rule_failed.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "CLOUD_PASSAGE",
     "cloudPassageSettings": {
       "authentication": {
         "user": "api_key_id",
         "secret": "api_key_secret",
       }
       "eventTypes": [
         "fim_target_integrity_changed",
         "lids_rule_failed",
         "sca_rule_failed"
       ],
     }
   }
}

Fox-IT

This section provides API reference details for the FOX_IT_STIX log type. For details about the data source, see the Fox-IT documentation.

Prerequisites

  • Get the values for all authentication and SSL fields.
  • Get the following required permissions:
    • None

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "FOX_IT_STIX",
     "foxItStixSettings": {
       "authentication": {
         "user": "USERNAME",
         "secret": "SECRET"
       },
       "ssl": {
         "sslCertificate": "<cert>",
         "encodedPrivateKey": "key"
       }
       "pollServiceURI": "https://stix.fox-it.com/services/poll",
       "collection": "mycollection"
     }
   }
}

Google Cloud Identity Devices

This section provides API reference details for the GCP_CLOUDIDENTITY_DEVICES log type. For details about the data source, see the Google Cloud Identity Devices documentation.

Data source Ingest schedule details.feedSourceType
cloudidentity.googleapis.com/v1/devices Every 24 hours API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.googleCloudIdentityDevicesSettings.authentication.tokenEndpoint Yes The endpoint to retrieve the OAuth JSON web token.
details.googleCloudIdentityDevicesSettings.authentication.claims.issuer Yes The JWT claims issuer, which is usually a client ID.
details.googleCloudIdentityDevicesSettings.authentication.claims.subject Yes The JWT claims subject, which is usually an email ID.
details.googleCloudIdentityDevicesSettings.authentication.claims.auidence Yes The JWT claims audience.
details.googleCloudIdentityDevicesSettings.authentication.rsCredentials.privateKey Yes An RSA private key in PEM format.
details.googleCloudIdentityDevicesSettings.apiVersion No The API version to use to fetch device information. The value must be either v1, v1beta1, or vx. If no version is specified, v1 version is used.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "GCP_CLOUDIDENTITY_DEVICES",
   "googleCloudIdentityDevicesSettings": {
     "authentication": {
       "tokenEndPoint": "jwt_token_uri",
       "claims": {
         "issuer": "jwt_client_email",
         "subject": "user_email",
         "audience": "jwt_token_uri"
       },
       "rsCredentials": {
         "private_key": "privatekey"
       }
     },
     "apiVersion": "v1",
   }
 }
}

Google Cloud Identity Device Users

This section provides API reference details for the GCP_CLOUDIDENTITY_DEVICEUSERS log type. For details about the data source, see the Google Cloud Identity Device Users documentation.

Data source Ingest schedule details.feedSourceType
cloudidentity.googleapis.com/v1/devices/-/deviceUsers Every 24 hours API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.googleCloudIdentityDeviceUsersSettings.authentication.tokenEndpoint Yes The endpoint to retrieve the OAuth JSON web token.
details.googleCloudIdentityDeviceUsersSettings.authentication.claims.issuer Yes The JWT claims issuer, which is usually a client ID.
details.googleCloudIdentityDeviceUsersSettings.authentication.claims.subject Yes The JWT claims subject, which is usually an email ID.
details.googleCloudIdentityDeviceUsersSettings.authentication.claims.auidence Yes The JWT claims audience.
details.googleCloudIdentityDeviceUsersSettings.authentication.rsCredentials.privateKey Yes An RSA private key in PEM format.
details.googleCloudIdentityDeviceUsersSettings.apiVersion No The API version to use to fetch device information. The value must be either v1, v1beta1, or vx. If no version is specified, v1 version is used.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "GCP_CLOUDIDENTITY_DEVICEUSERS",
   "googleCloudIdentityDeviceUsersSettings": {
     "authentication": {
       "tokenEndPoint": "jwt_token_uri",
       "claims": {
         "issuer": "jwt_client_email",
         "subject": "user_email",
         "audience": "jwt_token_uri"
       },
       "rsCredentials": {
         "private_key": "privatekey"
       }
     },
   }
 }
}

Google Workspace Activities

This section provides API reference details for the WORKSPACE_ACTIVITY log type. For details about the data source, see the Google Workspace Activities documentation.

Data source Ingest schedule details.feedSourceType details.logType
admin.googleapis.com Every hour API WORKSPACE_ACTIVITY

Prerequisites

In order for Google Security Operations to ingest Google Workspace activities, you must do the following:

  1. Enable the Admin SDK API on your Google Cloud project.
  2. Create a Service Account which will be used to authenticate against the Admin API.
  3. Generate a JSON key for the Service Account.
  4. Create a domain-wide delegation for the Service Account with the following OAuth scope:
  5. Create a Google Workspace user and assign it an admin role which includes the Reports admin privilege, or create a custom role which includes that privilege.
  6. Locate your Google Workspace customer ID.

To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.

Type-specific request fields

Field Required Description
details.workspaceActivitySettings.authentication.tokenEndpoint Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceActivitySettings.authentication.claims.issuer Yes The value of the client_email field in the JSON key for the service account created to access the admin API.
details.workspaceActivitySettings.authentication.claims.subject Yes The email address of the Google Workspace admin user with Reports privilege.
details.workspaceActivitySettings.authentication.claims.audience Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceActivitySettings.authentication.rsCredentials.privateKey Yes The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n) should be replaced with carriage returns. Also note that the field name is rsCredentials, and not rsaCredentials.
details.workspaceActivitySettings.workspaceCustomerId Yes The Google Workspace customer ID. The customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C', then prepend what you have with a 'C'.
details.workspaceActivitySettings.applications Yes The Google Workspace applications to gather activities for. See the following table for valid values.

Google Workspace applications

Activities are associated with one or more applications. The applications that Google Security Operations supports includes the following.

details.workspaceActivitySettings.applications Description
access_transparency Access Transparency log events
admin Admin log events
calendar Calendar log events
chat Chat log events
drive Drive log events
gcp Google Cloud activity events
gplus Currents log events
groups Groups log events
groups_enterprise Groups Enterprise log events
jamboard Jamboard log events
login User log events
meet Meet log events
mobile Device log events
rules Rule log events (beta)
saml SAML log events
token OAuth log events
user_accounts User log events
context_aware_access Context-Aware Access log events
chrome Chrome log events
data_studio Looker Studio log events
keep Keep log events

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "WORKSPACE_ACTIVITY",
   "workspaceActivitySettings": {
     "authentication": {
       "tokenEndpoint": "https://oauth2.googleapis.com/token",
       "claims": {
         "issuer": "[email protected]",
         "subject": "[email protected]",
         "audience": "https://oauth2.googleapis.com/token"
       },
       "rsCredentials": {
         "privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
         -----END PRIVATE KEY-----"
       },
     },
     "workspaceCustomerId": "C1e2x3ample",
     "applications": [
       "admin",
       "groups",
       "mobile"
     ],
   }
 }
}

Google Workspace Alerts

This section provides API reference details for the WORKSPACE_ALERTS log type. For details about the data source, see the Google Workspace Alerts documentation.

Data source Ingest schedule details.feedSourceType details.logType
alertcenter.googleapis.com Every hour API WORKSPACE_ALERTS

Prerequisites

For Google Security Operations to ingest Google Workspace alerts, complete the following steps:

  1. Enable the Alert Center API on your Google Cloud project.
  2. Create a Service Account which will be used to authenticate against the Alert Center API.
  3. Generate a JSON key for the Service Account.
  4. Create a domain-wide delegation for the Service Account with the following OAuth scope:
  5. Create a Google Workspace user and assign it an admin role which includes Alert Center view access, or create a custom role which includes that privilege.
  6. Locate your Google Workspace customer ID.

To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.

Type-specific request fields

Field Required Description
details.workspaceAlertsSettings.authentication.tokenEndpoint Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceAlertsSettings.authentication.claims.issuer Yes The value of the client_email field in the JSON key for the service account created to access the admin API.
details.workspaceAlertsSettings.authentication.claims.subject Yes The email address of the Google Workspace admin user with Alert Center view access.
details.workspaceAlertsSettings.authentication.claims.audience Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceAlertsSettings.authentication.rsCredentials.privateKey Yes The value of the private_key field in the JSON key for the service account created to access the Alert Center API. Note that literal newline characters (\n) should be replaced with carriage returns. Also note that the field name is rsCredentials, and not rsaCredentials.
details.workspaceAlertsSettings.workspaceCustomerId Yes The Google Workspace customer ID. Note that the customer ID must not have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have has a leading 'C', then remove it before including in your request.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "WORKSPACE_ALERTS",
   "workspaceAlertsSettings": {
     "authentication": {
       "tokenEndpoint": "https://oauth2.googleapis.com/token",
       "claims": {
         "issuer": "[email protected]",
         "subject": "[email protected]",
         "audience": "https://oauth2.googleapis.com/token"
       },
       "rsCredentials": {
         "privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
         -----END PRIVATE KEY-----"
       },
     },
     "workspaceCustomerId": "1e2x3ample",
   }
 }
}

Google Workspace ChromeOS Devices

This section provides API reference details for the WORKSPACE_CHROMEOS log type. For details about the data source, see the Google Workspace ChromeOS Devices documentation.

Data source Ingest schedule details.feedSourceType details.logType
admin.googleapis.com Every 24 hours API WORKSPACE_CHROMEOS

Prerequisites

For Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:

  1. Enable the Admin SDK API on your Google Cloud project.
  2. Create a Service Account which will be used to authenticate against the Admin API.
  3. Generate a JSON key for the Service Account.
  4. Create a domain-wide delegation for the Service Account with the following OAuth scope:
  5. Create a Google Workspace user and assign it an admin role which includes Chrome Management Settings access, or create a custom role which includes that privilege.
  6. Locate your Google Workspace customer ID.

To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.

Type-specific request fields

Field Required Description
details.workspaceChromeOsSettings.authentication.tokenEndpoint Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceChromeOsSettings.authentication.claims.issuer Yes The value of the client_email field in the JSON key for the service account created to access the admin API.
details.workspaceChromeOsSettings.authentication.claims.subject Yes The email address of the Google Workspace admin user with Reports privilege.
details.workspaceChromeOsSettings.authentication.claims.audience Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceChromeOsSettings.authentication.rsCredentials.privateKey Yes The value of the private_key field in the JSON key for the service account created to access the admin API. Replace the literal newline characters (\n) with carriage returns. The field name is rsCredentials, and not rsaCredentials.
details.workspaceChromeOsSettings.workspaceCustomerId Yes The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C', then prepend what you have with a 'C'.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "WORKSPACE_CHROMEOS",
   "workspaceChromeOsSettings": {
     "authentication": {
       "tokenEndpoint": "https://oauth2.googleapis.com/token",
       "claims": {
         "issuer": "[email protected]",
         "subject": "[email protected]",
         "audience": "https://oauth2.googleapis.com/token"
       },
       "rsCredentials": {
         "privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
         -----END PRIVATE KEY-----"
       },
     },
     "workspaceCustomerId": "C1e2x3ample",
   }
 }
}

Google Workspace Groups

This section provides API reference details for the WORKSPACE_GROUPS log type. For details about the data source, see the Google Workspace Groups documentation.

Data source Ingest schedule details.feedSourceType details.logType
admin.googleapis.com Every 24 hours API WORKSPACE_GROUPS

Prerequisites

For Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:

  1. Enable the Admin SDK API on your Google Cloud project.
  2. Create a Service Account which will be used to authenticate against the Admin API.
  3. Generate a JSON key for the Service Account.
  4. Create a domain-wide delegation for the Service Account with the following OAuth scope:
  5. Create a Google Workspace user and assign it an admin role which includes Admin API Group read privileges, or create a custom role which includes that privilege.
  6. Locate your Google Workspace customer ID.

To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.

Type-specific request fields

Field Required Description
details.workspaceGroupsSettings.authentication.tokenEndpoint Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceGroupsSettings.authentication.claims.issuer Yes The value of the client_email field in the JSON key for the service account created to access the admin API.
details.workspaceGroupsSettings.authentication.claims.subject Yes The email address of the Google Workspace admin user with the Admin API Group read privilege.
details.workspaceGroupsSettings.authentication.claims.audience Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceGroupsSettings.authentication.rsCredentials.privateKey Yes The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n) should be replaced with carriage returns. Note that the field name is rsCredentials, and not rsaCredentials.
details.workspaceGroupsSettings.workspaceCustomerId Yes The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "WORKSPACE_GROUPS",
   "workspaceGroupsSettings": {
     "authentication": {
       "tokenEndpoint": "https://oauth2.googleapis.com/token",
       "claims": {
         "issuer": "[email protected]",
         "subject": "[email protected]",
         "audience": "https://oauth2.googleapis.com/token"
       },
       "rsCredentials": {
         "privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
         -----END PRIVATE KEY-----"
       },
     },
     "workspaceCustomerId": "C1e2x3ample",
   }
 }
}

Google Workspace Mobile Devices

This section provides API reference details for the WORKSPACE_MOBILE log type. For details about the data source, see the Google Workspace Mobile Devices documentation.

Data source Ingest schedule details.feedSourceType details.logType
admin.googleapis.com Every 24 hours API WORKSPACE_GROUPS

Prerequisites

In order for Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:

  1. Enable the Admin SDK API on your Google Cloud project.
  2. Create a Service Account which will be used to authenticate against the Admin API.
  3. Generate a JSON key for the Service Account.
  4. Create a domain-wide delegation for the Service Account with the following OAuth scope:
  5. Create a Google Workspace user and assign it an admin role which includes Admin API Group read privileges, or create a custom role which includes that privilege.
  6. Locate your Google Workspace customer ID.

To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.

Type-specific request fields

Field Required Description
details.workspaceGroupsSettings.authentication.tokenEndpoint Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceGroupsSettings.authentication.claims.issuer Yes The value of the client_email field in the JSON key for the service account created to access the admin API.
details.workspaceGroupsSettings.authentication.claims.subject Yes The email address of the Google Workspace admin user with the Admin API Group read privilege.
details.workspaceGroupsSettings.authentication.claims.audience Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceGroupsSettings.authentication.rsCredentials.privateKey Yes The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n) should be replaced with carriage returns. Note that the field name is rsCredentials, and not rsaCredentials.
details.workspaceGroupsSettings.workspaceCustomerId Yes The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "WORKSPACE_MOBILE",
   "workspaceMobileSettings": {
     "authentication": {
       "tokenEndpoint": "https://oauth2.googleapis.com/token",
       "claims": {
         "issuer": "[email protected]",
         "subject": "[email protected]",
         "audience": "https://oauth2.googleapis.com/token"
       },
       "rsCredentials": {
         "privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
         -----END PRIVATE KEY-----"
       },
     },
     "workspaceCustomerId": "C1e2x3ample",
   }
 }
}

Google Workspace Privileges

This section provides API reference details for the WORKSPACE_PRIVILEGES log type. For details about the data source, see the Google Workspace Privileges documentation.

Data source Ingest schedule details.feedSourceType details.logType
admin.googleapis.com Every 24 hours API WORKSPACE_PRIVILEGES

Prerequisites

In order for Google Security Operations to ingest Google Workspace privileges, complete the following steps:

  1. Enable the Admin SDK API on your Google Cloud project.
  2. Create a Service Account which will be used to authenticate against the Admin API.
  3. Generate a JSON key for the Service Account.
  4. Create a domain-wide delegation for the Service Account with the following OAuth scope:
  5. Create a Google Workspace user and assign it a super admin role.
  6. Locate your Google Workspace customer ID.

To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.

Type-specific request fields

Field Required Description
details.workspacePrivilegesSettings.authentication.tokenEndpoint Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspacePrivilegesSettings.authentication.claims.issuer Yes The value of the client_email field in the JSON key for the service account created to access the admin API.
details.workspacePrivilegesSettings.authentication.claims.subject Yes The email address of the Google Workspace admin user.
details.workspacePrivilegesSettings.authentication.claims.audience Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspacePrivilegesSettings.authentication.rsCredentials.privateKey Yes The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n) should be replaced with carriage returns. Note that the field name is rsCredentials, and not rsaCredentials.
details.workspacePrivilegesSettings.workspaceCustomerId Yes The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "WORKSPACE_PRIVILEGES",
   "workspacePrivilegesSettings": {
     "authentication": {
       "tokenEndpoint": "https://oauth2.googleapis.com/token",
       "claims": {
         "issuer": "[email protected]",
         "subject": "[email protected]",
         "audience": "https://oauth2.googleapis.com/token"
       },
       "rsCredentials": {
         "privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
         -----END PRIVATE KEY-----"
       },
     },
     "workspaceCustomerId": "C1e2x3ample",
   }
 }
}

Google Workspace Users

This section provides API reference details for the WORKSPACE_USERS log type. For details about the data source, see the Google Workspace Users documentation.

Data source Ingest schedule details.feedSourceType details.logType
admin.googleapis.com Every 24 hours API WORKSPACE_USERS

Prerequisites

For Google Security Operations to ingest Google Workspace ChromeOS devices, complete the following steps:

  1. Enable the Admin SDK API on your Google Cloud project.
  2. Create a Service Account which will be used to authenticate against the Admin API.
  3. Generate a JSON key for the Service Account.
  4. Create a domain-wide delegation for the Service Account with the following OAuth scope:
  5. Create a Google Workspace user and assign it an admin role which includes Admin API User read privileges, or create a custom role which includes that privilege.
  6. Locate your Google Workspace customer ID.

To learn more about how to configure a feed to ingest Google Workspace logs, see Collect Google Workspace logs.

Type-specific request fields

Field Required Description
details.workspaceUserSettings.authentication.tokenEndpoint Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceUserSettings.authentication.claims.issuer Yes The value of the client_email field in the JSON key for the service account created to access the admin API.
details.workspaceUserSettings.authentication.claims.subject Yes The email address of the Google Workspace admin user with the Admin API User read privilege.
details.workspaceUserSettings.authentication.claims.audience Yes The value of the token_uri field in the JSON key for the service account created to access the admin API.
details.workspaceUserSettings.authentication.rsCredentials.privateKey Yes The value of the private_key field in the JSON key for the service account created to access the admin API. Note that literal newline characters (\n) should be replaced with carriage returns. Note that the field name is rsCredentials, and not rsaCredentials.
details.workspaceUserSettings.workspaceCustomerId Yes The Google Workspace customer ID. Note that the customer ID must have a leading 'C' character. The customer ID may appear differently depending on where in the Google admin console it is found. If the customer ID you have does not have a leading 'C' then prepend what you have with a 'C'.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "WORKSPACE_USERS",
   "workspaceUserSettings": {
     "authentication": {
       "tokenEndpoint": "https://oauth2.googleapis.com/token",
       "claims": {
         "issuer": "[email protected]",
         "subject": "[email protected]",
         "audience": "https://oauth2.googleapis.com/token"
       },
       "rsCredentials": {
         "privateKey": "-----BEGIN PRIVATE KEY-----
ABCDeFGHIJKLMnopqrsT0u1VWXY...z/abCdefgHIJK+lMN2o345P=
         -----END PRIVATE KEY-----"
       },
     },
     "workspaceCustomerId": "C1e2x3ample",
   }
 }
}

Imperva

This section provides API reference details for the IMPERVA_WAF log type. For details about the data source, see the Imperva documentation.

Data source Ingest schedule details.feedSourceType
api.imperva.com/audit-trail/v2/events Every 24 hours API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.impervaWafSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate api.imperva.com in key-value format.

Optional fields

initialStartTime

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "IMPERVA_WAF",
     "impervaWafSettings": {
       "authentication": {
         "headerKeyValues": [{
            "key": "key"
            "value": "value"
         }],
       }
     }
   }
}

Microsoft Azure Active Directory Audit

This section provides API reference details for the AZURE_AD_AUDIT log type. For details about the data source, see the Azure Active Directory Audit documentation.

Data source Ingest schedule details.feedSourceType details.logType
graph.microsoft.com Every minute API AZURE_AD_AUDIT

Prerequisites

  • Get an Azure AD Premium P1 or P2 license. For more information, see What licenses do I need.
  • Get the values for all required request fields. Note that the token endpoint for OAuth 2.0 is: https://login.microsoftonline.com/{tenantId}/oauth2/token
  • Get the following required permissions:
    • The user whose credentials are used to authenticate against the Microsoft Graph API to access directory audits must have the permissions AuditLog.Read.All and Directory.Read.All.

Type-specific request fields

Field Required Description
details.azureAdAuditSettings.authentication.clientId Yes Application ID (a UUID)
details.azureAdAuditSettings.authentication.clientSecret Yes Client Secret
details.azureAdAuditSettings.tenantId Yes Tenant ID (a UUID)
details.azureAdAuditSettings.hostname No API Full Path, default value : "graph.microsoft.com/v1.0/auditLogs/directoryAudits"

Test the API endpoint by using curl

Before you create the feed, you can test the Microsoft Graph API endpoint by using curl.

  1. Request an OAuth token to authenticate your request to the API resource. 
    curl 'https://login.microsoftonline.com/TENANT_ID/oauth2/token' \
    --data-urlencode 'grant_type=client_credentials' \
    --data-urlencode 'client_id=CLIENT_ID' \
    --data-urlencode 'client_secret=CLIENT_SECRET' \
    --data-urlencode 'resource=https://graph.microsoft.com'

    Replace the following:

    • CLIENT_ID: Application ID
    • CLIENT_SECRET: Client secret
    • TENANT_ID: Tenant ID

    The result of the curl request is a JSON response that contains the OAuth access token.

  2. Send a request to the Microsoft Graph API endpoint using the OAuth token. 
    curl 'https://graph.microsoft.com/v1.0/auditLogs/signIns' \
    --header 'Accept: application/json' \
    --header 'Authorization: Bearer ACCESS_TOKEN'

    Replace ACCESS_TOKEN with the value of the OAuth access token that you obtained from the previous step.

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "AZURE_AD_AUDIT",
   "azureAdAuditSettings": {
     "authentication": {
       "clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
       "clientSecret": "clientSecret",
     }
     "tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
     "hostname": "graph.microsoft.com/v1.0/auditLogs/directoryAudits",
   }
 }
}

Microsoft Azure Active Directory Organizational Context

This section provides API reference details for the AZURE_AD_CONTEXT log type. For details about the data source, see the Microsoft Graph API List users endpoint, which this feed uses to retrieve device and group data.

Data source Ingest schedule details.feedSourceType details.logType
graph.microsoft.com Every 24 hours API AZURE_AD_CONTEXT

Prerequisites

  • Get the values for all required request fields. The token endpoint for OAuth 2.0 is https://login.microsoftonline.com/{tenantId}/oauth2/token
  • Get the following required permissions:
    • The user whose credentials are used to authenticate against Microsoft Graph API to access organizational context must have permissions Directory.Read.All.

Type-specific request fields

Field Required Description
details.azureAdContextSettings.authentication.clientId Yes Application ID (a UUID)
details.azureAdContextSettings.authentication.clientSecret Yes Client secret
details.azureAdContextSettings.tenantId Yes Tenant ID (a UUID)
details.azureAdContextSettings.retrieveDevices No Whether to retrieve device information
details.azureAdContextSettings.retrieveGroups No Whether to retrieve user group information
details.azureAdContextSettings.hostname No API Full Path, default value : graph.microsoft.com/beta

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "AZURE_AD_CONTEXT",
   "azureAdContextSettings": {
     "authentication": {
       "clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
       "clientSecret": "clientSecret",
     }
     "tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
     "retrieveDevices": false,
     "retrieveGroups": false,
     "hostname": "graph.microsoft.com/beta",
   }
 }
}

Microsoft Azure Active Directory Sign-ins

This section provides API reference details for the AZURE_AD log type. For details about the data source, see the Azure Active Directory Sign-ins documentation.

Data source Ingest schedule details.feedSourceType details.logType
graph.microsoft.com Every minute API AZURE_AD

Prerequisites

  • Get an Azure AD Premium P1 or P2 license. For more information, see What licenses do I need.
  • Get the values for all required request fields. The token endpoint for OAuth 2.0 is https://login.microsoftonline.com/{tenantId}/oauth2/token
  • Get the following required permissions:
    • The user whose credentials are used to authenticate against Microsoft Graph API to access sign-ins must have permissions AuditLog.Read.All and Directory.Read.All.

Type-specific request fields

Field Required Description
details.azureAdSettings.authentication.clientId Yes Application ID (a UUID)
details.azureAdSettings.authentication.clientSecret Yes Client Secret
details.azureAdSettings.tenantId Yes Tenant ID (a UUID)
details.azureAdSettings.hostname No API Full Path, default value : graph.microsoft.com/v1.0/auditLogs/signIns

Sample Create Feed request

{
 "details": {
   "feedSourceType": "API",
   "logType": "AZURE_AD",
   "azureAdSettings": {
     "authentication": {
       "clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
       "clientSecret": "clientSecret",
     }
     "tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
     "hostname": "graph.microsoft.com/v1.0/auditLogs/signIns",
   }
 }
}

Microsoft Azure Microsoft Device Management Intune Audit Events

This section provides API reference details for the AZURE_MDM_INTUNE log type. For details about the data source, see the Azure Microsoft Device Management Intune Audit Events documentation.

Data source Ingest schedule details.feedSourceType
The Microsoft Graph REST API endpoint URL. The default value is graph.microsoft.com/beta/deviceManagement/auditEvents Every minute API

Prerequisites

  • Get an active Intune license.
  • Get the values for all authentication fields. The token endpoint for OAuth 2.0 is https://login.microsoftonline.com/{tenantId}/oauth2/token
  • Get the following required permission:
    • The provisioned OAuth client must have permission DeviceManagementApps.Read.All or DeviceManagementApps.ReadWrite.All.

Type-specific request fields

Field Required Description
details.azureMdmIntuneSettings.authentication.clientId Yes The application ID.
details.azureMdmIntuneSettings.authentication.clientSecret Yes The client secret.
details.azureMdmIntuneSettings.tenantId Yes The tenant ID, which is a UUID.
details.azureMdmIntuneSettings.hostname No The Microsoft Graph REST API endpoint URL. The following is the default value: graph.microsoft.com/beta/deviceManagement/auditEvents.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "AZURE_MDM_INTUNE",
     "azureMdmIntuneSettings": {
       "authentication": {
         "clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
         "clientSecret": "clientSecret",
       }
       "tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
       "hostname": "graph.microsoft.com/beta/deviceManagement/auditEvents",
     }
   }
}

Microsoft Graph Security API Alerts

This section provides API reference details for the MICROSOFT_GRAPH_ALERT log type. For details about the data source, see Microsoft Graph Security Legacy List alerts and List alerts_v2.

Data source Ingest schedule details.feedSourceType
  • graph.microsoft.com/v1.0/security/alerts
  • graph.microsoft.com/v1.0/security/alerts_v2
 Every minute API

Prerequisites

  • Get the values for authentication fields. The token endpoint for OAuth 2.0 is https://login.microsoftonline.com/{tenantId}/oauth2/token
  • Get the following required permissions; the API supports two data sources:
    • graph.microsoft.com/v1.0/security/alerts requires SecurityEvents.Read.All permissions
    • graph.microsoft.com/beta/security/alerts_v2 or graph.microsoft.com/v1.0/security/alerts_v2 requires SecurityAlert.Read.All permissions
    • The user whose credentials are used must have permissions SecurityEvents.Read.All.

Type-specific request fields

Field Required Description
details.microsoftGraphAlertSettings.authentication.clientId Yes Application ID (a UUID)
details.microsoftGraphAlertSettings.authentication.clientSecret Yes Client secret
details.microsoftGraphAlertSettings.tenantId Yes Tenant ID (a UUID)
details.microsoftGraphAlertSettings.authEndpoint Yes The Microsoft Active Directory authentication endpoint. The default value is login.microsoftonline.com.
details.microsoftGraphAlertSettings.hostname No The API full path. The default value is graph.microsoft.com/v1.0/security/alerts

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "MICROSOFT_GRAPH_ALERT",
     "microsoftGraphAlertSettings": {
       "authentication": {
         "clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
         "clientSecret": "clientSecret",
       }
       "tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789",
       "hostname": "graph.microsoft.com/v1.0/security/alerts",
       "authEndpoint": "login.microsoftonline.com",
     }
   }
}

Microsoft Office 365 Management Activity

This section provides API reference details for the OFFICE_365 log type. For details about the data source, see the Microsoft Office 365 Management Activity documentation.

Data source Ingest schedule details.feedSourceType details.logType
  • manage.office.com
  • manage-gcc.office.com
 Every minute API OFFICE_365

Prerequisites

  • Get the values for all required request fields. The token endpoint for OAuth 2.0 is https://login.microsoftonline.com/{tenantId}/oauth2/token
  • Get the following required permissions:
    • The user whose credentials are used to authenticate against the API must have permissions ActivityFeed.Read. If ingesting DLP data then the permission ActivityFeed.ReadDlp must be specified.

To learn more about how to configure a feed to ingest Microsoft Office 365 logs, see Collect Microsoft 365 logs.

Type-specific request fields

Field Required Description
details.office365Settings.authentication.clientId Yes Application ID (a UUID)
details.office365Settings.authentication.clientSecret Yes Client secret
details.office365Settings.tenantId Yes Tenant ID (a UUID)
details.office365Settings.contentType Yes The type of logs to fetch. See below to see the valid values for contentType.
details.office365Settings.hostname No API Full Path, default value: manage.office.com/api/v1.0

Office 365 Content Type

This section provides API reference details for the OFFICE_365 log type. For details about the data source, see the Office 365 Content Type documentation.

details.office365Settings.contentType Description
AUDIT_AZURE_ACTIVE_DIRECTORY Azure active directory audit logs.
AUDIT_EXCHANGE Azure exchange audit logs.
AUDIT_SHARE_POINT Azure share point audit logs.
AUDIT_GENERAL All other workloads not included in other Audit content types.
DLP_ALL DLP events only for all workloads.

Test the API endpoint by using curl

Before you create the feed, you can test the Office 365 Management Activity API endpoint by using curl.

  1. Request an OAuth token to authenticate your request to the API resource. 
    curl 'https://login.microsoftonline.com/TENANT_ID/oauth2/token' \
    --data-urlencode 'grant_type=client_credentials' \
    --data-urlencode 'client_id=CLIENT_ID' \
    --data-urlencode 'client_secret=CLIENT_SECRET' \
    --data-urlencode 'resource=https://manage.office.com'

    Replace the following:

    • CLIENT_ID: Application ID
    • CLIENT_SECRET: Client secret
    • TENANT_ID: Tenant ID

    The result of the curl request is a JSON response that contains the OAuth access token.

  2. Send a request to the Office 365 Management Activity API using the OAuth token. 
    curl 'https://manage.office.com/api/v1.0/TENANT_ID/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory' \
    --header 'Authorization: Bearer ACCESS_TOKEN'

    Replace ACCESS_TOKEN with the value of the OAuth access token that you obtained from the previous step.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "OFFICE_365",
     "office365Settings": {
       "authentication": {
         "clientId": "0ab12c34-d5ef-678g-9012-hi34j56k78l9",
         "clientSecret", "clientSecret",
       },
       "tenantId": "0ab123c4-de56-78fg-90h1-ijk2l3456789"",
       "contentType": "AUDIT_AZURE_ACTIVE_DIRECTORY",
       "hostname": "manage.office.com/api/v1.0",
     }
   }
}

Mimecast

This section provides API reference details for the MIMECAST_MAIL log type. For details about the data source, see the Mimecast documentation.

Data source Ingest schedule details.feedSourceType
The fully qualified domain name of your Mimecast API endpoint, such as us-api.mimecast.com. Every minute API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.mimecastMailSettings.authentication.headerKeyValues Yes The configuration in the key-value format that is used to construct the authentication header.
details.mimecastMailSettings.hostname Yes The fully qualified domain name of your Mimecast API endpoint, such as us-api.mimecast.com.

Test the endpoint

The API endpoint used to download Mimecast MTA logs is /api/audit/get-siem-logs. To use this endpoint, send a POST request to /api/audit/get-siem-logs.

For details on how to use the API, see the sample code in the Mimecast documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "MIMECAST_MAIL",
     "mimecastMailSettings": {
       "authentication": {
         "headerKeyValues": [
           {
             "key": "access_key",
             "value": "ACCESS_KEY"
           },
           {
             "key": "app_id",
             "value": "APP_ID"
           },
           {
             "key": "app_key",
             "value": "APP_KEY"
           },
           {
             "key": "secret_key",
             "value": "SECRET_KEY"
           }
         ]
       },
       "hostname": "xx-api.mimecast.com"
     }
   }
}

Netskope Alerts V1

This section provides API reference details for the NETSKOPE_ALERT log type. For details about the data source, see the Netskope Alerts documentation. Netskope REST API v1 data is supported.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/api/v1/alerts

Replace API_HOSTNAME with the fully qualified domain name of your Netskope REST API endpoint, such as myinstance.goskope.com.

 Every 10 minutes API

Prerequisites

  • Get the values for all authentication fields. Use auth tokens for the Netskope REST API v1.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.netskopeAlertSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate Netskope in key-value format.
details.netskopeAlertSettings.hostname Yes The fully qualified domain name of your Netskope REST API endpoint.
details.netskopeAlertSettings.feedname Yes The REST endpoint to connect to. This can be alerts or events.
details.netskopeAlertSettings.contentType Yes The value of the type query parameter that determines which type of event or alert is acquired.

Optional fields

initialStartTime

Test the API endpoint

Before you create the feed, you can test the Netskope alerts API endpoint by sending a POST request to https://TENANT_URL/api/v1/alerts. This endpoint returns alerts generated by Netskope.

The following is an example request using curl:

curl -X POST 'https://TENANT_URL/api/v1/alerts?' \
    -H 'Content-Type: application/json' \
    -d 'timeperiod=86400' \
    -d 'type=Security%20Assessment' \
    -d 'limit=1' \
    -d 'stimeperiod=2592000' \
    -d 'query=%28compliance_standards.standard%20eq%20%27CSA-CCM-3.0.1%27%29' \
    -d 'token=ACCESS_TOKEN'

Replace the following:

  • TENANT_URL: URL of your tenant
  • ACCESS_TOKEN: OAuth access token

To learn more about the different query parameters that can be used as a part of the request, see the Get Alerts Data page in the Netskope documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "NETSKOPE_ALERT",
     "netskopeAlertSettings": {
       "authentication": {
         "headerKeyValues": [{
          "key": "token",
          "value": secret
         }]
       },
       "hostname": hostname,
       "feedname": feedname,
       "contentType": contenttype
     }
   },
   "display_name": displayname
}

Netskope Alerts V2

This section provides API reference details for the NETSKOPE_ALERT_V2 log type. For details about the data source, see the Netskope Alerts V2 documentation. Netskope REST API v2 data is supported.

Data source Content Type Content Category
API_HOSTNAME/api/v2/events/dataexport/alerts/uba uba alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/securityassessment securityassessment alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/quarantine quarantine alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/remediation remediation alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/policy policy alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/malware malware alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/malsite malsite alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/compromisedcredential compromisedcredential alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/ctep ctep alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/dlp dlp alerts
API_HOSTNAME/api/v2/events/dataexport/alerts/watchlist watchlist alerts
API_HOSTNAME/api/v2/events/dataexport/events/application application events
API_HOSTNAME/api/v2/events/dataexport/events/audit audit events
API_HOSTNAME/api/v2/events/dataexport/events/connection connection events
API_HOSTNAME/api/v2/events/dataexport/events/incident incident events
API_HOSTNAME/api/v2/events/dataexport/events/infrastructure infrastructure events
API_HOSTNAME/api/v2/events/dataexport/events/network network events
API_HOSTNAME/api/v2/events/dataexport/events/page page events
  • Ingest schedule = Every 10 mins
  • details.feedSourceType = API

Replace API_HOSTNAME with the fully qualified domain name of your Netskope REST API v2 endpoint, such as myinstance.goskope.com.

Prerequisites

  • Get the values for all authentication fields. Use auth tokens for the Netskope REST API v2.
  • Create a Netskope access token following the steps on the REST API v2 Overview page. Note, when creating the Netskope token make sure to select all the relevant endpoint privileges.

Type-specific request fields

Field Required Description
details.netskopeAlertV2Settings.authentication.headerKeyValues Yes The HTTP header used to authenticate Netskope in key-value format.
details.netskopeAlertV2Settings.hostname Yes The fully qualified domain name of your Netskope REST API endpoint.
details.netskopeAlertV2Settings.contentCategory Yes The REST endpoint to connect to. This can be alerts or events.
details.netskopeAlertV2Settings.contentTypes Yes The type of event or alert. Allowed values for alerts are uba, securityassessment, quarantine, remediation, policy, malware, malsite, compromisedcredential, ctep, dlp and watchlist. Allowed values for events are application, audit, connection, incident, infrastructure, network and page.

Test the API endpoint

Before you create the feed, you can test the Netskope alerts V2 API endpoint by sending a GET request to https://TENANT_URL. This endpoint returns alerts generated by Netskope.

The following is an example request using curl:

curl -X 'GET' \
    'https://TENANT_URL' \
    -H 'accept: application/json' \
    -H 'Netskope-Api-Token: ACCESS_TOKEN'

Replace the following:

  • TENANT_URL: URL of one of the Data sources listed in the Data source table.
  • ACCESS_TOKEN: OAuth access token (See Prerequisites for details of creating the token.)

To learn more about the different query parameters that can be used as a part of the request, see the Get Alerts Data page in the Netskope documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "NETSKOPE_ALERT_V2",
     "netskopeAlertV2Settings": {
       "authentication": {
         "headerKeyValues": [{
          "key": "Netskope-Api-Token",
          "value": "token_value"
         }]
       },
       "contentTypes": [
          "uba",
          "securityassessment"
       ],
       "hostname": "myinstance.goskope.com",
       "contentCategory": "alerts"
     }
   }
}

Okta System Log

This section provides API reference details for the OKTA log type. For details about the data source, see the Okta System Log documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/api/v1/logs

Replace API_HOSTNAME with the fully qualified domain name of your Okta instance, such as example.okta.com.

 Every minute API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.oktaUserContextSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate Okta in key-value format.
details.oktaUserContextSettings.hostname Yes The fully qualified domain name of your Okta instance.

Test the API endpoint

Before you create the feed, you can test the Okta System Log API endpoint by sending a GET request to OKTA_URL/api/v1/logs. This endpoint returns system log events that can be ingested into a SIEM platform.

The following is an example request to obtain system log events from a particular point of time in the past:

curl -v -X GET \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -H "Authorization: SSWS API_TOKEN" \
    "https://OKTA_URL/api/v1/logs?since=DATETIME"

Replace the following:

  • API_TOKEN: OAuth access token
  • OKTA_URL: fully qualified domain name of your Okta instance, such as example.okta.com
  • DATETIME: timestamp in UTC format according to ISO 8601, separating date and time with a T. For example: 2024-01-31T00:00:00Z. The API will fetch the logs recorded after the specified timestamp.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "OKTA",
     "oktaSettings": {
       "authentication": {
         "headerKeyValues": [{
            "key": "Authorization",
            "value": "APITOKEN"
          }]
       },
       "hostname": "hostname"
     }
   }
}

Okta Users

This section provides API reference details for the OKTA_USER_CONTEXT log type. For details about the data source, see the Okta Users documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/api/v1/logs

Replace API_HOSTNAME with the fully qualified domain name of your Okta instance, such as example.okta.com.

 Every minute API

Prerequisites

  • Get the values for hostname and all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.oktaUserContextSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate Okta in key-value format.
details.oktaUserContextSettings.hostname Yes The fully qualified domain name of your Okta instance.
details.oktaUserContextSettings.managerIdReferenceField No This ID is required when you use a non Okta ID to reference managers.

Sample Create Feed request

managerIdReferenceField is required when you use a non-Okta ID to reference managers. It should be a JSON field path pointing to the field that contains the manager ID in the result of a call to the "users" Okta API.

{
   "details": {
     "feedSourceType": "API",
     "logType": "OKTA_USER_CONTEXT",
     "oktaSettings": {
       "authentication": {
         "headerKeyValues": [{
            "key": "Authorization",
            "value": "APITOKEN"
          }]
       },
       "hostname": "hostname",
       "managerIdReferenceField": "fooId"
     }
   }
}

Palo Alto Networks AutoFocus

This section provides API reference details for the PAN_IOC log type. For details about the data source, see the Palo Alto Networks AutoFocus documentation.

Data source Ingest schedule details.feedSourceType
autofocus.paloaltonetworks.com/api/v1.0/IOCFeed/FEED_ID/FEED_NAME

Replace FEED_ID and FEED_NAME with the Google Security Operations feed ID and feed name respectively.

 Every five minutes API

Prerequisites

  • Get the values for feedId, feed, and all authentication fields.
  • Get the following required permissions:
    • None

Test the endpoint

To get the results for a custom threat indicator feed, you need to send a request to the custom feed resource of the AutoFocus API. The custom feed endpoint is as follows: /IOCFeed/OUTPUT_FEED_ID/OUTPUT_FEED_NAME.

The following is an example request to retrieve threat intelligence:

curl -X GET \
-H "apiKey:API_KEY" \
https://autofocus.paloaltonetworks.com/api/v1.0/IOCFeed/OUTPUT_FEED_ID/OUTPUT_FEED_NAME?limit=MAX_ENTRIES

Replace the following:

  • API_KEY: API key tied to your license

  • OUTPUT_FEED_ID: custom threat feed ID number

  • OUTPUT_FEED_NAME: name of the custom feed

  • MAX_ENTRIES: maximum number of indicator entries displayed in the output

For details on how to use the Palo Alto AutoFoucs API, see the Get Custom Threat Indicator Feed documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "PAN_IOC",
     "panIocSettings": {
       "authentication": {
         "headerKeyValues": [{
            "key": "key"
            "value": "value"
         }],
       }
       "feedId": "ID",
       "feed": "feed"
     }
   }
}

Palo Alto Networks Cortex XDR

This section provides API reference details for the CORTEX_XDR log type. For details about the data source, see the Palo Alto Networks Cortex XDR documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/public_api/v1/incidents/get_incidents

Replace API_HOSTNAME with the fully qualified domain name of your instance, such as api-abcd.xdr.ab.paloaltonetworks.com.

 Every five minutes API

Prerequisites

  • Get the values for all authentication fields.
  • Make sure the API key is an advanced key, not a standard key.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.cortexXdrSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate Cortex XDR API in key-value format.
details.cortexXdrSettings.hostname Yes The fully qualified domain name of your Cortex XDR instance.
details.cortexXdrSettings.endpoint No The API endpoint to connect to retrieve logs, which include incidents or alerts.

Optional fields

initialStartTime

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "CORTEX_XDR",
     "cortexXdrSettings": {
       "authentication": {
         "headerKeyValues": [{
            "key": "Authorization"
            "value": "api_key"
         },
         {
            "key": "x-xdr-auth-id"
            "value": "api_key_id"
         }
         ],
       },
       "hostname": "api-abcd.xdr.ab.paloaltonetworks.com",
       "endpoint": "incidents"
     }
   }
}

Palo Alto Networks Prisma Cloud Audit Logs

This section provides API reference details for the PAN_PRISMA_CLOUD log type. For details about the data source, see the Palo Alto Networks Prisma Cloud Audit Logs documentation.

Data source Ingest schedule details.feedSourceType
api.prismacloud.io/audit/redlock Every five minutes API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.panPrismaCloudSettings.authentication.user Yes The Prisma Cloud username.
details.panPrismaCloudSettings.authentication.password Yes The Prisma Cloud password.
details.panPrismaCloudSettings.hostname Yes The Palo Alto Prisma Cloud API hostname.

Test the endpoints by using curl

Before you create the feed, you can test the API endpoints by using curl. Send a GET request to https://api.prismacloud.io/audit/redlock

The following example returns audit logs for events that took place on the Prisma Cloud platform:

curl -L 'https://api.prismacloud.io/audit/redlock' \
-H 'Accept: application/json; charset=UTF-8' \
-H 'x-redlock-auth: API_KEY_VALUE'

Replace the following:

  • API_KEY_VALUE: The Prisma Cloud authentication value is a JSON Web Token (JWT).

Optional fields

timeType, timeAmount, timeUnit

For details about the data source, see the Palo Alto Networks Prisma Cloud Audit Logs documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "PAN_PRISMA_CLOUD",
     "panPrismaCloudSettings": {
       "authentication": {
         "user": "user",
         "password": "password"
       },
       "hostname": "api2.prismacloud.io"
     }
   }
}

Proofpoint on Demand

This section provides API reference details for the PROOFPOINT_ON_DEMAND log type. For details about the data source, see the Proofpoint on Demand documentation.

Data source Ingest schedule details.feedSourceType
The default data endpoint is used. Every hour API

Prerequisites

  • Get the values for all authentication fields.
    • Make sure that the token is not used in any other instance or connection, whether inside or outside Google SecOps, as Proofpoint limits tokens to one active session.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.proofpointOnDemandSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate logstream.proofpoint.com in the key-value format.
details.proofpointOnDemandSettings.clusterId Yes The cluster ID, which is a user group string.

Other fields

proofpointOnDemandSourceDetails

Optional fields

initialStartTime

Test the endpoint

Before you create a real-time email processing log feed, you can test connectivity between your system and the Proofpoint on Demand (PoD) Log API.

The following is an example request to receive uncompressed data:

curl -i --no-buffer \
-H "Connection: Upgrade" \
-H "Upgrade: websocket" \
-H "Host: logstream.proofpoint.com:443" \
-H "Authorization: Bearer ACCESS_TOKEN " \
-H "Sec-WebSocket-Key: KEY" \
-H "SecWebSocket-Version: 13" \
"https://logstream.proofpoint.com:443/v1/stream?cid=CLUSTER_ID&type=message&sinceTime=DATE_TIME"

Replace the following:

  • ACCESS_TOKEN: a token provided by Proofpoint for a customer cluster to authenticate with the service.

  • KEY: a base64-encoded key used in the WebSocket opening handshake.

  • CLUSTER_ID: the cluster ID assigned by Proofpoint.

  • DATE_TIME: start time to begin streaming log data, in ISO 8601 format, which includes timezone information. For example: 2018-08-31T00:00:00-0800. The API fetches the logs recorded after the specified timestamp.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "PROOFPOINT_ON_DEMAND",
     "proofpointOnDemandSettings": {
       "authentication": {
         "user": "user",
         "secret": "secret"
       },
       "clusterId": "ID"
     }
   }
}

Proofpoint TAP

This section provides API reference details for the PROOFPOINT_MAIL log type. For details about the data source, see the Proofpoint SIEM API documentation.

Data source Ingest schedule details.feedSourceType
The default data endpoint is used. Every 24 hours API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.proofpointMailSettings.authentication.user Yes The user account required for authentication.
details.proofpointMailSettings.authentication.secret Yes The secret required for authentication.

Test the endpoint

Before you create the feed, you can test the Proofpoint TAP SIEM API endpoint by sending a GET request to /v2/siem/all.

To fetch events for all clicks and messages relating to known threats within the specified time period, use a GET request as follows:

curl \
"https://tap-api-v2.proofpoint.com/v2/siem/all?format=syslog&sinceSeconds=SECONDS" \
--user "PRINCIPAL:SECRET" \
-s

Replace the following:

  • SECONDS: an integer representing a time window in seconds from the current API server time. For example, 3600.

  • PRINCIPAL: Proofpoint service principal to authenticate to the SIEM API.

  • SECRET: Proofpoint API secret to authenticate to the SIEM API.

To learn more about the different query parameters that can be used as a part of the request, see the Proopoint TAP SIEM API documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "PROOFPOINT_MAIL",
     "proofpointMailSettings": {
       "authentication": {
         "user": "user",
         "secret": "secret"
       }
     }
   }
}

Qualys VM

This section provides API reference details for the QUALYS_VM log type. For details about the data source, see the Qualys VM documentation (PDF).

Data source Ingest schedule details.feedSourceType
The domain and full path of the resource, such as qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list. Every minute API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.qualysVmSettings.authentication.user Yes The user account required for authentication.
details.qualysVmSettings.authentication.secret Yes The secret required for authentication.
details.qualysVmSettings.hostname Yes The domain and full path of the resource, such as qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list.

Test the endpoints by using curl

Before you create the feed, you can test the API endpoints by using curl.

  • To test the endpoint for the Qualys VM Host List API, use the following curl command: 
    curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \
"https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/asset/host/?action=list"
  • To test the endpoint for the Qualys VM Host List Detection API, use the following curl command: 
    curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \
"https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=list"

Replace the following:

  • USERNAME: username of your Qualys account
  • PASSWORD: password of your Qualys account

Sample Create Feed request for Qualys VM Host List API

{
   "details": {
     "feedSourceType": "API",
     "logType": "QUALYS_VM",
     "qualysVmSettings": {
       "authentication": {c
         "user": "USERNAME",
         "secret": "PASSWORD"
       },
       "hostname": "qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list"
     }
   }
}

Sample Create Feed request for Qualys VM Host List Detection API

{
   "details": {
     "feedSourceType": "API",
     "logType": "QUALYS_VM",
     "qualysVmSettings": {
       "authentication": {
         "user": "USERNAME",
         "secret": "PASSWORD"
       },
       "hostname": "qualysapi.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=list"
     }
   }
}

Qualys Scan

This section provides API reference details for the QUALYS_SCAN log type. For details about the data source, see the Qualys VM documentation (PDF).

Data source Ingest schedule details.feedSourceType details.logType
qualysapi.qualys.com Every Day API QUALYS_SCAN

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • Ensure API access is enabled for the user.

Scan APIs

The Qualys Scan APIs that Google Security Operations supports include the following.

details.qualysScanSettings.api_type Description
SCAN_SUMMARY_OUTPUT Scan Summaries API to identify which hosts were scanned or not scanned and why.
SCAN_COMPLIANCE_OUTPUT Scan Compliance API to list out the compliance scans in your Qualys account.
SCAN_COMPLIANCE_CONTROL_OUTPUT Compliance Control API to view a list of compliance controls which are visible to the user.

Test the endpoints by using curl

Before you create the feed, you can test the API endpoints by using curl.

  • To test the endpoint for the API type SCAN_SUMMARY_OUTPUT, use the following curl command: 
    curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \
"https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/scan/vm/summary/?action=list&scan_datetime_since=DATETIME"
  • To test the endpoint for the API type SCAN_COMPLIANCE_OUTPUT, use the following curl command: 
    curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \
"https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/scan/compliance/?action=list&launched_after_datetime=DATETIME"
  • To test the endpoint for the API type SCAN_COMPLIANCE_CONTROL_OUTPUT, use the following curl command: 
    curl -H "X-Requested-With: Curl Sample" -u "USERNAME:PASSWORD" \
"https://qualysapi.qg3.apps.qualys.com/api/2.0/fo/compliance/control/?action=list&updated_after_datetime=DATETIME"

Replace the following:

  • USERNAME: username of your Qualys account
  • PASSWORD: password of your Qualys account
  • DATETIME: timestamp in UTC format according to ISO 8601, separating date and time with a T. For example: 2024-01-31T18:00:42Z. The API will fetch the logs recorded after the specified timestamp.

Sample Create Feed request for Qualys Scan API

{
   "details": {
     "feedSourceType": "API",
     "logType": "QUALYS_SCAN",
     "qualysScanSettings": {
       "authentication": {
         "user": "USERNAME",
         "secret": "PASSWORD"
       },
       "hostname": "qualysapi.qualys.com",
       "api_type": "SCAN_SUMMARY_OUTPUT"
     }
   }
}

Rapid7 InsightVM

This section provides API reference details for the RAPID7_INSIGHT log type. For details about the data source, see the Rapid7 InsightVM documentation.

Data source Ingest schedule details.feedSourceType
The REST endpoint, which must be either vulnerabilities or assets. Every minute API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.rapid7InsightSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate us.api.insight.rapid7.com in key-value format.
details.rapid7InsightSettings.endpoint Yes The REST endpoint to connect to. The endpoint value must be either vulnerabilities or assets.
details.rapid7InsightSettings.hostname No The fully qualified domain name of the Rapid7 endpoint, such as us.api.insight.rapid7.com.

Optional fields

initialStartTime

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "RAPID7_INSIGHT",
     "rapid7InsightSettings": {
       "authentication": {
         "headerKeyValues": [{
            "key": "X-Api-Key",
            "value": ApiToken "API_TOKEN"
         }],
       },
       "endpoint": "assets"
       "hostname": "us.api.insight.rapid7.com"
     }
   }
}

Replace API_TOKEN with your API token.

Recorded Future

This section provides API reference details for the RECORDED_FUTURE_IOC log type. For details about the data source, see the Recorded Future documentation.

Data source Ingest schedule details.feedSourceType
api.recordedfuture.com/v2/fusion/files. Every two hours API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.recordedFutureIocSettings.authentication.headerKeyValues Yes The HTTP header used to authenticate to api.recordedfuture.com in key-value format.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "RECORDED_FUTURE_IOC",
     "recordedFutureIocSettings": {
       "authentication": {
         "user": "user",
         "secret": "secret"
       },
     }
   }
}

RH-ISAC

This section provides API reference details for the RH_ISAC_IOC log type. For details about the data source, see the RH-ISAC documentation.

Data source Ingest schedule details.feedSourceType
api.trustar.co/api/1.3/indicators/search. Every 24 hours API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.rhIsacIocSettings.authentication.tokenEndpoint Yes The endpoint to retrieve the OAuth token.
details.rhIsacIocSettings.authentication.clientId Yes The application ID.
details.rhIsacIocSettings.authentication.clientSecret Yes The client secret.

Other fields

tags, queueDelay

Optional fields

initialStartTime

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "RH_ISAC_IOC",
     "rhIsacIocSettings": {
       "authentication": {
         "tokenEndPoint": "endpoint",
         "clientId": "clientId",
         "clientSecret": "clientSecret"
       }
     }
   }
}

Salesforce

This section provides API reference details for the SALESFORCE log type. For details about the data source, see the Salesforce documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/services/data/v50.0/query

Replace API_HOSTNAME with the fully qualified domain name of your Salesforce REST API endpoint, such as myinstance.salesforce.com.

 Every minute API

Prerequisites

Type-specific request fields

Field Required Description
details.salesforceSettings.hostname Yes The fully qualified domain name of your Salesforce REST API endpoint, such as myinstance.salesforce.com.
details.salesforceSettings.oauthPasswordGrantAuth.tokenEndpoint No The endpoint to retrieve the OAuth token. This field must be specified in the following format: https://SF_INSTANCE.my.salesforce.com/services/oauth2/token?grant_type=password. Replace SF_INSTANCE with your Salesforce instance name. This field is required only for OAuth password grant.
details.salesforceSettings.oauthPasswordGrantAuth.clientId No The application ID. This field is required only for OAuth password grant.
details.salesforceSettings.oauthPasswordGrantAuth.clientSecret No The client secret. This field is required only for OAuth password grant.
details.salesforceSettings.oauthPasswordGrantAuth.user No The username used for authentication. This field is required only for OAuth password grant.
details.salesforceSettings.oauthPasswordGrantAuth.password No The password used for authentication. This field is required only for OAuth password grant.
details.salesforceSettings.oauthJwtCredentials.tokenEndpoint No The endpoint to retrieve the OAuth JSON web token. This field is required only for OAuth JWT grant.
details.salesforceSettings.oauthJwtCredentials.claims.issuer No The JWT claims issuer, which is usually a client ID. This field is required only for OAuth JWT grant.
details.salesforceSettings.oauthJwtCredentials.claims.subject No The JWT claims subject, which is usually an email ID. This field is required only for OAuth JWT grant.
details.salesforceSettings.oauthJwtCredentials.claims.audience No The JWT claims audience. This field is required only for OAuth JWT grant.
details.salesforceSettings.oauthJwtCredentials.rsCredentials.privateKey No An RSA private key in PEM format. This field is required only for OAuth JWT grant.

Optional fields

initialStartTime

Test the endpoint

Before you create the feed, you can test the REST API endpoint by sending a GET request to /services/data/vAPI_VERSION/query. The Query resource is used to retrieve field values from a record.

To query event monitoring records based on fields, such as LogDate and EventType, use a GET request as follows:

curl https://SUBDOMAIN.my.salesforce.com/services/data/vAPI_VERSION/query \
    -X GET \
    -H "Authorization: Bearer AUTH_TOKEN" \
    -G \
    --data-urlencode "q=SELECT Id, EventType, LogFile, LogDate, LogFileLength FROM EventLogFile WHERE LogDate > Yesterday AND EventType = 'API'"

Replace the following:

  • SUBDOMAIN: the subdomain name relevant to the Salesforce instance being accessed.
  • API_VERSION: version number of the API endpoint. For example, 60.0.
  • AUTH_TOKEN: OAuth access token.

Sample Create Feed request using OAuth password grant

{
   "details": {
     "feedSourceType": "API",
     "logType": "SALESFORCE",
     "salesforceSettings": {
       "authentication": {
         "tokenEndpoint": "endpoint",
         "clientId": "clientId",
         "clientSecret": "clientSecret",
         "user": "user",
         "password": "password"
       },
       "hostname": "hostname"
     }
   }
}

Sample Create Feed request using OAuth JWT grant

{
   "details": {
     "feedSourceType": "API",
     "logType": "SALESFORCE",
     "salesforceSettings": {
       "authentication": {
         "tokenEndpoint": "endpoint",
         "issuer": "clientId",
         "subject": "emailID",
         "audience": "audience",
         "privateKey": "RSAKey"
       },
       "hostname": "hostname"
     }
   }
}

SentinelOne Alert

This section provides API reference details for the SENTINELONE_ALERT log type. For details about the data source, see the SentinelOne Alert documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/web/api/v2.1/cloud-detection/alerts

Replace API_HOSTNAME with the fully qualified domain name of SentinelOne API.

 Every five minutes API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.sentineloneAlertSettings.authentication.headerKeyValues Yes The HTTP headers to authenticate the SenitnelOne alerts, threats, and static-indicator API in key-value format.
details.sentineloneAlertSettings.hostname Yes The fully qualified domain name of the SenitnelOne API.
details.sentineloneAlertSettings.initialStartTime No The time when the alerts must be fetched.
details.sentineloneAlertSettings.isAlertApiSubscribed No Indicates whether the alerts API is subscribed.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "SENTINELONE_ALERT",
     "sentineloneAlertSettings": {
       "authentication": {
         "headerKeyValues": [{
            "key": "Authorization",
            "value": "ApiToken"
          }]
       },
       "hostname": "hostname",
       "isAlertApiSubscribed": false
     }
   }
}

ServiceNow CMDB

This section provides API reference details for the SERVICENOW_CMDB log type. For details about the data source, see the ServiceNow CMDB documentation.

Data source Ingest schedule details.feedSourceType
The fully qualified domain name of your ServiceNow REST API endpoint, such as myinstance.servicenow.com. Every 24 hours API

Prerequisites

  • Get the values for all the required fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.serviceNowCmdbSettings.authentication.user Yes The username required for authentication.
details.serviceNowCmdbSettings.authentication.secret Yes The secret required for authentication.
details.serviceNowCmdbSettings.hostname Yes The fully qualified domain name of your ServiceNow REST API endpoint, such as myinstance.servicenow.com.
details.serviceNowCmdbSettings.feedname Yes The ServiceNow table, which corresponds to a collection of records.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "SERVICENOW_CMDB",
     "servicenowCmdbSettings": {
       "authentication": {
         "user": "user",
         "secret": "secret"
       },
       "hostname": "hostname",
       "feedname": "feedname"
     }
   }
}

Symantec Event Export

This section provides API reference details for the SYMANTEC_EVENT_EXPORT log type. For details about the data source, see the Symantec Event Export documentation.

Data source Ingest schedule details.feedSourceType
The Symantec Event Export API. Every hour API

Prerequisites

  • Get the values for all authentication fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.symantecEventExportSettings.authentication.tokenEndpoint Yes The endpoint to retrieve the OAuth token.
details.symantecEventExportSettings.authentication.clientId Yes The application ID.
details.symantecEventExportSettings.authentication.clientSecret Yes The client secret.
details.symantecEventExportSettings.authentication.refreshToken Yes An OAuth 2.0 token used to refresh access tokens when they expire.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "SYMANTEC_EVENT_EXPORT",
     "symantecEventExportSettings ": {
       "authentication": {
         "tokenEndPoint": "REFRESH TOKEN URI",
         "clientId": "CLIENT ID",
         "clientSecret": "CLIENT SECRET",
         "refreshToken": "REFRESH TOKEN",
       }
     }
   }
}

Thinkst Canary

This section provides API reference details for the THINKST_CANARY log type. For details about the data source, see the Thinkst Canary documentation.

Data source Ingest schedule details.feedSourceType
API_HOSTNAME/api/v1/incidents/all

Replace API_HOSTNAME with the domain name of Thinkst Canary REST API endpoint.

 Every 30 minutes API

Prerequisites

  • Get the values for all the required fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.thinkstCanarySettings.authentication.headerKeyValues Yes The HTTP headers in key-value format.
details.thinkstCanarySettings.hostname Yes The fully qualified domain name of the Thinkst Canary REST API endpoint, such as myinstance.canary.tools.

Test the API endpoint

Before you create the feed, you can test the Canary API endpoint by sending a GET request to DOMAIN/api/v1/incidents/all.

The following is an example request to get all incidents:

curl https://DOMAIN.canary.tools/api/v1/incidents/all \
  -d auth_token=AUTH_TOKEN \
  -d limit=1 \
  -G

Replace the following:

  • DOMAIN: unique hash identifying your Canary Console
  • AUTH_TOKEN: OAuth access token

To learn more about the different query parameters that can be used as a part of the request, see the Canary API documentation.

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "THINKST_CANARY",
     "thinkstCanarySettings": {
       "authentication": {
         "user": "user",
         "secret": "secret"
       },
       "hostname": "hostname"
     }
   }
}

ThreatConnect

This section provides API reference details for the THREATCONNECT_IOC log type. For details about the data source, see the ThreatConnect documentation.

Data source Ingest schedule details.feedSourceType
The fully qualified domain name of the ThreatConnect REST API endpoint, such as myinstance.threatconnect.com. Every five minutes API

Prerequisites

  • Get the values for all the required fields.
  • Get the following required permissions:
    • None

Type-specific request fields

Field Required Description
details.threatConnectIocSettings.authentication.user Yes The username required for authentication.
details.threatConnectIocSettings.authentication.password Yes The password required for authentication
details.threatConnectIocSettings.hostname Yes The fully qualified domain name of the ThreatConnect REST API endpoint, such as myinstance.threatconnect.com.
details.threatConnectIocSettings.owners Yes All the owner names, where an owner identifies a collection of IoCs.

Other fields

queueDelay

Optional fields

initialStartTime

Sample Create Feed request

{
   "details": {
     "feedSourceType": "API",
     "logType": "THREATCONNECT_IOC",
     "threatConnectIocSettings": {
       "authentication": {
         "user": "user",
         "secret": "secret"
       },
       "hostname": "hostname",
       "owners": [{
         "owner"
       }]
     }
   }
}

Workday

This section provides API reference details for the WORKDAY log type. For details about the data source, see the Workday Administrator Guide (Integrations > Workday REST API).

Data source Ingest schedule details.feedSourceType details.logType
  • {hostname}/ccx/api/v1/{mytentant}/workers
  • {hostname}/ccx/api/v1/{mytentant}/workers/{id}/timeOffEntries
  • {hostname}/ccx/api/v1/{mytentant}/workers/{id}/history
  • {hostname}/ccx/api/v1/{mytentant}/supervisoryOrganizations
 Every 24 hours API WORKDAY

Prerequisites

  • In the Workday documentation for configuring OAuth 2.0 for your REST API client, follow the steps in Register API Clients.
  • Ensure that the Workday administrator provides you the Get and View permissions for the required security domain policies and provides access to the Workday API endpoints.

Type-specific request fields

Field Required Description
details.workdaySettings.authentication.secret Yes The access token generated by Workday after completing the steps to register OAuth 2.0 clients with Workday.

To set up authentication, you must either specify the access token or all of the following OAuth fields: token endpoint, client ID, client secret, and refresh token.
details.workdaySettings.authentication.tokenEndpoint" Yes The endpoint from which to get the access token.
details.workdaySettings.authentication.user Yes The client ID generated by Workday after completing the steps to register OAuth 2.0 clients with Workday.
details.workdaySettings.authentication.secret Yes The client secret generated by Workday after completing the steps to register OAuth 2.0 clients with Workday.
details.workdaySettings.authentication.refreshToken Yes The refresh token generated by Workday after completing the steps to register OAuth 2.0 clients with Workday.
details.workdaySettings.hostname Yes The hostname of the Workday REST web service. Example: services1.workday.com.
details.workdaySettings.tenantId Yes The name of the tenant.

Sample Create Feed request

The following sample uses a token endpoint, client ID, client secret, and refresh token:

{
   "details": {
     "feedSourceType": "API",
     "logType": "WORKDAY",
     "workdaySettings": {
       "authentication": {
         "tokenEndpoint": "TokenEndpoint",
         "user": "ClientID",
         "clientSecret": "ClientSecret"
         "refreshToken": "RefreshToken"
       },
       "hostname": "hostname",
       "tenantId": "ID"
     }
   }
}

The following sample uses an access token:

{
   "details": {
     "feedSourceType": "API",
     "logType": "WORKDAY",
     "workdaySettings": {
       "authentication": {
         "secret": "AccessToken"
       },
       "hostname": "hostname",
       "tenantId": "ID"
     }
   }
}