Datadog Integration

Learn how to use Datadog together with Observability Kit.

Datadog is a commercial service for collecting and viewing telemetry data, with support for metrics, traces, and limited support for logs. It can be integrated with Observability Kit.

Account Setup

To set up an account with Datadog, go to the Datadog website and click on Free Trial. Select the appropriate region for storing data. Then fill in the remaining details and follow their instructions.

To complete the sign-up process, you’ll have to install one of their agents and have it send data. An agent isn’t required for the Observability Kit integration, so it can be removed immediately afterwards.

The last sign-up step provides instructions for installing the agent. An easy way to run the agent once is to use the Docker option. This allows you to create a container, send some data, and then stop and remove the container immediately afterwards.

OpenTelemetry Collector Setup

Datadog doesn’t support the OpenTelemetry APIs, directly. Instead, the OpenTelemetry collector must be used as a proxy for converting data into the Datadog format and forwarding it to their API endpoints.

Start by creating a collector.yaml configuration file with the following content:

receivers:
  otlp:
    protocols:
      http:
      grpc:

processors:
  batch:

exporters:
  datadog:
    api:
      site: DD_SITE
      key: DD_API_KEY

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

Replace the following placeholders in the configuration:

DD_SITE: The domain of the Datadog region for which you registered. You can get this from the browsers address bar when logged into the Datadog UI. For example, for the US1 region the value should be datadoghq.com; for the EU region, datadoghq.eu.
DD_API_KEY: An API key for authenticating against the Datadog API. To create an API key, open the Datadog web UI, click on Organization Setting and then API Keys. Create a new key. Make sure to save it, and then insert it into the configuration here.

You can run the collector either by downloading and running the binary manually, or by using Docker.

Download the binary from the latest release on GitHub. Make sure to download the otelcol_contrib_ variant. It includes support for Datadog and the correct variant for your operating system.

Next, run the collector using the following command:

otelcontribcol_linux_amd64 --config collector.yaml

Logging Setup

Datadog doesn’t support collecting logs in the OpenTelemetry format. You can use the following work-arounds for collecting log data, and correlating them with the traces collected by Observability Kit.

Logback Configuration

This method uses a Logback appender to send logs to the Datadog TCP logs API, while also adding the necessary Datadog metadata to correlate logs to traces.

First, add the following dependency to your Maven configuration to include a TCP appender for Logback:

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>7.2</version>
</dependency>

To correlate logs to traces collected by Observability Kit, you need to customize the JSON output of Logback. To do that, create a DataDogContextProvider class with the following content:

package com.example.application;

import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.core.JsonGenerator;
import net.logstash.logback.composite.AbstractJsonProvider;

import java.io.IOException;
import java.util.Map;

public class DataDogContextProvider extends AbstractJsonProvider<ILoggingEvent> {
    @Override
    public void writeTo(JsonGenerator generator, ILoggingEvent iLoggingEvent) throws IOException {
        Map<String, String> mdcPropertyMap = iLoggingEvent.getMDCPropertyMap();

        if (mdcPropertyMap.containsKey("trace_id")) {
            String traceId = mdcPropertyMap.get("trace_id");
            String traceIdHexString = traceId.substring(traceId.length() - 16 );
            long datadogTraceId = Long.parseUnsignedLong(traceIdHexString, 16);
            String datadogTraceIdString = Long.toUnsignedString(datadogTraceId);

            generator.writeStringField("dd.trace_id", datadogTraceIdString);
            System.out.println("dd.trace_id: " + datadogTraceIdString);
        }

        if (mdcPropertyMap.containsKey("span_id")) {
            String spanId = mdcPropertyMap.get("span_id");
            String spanIdHexString = spanId.substring(spanId.length() - 16 );
            long datadogSpanId = Long.parseUnsignedLong(spanIdHexString, 16);
            String datadogSpanIdString = Long.toUnsignedString(datadogSpanId);

            generator.writeStringField("dd.span_id", datadogSpanIdString);
        }
    }
}

This class checks if the log event has the trace_id and span_id attributes. It writes corresponding Datadog trace and span IDs to the JSON output.

Next, create a custom Logback encoder that makes use of this JSON provider. The encoder extends from LogstashEncoder, which is an encoder that generates JSON output that can be processed by the Datadog logging endpoint:

package com.example.application;

import ch.qos.logback.classic.spi.ILoggingEvent;
import net.logstash.logback.LogstashFormatter;
import net.logstash.logback.composite.AbstractCompositeJsonFormatter;
import net.logstash.logback.encoder.LogstashEncoder;

public class DataDogLogstashEncoder extends LogstashEncoder {
    @Override
    protected AbstractCompositeJsonFormatter<ILoggingEvent> createFormatter() {
        AbstractCompositeJsonFormatter<ILoggingEvent> formatter = super.createFormatter();

        ((LogstashFormatter)formatter).addProvider(new DataDogContextProvider());

        return formatter;
    }
}

Next add a new TCP appender to your Logback configuration, using the encoder created above, like so:

<appender name="JsonTcp" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>tcp-intake.logs.${DD_SITE}:443</destination>
    <keepAliveDuration>20 seconds</keepAliveDuration>
    <encoder class="com.example.application.DataDogLogstashEncoder">
        <prefix class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="ch.qos.logback.classic.PatternLayout">
                <pattern>${DD_API_KEY} %mdc{keyThatDoesNotExist}</pattern>
            </layout>
        </prefix>
    </encoder>
    <ssl />
</appender>

The configuration has two variables that need to be passed as system variables when running the Java agent:

DD_SITE: The domain of the Datadog region for which you registered. This should have the same value as in the collector configuration from earlier.
DD_API_KEY: An API key for authenticating against the Datadog API. This should have the same value as in the collector configuration from earlier.

For example, to pass these system variables when running the application, use this:

java -DDD_SITE=datadoghq.eu -DDD_API_KEY=1234ABCD ...

Finally, register the appender for the root logger to take it into effect. You would do that like so:

<root level="info">
    <appender-ref ref="JsonTcp" />
    <!-- ...other appenders -->
</root>

Observability Kit Configuration

To configure Observability Kit, create an agent.properties file with the following content:

otel.service.name=vaadin
otel.traces.exporter=otlp
otel.metrics.exporter=otlp

In the configuration above, the service name is defined as vaadin, which is also the default of the agent. The service name can be customized.

By default, the OpenTelemetry Protocol (OTLP) exporters assume that the OpenTelemetry collector is running locally. Therefore, you don’t need to configure endpoints. If the collector is running on a different system, you need to configure an endpoint for the exporter:

otel.exporter.otlp.endpoint=https://collector.my-domain.net:4317

Running the Application

Run the application using the Java binary and pass the respective arguments for the Agent and configuration like so:

java -javaagent:PATH/TO/observability-kit-agent-VERSION.jar \
     -Dotel.javaagent.configuration-file=PATH/TO/agent.properties \
     -jar myapp.jar

Note	Replace Placeholder Paths & Version Remember to correct the path to the `agent.properties` file, as well as the path and version of the Agent `.jar` file.

Viewing Data

Viewing Traces

In the Datadog web UI, select APM and then Traces from the menu on the left. The view provides graphs of the number of requests, failed requests and latency, as well as a list of the latest traces.

The search bar allows filtering for tags, which are attribute values of spans. To get an idea of what kind of tags can be filtered, click on a trace. This displays a detailed view with all tags recorded for the span. Clicking on a tag will present an option on which to filter that specific tag value.

Clicking on a trace ID will display a side panel that shows detailed information about a specific trace, such as nested spans, and attributes and related logs.

See Datadog documentation for Trace Explorer for more information on how to use this view.

Viewing Metrics

In the Datadog web UI, select Metrics and then Explorer from the menu on the left.

In the metrics input field, enter one of the metrics collected by Observability Kit. For example, use otel.process.runtime.jvm.memory.usage to view the JVM memory usage.

See Datadog documentation for the Metrics Explorer for more information on how to use this view.

Viewing Logs

Logging integration is limited to the Logback logging framework.

In the Datadog web UI, select Logs from the menu on the left. The view shows a list of recent log entries, which can be filtered by text or log level.

See Datadog documentation for the Logs Explorer for more information on how to use this view.

Docs