Skip to content

How to use Sentry with OTLP #14521

New issue
New issue
Open
Open
How to use Sentry with OTLP#14521

Description

@blephy

blephy
openedon Nov 28, 2024

Environment

SaaS (https://sentry.io/)

What are you trying to accomplish?

I'm trying to get OTLP working with Sentry.

I went through all the documentation on the subject, as well as several videos provided by sentry without ever succeeding.

The OTLP setup works, I have the Pino instrumentation for example. But the Sentry sdk no longer sends errors to the sentry cloud.

What I would like is:

  • use OTLP
  • only use Sentry for error management (no tracing / no profiling)

Version :

  • Sentry@latest
  • all opentelemetry packages : latest
  • node 22.11.0

How are you getting stuck?

Here is my code (imported with node --import)

// eslint-disable-next-line n/no-unsupported-features/node-builtins
import { register } from 'node:module';

// eslint-disable-next-line import-x/order
import { createAddHookMessageChannel } from 'import-in-the-middle';

// Yes, createAddHookMessageChannel is new. See below.
const { registerOptions, waitForAllMessagesAcknowledged } = createAddHookMessageChannel();

// @ts-expect-error -- wrong type from import-in-the-middle
register('import-in-the-middle/hook.mjs', import.meta.url, registerOptions);

// eslint-disable-next-line unicorn/no-single-promise-in-promise-methods
await Promise.all([waitForAllMessagesAcknowledged]);

/* eslint-disable import-x/first */
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { Resource } from '@opentelemetry/resources';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
import * as Sentry from '@sentry/node';
import { SentryPropagator, SentrySampler } from '@sentry/opentelemetry';

import type { NodeSDKConfiguration } from '@opentelemetry/sdk-node';

const sentryClient = Sentry.init({
  debug: true,
  dsn: 'https://....ingest.us.sentry.io/...',
  environment: 'staging',
  release: '[email protected]',
  skipOpenTelemetrySetup: true,
  registerEsmLoaderHooks: false,
  enableTracing: false,
  attachStacktrace: true,
  tracesSampleRate: 0,
  profilesSampleRate: 0,
  integrations: [
    ...Sentry.getDefaultIntegrationsWithoutPerformance(),
    Sentry.httpIntegration({ spans: false }),
    Sentry.postgresIntegration(),
    Sentry.fastifyIntegration(),
    Sentry.zodErrorsIntegration(),
  ],
});

// diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

// const traceExporter = new ConsoleSpanExporter();
const nodeTraceConfig: Partial<NodeSDKConfiguration> = {
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'my-service',
    [ATTR_SERVICE_VERSION]: '1.5.1',
  }),
  autoDetectResources: true,
  mergeResourceWithDefaults: true,
  serviceName: 'my-service',
  // traceExporter: traceExporter,
  // metricReader: new PeriodicExportingMetricReader({
  //   exporter: new ConsoleMetricExporter(),
  // }),
  contextManager: new Sentry.SentryContextManager(),
  // spanProcessors: [new BatchSpanProcessor(traceExporter)],
  textMapPropagator: new SentryPropagator(),
};

if (sentryClient !== undefined) {
  nodeTraceConfig.sampler = new SentrySampler(sentryClient);
}

const sdk = new NodeSDK(nodeTraceConfig);

registerInstrumentations({
  instrumentations: getNodeAutoInstrumentations(),
});

sdk.start();

// Validate that the setup is correct
Sentry.validateOpenTelemetrySetup();

process.on('SIGTERM', async () => {
  try {
    // eslint-disable-next-line no-console -- this is intended
    console.log('[OTEL] shuting down sdk');

    await sdk.shutdown();
    // eslint-disable-next-line no-console -- this is intended
    console.log('[OTEL] sdk shutdown successfully');
  } catch (error) {
    // eslint-disable-next-line no-console -- this is intended
    console.log('[OTEL] error occurred when terminating sdk', error);
  }

  // eslint-disable-next-line n/no-process-exit -- this is intended
  process.exit(0);
});

logs :

Sentry Logger [log]: Initializing Sentry: process: 20346, thread: main.
Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: NodeFetch
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariablesAsync
Sentry Logger [log]: Integration installed: Context
Sentry Logger [log]: Integration installed: ProcessAndThreadBreadcrumbs
Sentry Logger [log]: Integration installed: Express
Sentry Logger [log]: Integration installed: Fastify
Sentry Logger [log]: Integration installed: Graphql
Sentry Logger [log]: Integration installed: Mongo
Sentry Logger [log]: Integration installed: Mongoose
Sentry Logger [log]: Integration installed: Mysql
Sentry Logger [log]: Integration installed: Mysql2
Sentry Logger [log]: Integration installed: Redis
Sentry Logger [log]: Integration installed: Postgres
Sentry Logger [log]: Integration installed: Nest
Sentry Logger [log]: Integration installed: Hapi
Sentry Logger [log]: Integration installed: Koa
Sentry Logger [log]: Integration installed: Connect
Sentry Logger [log]: Integration installed: Tedious
Sentry Logger [log]: Integration installed: GenericPool
Sentry Logger [log]: Integration installed: Kafka
Sentry Logger [log]: Integration installed: Amqplib
Sentry Logger [log]: Integration installed: LruMemoizer
Sentry Logger [log]: Integration installed: ZodErrors
Sentry Logger [log]: Running in ESM mode.
Sentry Logger [error]: You have to set up the SentryPropagator. Without this, the OpenTelemetry & Sentry integration will not work properly.
Sentry Logger [error]: You have to set up the SentrySpanProcessor. Without this, the OpenTelemetry & Sentry integration will not work properly.
Sentry Logger [warn]: You have to set up the SentrySampler. Without this, the OpenTelemetry & Sentry integration may still work, but sample rates set for the Sentry SDK will not be respected. If you use a custom sampler, make sure to use `wrapSamplingDecision`.
(node:20346) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
{"level":20,"time":1732794515247,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"initializing server"}
{"level":20,"time":1732794515248,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"adding sentry fastify error handler"}
{"level":20,"time":1732794515249,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"adding zod validator and serialized"}
{"level":20,"time":1732794515249,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"zod validator and serialized added"}
{"level":20,"time":1732794515249,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"adding global error handler"}
{"level":20,"time":1732794515249,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"global error handler added"}
{"level":20,"time":1732794515250,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"decorating server with config"}
{"level":20,"time":1732794515250,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"decorating server with googleJwksClient"}
{"level":20,"time":1732794515250,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"decorating server with hiddenData"}
{"level":20,"time":1732794515250,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"decorating server with guides"}
{"level":20,"time":1732794515250,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"adding readyness and liveness routes"}
{"level":20,"time":1732794515255,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"readyness and liveness routes added"}
{"level":20,"time":1732794515255,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"server initialized"}
{"level":30,"time":1732794515277,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"🚥 starting server"}
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
{"level":30,"time":1732794515287,"app_name":"my-service","app_version":"1.5.1","module":"server","msg":"🚀 server started. listening at http://127.0.0.1:3000"}
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
{"level":30,"time":1732794536349,"app_name":"my-service","app_version":"1.5.1","module":"server","reqId":"req-1","trace_id":"d8c10b42c9284ed331e8644026a22bbc","span_id":"019b57797824f2fb","trace_flags":"00","req":{"method":"GET","url":"/api/partners/patient/qrcode?patient=90432","host":"localhost:3000","remoteAddress":"127.0.0.1","remotePort":56789},"msg":"incoming request"}
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for middleware - handleCors: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for middleware - fastify -> @fastify/cors -> @fastify/view -> @fastify/view/cache: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for middleware - fastify -> @fastify/cors -> @fastify/view -> @fastify/view/cache -> sentry-fastify-error-handler: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for middleware - fastify -> @fastify/cors -> @fastify/view -> @fastify/view/cache -> sentry-fastify-error-handler: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for POST: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for tls.connect: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for tcp.connect: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for dns.lookup: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
{"level":30,"time":1732794536527,"app_name":"my-service","app_version":"1.5.1","trace_id":"29d9328d95a3a360b952720a0423cc51","span_id":"636d72c40c71b4dc","trace_flags":"00","msg":"61a18228-dfa9-4859-9808-3da874b76277 | Requesting API 'POST /scanmonitoring/api/v1/scan_sessions/31bb0d7c-f9c0-4388-a429-1361c1f6f625/scan_events' with body:true"}
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for POST: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for tls.connect: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for tcp.connect: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for dns.lookup: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
{"level":20,"time":1732794536527,"app_name":"my-service","app_version":"1.5.1","trace_id":"29d9328d95a3a360b952720a0423cc51","span_id":"636d72c40c71b4dc","trace_flags":"00","msg":"{\"data\":{\"name\":\"generate-qr-code\",\"status\":\"error\",\"message\":{\"patient_id\":\"90432\",\"error\":\"querystring/slug Required\"},\"app_name\":\"my-service\",\"app_version\":\"1.5.1\"}}"}
{"level":30,"time":1732794536754,"app_name":"my-service","app_version":"1.5.1","trace_id":"29d9328d95a3a360b952720a0423cc51","span_id":"636d72c40c71b4dc","trace_flags":"00","msg":"61a18228-dfa9-4859-9808-3da874b76277 | API answered successfully in 227 ms"}
{"level":20,"time":1732794536754,"app_name":"my-service","app_version":"1.5.1","trace_id":"29d9328d95a3a360b952720a0423cc51","span_id":"636d72c40c71b4dc","trace_flags":"00","msg":"{\"success\":true}"}
{"level":30,"time":1732794536754,"app_name":"my-service","app_version":"1.5.1","module":"server","reqId":"req-1","trace_id":"29d9328d95a3a360b952720a0423cc51","span_id":"636d72c40c71b4dc","trace_flags":"00","res":{"statusCode":400},"responseTime":9.595392003655434,"msg":"request completed"}

Seems OTLP instrumentation is working, my pino instance is instrumented. I don't know about fastify / http, server response does not include traceId an so on headers.

You have to set up the SentryPropagator : i deep dive into the Sentry.validateOpenTelemetrySetup(); function's code, and i think this is a false positive.

As you can see, i'm triggering the fastify sentry error handler, but no error is sent to sentry.

I tried :

  • run the same config with the Sentry OTLP instrumentation and not OTLP custom : this is working
  • use NodeTracerProvider instead of NodeSDK : same, does not work
  • use the esm hook from Sentry and not mine : same, does not work

I also tried this an this does not work

Sentry.init({
  debug: true,
  dsn: '...',
  environment: 'staging',
  release: '[email protected]',
  // skipOpenTelemetrySetup: true,
  // registerEsmLoaderHooks: false,
  // enableTracing: false,
  attachStacktrace: true,
  tracesSampleRate: 1,
  profilesSampleRate: 1,
  integrations: [
    ...Sentry.getDefaultIntegrationsWithoutPerformance(),
    Sentry.postgresIntegration(),
    Sentry.fastifyIntegration(),
    Sentry.zodErrorsIntegration(),
  ],
  openTelemetryInstrumentations: [new GenericPoolInstrumentation()],
});

Where in the product are you?

APIs

Link

No response

DSN

No response

Version

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    GitHub Issues with 👀 3

    • Status

      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions