Observability

Synapse is instrumented with OpenTelemetry metrics and traces. To obtain the data via OTLP, configure the exporters using OpenTelemetry environment variables.

Metrics

Environment variable Description

OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT

Address of the OTLP metrics receiver (for example: https://localhost:9090/api/v1/otlp/v1/metrics). If not defined, OTLP metrics are disabled.

OTEL_EXPORTER_OTLP_METRICS_INSECURE or OTEL_EXPORTER_OTLP_INSECURE

Skip validating the TLS certificate of the endpoint

OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE or OTEL_EXPORTER_OTLP_CERTIFICATE

Path to the certificate to use for validating the server’s TLS credentials.

OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE or OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE

Path to the client certificate to use for mTLS

OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY or OTEL_EXPORTER_OTLP_CLIENT_KEY

Path to the client key to use for mTLS

OTEL_EXPORTER_OTLP_METRICS_PROTOCOL or OTEL_EXPORTER_OTLP_PROTOCOL

OTLP protocol. Supported values are grpc and http/protobuf. Defaults to grpc.

OTEL_METRIC_EXPORT_INTERVAL

The export interval in milliseconds. Defaults to 60000.

OTEL_METRIC_EXPORT_TIMEOUT

Timeout for exporting the data in milliseconds. Defaults to 30000.

OTEL_METRICS_EXPORTER

Set to otlp to enable the OTLP exporter. Defaults to prometheus which enables the scrape endpoint at /_cerbos/metrics. Set to none to disable metrics.

Refer to https://opentelemetry.io/docs/specs/otel/protocol/exporter/ for more information about exporter configuration through environment variables. Note that the OpenTelemetry Go SDK used by Synapse might not have full support for some of the environment variables listed on the OpenTelemetry specification.

OTEL_METRICS_EXPORTER and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT are the only required environment variables to enable OTLP metrics.

Traces

Environment variable Description

OTEL_SERVICE_NAME

Service name reported in the traces. Defaults to Synapse.

OTEL_TRACES_EXPORTER

Set to otlp to enable the OTLP exporter. Defaults to none.

OTEL_TRACES_SAMPLER

Trace sampler. Defaults to parentbased_always_off. Supported values:

always_on

Record every trace.

always_off

Don’t record any traces.

traceidratio

Record a fraction of traces based on ID. Set OTEL_TRACES_SAMPLER_ARG to a value between 0 and 1 to define the fraction.

parentbased_always_on

Record all traces except those where the parent span is not sampled.

parentbased_always_off

Don’t record any traces unless the parent span is sampled.

parentbased_traceidratio

Record a fraction of traces where the parent span is sampled. Set OTEL_TRACES_SAMPLER_ARG to a value between 0 and 1 to define the fraction.

OTEL_TRACES_SAMPLER_ARG

Set the sampling ratio when OTEL_TRACES_SAMPLER is a ratio-based sampler. Defaults to 0.1.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT

Address of the OTLP collector (for example: https://localhost:4317). If not defined, traces are disabled.

OTEL_EXPORTER_OTLP_TRACES_INSECURE or OTEL_EXPORTER_OTLP_INSECURE

Skip validating the TLS certificate of the endpoint

OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE or OTEL_EXPORTER_OTLP_CERTIFICATE

Path to the certificate to use for validating the server’s TLS credentials.

OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE or OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE

Path to the client certificate to use for mTLS

OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY or OTEL_EXPORTER_OTLP_CLIENT_KEY

Path to the client key to use for mTLS

OTEL_EXPORTER_OTLP_TRACES_PROTOCOL or OTEL_EXPORTER_OTLP_PROTOCOL

OTLP protocol. Supported values are grpc and http/protobuf. Defaults to grpc.

Refer to https://opentelemetry.io/docs/specs/otel/protocol/exporter/ for more information about exporter configuration through environment variables. Note that the OpenTelemetry Go SDK used by Synapse might not have full support for some of the environment variables listed on the OpenTelemetry specification.

OTEL_TRACES_EXPORTER and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT are the only required environment variables to enable OTLP trace exports.

Remote span context

By default, Synapse does not trust the trace context of incoming API requests. Spans that Synapse creates for a request are linked to the caller’s span instead of becoming its children. Synapse applies its own sampler to these spans rather than following the caller’s sampling decision.

When the caller is a trusted component, such as an Envoy proxy in the same deployment, set server.trace.trustRemoteSpans to true. Synapse spans then become children of the caller’s span, and the request appears as a single trace.

server:
  trace:
    trustRemoteSpans: true