Enabling Tracing

Learn how to enable distributed tracing in Keycloak

This guide explains how you can enable and configure distributed tracing in Keycloak by utilizing OpenTelemetry (OTel). Tracing allows for detailed monitoring of each request’s lifecycle, which helps quickly identify and diagnose issues, leading to more efficient debugging and maintenance.

It also provides valuable insights into performance bottlenecks and can help optimize the system’s overall efficiency. Keycloak uses a supported Quarkus OTel extension that provides smooth integration and exposure of application traces.

Enable tracing

It is possible to enable exposing traces using the build time option tracing-enabled as follows:

bin/kc.[sh|bat] start --tracing-enabled=true

By default, the trace exporters send out data in batches, using the gRPC protocol and endpoint http://localhost:4317.

The default service name is keycloak, specified via the tracing-service-name property, which takes precedence over service.name defined in the tracing-resource-attributes property.

For more information about resource attributes that can be provided via the tracing-resource-attributes property, see the Quarkus OpenTelemetry Resource guide.

Tracing can be enabled only when the opentelemetry feature is enabled (by default).

For more tracing settings, see all possible configurations below.

Development setup

In order to see the captured Keycloak traces, the basic setup with leveraging the Jaeger tracing platform might be used. For development purposes, the Jaeger-all-in-one can be used to see traces as easily as possible.

Jaeger-all-in-one includes the Jaeger agent, an OTel collector, and the query service/UI. You do not need to install a separate collector, as you can directly send the trace data to Jaeger. 
podman|docker run --name jaeger \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
jaegertracing/all-in-one

Exposed ports

16686

Jaeger UI

4317

OpenTelemetry Protocol gRPC receiver (default)

4318

OpenTelemetry Protocol HTTP receiver

You can visit the Jaeger UI on http://localhost:16686/ to see the tracing information. The Jaeger UI might look like this with an arbitrary Keycloak trace:

Jaeger UI

Information in traces

Spans

Keycloak creates spans for the following activities:

  • Incoming HTTP requests

  • Outgoing Database including acquiring a database connections

  • Outgoing LDAP requests including connecting to the LDAP server

  • Outgoing HTTP requests including IdP brokerage

Tags

Keycloak adds tags to traces depending on the type of the request. All tags are prefixed with kc..

Example tags are:

kc.clientId

Client ID

kc.realmName

Realm name

kc.sessionId

User session ID

kc.token.id

id as mentioned in the token

kc.token.issuer

issuer as mentioned in the token

kc.token.sid

sid as mentioned in the token

kc.authenticationSessionId

Authentication session ID

kc.authenticationTabId

Authentication Tab ID

Logs

If a trace is being sampled, it will contain any user events created during the request. This includes, for example, LOGIN, LOGOUT or REFRESH_TOKEN events with all details and IDs found in user events.

LDAP communication errors are shown as log entries in recorded traces as well with a stack trace and details of the failed operation.

Trace IDs in logs

When tracing is enabled, the trace IDs are included in the log messages of all enabled log handlers (see more in Configuring logging). It can be useful for associating log events to request execution, which might provide better traceability and debugging. All log lines originating from the same request will have the same traceId in the log.

The log message also contains a sampled flag, which relates to the sampling described below and indicates whether the span was sampled - sent to the collector.

The format of the log records may start as follows:

2024-08-05 15:27:07,144 traceId=b636ac4c665ceb901f7fdc3fc7e80154, parentId=d59cea113d0c2549, spanId=d59cea113d0c2549, sampled=true WARN  [org.keycloak.events] ...

Hide trace IDs in logs

You can hide trace IDs in specific log handlers by specifying their associated Keycloak option log-<handler-name>-include-trace, where <handler-name> is the name of the log handler. For instance, to disable trace info in the console log, you can turn it off as follows:

bin/kc.[sh|bat] start --tracing-enabled=true --log=console --log-console-include-trace=false
When you explicitly override the log format for the particular log handlers, the *-include-trace options do not have any effect, and no tracing is included.

Sampling

Sampler decides whether a trace should be discarded or forwarded, effectively reducing overhead by limiting the number of collected traces sent to the collector. It helps manage resource consumption, which leads to avoiding the huge storage costs of tracing every single request and potential performance penalty.

For a production-ready environment, sampling should be properly set to minimize infrastructure costs.

Keycloak supports several built-in OpenTelemetry samplers, such as:

  • always_on

  • always_off

  • traceidratio (default)

  • parentbased_always_on

  • parentbased_always_off

  • parentbased_traceidratio

The used sampler can be changed via the tracing-sampler-type property.

Default sampler

The default sampler for Keycloak is traceidratio, which controls the rate of trace sampling based on a specified ratio configurable via the tracing-sampler-ratio property.

Trace ratio

The default trace ratio is 1.0, which means all traces are sampled - sent to the collector. The ratio is a floating number in the range (0,1]. For instance, when the ratio is 0.1, only 10% of the traces are sampled.

For a production-ready environment, the trace ratio should be a smaller number to prevent the massive cost of trace store infrastructure and avoid performance overhead.

Rationale

The sampler makes its own sampling decisions based on the current ratio of sampled spans, regardless of the decision made on the parent span, as with using the parentbased_traceidratio sampler.

The parentbased_traceidratio sampler could be the preferred default type as it ensures the sampling consistency between parent and child spans. Specifically, if a parent span is sampled, all its child spans will be sampled as well - the same sampling decision for all. It helps to keep all spans together and prevents storing incomplete traces.

However, it might introduce certain security risks leading to DoS attacks. External callers can manipulate trace headers, parent spans can be injected, and the trace store can be overwhelmed. Proper HTTP headers (especially tracestate) filtering and adequate measures of caller trust would need to be assessed.

For more information, see the W3C Trace context document.

Tracing in Kubernetes environment

When the tracing is enabled when using the Keycloak Operator, certain information about the deployment is propagated to the underlying containers.

Configuration via Keycloak CR

You can change tracing configuration via Keycloak CR. For more information, see the Advanced configuration.

Filter traces based on Kubernetes attributes

You can filter out the required traces in your tracing backend based on their tags:

  • service.name - Keycloak deployment name

  • k8s.namespace.name - Namespace

  • host.name - Pod name

Keycloak Operator automatically sets the KC_TRACING_SERVICE_NAME and KC_TRACING_RESOURCE_ATTRIBUTES environment variables for each Keycloak container included in pods it manages.

The KC_TRACING_RESOURCE_ATTRIBUTES variable always contains (if not overridden) the k8s.namespace.name attribute representing current namespace.

Relevant options

Value

log-console-include-trace

Include tracing information in the console log.

If the log-console-format option is specified, this option has no effect.

CLI: --log-console-include-trace
Env: KC_LOG_CONSOLE_INCLUDE_TRACE

Available only when Console log handler and Tracing is activated

true (default), false

log-file-include-trace

Include tracing information in the file log.

If the log-file-format option is specified, this option has no effect.

CLI: --log-file-include-trace
Env: KC_LOG_FILE_INCLUDE_TRACE

Available only when File log handler and Tracing is activated

true (default), false

log-syslog-include-trace

Include tracing information in the Syslog.

If the log-syslog-format option is specified, this option has no effect.

CLI: --log-syslog-include-trace
Env: KC_LOG_SYSLOG_INCLUDE_TRACE

Available only when Syslog handler and Tracing is activated

true (default), false

tracing-compression

OpenTelemetry compression method used to compress payloads.

If unset, compression is disabled.

CLI: --tracing-compression
Env: KC_TRACING_COMPRESSION

Available only when Tracing is enabled

gzip, none (default)

tracing-enabled

Enables the OpenTelemetry tracing.

CLI: --tracing-enabled
Env: KC_TRACING_ENABLED

Available only when 'opentelemetry' feature is enabled

true, false (default)

tracing-endpoint

OpenTelemetry endpoint to connect to.

CLI: --tracing-endpoint
Env: KC_TRACING_ENDPOINT

Available only when Tracing is enabled

http://localhost:4317 (default)

tracing-jdbc-enabled

Enables the OpenTelemetry JDBC tracing.

CLI: --tracing-jdbc-enabled
Env: KC_TRACING_JDBC_ENABLED

Available only when Tracing is enabled

true (default), false

tracing-protocol

OpenTelemetry protocol used for the telemetry data.

CLI: --tracing-protocol
Env: KC_TRACING_PROTOCOL

Available only when Tracing is enabled

grpc (default), http/protobuf

tracing-resource-attributes

OpenTelemetry resource attributes present in the exported trace to characterize the telemetry producer.

Values in format key1=val1,key2=val2. For more information, check the Tracing guide.

CLI: --tracing-resource-attributes
Env: KC_TRACING_RESOURCE_ATTRIBUTES

Available only when Tracing is enabled

tracing-sampler-ratio

OpenTelemetry sampler ratio.

Probability that a span will be sampled. Expected double value in interval <0,1).

CLI: --tracing-sampler-ratio
Env: KC_TRACING_SAMPLER_RATIO

Available only when Tracing is enabled

1.0 (default)

tracing-sampler-type

OpenTelemetry sampler to use for tracing.

CLI: --tracing-sampler-type
Env: KC_TRACING_SAMPLER_TYPE

Available only when Tracing is enabled

always_on, always_off, traceidratio (default), parentbased_always_on, parentbased_always_off, parentbased_traceidratio

tracing-service-name

OpenTelemetry service name.

Takes precedence over service.name defined in the tracing-resource-attributes property.

CLI: --tracing-service-name
Env: KC_TRACING_SERVICE_NAME

Available only when Tracing is enabled

keycloak (default)
On this page
Edit this guide