Manual Java Flight Recorder recording in Kubernetes

This describes how to create a Java Flight Recorder recording for Keycloak in a containerized environment when no additional tools are available.

Overview

Java Flight Recorder (JFR) records events of the Java Virtual Machine including thread dumps that can then be assembled to flame graphs and that can be used to analyze the performance. In a fully-fledged setup, Capturing performance metrics with Cryostat offers an automated way, and no custom Keycloak image is required. If such a setup is not available, follow these instructions to capture a JFR.

This is not using async profiling as this is not available inside OpenShift AFAIK. Therefore, the recordings will have the Safepoint bias problem. See Profiling Java in a container.

Read more on JDK Mission Control at Oracle, and on Monitoring Java Applications with Flight Recorder on Baeldung.

Preparing the Keycloak image

The following procedure requires jcmd to be present in the container to start the Java Flight Recording, and tar to be able to use kubectl cp to retrieve the recording from the container.

While older versions of Keycloak contain these tools, newer Keycloak image versions don’t contain them to make the images smaller and more secure. Therefore, the first step is to create a custom Keycloak image with these tools. There are two ways to do this: Creating a Keycloak image from scratch, or updating a Keycloak image with the necessary packages.

Building Keycloak from scratch

If you’re building a custom distribution of Keycloak from Keycloak’s main repository, change the file quarkus/container/Dockerfile and exchange line

RUN bash /tmp/ubi-null.sh java-17-openjdk-headless glibc-langpack-en

with

RUN bash /tmp/ubi-null.sh java-17-openjdk-devel tar glibc-langpack-en

Then proceed as described in Using a custom Keycloak image for deployment in Kubernetes to build the image.

Adding additional RPM packages to an image

The Keycloak docs on containers contain a section on how to add packages. To add the two packages java-17-openjdk-devel tar, proceed with a Dockerfile like the following:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs java-17-openjdk-devel tar --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

Then proceed as described in Using a custom Keycloak image for deployment in Kubernetes to use the image.

Updating JVM options

Starting from Keycloak 23 it is no longer necessary to override the -XX:FlightRecorderOptions=stackdepth JVM option as Keycloak uses 512 by default.

Keycloak uses very deep stack traces for its invocations. To be able to use the flame graphs, increase the number of stack frames by adding the following JVM option.

-XX:FlightRecorderOptions=stackdepth=512

When using the Keycloak operator, this can be passed to the Keycloak image via Keycloak’s CustomResource as follows:

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
spec:
  unsupported:
    podTemplate:
      spec:
        containers:
          - env:
              - name: JAVA_OPTS_APPEND
                value: >
                  -XX:FlightRecorderOptions=stackdepth=512

If you’re running a StatefulSet or Deployment that is managed by an Operator, consider stopping the Operator and updating the StatefulSet or Deployment manually to add or extend the Java options.

Starting the recording

To start the recording, issue a command in the container:

kubectl exec -n namespace pod -- jcmd 1 JFR.start duration=60s filename=/tmp/recording.jfr settings=/usr/lib/jvm/java/lib/jfr/profile.jfc

The value 1 is the process ID of the Java process that is the default for all Quarkus-based Keycloak containers. For Wildfly based distributions, this might be a different process ID. Use jcmd without parameters to list all Java process IDs to find the one you’re looking for.

Add the CLI option -c container if there is more than one container running in the Pod.

The profile.jfc contains instructions on what to capture. profile.jfc is one of the standard profiles shipped by the JVM and stands for “Profiling”: It collects a lot of information and is supposed to collect information for some minutes. A one-minute recording will collect already about 5 megabytes of data, so seek the time spans short. Adjust it as needed to collect the information you need.

Retrieving the recording

To retrieve the recording, issue the following command:

kubectl cp -n namespace keycloak pod:/tmp/recording.jfr recording.jfr --retries 999

Add the CLI option -c container if there is more than one container running in the Pod.

The CLI option --retries 999 helps to resume downloads for large files which might fail otherwise.

Analyzing the recording

See Analyzing a Java Flight Recorder recording for details.