Keycloak Cluster Setup

May 10 2019 by 张立强 liqiang@fit2cloud.com

This post is more than one year old. The contents within the blog is likely to be out of date.

This post shares some solutions to setup Keycloak cluster in various scenarios (e.g. cross-DC, docker cross-host, Kubernetes).

If you'd like to setup Keycloak cluster, this blog may give you some reference.

Two cli script files are added to the Keycloak image as per the guide.

The Dockerfile is below and these two files are the most important matter for this blog, you can find them from TCPPING.cli and JDBC_PING.cli.

FROM jboss/keycloak:latest

ADD cli/TCPPING.cli /opt/jboss/tools/cli/jgroups/discovery/
ADD cli/JDBC_PING.cli /opt/jboss/tools/cli/jgroups/discovery/

First of all we should know that for a Keycloak cluster, all keycloak instances should use same database and this is very simple, another thing is about cache(generally there are two kinds of cache in Keycloaks, the 1st is persistent data cache read from database aim to improve performance like realm/client/user, the 2nd is the non-persistent data cache like sessions/clientSessions, the 2nd is very important for a cluster) which is a little bit complex to configure, we have to make sure the consistent of cache in a cluster view.

Totally here are 3 solutions for clustering, and all of the solutions are base on the discovery protocols of JGroups (Keycloak use Infinispan cache and Infinispan use JGroups to discover nodes).

1. PING

PING is the default enabled clustering solution of Keycloak using UDP protocol, and you don't need to do any configuration for this.

But PING is only available when multicast network is enabled and port 55200 should be exposed, e.g. bare metals, VMs, docker containers in the same host.

We tested this by two Keycloak containers in same host.

The logs show that the two Keycloak instances discovered each other and clustered.

2. TCPPING

TCPPING use TCP protocol with 7600 port. This can be used when multicast is not available, e.g. deployments cross DC, containers cross host.

We tested this by two Keycloak containers cross host.

And in this solution we need to set three below environment variables for containers.

#IP address of this host, please make sure this IP can be accessed by the other Keycloak instances
JGROUPS_DISCOVERY_EXTERNAL_IP=172.21.48.39
#protocol
JGROUPS_DISCOVERY_PROTOCOL=TCPPING
#IP and Port of all host
JGROUPS_DISCOVERY_PROPERTIES=initial_hosts="172.21.48.4[7600],172.21.48.39[7600]"

The logs show that the two Keycloak instances discovered each other and clustered.

3. JDBC_PING

JDBC_PING use TCP protocol with 7600 port which is similar as TCPPING, but the difference between them is, TCPPING requires you configure the IP and port of all instances, for JDBC_PING you just need to configure the IP and port of current instance, this is because in JDBC_PING solution each instance insert its own information into database and the instances discover peers by the ping data read from database.

We tested this by two Keycloak containers cross host.

And in this solution we need to set two below environment variables for containers.

#IP address of this host, please make sure this IP can be accessed by the other Keycloak instances
JGROUPS_DISCOVERY_EXTERNAL_IP=172.21.48.39
#protocol
JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING

The ping data of all instances haven been saved in database after instances started.

The logs show that the two Keycloak instances discovered each other and clustered.

One more thing

The above solutions are available for most scenarios, but they are still not enough for some others, e.g.Kubernetes.

The typical deployment on Kubernetes is one Deployment/ReplicateSet/StatefulSet contains multi Keycloak Pods, the Pods are really dynamic as they can scale up and down or failover to another node, which requires the cluster to discover and remove these dynamic members.

On Kubernetes we can use DNS_PING or KUBE_PING which work quite well in practice.

Besides DNS_PING and KUBE_PING, JDBC_PING is another option for Kubernetes.

On Kubernetes multicast is available only for the containers in the same node and a pod has no static ip which can be used to configure TCPPING or JDBC_PING. But in the JDBC_PING.cli mentioned above we have handled this, if you don't set the JGROUPS_DISCOVERY_EXTERNAL_IP env, the pod ip will be used, that means on Kubernetes you can simply set JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING then your keycloak cluster is ok.

Discussion

Suggestions and comments can be discussed via Keycloak User Mail List or this GitHub Repository.