Recover from site failures with a Multi-Site Setup

These guides include step-by-step instructions to bring up different components of the Keycloak multi-site architecture such as:

A series of guides around how to deploy Keycloak in various configurations on Amazon Web Service.

These guides include detailed operational procedures, ensuring that users can set up and operate their multi-site Keycloak instances efficiently.

Install this into a Keycloak server in a test environment, and create as many users, clients, groups, etc. as you need to run your performance benchmark. Keycloak caches a lot of information in its internal caches, and so does the database, so you will be able to spot some problems only when you have the right amount of data in your database.

This contains ready-to-be used scenarios for authentication flows and for Keycloak’s admin REST endpoints. If it does not fit your needs yet, use it as a library to create your own Gatling scenarios based on existing and custom steps. These tests are deployed as a JAR and a shell script wrapper, so you will only need to install Java on your load runners and you are ready to go.

Use these Ansible playbooks to spin up a set of EC2 instances to drive load against a Keycloak test installation, and aggregate the results.

Based on Red Hat OpenShift Service on AWS (ROSA), use the scripts to provision an instance with monitoring, logging and useful Operators preconfigured, ready to deploy Keycloak.

Set up an Aurora in different variants regional or global, and connect it to a ROSA environment.

This deploys Keycloak with additional monitoring and debugging tools so we can look at metrics, logs and traces as needed

Set up Route 53 for an active-passive setup to distribute the load to two Keycloak deployments in different OpenShift clusters

Every weekday we create a new Multi-AZ setup from scratch using GitHub actions, a performance testsuite, and record the results. This way we catch functional and performance regressions as they occur.

When running Keycloak under a high load, requests might queue up in a Keycloak instance. The more requests queue up, the longer it takes to reply to the requests. In previous versions also the requests to the liveness probe (/health/live) were queued, and the probe eventually timed out, and then Kubernetes restarted the Pod. In the latest version of Keycloak, the probe is re-implemented to be non-blocking, so it will not queue, and therefore will not time out and the Pod is not restarted under a high load.

When requests are queued as described above, the caller will not get a response in time, and the Pod might eventually run out of resources like memory or network connections. The recommended recipe is to drop requests early when an instance will not be able to serve the requests in time, which is called load shedding. Keycloak 23 now supports the new option http-max-queued-requests that can limit the number of concurrent blocking requests. When the number is exceeded, Keycloak immediately returns the response 503 Server not Available. This has two benefits: The caller receives an immediate response and can retry later, and resources are freed on the server side immediately.

When a new Keycloak instance starts or restarts, its caches are empty. If under high load parallel requests arrive for the same realm or the same client on a node of Keycloak, previous versions of Keycloak loaded the data from the database in each parallel request. This caused a spike in database connection usage and an initial response delay. The same happens when a cache or realm entry in the cache is evicted, for example, because it was modified. The latest version of Keycloak prevents this so that each Keycloak instance will fetch the data from the database once, and all other parallel requests then use this data without querying the database again (see #21521 and #22988, #24202).

The more Keycloak instances run in a cluster, and the more requests are processed in parallel, the higher is the load on the JGroups thread pool. The JGroups thread pool ensures smooth communication for the embedded Infinispan of Keycloak, and could lead to timeouts on the internal Infinispan communications if its capacity is exceeded. The high-availability docs now contain documentation on how to set the Quarkus thread pool to not exceed the JGroup thread pool.

The embedded Infinispan provides improved metrics that allow you to monitor your cluster. The metrics exposed by the Keycloak’s metrics endpoint now contain only Infinispan metrics for the current node, so they will not block if another Pod is currently starting up or shutting down (ISPN-15042 and ISPN-15072). This way you have better visibility of your cluster during those critical moments. The metrics can now expose the cache names as labels, so they can be plotted simpler in dashboards by adding a <metrics names-as-tags="true" /> to the Infinispan XML configuration. Additional metrics are available for the latencies between sites.

We tested Infinispan and its communication layer JGroups thoroughly, and we were able to fix situations where a state transfer stalled (ISPN-14982), or an initial state transfer failed. The Gossip router used in the multi-site setup now works even in situations where a load balancer has multiple IP addresses (JGRP-2722, JGRP-2721, infinispan-operator#1857, and infinispan-operator#1856).