Simulate Keycloak Site Failover

Prerequisites:

  • A Keycloak instance replicated across two ROSA clusters with Infinispan xsite and an Aurora DB

  • Task

Running the failure test from the CLI

Installation steps

  1. Check out the Git repository from https://github.com/keycloak/keycloak-benchmark/.

  2. Change to the folder provision/rosa-cross-dc.

  3. Provide the required variables for the desired task in a .env file in the folder.

Split Brain

Two tasks are provided for split-brain failover, one to cause a split-brain and the other to recover from it:

* split-heal:                             Heal a split-brain simulation by reenabling GossipRouters and reseting the global accelerator endpoints
* split-trigger:                          Disables Infinispan GossipRouters on each cluster to simulate a split-brain scenario
Variable Details

ACCELERATOR_NAME

The unique name of the AWS Global Accelerator used by the deployment.

ROSA_CLUSTER_NAME_1

The name of one ROSA cluster.

ROSA_CLUSTER_NAME_2

The name of the other ROSA cluster.

Cluster Failure

Two tasks are provided for simulating cluster failures. One to aggressively delete the Keycloak and Infinispan StatefulSets, the other to recreate them:

* kill-cluster:                           Delete the Keycloak and Infinispan StatefulSets in the specified cluster
* revive-cluster:                         Recreate the Keycloak and Infinispan StatefulSets in the specified cluster

The kill-cluster task disables both the Keycloak and Infinispan Operators to prevent the StatefulSets being recreated automatically. Conversely, the revive-cluster re-enables the Operators to allow the StatefulSets to be recreated.

Variable Details

ROSA_CLUSTER_NAME

The name of the ROSA cluster.