Deploy Infinispan for HA with the Infinispan Operator

Building block for an Infinispan deployment on Kubernetes

This guide describes the procedures required to deploy Infinispan in a multiple-cluster environment (cross-site). For simplicity, this topic uses the minimum configuration possible that allows Keycloak to be used with an external Infinispan.

This guide assumes two OpenShift clusters named Site-A and Site-B.

This is a building block following the concepts described in the Concepts for multi-site deployments guide. See the Multi-site deployments guide for an overview.

Only Infinispan version 15.0.11.Final or more recent patch releases are supported for external Infinispan deployments.

Architecture

This setup deploys two synchronously replicating Infinispan clusters in two sites with a low-latency network connection. An example of this scenario could be two availability zones in one AWS region.

Keycloak, loadbalancer and database have been removed from the following diagram for simplicity.

infinispan crossdc az.dio

Prerequisites

Procedure

  1. Install the Infinispan Operator

  2. Configure the credential to access the Infinispan cluster.

    Keycloak needs this credential to be able to authenticate with the Infinispan cluster. The following identities.yaml file sets the username and password with admin permissions

    credentials:
  - username: developer
    password: strong-password
    roles:
      - admin

    The identities.yaml could be set in a secret as one of the following:

    • As a Kubernetes Resource:

      Credential Secret
      apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: connect-secret
  namespace: keycloak
data:
  identities.yaml: Y3JlZGVudGlhbHM6CiAgLSB1c2VybmFtZTogZGV2ZWxvcGVyCiAgICBwYXNzd29yZDogc3Ryb25nLXBhc3N3b3JkCiAgICByb2xlczoKICAgICAgLSBhZG1pbgo= (1)
      1 The identities.yaml from the previous example base64 encoded.

    • Using the CLI

      kubectl create secret generic connect-secret --from-file=identities.yaml

      Check the Configuring Authentication documentation for more details.

      These commands must be executed on both OpenShift clusters.

  3. Create a service account.

    A service account is required to establish a connection between clusters. The Infinispan Operator uses it to inspect the network configuration from the remote site and to configure the local Infinispan cluster accordingly.

    For more details, see the Managing Cross-Site Connections documentation.

    1. Create a service-account-token secret type as follows. The same YAML file can be used in both OpenShift clusters.

      xsite-sa-secret-token.yaml
      apiVersion: v1
kind: Secret
metadata:
  name: ispn-xsite-sa-token (1)
  annotations:
    kubernetes.io/service-account.name: "xsite-sa" (2)
type: kubernetes.io/service-account-token
      1 The secret name.
      2 The service account name.

    2. Create the service account and generate an access token in both OpenShift clusters.

      Create the service account in Site-A
      kubectl create sa -n keycloak xsite-sa
oc policy add-role-to-user view -n keycloak -z xsite-sa
kubectl create -f xsite-sa-secret-token.yaml
kubectl get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > Site-A-token.txt
      Create the service account in Site-B
      kubectl create sa -n keycloak xsite-sa
oc policy add-role-to-user view -n keycloak -z xsite-sa
kubectl create -f xsite-sa-secret-token.yaml
kubectl get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > Site-B-token.txt

    3. The next step is to deploy the token from Site-A into Site-B and the reverse:

      Deploy Site-B token into Site-A
      kubectl create secret generic -n keycloak xsite-token-secret \
  --from-literal=token="$(cat Site-B-token.txt)"
      Deploy Site-A token into Site-B
      kubectl create secret generic -n keycloak xsite-token-secret \
  --from-literal=token="$(cat Site-A-token.txt)"

  4. Create TLS secrets

    In this guide, Infinispan uses an OpenShift Route for the cross-site communication. It uses the SNI extension of TLS to direct the traffic to the correct Pods. To achieve that, JGroups use TLS sockets, which require a Keystore and Truststore with the correct certificates.

    For more information, see the Securing Cross Site Connections documentation or this Red Hat Developer Guide.

    Upload the Keystore and the Truststore in an OpenShift Secret. The secret contains the file content, the password to access it, and the type of the store. Instructions for creating the certificates and the stores are beyond the scope of this guide.

    To upload the Keystore as a Secret, use the following command:

    Deploy a Keystore
    kubectl -n keycloak create secret generic xsite-keystore-secret \
  --from-file=keystore.p12="./certs/keystore.p12" \ (1)
  --from-literal=password=secret \ (2)
  --from-literal=type=pkcs12 (3)
    1 The filename and the path to the Keystore.
    2 The password to access the Keystore.
    3 The Keystore type.

    To upload the Truststore as a Secret, use the following command:

    Deploy a Truststore
    kubectl -n keycloak create secret generic xsite-truststore-secret \
        --from-file=truststore.p12="./certs/truststore.p12" \  (1)
        --from-literal=password=caSecret \  (2)
        --from-literal=type=pkcs12  (3)
    1 The filename and the path to the Truststore.
    2 The password to access the Truststore.
    3 The Truststore type.
    Keystore and Truststore must be uploaded in both OpenShift clusters.

  5. Create a Cluster for Infinispan with Cross-Site enabled

    The Setting Up Cross-Site documentation provides all the information on how to create and configure your Infinispan cluster with cross-site enabled, including the previous steps.

    A basic example is provided in this guide using the credentials, tokens, and TLS Keystore/Truststore created by the commands from the previous steps.

    The Infinispan CR for Site-A
    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan (1)
  namespace: keycloak
  annotations:
    infinispan.org/monitoring: 'true' (2)
spec:
  replicas: 3
  jmx:
    enabled: true
  security:
    endpointSecretName: connect-secret (3)
  service:
    type: DataGrid
    sites:
      local:
        name: site-a (4)
        expose:
          type: Route (5)
        maxRelayNodes: 128
        encryption:
          transportKeyStore:
            secretName: xsite-keystore-secret (6)
            alias: xsite (7)
            filename: keystore.p12 (8)
          routerKeyStore:
            secretName: xsite-keystore-secret (6)
            alias: xsite (7)
            filename: keystore.p12 (8)
          trustStore:
            secretName: xsite-truststore-secret (9)
            filename: truststore.p12 (10)
      locations:
        - name: site-b (11)
          clusterName: infinispan
          namespace: keycloak (12)
          url: openshift://api.site-b (13)
          secretName: xsite-token-secret (14)
    1 The cluster name
    2 Allows the cluster to be monitored by Prometheus.
    3 If using a custom credential, configure here the secret name.
    4 The name of the local site, in this case Site-A.
    5 Exposing the cross-site connection using OpenShift Route.
    6 The secret name where the Keystore exists as defined in the previous step.
    7 The alias of the certificate inside the Keystore.
    8 The secret key (filename) of the Keystore as defined in the previous step.
    9 The secret name where the Truststore exists as defined in the previous step.
    10 The Truststore key (filename) of the Keystore as defined in the previous step.
    11 The remote site’s name, in this case Site-B.
    12 The namespace of the Infinispan cluster from the remote site.
    13 The OpenShift API URL for the remote site.
    14 The secret with the access token to authenticate into the remote site.

    For Site-B, the Infinispan CR looks similar to the above. Note the differences in point 4, 11 and 13.

    The Infinispan CR for Site-B
    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan (1)
  namespace: keycloak
  annotations:
    infinispan.org/monitoring: 'true' (2)
spec:
  replicas: 3
  jmx:
    enabled: true
  security:
    endpointSecretName: connect-secret (3)
  service:
    type: DataGrid
    sites:
      local:
        name: site-b (4)
        expose:
          type: Route (5)
        maxRelayNodes: 128
        encryption:
          transportKeyStore:
            secretName: xsite-keystore-secret (6)
            alias: xsite (7)
            filename: keystore.p12 (8)
          routerKeyStore:
            secretName: xsite-keystore-secret (6)
            alias: xsite (7)
            filename: keystore.p12 (8)
          trustStore:
            secretName: xsite-truststore-secret (9)
            filename: truststore.p12 (10)
      locations:
        - name: site-a (11)
          clusterName: infinispan
          namespace: keycloak (12)
          url: openshift://api.site-a (13)
          secretName: xsite-token-secret (14)

  6. Creating the caches for Keycloak.

    Keycloak requires the following caches to be present: actionTokens, authenticationSessions, loginFailures, and work.

    The Infinispan Cache CR allows deploying the caches in the Infinispan cluster. Cross-site needs to be enabled per cache as documented by Cross Site Documentation. The documentation contains more details about the options used by this guide. The following example shows the Cache CR for Site-A.

    1. In Site-A create a Cache CR for each of the caches mentioned above with the following content. This is an example for the authenticationSessions cache:

    apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
  name: authenticationsessions
  namespace: keycloak
spec:
  clusterName: infinispan
  name: authenticationSessions
  template: |-
    distributedCache:
      mode: "SYNC"
      owners: "2"
      statistics: "true"
      remoteTimeout: "5000"
      encoding:
        media-type: "application/x-protostream"
      locking:
        acquireTimeout: "4000"
      transaction:
        mode: "NON_XA" (1)
        locking: "PESSIMISTIC" (2)
      stateTransfer:
        chunkSize: "16"
      backups:
        site-b: (3)
          backup:
            strategy: "SYNC" (4)
            timeout: "4500" (5)
            failurePolicy: "FAIL" (6)
            stateTransfer:
              chunkSize: "16"
    1 The transaction mode.
    2 The locking mode used by the transaction.
    3 The remote site name.
    4 The cross-site communication strategy, in this case, SYNC.
    5 The cross-site replication timeout.
    6 The cross-site replication failure policy.

    The example above is the recommended configuration to achieve the best data consistency.

    Background information

    Deadlocks may occur in an active-active setup as entries are modified concurrently in both sites.

    The transaction.mode: NON_XA ensures that the transaction is rolled back keeping the data consistent if this occurs. The setting backup.failurePolicy: FAIL is required in this case. It will throw an error that allows the transaction to be safely rolled back. When this occurs, Keycloak will attempt a retry.

    The transaction.locking: PESSIMISTIC is the only supported locking mode; OPTIMISTIC is not recommended due to its network costs. The same settings also prevent that one site is updated while the other site is unreachable.

    The backup.strategy: SYNC ensures the data is visible and stored in the other site when the Keycloak request is completed.

    The locking.acquireTimeout can be reduced to fail fast in a deadlock scenario. The backup.timeout must always be higher than the locking.acquireTimeout.

    For Site-B, the Cache CR is similar, except for the backups.<name> outlined in point 3 of the above diagram.

    authenticationSessions Cache CR in Site-B
    apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
  name: authenticationsessions
  namespace: keycloak
spec:
  clusterName: infinispan
  name: authenticationSessions
  template: |-
    distributedCache:
      mode: "SYNC"
      owners: "2"
      statistics: "true"
      remoteTimeout: "5000"
      encoding:
        media-type: "application/x-protostream"
      locking:
        acquireTimeout: "4000"
      transaction:
        mode: "NON_XA" (1)
        locking: "PESSIMISTIC" (2)
      stateTransfer:
        chunkSize: "16"
      backups:
        site-a: (3)
          backup:
            strategy: "SYNC" (4)
            timeout: "4500" (5)
            failurePolicy: "FAIL" (6)
            stateTransfer:
              chunkSize: "16"

Verifying the deployment

Confirm that the Infinispan cluster is formed, and the cross-site connection is established between the OpenShift clusters.

Wait until the Infinispan cluster is formed
kubectl wait --for condition=WellFormed --timeout=300s infinispans.infinispan.org -n keycloak infinispan
Wait until the Infinispan cross-site connection is established
kubectl wait --for condition=CrossSiteViewFormed --timeout=300s infinispans.infinispan.org -n keycloak infinispan

Connecting Infinispan with Keycloak

Now that the Infinispan server is running, here are the relevant Keycloak CR changes necessary to connect it to Keycloak. These changes will be required in the Deploy Keycloak for HA with the Keycloak Operator guide.

  1. Create a Secret with the username and password to connect to the external Infinispan deployment:

    apiVersion: v1
kind: Secret
metadata:
  name: remote-store-secret
  namespace: keycloak
type: Opaque
data:
  username: ZGV2ZWxvcGVy # base64 encoding for 'developer'
  password: c2VjdXJlX3Bhc3N3b3Jk # base64 encoding for 'secure_password'

  2. Extend the Keycloak Custom Resource with additionalOptions as shown below.

    All the memory, resource and database configurations are skipped from the CR below as they have been described in the Deploy Keycloak for HA with the Keycloak Operator guide already. Administrators should leave those configurations untouched.

    		 
    apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  labels:
    app: keycloak
  name: keycloak
  namespace: keycloak
spec:
  additionalOptions:
    - name: cache-remote-host (1)
      value: "infinispan.keycloak.svc"
    - name: cache-remote-port (2)
      value: "11222"
    - name: cache-remote-username (3)
      secret:
        name: remote-store-secret
        key: username
    - name: cache-remote-password (4)
      secret:
        name: remote-store-secret
        key: password
    - name: spi-connections-infinispan-quarkus-site-name (5)
      value: keycloak
    1 The hostname of the remote Infinispan cluster.
    2 The port of the remote Infinispan cluster. This is optional and it defaults to 11222.
    3 The Secret name and key with the Infinispan username credential.
    4 The Secret name and key with the Infinispan password credential.
    5 The spi-connections-infinispan-quarkus-site-name is an arbitrary Infinispan site name which Keycloak needs for its Infinispan caches deployment when a remote store is used. This site-name is related only to the Infinispan caches and does not need to match any value from the external Infinispan deployment. If you are using multiple sites for Keycloak in a cross-DC setup such as Deploy Infinispan for HA with the Infinispan Operator, the site name must be different in each site.

Architecture

This connects Keycloak to Infinispan using TCP connections secured by TLS 1.3. It uses the Keycloak’s truststore to verify Infinispan’s server certificate. As Keycloak is deployed using its Operator on OpenShift in the prerequisites listed below, the Operator already added the service-ca.crt to the truststore which is used to sign Infinispan’s server certificates. In other environments, add the necessary certificates to Keycloak’s truststore.

Next steps

After the Aurora AWS database and Infinispan are deployed and running, use the procedure in the Deploy Keycloak for HA with the Keycloak Operator guide to deploy Keycloak and connect it to all previously created building blocks.

Relevant options

Value

cache-remote-host

The hostname of the external Infinispan cluster.

Available only when feature multi-site, clusterless or cache-embedded-remote-store is set.

CLI: --cache-remote-host
Env: KC_CACHE_REMOTE_HOST

cache-remote-password

The password for the authentication to the external Infinispan cluster.

It is optional if connecting to an unsecure external Infinispan cluster. If the option is specified, cache-remote-username is required as well.

CLI: --cache-remote-password
Env: KC_CACHE_REMOTE_PASSWORD

Available only when remote host is set

cache-remote-port

The port of the external Infinispan cluster.

CLI: --cache-remote-port
Env: KC_CACHE_REMOTE_PORT

Available only when remote host is set

11222 (default)

cache-remote-tls-enabled

Enable TLS support to communicate with a secured remote Infinispan server.

Recommended to be enabled in production.

CLI: --cache-remote-tls-enabled
Env: KC_CACHE_REMOTE_TLS_ENABLED

Available only when remote host is set

true (default), false

cache-remote-username

The username for the authentication to the external Infinispan cluster.

It is optional if connecting to an unsecure external Infinispan cluster. If the option is specified, cache-remote-password is required as well.

CLI: --cache-remote-username
Env: KC_CACHE_REMOTE_USERNAME

Available only when remote host is set
On this page
Edit this guide