Keycloak on ROSA Benchmark Key Results

This summarizes a benchmark run with Keycloak 25 release candidate build performed in May 2024. Use this as a starting point to calculate the requirements of a Keycloak environment. Use them to perform a load testing in your environment.

Collecting the CPU usage for refreshing a token is currently performed manually and is expected to be automated in the near future (keycloak/keycloak-benchmark#517).

Setup

This setup is run daily on a GitHub action schedule:

  • OpenShift 4.15.x deployed on AWS via ROSA with two AWS availability zones in AWS one region.

  • Machinepool with m5.4xlarge instances.

  • Keycloak 25 release candidate build deployed with Operator and 3 pods in each site as an active/passive setup, and Infinispan connecting the two sites.

  • Default user password hashing with Argon2 and 5 hash iterations and minimum memory size 7 MiB as recommended by OWASP.

  • Database seeded with 100,000 users and 100,000 clients.

  • Infinispan caches at default of 10,000 entries, so not all clients and users fit into the cache, and some requests will need to fetch the data from the database.

  • All sessions in distributed caches as per default, with two owners per entries, allowing one failing pod without losing data.

  • Database Amazon Aurora PostgreSQL in a multi-AZ setup, with the writer instance in the availability zone of the primary site.

Installation

Deploy OpenShift and ROSA as described in ROSA and OpenShift with

OpenShift .env file
# no KC_CPU_LIMITS set for this scenario
KC_CPU_REQUESTS=6
KC_INSTANCES=3
KC_DISABLE_STICKY_SESSION=true
KC_PERSISTENT_SESSIONS=false
KC_MEMORY_REQUESTS_MB=3000
KC_MEMORY_LIMITS_MB=4000
KC_DB_POOL_INITIAL_SIZE=30
KC_DB_POOL_MAX_SIZE=30
KC_DB_POOL_MIN_SIZE=30

Results

Refer to concepts-memory-and-cpu-sizing in main Keycloak docs for results on the latest release, calculation example, observations and recommendations. Updated information about the upcoming release are available in the main branch of the GitHub repository.

Tests performed

Each test ran for 10 minutes.

  1. Setup ROSA cluster as default.

  2. Scale machine pool.

    rosa edit machinepool -c  <clustername> --min-replicas 3 --autorepair scaling

  3. Deploy Keycloak and Monitoring

    cd provision/openshift
task
task monitoring

  4. Create dataset

    task dataset-import -- -a create-realms -u 100000
# wait for first task to complete
task dataset-import -- -a create-clients -c 100000 -n realm-0

  5. Prepare environment for running the benchmark via Ansible

    See Running benchmarks via Ansible and EC2 for details.

    Contents of env.yml used here
    cluster_size: 5
instance_type: t4g.small
instance_volume_size: 30
kcb_zip: ../benchmark/target/keycloak-benchmark-0.14-SNAPSHOT.zip
kcb_heap_size: 1G

  6. Create load runners

    cd ../../ansible
./aws_ec2.sh start <region of ROSA cluster>

  7. Run different load tests

    • Testing memory for creating sessions

      ./benchmark.sh eu-west-1 \
--scenario=keycloak.scenario.authentication.AuthorizationCode \
--server-url=${KEYCLOAK_URL} \
--realm-name=realm-0 \
--users-per-sec=<number of users per second> \
--ramp-up=20 \
--logout-percentage=0 \
--measurement=600 \
--users-per-realm=100000 \
--log-http-on-failure

    • Testing CPU usage for user logins

      ./benchmark.sh eu-west-1 \
--scenario=keycloak.scenario.authentication.AuthorizationCode \
--server-url=${KEYCLOAK_URL} \
--realm-name=realm-0 \
--users-per-sec=<number of users per second> \
--ramp-up=20 \
--logout-percentage=100 \
--measurement=600 \
--users-per-realm=100000 \
--log-http-on-failure

    • Testing CPU usage for logins and refreshing tokens with a ratio of 10 refreshes per one login. Use the previous test to deduct the CPU usage of logins only to get the CPU usage of token refreshes.

      ./benchmark.sh eu-west-1 \
--scenario=keycloak.scenario.authentication.AuthorizationCode \
--server-url=${KEYCLOAK_URL} \
--realm-name=realm-0 \
--users-per-sec=<number of users per second> \
--ramp-up=20 \
--logout-percentage=100 \
--refresh-token-count=10 \
--measurement=600 \
--users-per-realm=100000 \
--log-http-on-failure

    • Testing CPU usage for client credential grants

      ./benchmark.sh eu-west-1 \
--scenario=keycloak.scenario.authentication.ClientSecret \
--server-url=${KEYCLOAK_URL} \
--realm-name=realm-0 \
--users-per-sec=<number of clients per second> \
--ramp-up=20 \
--logout-percentage=100 \
--measurement=600 \
--users-per-realm=100000 \
--log-http-on-failure