Results summary

There shell script creates a summary JSON which can be used to compare different runs.

JSON created for each run

For every run, kcb.sh creates a file results.json which contains the input parameters to Gatling and also an output of the results.

Each result summarizes a run, and can be processed further in a pipeline. Each run has a descriptive name and a UUID which identifies the run. The file name contains a timestamp and the UUID, so it is simple to put multiple result files in a single folder for further processing.

A user of the kcb.sh script might want to capture information about the system under test (SUT) as part of the result file to record the information for later use. To do this, the user has to provide a script which collects the information and prints them on the console. For a ROSA environment, there is rosa_describe.sh which would collect the information about the cluster. When the environment variable SUT_DESCRIBE is set, kcb.sh calls the executable and CLI arguments given in the environment variable SUT_DESCRIBE to retrieve the JSON description for the system under test. It then adds it to the file with the key system_under_test. See oc-keycloak-login/action.yml how the environment variable is set for a GitHub workflow.

When running this from a GitHub Workflow, the file name is recorded in the GitHub Action output kcb_result so it can be picked up in later steps in the GitHub Workflow.

Example excerpt from a results JSON file.
{
  "uuid": ...,
  "name": ...,
  "grafana_input": {
    "start": ...,
    "end": ...,
    "input": ...
  },
  "grafana_output": ...,
  "system_under_test": ...
}

Summarizing report for the scalability test

Until August 2023, we’ve been running these tests continuously. We found that the numbers weren’t as actionable as we would like them to be. So we’re looking towards Keycloak on ROSA Benchmark Key Results and having those automated in the future. This content will soon be removed.

For the scalability test, there is a report visualizing the latest runs here.

The process that leads to this diagram is as follows:

  1. The Keycloak - Scalability benchmark GitHub workflow commits a result file to the result-data branch of the repository on each successful run.

  2. For Dashbuilder, the data needs to be available as a single JSON file. This is generated using the Aggregate Benchmark results GitHub workflow in the result-data branch of the repository.

  3. The data is then published to GitHub pages in the Publishing Documentation Site GitHub workflow, as the file created for the releases can’t be fetched by Dashbuilder, as the initial response is an HTTP redirect.

  4. The dashboard is built using Dashbuilder using this dashboard.yaml file. There is an interactive dashboard builder to interactively change the diagram. See the Dashbuilder docs on how to update the YAML file.