Analyzing the Continuous Performance Test results

A utility to perform custom analysis on the generated results from continuous performance tests.

Continuous Performance Tests

The ROSA Daily Scheduled Run workflow is an automated process that ensures continuous performance testing which is scheduled to run every weekday.

This workflow initiates by deploying a multi-AZ cluster, activating features such as external Infinispan and persistent sessions. It executes a series of functional tests to verify the system’s performance and stability in an active-active configuration.

Following these tests, a scaling benchmark assesses the system’s ability to handle varying loads, providing crucial insights into performance under real-world conditions and the results are then persisted to a result-data branch in the benchmark’s GitHub repository for further analysis.

This automated schedule ensures consistent testing, early detection of potential issues, and continuous improvement of Keycloak’s performance.

Analyze the results

We have a Python script perfInsights.py which allows us to analyze the results generated from the Continuous Performance tests mentioned above.

Installing needed python libraries

pip3 install -U -r requirements.txt

And we can check if all the requirements are satisfied using the below command.

python3 -m pip check

Usage

Run the below command to access the results from the result-data branch and save the analysis to a local dir.

python3 perfInsights.py <result_json_dir_path_on_github> <output_dir_path>
Example:
python3 perfInsights.py rosa_scalability/2024/07/23 results

Visualize the results

We can use the generate_timeseries_plots.py script to visualize the data generated from the above steps.

Usage

Run the below command to visualize the results from the results local directory.

python3 generate_timeseries_plots.py <results_dir_path>
Example:
python3 generate_timeseries_plots.py results
Saved combined plot for memoryUsageTest to results/memoryUsageTest_combined_timeseries_plot.png
Saved combined plot for cpuUsageForLoginsTest to results/cpuUsageForLoginsTest_combined_timeseries_plot.png
Saved combined plot for cpuUsageForCredentialGrantsTest to results/cpuUsageForCredentialGrantsTest_combined_timeseries_plot.png