November 14 2024 by Lukas Hanusovsky
The idea to replace the current test suite has been on the table for multiple years. Initially, it was meant to be only a refactoring of the current approach on how to write tests, but after a few internal discussions and refactor updates it turned out a new test suite, based on a new framework would be a better solution.
It would be good to mention a few drawbacks, that stand out when working with the current test suite. First of all, is the complexity of various configurations and additions made on top of the Arquillian framework. These changes make the test suite powerful, but the cons is that without proper documentation for beginners is almost unreadable. The second thing has the same importance, the Arquillian framework is not fully supported anymore. Other things to mention are a complicated execution system, where you want to specify what exactly should be tested, then abstract classes with shared configurations and missing the option to add a custom extension.
The Keycloak team began an effort to design a new test framework in May 2024. It started with a prototype to verify if our ideas were feasible. The prototype is a JUnit5 Extension based on the JUnit5 testing framework, specifically to implement JUnit5 callback classes which extend the default test lifecycle functionality and provide custom inject annotations, like @KeycloakIntegrationTest, @InjectWebDriver or @InjectRealm.
After a successful test round, we’ve continued with a proof of concept extending features list to support multiple server modes, different databases and WebDrivers, clients and users setup, SmallRye configuration support, OAuthClient based on Nimbus SDK (this feature is a preview only) etc. The full list of currently implemented features is:
Server lifecycle
Database lifecycle
Admin client injection
Realm, User, Client lifecycle and injection
Event and Admin event listener and injection
OAuth client injection
WebDriver lifecycle and injection
Page injection
Support for the Chrome, Firefox and HtmlUnit4 browsers
Database modules
It is already present in the main branch and Keycloak nightly builds.
We suggest reading the user guide, which will provide a basic overview of how the framework works and should be used. If this is not enough, you can also check test examples.
For extension developers we recommend to look into an example on how to start Keycloak with their custom provider: provider example, pom.xml test dependency and test example.
If you find a bug, want to discuss something, or propose a new enhancement, please follow this GitHub feedback discussion link.
We already have enough capabilities in the new test framework to start migrating some tests from the old testsuite; and in fact already have our very first test migrated. We plan to migrate one package at a time from the old testsuite starting with the admin tests, then moving on to the forms and oauth packages. As we are doing this we will expand on the capabilities of the test framework.
Some features we know will be coming soon included:
An easier way to deploy custom providers, not requiring a Maven build of the provider first
Improved logging, making it easy to configure logging from tests as well as Keycloak
Easy testing of OAuth and OpenID Connect, including a mock application
Extension to allow running code on the tested server when it’s not possible to easily test through only remote interfaces
We also have some more long term plans to deliver:
Provider tests that can be used to easily test a provider by invoking the provider directly
Parallel execution of tests, to take full advantage of multiple cores to reduce test execution time
I would like to thank all the people who put the proof of concept together and made it real: Miquel, Simon, Filip, Moises, Jon, and Pedro. A special thank-you goes to Stian, who led the technical design and proposed very nifty things that raised the project to another level.
Thank you for your feedback.
Enjoy!