1. Guide Overview

The purpose of this guide is to walk through the steps that need to be completed prior to booting up the Aerobase server for the first time. If you just want to test drive Aerobase, it pretty much runs out of the box with its own embedded and local-only database. For actual deployments that are going to be run in production you’ll need to decide how you want to manage server configuration at runtime (standalone or cluster mode), configure a shared database for Aerobase storage, set up encryption and HTTPS, and finally set up Aerobase to run in a cluster. This guide walks through each and every aspect of any pre-boot decisions and setup you must do prior to deploying the server.

2. Installation

Installing Aerobase is as simple as downloading it and installing OS package (deb|rpm|msi). This chapter reviews system requirements as well as the directory structure of the distribution.

2.1. System requirements

These are the requirements to run the Aerobase authentication server:

  • Can run on any operating system that runs Java

  • Java 8 JDK

  • At least 512M of RAM

  • At least 1G of diskspace

  • A shared external database like PostgreSQL, MySQL, Oracle, etc. Aerobase requires an external shared database if you want to run in a cluster. Please see the database configuration section of this guide for more information.

  • On Linux, it is recommended to use /dev/urandom as a source of random data to prevent Aerobase hanging due to lack of available entropy, unless /dev/random usage is mandated by your security policy. To achieve that on Oracle JDK 8 and OpenJDK 8, set the java.security.egd system property on startup to file:/dev/urandom.

2.2. Installing distribution files

Please see the Installing Distribution Files section of this guide for more information.

2.3. Distribution directory structure

This chapter walks you through the directory structure of the server distribution.

distribution directory structure

distribution

Let’s examine the purpose of some of the directories:

/opt/aerobase/ - C:/Aerobase/

This contains Aerobase core software and its dependencies.

/var/opt/aerobase/ - C:/Aerobase/Data/

This contains application data and configuration files that aerobase-ctl reconfigure writes to.

/etc/aerobase/ - C:/Aerobase/Configuration/

This contains configuration files for aerobase-ctl. These are the only files that you should ever have to edit manually.

/var/log/aerobase - C:/Aerobase/Logs/

This contains all log data generated by components of aerobase-ctl.

/etc/aerobase/ssl/ - C:/Aerobase/Configuration/ssl

This contains ssl configuration files for running Aerobase with https.

/var/opt/aerobase/aerobase-server/themes/ - C:/Aerobase/Data/aerobase-server/themes/

This directory contains all the html, style sheets, JavaScript files, and images used to display any UI screen displayed by the server. Here you can modify an existing theme or create your own. See the Server Developer Guide for more information on this.

3. Choosing an Operating Mode

Before deploying Aerobase in a production environment you need to decide which type of operating mode you are going to use. Will you run Aerobase within a cluster? Do you want a centralized way to manage your server configurations? Your choice of operating mode effects how you configure databases, configure caching and even how you boot the server.

The Aerobase is built on top of the Infinispan distributed in-memory key/value data store. Runtime Data is automatically replicated to multiple nodes for fault-tolerance and linear scalability. Replication across multiple data centers is supported. This guide will only go over the basics for deployment within a specific mode.

3.1. Standalone Mode

Standalone operating mode is only useful when you want to run one, and only one Aerobase server instance. It is not usable for clustered deployments and all caches are non-distributed and local-only. It is not recommended that you use standalone mode in production as you will have a single point of failure. If your standalone mode server goes down, users will not be able to log in. This mode is really only useful to test drive and play with the features of Aerobase

3.1.1. Standalone Boot

You can start, stop or restart Aerobase Server and all of its components with the following commands. These scripts live in the bin/ directory of the server distribution.

aerobase-ctl

bin files

To boot the server:

Linux/Unix
$ sudo aerobase-ctl start
Windows
> aerobase-ctl.bat start

3.1.2. Aerobase Configuration

The bulk of this guide walks you through how to configure infrastructure level aspects of Aerobase. These aspects are configured in a configuration file that is specific to the application server that Aerobase is a derivative of. In the standalone operation mode, this file lives in /etc/aerobase/aerobase.rb. This file is also used to configure non-infrastructure level things that are specific to Aerobase components.

Aerobase Config File

config file

Any changes you make to this file while the server is running will not take effect. Run 'sudo aerobase-ctl reconfigure' for the change to take effect.
Linux/Unix
$ sudo aerobase-ctl reconfigure
Windows
> aerobase-ctl.bat reconfigure

3.2. Clustered Mode

Clustered operation mode is for when you want to run Aerobase within a cluster. This mode requires that you have a copy of the Aerobase distribution on each machine you want to run a server instance. This mode is very easy to deploy, and require a few configuration cahnge.

The recommended network architecture for deploying Aerobase is to set up an HTTP/HTTPS load balancer on a public IP address that routes requests to Aerobase servers sitting on a private network. This isolates all clustering connections and provides a nice means of protecting the servers.

3.2.2. Clustered Configuration

The distribution has a mostly pre-configured app server configuration file for running within a cluster. It has all the specific infrastructure settings for networking, databases, caches, and discovery. This file resides in /etc/aerobase/aerobase.rb. There’s a few things missing from this configuration. You can’t run Aerobase in a cluster without configuring a shared database connection. You also need to deploy some type of load balancer in front of the cluster. The clustering and database sections of this guide walk you though these things.

cat aerobase.rb
global['contactpoints'] = 'IP1, IP2'
Any changes you make to this file while the server is running will not take effect. When using host names as 'contactpoints', make sure hostname is resolved to a network address. See the WildFly 16 Documentation for more information.

3.2.3. Clustered Boot Script

You use the same boot scripts to start Aerobase as you do in standalone mode.

To boot the server:

Linux/Unix
$ aerobase-ctl start
Windows
> aerobase-ctl.bat start

4. Relational Database Setup

Aerobase comes with its own embedded relational database called PostgreSQL. This is the default database that Aerobase will use to persist data and really only exists so that you can run the authentication server out of the box. We highly recommend that you replace it with a more production ready external database.

Aerobase uses two layered technologies to persist its relational data. The bottom layered technology is JDBC. JDBC is a Java API that is used to connect to a RDBMS. There are different JDBC drivers per database type that are provided by your database vendor. This chapter discusses how to configure Aerobase to use one of these vendor-specific drivers.

The top layered technology for persistence is Hibernate JPA. This is a object to relational mapping API that maps Java Objects to relational data. Most deployments of Aerobase will never have to touch the configuration aspects of Hibernate, but we will discuss how that is done if you run into that rare circumstance.

4.1. Supported databases

Due to variations in data types and SQL syntax, the following databases are currently supported out of the box. Additional databases as well as enhancements to support new databases are available through Aerobase support site.

  1. PostgreSQL

  2. MySQL

  3. MSSQL

  4. Oracle

This chapter will use PostgresSQL for all its examples. Other databases follow the same steps for installation.

4.2. Connecting to external database

By default Aerobase reconfigure process will generate SQL Users/Schema/Tables required for runtime. If required object is already exists in database, it will be skiped. Therefore it is safe to run reconfigure command multiple times.

Configuration username/password are used only during the reconfigure process.

4.2.1. Connecting to PostgreSQL

# Shutdown internal embedded database
postgresql['enable'] = false

# Point to external database
postgresql['server'] = "DATABASE-IP"
postgresql['username'] = "USER_NAME"
postgresql['password'] = "PASSWORD"

4.2.2. Connecting to MSSQL

# Shutdown internal embedded database
postgresql['enable'] = false

# Change adapter type
aerobase_server['db_adapter'] = "mssql"

# Point to MSSQL host and port (default 1433)
mssql['server'] = "localhost"
mssql['port'] = 1433
mssql['instance'] = nil
If both port and instance are specified, port will be preferred.
Connecting using Windows Logon
mssql['logon'] = false
Any changes you make to this file while the server is running will not take effect. Run 'sudo aerobase-ctl reconfigure' for the change to take effect.

4.3. Unicode considerations for Databases

Database schema in Aerobase only accounts for Unicode strings in the following special fields:

  • Realms: display name, HTML display name

  • Federation Providers: display name

  • Users: username, given name, last name, attribute names and values

  • Groups: name, attribute names and values

  • Roles: name

  • Descriptions of objects

Otherwise, characters are limited to those contained in database encoding which is often 8-bit. However, for some database systems, it is possible to enable UTF-8 encoding of Unicode characters and use full Unicode character set in all text fields. Often, this is counterbalanced by shorter maximum length of the strings than in case of 8-bit encodings.

Some of the databases require special settings to database and/or JDBC driver to be able to handle Unicode characters. Please find the settings for your database below. Note that if a database is listed here, it can still work properly provided it handles UTF-8 encoding properly both on the level of database and JDBC driver.

Technically, the key criterion for Unicode support for all fields is whether the database allows setting of Unicode character set for VARCHAR and CHAR fields. If yes, there is a high chance that Unicode will be plausible, usually at the expense of field length. If it only supports Unicode in NVARCHAR and NCHAR fields, Unicode support for all text fields is unlikely as Keycloak schema uses VARCHAR and CHAR fields extensively.

4.3.1. Oracle

Unicode characters are properly handled provided the database was created with Unicode support in VARCHAR and CHAR fields (e.g. by using AL32UTF8 character set as the database character set). No special settings is needed for JDBC driver.

If the database character set is not Unicode, then to use Unicode characters in the special fields, the JDBC driver needs to be configured with the connection property oracle.jdbc.defaultNChar set to true. It might be wise, though not strictly necessary, to also set the oracle.jdbc.convertNcharLiterals connection property to true. These properties can be set either as system properties or as connection properties. Please note that setting oracle.jdbc.defaultNChar may have negative impact on performance. For details, please refer to Oracle JDBC driver configuration documentation.

4.3.2. Microsoft SQL

Unicode characters are properly handled only for the special fields. No special settings of JDBC driver or database is necessary.

4.3.3. MySQL

Unicode characters are properly handled provided the database was created with Unicode support in VARCHAR and CHAR fields in the CREATE DATABASE command (e.g. by using utf8 character set as the default database character set in MySQL 5.5. Please note that utf8mb4 character set does not work due to different storage requirements to utf8 character set. Note that in this case, length restriction to non-special fields does not apply because columns are created to accommodate given amount of characters, not bytes. If the database default character set does not allow storing Unicode, only the special fields allow storing Unicode values.

At the side of JDBC driver settings, it is necessary to add a connection property characterEncoding=UTF-8 to the JDBC connection settings.

4.3.4. PostgreSQL

Unicode is supported when the database character set is UTF8. In that case, Unicode characters can be used in any field, there is no reduction of field length for non-special fields. No special settings of JDBC driver is necessary.

5. HTTP/HTTPS Access

Aerobase comes with its own embedded reverse proxy named NGINX.

5.1. Enable HTTPS

The Nginx config will tell browsers and clients to only communicate with your Aerobase instance over a secure connection for the next 24 months. By enabling HTTPS you’ll need to provide a secure connection to your instance for at least the next 24 months. By default, Aerobase does not use HTTPS. If you want to enable HTTPS for aerobase.example.com, add the following statement to /etc/aerobase/aerobase.rb:

# note the 'https' below
external_url "https://aerobase.example.com"

Because the hostname in our example is 'aerobase.example.com', aerobase will look for key and certificate files called /etc/aerobase/ssl/aerobase.example.com.key and /etc/aerobase/ssl/aerobase.example.com.crt respectively. Create the /etc/aerobase/ssl directory and copy your key and certificate there.

sudo mkdir -p /etc/aerobase/ssl
sudo chmod 700 /etc/aerobase/ssl
sudo cp aerobase.example.com.key aerobase.example.com.crt /etc/aerobase/ssl/

For self-signed certificate, we can create the SSL key and certificate files in one motion by typing:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/aerobase/ssl/aerobase.example.com.key -out /etc/aerobase/ssl/aerobase.example.com.crt

5.2. Redirect HTTP to HTTPS

By default, when you specify an external_url starting with 'https', Nginx will no longer listen for unencrypted HTTP traffic on port 80. If you want to redirect all HTTP traffic to HTTPS you can use the redirect_http_to_https setting.

external_url "https://aerobase.example.com"
nginx['redirect_http_to_https'] = true

5.3. Change The Default Port

If you need to use an HTTPS port other than the default (443), just specify it as part of the external_url.

external_url "https://aerobase.example.com:2443"

5.3.1. Change SSL certificate locations

To set the location of ssl certificates create /etc/aerobase/ssl directory, place the .crt and .key files in the directory and specify the following configuration:

nginx['ssl_certificate'] = "/etc/aerobase/ssl/other.example.crt"
nginx['ssl_certificate_key'] = "/etc/aerobase/ssl/other.example.com.key"

5.3.2. Setting the listen addresses

By default NGINX will accept incoming connections on all local IPv4 addresses. You can change the list of addresses in /etc/aerobase/aerobase.rb.

nginx['listen_addresses'] = ["0.0.0.0", "[::]"] # listen on all IPv4 and IPv6 addresses

5.3.3. Setting the NGINX listen port

By default NGINX will listen on the port specified in external_url or implicitly use the right port (80 for HTTP, 443 for HTTPS). If you are running Aerobase behind a reverse proxy, you may want to override the listen port to something else. For example, to use port 8181:

nginx['listen_port'] = 8181

5.3.4. Using a non-bundled web-server

By default, Aerobase is installed with bundled Nginx. Aerobase allows webserver access through user aerobase-www which resides in the group with the same name. To allow an external webserver access to Aerobase, external webserver user needs to be added aerobase-www group.

To use another web server like Apache or an existing Nginx installation you will have to perform the following steps:

nginx['enable'] = false

5.4. Additional Config

5.4.1. Adding additional Directives to nginx config

By default, Aerobase config file contains global nginx directives, e.g: worker_processes, events, etc. You can pugin additional directives to Aerobase server by setting a value to custom_nginx_config in /etc/aerobase/aerobase.rb; as it does in the server block below.

nginx['custom_nginx_config'] = "include /var/opt/aerobase/nginx/conf.d/additional-nginx-directives.import;"

5.4.2. Adding additional Directives to Http Blocks

By default, Aerobase http config block contains an include directive which tells NGINX where additional website configuration files are located. You can pugin additional Directives to Aerobase Http by setting a value to custom_http_config in /etc/aerobase/aerobase.rb; as it does in the server block below.

nginx['custom_http_config'] = "include /var/opt/aerobase/nginx/conf.d/additional-http-directives.import;"

5.4.3. Adding additional Location Blocks to Aerobase

The location setting lets you configure how NGINX will respond to requests for resources within the server. You can pugin additional locations file to Aerobase server by setting a value to custom_aerobase_config in /etc/aerobase/aerobase.rb; as it does in the server block below.

nginx['custom_aerobase_config'] = "include /var/opt/aerobase/nginx/conf.d/additional-location.import;"

6. Configuration Options

Aerobase Server is configured by setting relevant options in /etc/aerobase/aerobase.rb. For a complete list of available options, visit the aerobase.rb.template.

6.1. Configuring the external URL

In order for Aerobase to properly validete oauth requests it needs to know the URL under which it is reached by your users, e.g. http://idp.aerobase.com. Add or edit the following line in /etc/aerobase/aerobase.rb:

external_url "http://idp.aerobase.com"
Any changes you make to this file while the server is running will not take effect. Run sudo aerobase-ctl reconfigure for the change to take effect.

6.2. Rename Aerobase services

Aerobase service names can be labled with your company name, Add or edit the following line in C:\Aerobase\Configuration\aerobase.rb

global['srv_label'] = "YOURCOMPNAME"
Service renaming is supported only for Windows installation.

6.3. External configuration file

Aerobase packages loads all configuration from /etc/aerobase/aerobase.rb file. This file has strict file permissions and is owned by the root user. The reason for strict permissions and ownership is that /etc/aerobase/aerobase.rb is being executed as Ruby code by the root user during aerobase-ctl reconfigure. This means that users who have write access to /etc/aerobase/aerobase.rb can add configuration that will be executed as code by root.

You can include an external configuration file overrides.rb inside /etc/aerobase/. Any configuration that is set in /etc/aerobase/aerobase.rb will take precedence over the configuration from the included overrides.rb file.

6.4. Changing user / group

6.4.1. Changing the name of the Aerobase user / group

By default, Aerobase uses the user name aerobase for ownership of the Aerobase data itself.

We do not recommend changing the user/group of an existing installation because it can cause unpredictable side-effects. If you still want to do change the user and group, you can do so by adding the following lines to /etc/aerobase/aerobase.rb.

user['username'] = "aerobase"
user['group'] = "aerobase"

6.4.2. Specify numeric user and group identifiers

Aerobase creates users for Aerobase-server, PostgreSQL and NGINX. You can specify the numeric identifiers for these users in /etc/aerobase/aerobase.rb as follows.

user['uid'] = 1234
user['gid'] = 1234

postgresql['uid'] = 1235
postgresql['gid'] = 1235

web_server['uid'] = 1237
web_server['gid'] = 1237

6.5. Aerobase Delayed start

6.5.1. Linux delayed start

If you want to prevent Aerobase services (Aerobase-server, NGINX, PostgresSQL) from starting before a given filesystem is mounted, add the following to /etc/aerobase/aerobase.rb:

# wait for /var/opt/aerobase to be mounted
high_availability['mountpoint'] = '/var/opt/aerobase'

6.5.2. Windows delayed start

# Delayed start service on windows
aerobase_server['delayed_start'] = false

postgresql['delayed_start'] = false

nginx['delayed_start'] = false

6.6. Load additional realm

Place your custom realm.json file under /etc/aerobase/

In order for Aerobase to load custom realm it needs to know the PATH under which it is located, e.g. /etc/aerobase/your-realm.json. Add or edit the following line in /etc/aerobase/aerobase.rb:

keycloak_server['realm_files'] = ['/etc/aerobase/your-realm.json']

6.7. Disable theme caching

Add or edit the following line in /etc/aerobase/aerobase.rb:

keycloak_server['theme_cache'] = false

7. Clustering

This section covers configuring Aerobase to run in a cluster. There’s a number of things you have to do when setting up a cluster, specifically:

Picking an operation mode and configuring a shared database have been discussed earlier in this guide. In this chapter we’ll discuss setting up a load balancer and supplying a private network. We’ll also discuss some issues that you need to be aware of when booting up a host in the cluster.

The recommended network architecture for deploying Aerobase is to set up an HTTP/HTTPS load balancer on a public IP address that routes requests to Aerobase servers sitting on a private network. This isolates all clustering connections and provides a nice means of protecting the servers.

By default, there is nothing to prevent unauthorized nodes from joining the cluster and broadcasting multicast messages. This is why cluster nodes should be in a private network, with a firewall protecting them from outside attacks.

7.2. Setting Up a Load Balancer or Proxy

This section discusses a number of things you need to configure before you can put a reverse proxy or load balancer in front of your clustered Aerobase deployment. It also covers configuring the built in load balancer that was Clustered Domain Example.

7.2.1. Identifying Client IP Addresses

A few features in Aerobase rely on the fact that the remote address of the HTTP client connecting to the authentication server is the real IP address of the client machine. Examples include:

  • Event logs - a failed login attempt would be logged with the wrong source IP address

  • SSL required - if the SSL required is set to external (the default) it should require SSL for all external requests

  • Authentication flows - a custom authentication flow that uses the IP address to for example show OTP only for external requests

  • Dynamic Client Registration

7.2.2. Verify Configuration

You can verify the reverse proxy or load balancer configuration by opening the path /auth/realms/aerobase/.well-known/openid-configuration through the reverse proxy. For example if the reverse proxy address is https://acme.com/ then open the URL https://acme.com/auth/realms/aerobase/.well-known/openid-configuration. This will show a JSON document listing a number of endpoints for Aerobase. Make sure the endpoints starts with the address (scheme, domain and port) of your reverse proxy or load balancer. By doing this you make sure that Aerobase is using the correct endpoint.

7.3. Sticky sessions

Typical cluster deployment consists of the load balancer (reverse proxy) and 2 or more Aerobase servers on private network. For performance purposes, it may be useful if load balancer forwards all requests related to particular browser session to the same Aerobase backend node.

The reason is, that Aerobase is using Infinispan distributed cache under the covers for save data related to current authentication session and user session. The Infinispan distributed caches are configured with one owner by default. That means that particular session is saved just on one cluster node and the other nodes need to lookup the session remotely if they want to access it.

For example if authentication session with ID 123 is saved in the Infinispan cache on node1, and then node2 needs to lookup this session, it needs to send the request to node1 over the network to return the particular session entity.

It is beneficial if particular session entity is always available locally, which can be done with the help of sticky sessions. The workflow in the cluster environment with the public frontend load balancer and two backend Aerobase nodes can be like this:

  • User sends initial request to see the Aerobase login screen

  • This request is served by the frontend load balancer, which forwards it to some random node (eg. node1). Strictly said, the node doesn’t need to be random, but can be chosen according to some other criterias (client IP address etc). It all depends on the implementation and configuration of underlying load balancer (reverse proxy).

  • Aerobase creates authentication session with random ID (eg. 123) and saves it to the Infinispan cache.

  • Infinispan distributed cache assigns the primary owner of the session based on the hash of session ID. See Infinispan documentation for more details around this. Let’s assume that Infinispan assigned node2 to be the owner of this session.

  • Aerobase creates the cookie AUTH_SESSION_ID with the format like <session-id>.<owner-node-id> . In our example case, it will be 123.node2 .

  • Response is returned to the user with the Aerobase login screen and the AUTH_SESSION_ID cookie in the browser

From this point, it is beneficial if load balancer forwards all the next requests to the node2 as this is the node, who is owner of the authentication session with ID 123 and hence Infinispan can lookup this session locally. After authentication is finished, the authentication session is converted to user session, which will be also saved on node2 because it has same ID 123 .

The sticky session is not mandatory for the cluster setup, however it is good for performance for the reasons mentioned above. You need to configure your loadbalancer to sticky over the AUTH_SESSION_ID cookie. How exactly do this is dependent on your loadbalancer.

7.4. Securing Cluster Communication

When cluster nodes are isolated on a private network it requires access to the private network to be able to join a cluster or to view communication in the cluster. In addition you can also enable authentication and encryption for cluster communication. As long as your private network is secure it is not necessary to enable authentication and encryption. Aerobase does not send very sensitive information on the cluster in either case.

7.5. Serialized Cluster Startup

Aerobase cluster nodes are allowed to boot concurrently. When Aerobase server instance boots up it may do some database migration, importing, or first time initializations. A DB lock is used to prevent start actions from conflicting with one another when cluster nodes boot up concurrently.

By default, the maximum timeout for this lock is 900 seconds. If a node is waiting on this lock for more than the timeout it will fail to boot. Typically you won’t need to increase/decrease the default value, but just in case it’s possible to configure it in /etc/aerobase/aerobase.rb

keycloak_server['start_timeout'] = "600"

7.6. Troubleshooting

  • Note that when you run a cluster, you should see message similar to this in the log of both cluster nodes:

    INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-10,shared=udp)
    ISPN000094: Received new cluster view: [node1/keycloak|1] (2) [node1/keycloak, node2/keycloak]

    If you see just one node mentioned, it’s possible that your cluster hosts are not joined together.

    Usually it’s best practice to have your cluster nodes on private network without firewall for communication among them. Firewall could be enabled just on public access point to your network instead. If for some reason you still need to have firewall enabled on cluster nodes, you will need to open some ports. Default values are TCP port 7600. Note that you may need more ports opened if you want to enable additional features like diagnostics for your JGroups stack. Aerobase delegates most of the clustering work to Infinispan/JGroups. For more information, see JGroups in the WildFly 16 Documentation.

  • If you are interested in failover support (high availability), evictions, expiration and cache tuning, see Server Cache Configuration.

8. Server Cache Configuration

Aerobase has two types of caches. One type of cache sits in front of the database to decrease load on the DB and to decrease overall response times by keeping data in memory. Realm, client, role, and user metadata is kept in this type of cache. This cache is a local cache. Local caches do not use replication even if you are in the cluster with more Aerobase servers. Instead, they only keep copies locally and if the entry is updated an invalidation message is sent to the rest of the cluster and the entry is evicted. There is separate replicated cache work, which task is to send the invalidation messages to the whole cluster about what entries should be evicted from local caches. This greatly reduces network traffic, makes things efficient, and avoids transmitting sensitive metadata over the wire.

The second type of cache handles managing user sessions, offline tokens, and keeping track of login failures so that the server can detect password phishing and other attacks. The data held in these caches is temporary, in memory only, but is possibly replicated across the cluster.

This chapter discusses some configuration options for these caches for both clustered a non-clustered deployments.

More advanced configuration of these caches can be found in the Infinispan section of the WildFly 16 Documentation.

8.1. Eviction and Expiration

There are multiple different caches configured for Aerobase. There is a realm cache that holds information about secured applications, general security data, and configuration options. There is also a user cache that contains user metadata. Both caches default to a maximum of 10000 entries and use a least recently used eviction strategy. Each of them is also tied to an object revisions cache that controls eviction in a clustered setup. This cache is created implicitly and has twice the configured size. The same applies for the authorization cache, which holds the authorization data. The keys cache holds data about external keys and does not need to have dedicated revisions cache. Rather it has expiration explicitly declared on it, so the keys are periodically expired and forced to be periodically downloaded from external clients or identity providers.

The eviction policy and max entries for these caches can be configured in the /etc/aerobase/aerobase.rb.

keycloak_server['cache_size'] = 10000

In addition, there are also separate caches sessions, clientSessions, offlineSessions, offlineClientSessions, loginFailures and actionTokens. These caches are distributed in cluster environment and they are unbounded in size by default. If they are bounded, it would then be possible that some sessions will be lost. Expired sessions are cleared internally by Aerobase itself to avoid growing the size of these caches without limit. If you see memory issues due to a large number of sessions, you can try to:

  • Increase the size of cluster (more nodes in cluster means that sessions are spread more equally among nodes)

  • Increase the memory for Aerobase server process

  • Decrease the number of owners to ensure that caches are saved in one single place. See Replication and Failover for more details

  • Disable l1-lifespan for distributed caches. See Infinispan documentation for more details

  • Decrease session timeouts, which could be done individually for each realm in Aerobase admin console. But this could affect usability for end users. See Timeouts for more details.

There is an additional replicated cache, work, which is mostly used to send messages among cluster nodes; it is also unbounded by default. However, this cache should not cause any memory issues as entries in this cache are very short-lived.

8.2. Replication and Failover

There are caches like sessions, authenticationSessions, offlineSessions, loginFailures and a few others (See Eviction and Expiration for more details), which are configured as distributed caches when using a clustered setup. Entries are not replicated to every single node, but instead one or more nodes is chosen as an owner of that data. If a node is not the owner of a specific cache entry it queries the cluster to obtain it. What this means for failover is that if all the nodes that own a piece of data go down, that data is lost forever. By default, Aerobase only specifies one owner for data. So if that one node goes down that data is lost. This usually means that users will be logged out and will have to login again.

You can change the number of nodes that replicate a piece of data by change the cache_owners attribute in the /etc/aerobase/aerobase.rb declaration.

cache_owners
keycloak_server['cache_owners'] = 2

Here we’ve changed it so at least two nodes will replicate one specific user login session.

The number of owners recommended is really dependent on your deployment. If you do not care if users are logged out when a node goes down, then one owner is good enough and you will avoid replication.
It is generally wise to configure your environment to use loadbalancer with sticky sessions. It is beneficial for performance as Aerobase server, where the particular request is served, will be usually the owner of the data from the distributed cache and will therefore be able to look up the data locally. See Sticky sessions for more details.

8.3. Clearing Caches at Runtime

To clear the realm or user cache, go to the Aerobase admin console Realm Settings→Cache Config page. On this page you can clear the realm cache, the user cache or cache of external public keys.

The cache will be cleared for all realms!