Extend Failover Checks to Back-end Servers

With two Airlock nodes in a failover (High Availability) cluster, a failover is triggered by a malfunctioning external interface. This article describes how to trigger a failover based on missing back-end connectivity. The later is useful if Airlock cluster nodes have redundant physical connections to the back-end servers. That is, if the active Airlock looses connection to the back-ends, the cluster partner still reaches the back-ends through its redundant connection.

How it works

The failover mechanism has been extended to perform reachability checks on a set of back-end hosts. These additional checks are implemented in a shell script on the Airlock host. This script is fully configurable as it can be written by the customer. A sample script is available on the system.

Enable the script on AIrlock WAF 7.0 and newer

The backend check script is located here:

1. To enable monitoring, edit following file:

   # vi /opt/airlock/custom-settings/monitor/airlock-backend-sensor.sh

    

2. Set variable to ENABLED="yes" and save the script

3. Restart the script:

   # /etc/init.d/airlock-backend-check-nurse restart

    

Enable the Script on Airlock WAF 5.x - 6.x

Select the hosts to be monitored

By default, the script pings a set of hosts and triggers a failover if any host is not reachable. The set of hosts checked is defined in a variable at the beginning of the script:

You must separate the hosts with whitespaces. Don't forget the enclosing quotation marks! Make sure not to add too many hosts as this might slow down the failover process (hosts are checked sequentially).

If one of the defined hosts does not respond within a given time (2 seconds is the preconfigured value) the script exits with an error code. As a consequence, that instance of Airlock will switch to the new state "Passive (Back-end Lost)" and the partner becomes active. You will see the state information in the Configuration Center.

If one of the two Airlocks is already in the "Back-end Lost" state, the back-end checks are discarded to assure that at least the error page 500 can be delivered even if the back-end connection of the second instance becomes corrupt.

Advanced configuration with custom script

The script airlock-backend-sensor.sh is a working sample and also thought as an inspiration for your own test script. You can easily write your own checker. The API for integration in the failover mechanism is straight-forward:

If the script returns 0, the test is interpreted as passed. If the script exits with error code 1, the test has failed and a failover is triggered. If the script exits with error code 2, the back-end check nurse ist stopped and must be restarted to enable monitoring (see above). If the script exists with error code 3, the back-end check nurse script is disabled.

An additional example script using curl to check application URLs is available as an attachment.

Be aware that your script is executed during the failover checks and may corrupt the HA mechanism if it's not working as expected. Please test your script carefully on an integration system before running it in production mode!

The script must be installed on both nodes of the Airlock cluster to enable failover in both directions.