Firebird: The Simplest High Availability Cluster with Synchronous Replication and Failover

Installation of a SafeKit / Firebird high availability cluster with real-time synchronous replication and failover between two redundant servers

Prerequisites

• you need Firebird installed on 2 nodes (virtual machines or physical servers)
• on Windows, with Windows services manager, put Firebird services with Boot Startup Type = Manual on both nodes. SafeKit controls start of Firebird services in start_prim. Edit start_prim during the configuration to check if you have put all services in Manual boot including the new ones that you can add.
• if it is not already done, enable execution of powershell scripts on both nodes

Package installation on Windows

• Install the free version of SafeKit on 2 Windows nodes with Firebird services installed on both nodes
  
  SafeKit free trial >
• Download the Windows module
  
  Module firebird.safe for Windows >
  
  The module contains the files userconfig.xml, start_prim and stop_prim described in the internals tab
• Put firebird.safe under C:\safekit\Application_Modules\demo\ (create the demo directory if it does not exist)
• To open the Windows firewall, on both nodes start a command line as administrator, goto C:\safekit\private\bin and type  .\firewallcfg.cmd add
• For synchronizing SafeKit at boot and at shutdown, on both nodes start a command line as administrator, goto C:\safekit\private\bin and type  .\addStartupShutdown.cmd only once

Package installation on Linux

• Install the free version of SafeKit on 2 Linux nodes with Firebird services installed on both nodes
  
  SafeKit free trial >
• After the download of safekit_xx.bin package, execute it to extract the rpm and the safekitinstall script and then execute the safekitinstall script
• Answer yes to firewall automatic configuration
• Download the Linux module
  
  Module firebird.safe for Linux >
  
  The module contains the files userconfig.xml, start_prim and stop_prim described in the internals tab
• Put firebird.safe under /opt/safekit/Application_Modules/demo/ (create the demo directory if it does not exist)

   User's guide available here

Go to the Configuration tab (above)

Configuration of a SafeKit / Firebird high availability cluster with real-time synchronous replication and failover between two redundant servers

Configuration instructions

  Important: all the SafeKit configuration is made from a single browser.

Launch the web console in a browser on one node by connecting to http://localhost:9010 (next image). Note that you can run the console in a browser on an external workstation.

Enter IP address of the first node and click on Confirm (next image)

Click on New node and enter IP address of the second node (next image)
Then click on the red floppy disk to save the configuration (next image)

   In the previous image, the save operation put the IP addresses in the file c:\safekit\cluster\cluster.xml ou /var/safekit/cluster/cluster.xml and apply the distributed command c:\safekit\safekit –H "*" -G ou /opt/safekit/safekit –H "*" -G which configures node 1 and node 2. So do not redo this operation on the other node. Check the connectivity and firewall between both nodes for troubleshooting.

In the Configuration tab (next image), click on firebird.safe (xxx.safe in the next image) then enter firebird as the module name and Confirm (the console finds firebird.safe in the 'demo/' directory that you previously filled during the installation)

Click on Validate (next image)

After the click on Validate, this new window appears (next image)

On the previous image:

• Check that replicated directories are installed on both nodes and contain application data.
  Replication of data and also logs is required for a database.
  You can add new directories to replicate if necessary.
• Enter a virtual IP address. A virtual IP address is a standard IP address in the same IP network (same subnet) as the IP addresses of both nodes.
  Clients of the application must be configured with the virtual IP address (or the DNS name associated to the virtual IP address).
  The virtual IP address is automatically switched in case of failover.
• start_prim and stop_prim must contain the start and the stop of the Firebird application.
  Edit the scripts and check that the names of services in these scripts are those installed on your both machines (else modify them in the scripts).
  You can add new services in the scripts if necessary.
• Stop the services configured in start_prim on both nodes.
• On Windows and on both nodes, with Windows services manager, put Firebird services set inside start_prim with Boot Startup Type = Manual on both servers (SafeKit controls start of Firebird services in start_prim).
• Note that if a process name is displayed in Process checker, it will be monitored with the action restart in case of failure. The services will be stopped an restarted locally on the primary server if this process disappears from the list of running processes. After 3 unsuccessful local restarts, the module is stopped on the primary server and the services are restarted on the secondary server.

Click on Validate (previous image)

Click on Configure (previous image)

Check the success green message on both servers and click on Next (previous image). On Linux, you may have an error at this step if replicated directories are mount points. See this article to solve the problem.

We assume that node 1 is the referencial with the up-to-date data (otherwise do the following operation on node 2).
We assume that the services started automatically by SafeKit in the start_prim script are stopped on node 1.
Select node 1 and click on start it (next image).
When you will start node 2, the first resynchronization will be made from node1 to node 2.
If in doubt, before this operation, we suggest you to make a copy of replicated directories before starting the cluster to avoid any errors.

node 1 must reach the ALONE (green) state (next image) meaning that the start_prim script has been executed on node 1.
When node 1 reaches the ALONE (green) state, you can check that the Firebird application is started on node 1.
If the state is ALONE (green) and if the application is not started, check the start_prim script: you can find output messages of the script in the Application Log available in the Control tab.
If node 1 does not reach the ALONE (green) state, analyze why with the Module Log available in the Control tab.
Note: in the next image, if the cluster is in the state [WAIT (red) not uptodate - STOP (red) not uptodate], stop the WAIT server and force its start as primary (click here to see the operation).

We assume that the services started automatically by SafeKit in the start_prim script are stopped on node 2.
Start node 2 (previous image) which becomes SECOND green (next image) after resynchronisation of all replicated directories (binary copy from node 1 to node 2).

The cluster is operational with Firebird services running on the PRIM node and nothing running on the SECOND node (previous image). Only modifications inside files are replicated in real-time in this state.

Be careful, components which are clients of the Firebird services must be configured with the virtual IP address. The configuration can be made with a DNS name (if a DNS name has been created and associated with the virtual IP address).

Test

Check with Windows Microsoft Management Console (MMC) or with Linux command lines that the Firebird services have been started on the primary server by the start_prim script and stopped on the secondary server by the stop_prim script.

Stop the PRIM node by scrolling down the menu of the primary node and by clicking on Stop. Check that there is a failover on the SECOND node. And with Windows Microsoft Management Console (MMC) or with Linux command lines, check the failover of Firebird services (stopped and started in the stop_prim and start_prim scripts).

More information on tests in the User's Guide

Automatic start of the module at boot

Configure boot start (next image on the right side) configures the automatic boot of the module when the server boots. Do this configuration on node1 and node2 once the high availability solution is correctly running.

Note that on Windows, with Windows services manager, we assume that Firebird services are with Boot Startup Type = Manual on both nodes. SafeKit controls start of Firebird services when starting the module in start_prim.

Note that for synchronizing SafeKit at boot and at shutdown, on both nodes, we assume that a command line has been run during installation as administrator: .\addStartupShutdown.cmd in C:\safekit\private\bin (otherwise dot it now only once).

For reading the SafeKit logs, go to the Troubleshooting tab

For editing userconfig.xml, start_prim and stop_prim, go to the Advanced Configuration tab

Troubleshooting of a SafeKit / Firebird high availability cluster with real-time synchronous replication and failover between two redundant servers

Module log

Read the module log to understand the reasons of a failover, of a waiting state on the availability of a resource etc...
To see the module log of the primary server (next image):

• click on the Control tab
• click on node 1/PRIM (it becomes blue) on the left side to select the server
• click on Module Log
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the module log in a .txt file and to analyze in a text editor

Repeat the same operation to see the module log of the secondary server.

Application log

Read the application log to see the output messages of the stat_prim and stop_prim restart scripts.
To see the application log of the primary server (next image):

• click on the Control tab
• click on node 1/PRIM (it becomes blue) on the left side to select the server
• click on Application Log to see messages when starting and stopping Firebird services
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the application log in a .txt file and to analyze in a text editor

Repeat the same operation to see the application log of the secondary server.

More information on troubleshooting in the User's Guide

For support, go to the Support tab

Advanced configuration of a SafeKit / Firebird high availability cluster with real-time synchronous replication and failover between two redundant servers

In Advanced Configuration tab (next image), you can edit internal files of the module: bin/start_prim and bin/stop_prim and conf/userconfig.xml (next image on the left side). If you make change in the internal files here, you must apply the new configuration by a right click on the icon/xxx on the left side (next image): the interface will allow you to redeploy the modified files on both servers.

More information on userconfig.xml in the User's Guide

For an example of userconfig.xml, start_prim and stop_prim, go to the Internals tab

Support of a SafeKit / Firebird high availability cluster with real-time synchronous replication and failover between two redundant servers

For getting support on the call desk of https://support.evidian.com, get 2 Snaphots (2 .zip files), one for each server and upload them in the call desk tool (next image).

More information on support in the User's Guide

Internals of a SafeKit / Firebird high availability cluster with synchronous replication and failover

Go to the Advanced Configuration tab, for editing these files

Internal files of the Windows firebird.safe module

userconfig.xml on Windows (description in the User's Guide)

<!DOCTYPE safe>
<safe>
<service mode="mirror" defaultprim="alone" maxloop="3" loop_interval="24" failover="on">
  <!-- Heartbeat Configuration -->
  <!-- Names or IP addresses on the default network are set during initialization in the console -->
  <heart pulse="700" timeout="30000">
    <heartbeat name="default" ident="flow">
    </heartbeat>
  </heart>
  <!-- Virtual IP Configuration -->
  <!-- Replace
     * VIRTUAL_TO_BE_DEFINED by the name of your virtual server 
  -->
  <vip>
    <interface_list>
        <interface check="on" arpreroute="on"> 
	  <real_interface>
               <virtual_addr addr="VIRTUAL_TO_BE_DEFINED" where="one_side_alias" />
          </real_interface>
        </interface>
    </interface_list>
  </vip>
  <!-- Software Error Detection Configuration -->
  <errd polltimer="10">
    <!-- Firebird Server -->
    <proc name="firebird.exe" atleast="1" action="restart" class="prim" />
  </errd>
  <!-- File Replication Configuration -->
  <!-- 
     * C:\Program Files\Firebird\Firebird_3_0\examples\empbuild: directory of employee.fdb sample database
     * replace the path with your database directory
  -->
  <rfs async="second" acl="off" nbrei="3">
	<replicated dir="C:\Program Files\Firebird\Firebird_3_0\examples\empbuild" mode="read_only" />
  </rfs>
  <!-- User scripts activation -->
  <user nicestoptimeout="300" forcestoptimeout="300" logging="userlog" />
</service>
</safe>

start_prim.cmd on Windows

@echo off
rem Script called on the primary server for starting application services 

rem For logging into SafeKit log use:
rem "%SAFE%\safekit" printi | printe "message"

rem stdout goes into Application log
echo "Running start_prim %*" 

set res=0

net start "Firebird Server - DefaultInstance" > nul
if not %errorlevel% == 0 (
  %SAFE%\safekit printi "Firebird start failed"
) else (
  %SAFE%\safekit printi "Firebird started"
)

if %res% == 0 goto end

:stop
set res=%errorlevel%
"%SAFE%\safekit" printe "start_prim failed"

rem uncomment to stop SafeKit when critical
rem "%SAFE%\safekit" stop -i "start_prim"

:end

stop_prim.cmd on Windows

@echo off
rem Script called on the primary server for stopping application services 

rem ----------------------------------------------------------
rem
rem 2 stop modes:
rem
rem - graceful stop
rem   call standard application stop with net stop
rem
rem - force stop (%1=force)
rem   kill application's processes
rem
rem ----------------------------------------------------------

rem For logging into SafeKit log use:
rem "%SAFE%\safekit" printi | printe "message"

rem stdout goes into Application log
echo "Running stop_prim %*" 

set res=0

rem default: no action on forcestop
if "%1" == "force" goto end

net stop "Firebird Server - DefaultInstance" > nul
%SAFE%\safekit printi "Firebird stopped"

rem wait a little for a real stop of services
%SAFEBIN%\sleep 10

:end

Internal files of the Linux firebird.safe module

userconfig.xml on Linux (description in the User's Guide)

<!DOCTYPE safe>
<safe>
<service mode="mirror" defaultprim="alone" maxloop="3" loop_interval="24" failover="on">
  <!-- Heartbeat Configuration -->
  <!-- Names or IP addresses on the default network are set during initialization in the console -->
  <heart pulse="700" timeout="30000">
    <heartbeat name="default" ident="flow">
    </heartbeat>
  </heart>
  <!-- Virtual IP Configuration -->
  <!-- Replace
     * VIRTUAL_TO_BE_DEFINED by the name of your virtual server 
  -->
  <vip>
    <interface_list>
        <interface check="on" arpreroute="on"> 
	  <real_interface>
               <virtual_addr addr="VIRTUAL_TO_BE_DEFINED" where="one_side_alias"/>
          </real_interface>
        </interface>
    </interface_list>
  </vip>
  <!-- Software Error Detection Configuration -->
  <errd polltimer="10">
    <!-- Firebird Server -->
    <proc name="firebird" atleast="1" action="restart" class="prim" />
  </errd>
  <!-- File Replication Configuration -->
  <!--
     * /opt/firebird/examples/empbuild: directory of employee.fdb sample database
     * replace the path with your database directory
  -->
  <rfs mountover="off" async="second" acl="off" nbrei="3">
	<replicated dir="/opt/firebird/examples/empbuild" mode="read_only" />
  </rfs>
  <!-- User scripts activation -->
  <user nicestoptimeout="300" forcestoptimeout="300" logging="userlog" />
</service>
</safe>

start_prim on Linux

#!/bin/sh 
# Script called on the primary server for starting applications 

# For logging into SafeKit log use:
# $SAFE/safekit printi | printe "message" 

#---------- Clean Firebird residual processes 
# Call this function before starting any Firebird databases 
# to clean eventual resual Firebird processes
clean_Firebird()
{
  retval=0

  $SAFE/safekit printw "Cleaning Firebird processes"

  # kill started Firebird processes
  ps -e -o pid,comm | grep firebird | $AWK '{print "kill " $1}'| sh >/dev/null 2>&1

  return $retval
}

#---------- Firebird Databases
# Call this function for starting Firebird Server
start_Firebird()
{
  retval=0

  $SAFE/safekit printw "Starting Firebird Server"

  # Firebird - Database Starting 
  service firebird start   
  if [ $? -ne 0 ] ; then 
    $SAFE/safekit printw "Firebird server start failed"
  else
    $SAFE/safekit printw "Firebird server started"
  fi

  return $retval
}

# stdout goes into Application log
echo "Running start_prim $*" 

res=0

[ -z "$OSNAME" ] && OSNAME=`uname -s`
OSNAME=`uname -s`
case "$OSNAME" in
    Linux)
	AWK=/bin/awk
	;;
    *)
	AWK=/usr/bin/awk
	;;
esac

# TODO
# remove Firebird boot start 

# Clean Firebird residual processes 
clean_Firebird || res=$?

# Start Firebird databases
start_Firebird || res=$?

if [ $res -ne 0 ] ; then
  $SAFE/safekit printi "start_prim failed"

  # uncomment to stop SafeKit when critical
  # $SAFE/safekit stop -i "start_prim"
fi

exit 0

stop_prim on Linux

#!/bin/sh
# Script called on the primary server for stopping application services

# For logging into SafeKit log use:
# $SAFE/safekit printi | printe "message" 

#----------------------------------------------------------
#
# 2 stop modes:
#
# - graceful stop
#   call standard application stop
#
# - force stop ($1=force)
#   kill application's processes
#
#----------------------------------------------------------

#---------- Clean Firebird residual processes
# Call this function on force stop 
# to clean eventual resual Firebird processes 
clean_Firebird()
{
  retval=0

  $SAFE/safekit printw "Cleaning Firebird processes "

  # kill started Firebird 
  ps -e -o pid,comm | grep firebird | $AWK '{print "kill -9 " $1}'| sh >/dev/null 2>&1

  return $retval
}

#---------- Firebird databases
# Call this function for stopping Firebird databases
stop_Firebird()
{
  retval=0

  if [ "$1" = "force" ] ; then
    # Firebird databases force stop
    clean_Firebird
    return $retval
  fi

  # Firebird databases graceful stop
  $SAFE/safekit printw "Stopping Firebird server"  
  service firebird stop
  if [ $? -ne 0 ] ; then 
    $SAFE/safekit printw "Firebird server stop failed"
  else
    $SAFE/safekit printw "Firebird server stopped"  
  fi

  return $retval
}

# stdout goes into Application log
echo "Running stop_prim $*" 

res=0

[ -z "$OSNAME" ] && OSNAME=`uname -s`
case "$OSNAME" in
    Linux)
	AWK=/bin/awk
	;;
    *)
	AWK=/usr/bin/awk
	;;
esac

mode=
if [ "$1" = "force" ] ; then
  mode=force
  shift
fi

# Stop Firebird  server 
stop_Firebird $mode || res=$?

[ $res -ne 0 ] && $SAFE/safekit printi "stop_prim failed"

exit 0