Apache: The Simplest Load Balancing Cluster with Failover

Apache: The Simplest High Availability Cluster with Synchronous Replication and Failover

Evidian SafeKit brings load balancing and failover to Apache (HTTP Server). This article explains how to implement quickly a Apache cluster without network load balancers or dedicated proxy servers. SafeKit is installed directly on the Apache servers. The high availabity module apache_farm.safe and a free trial are offered in the installation section.

Note that you can implement with the same product real-time replication and failover of the back-end application like Microsoft SQL Server, PostgreSQL or other applications: see examples of mirror modules here.

This clustering solution is recognized as the simplest to implement by our customers and partners. It is also a complete solution that solves hardware failures (20% of problems) including the complete failure of a computer room, software failures (40% of problems) including smooth upgrade server by server and human errors (40% of problems) thanks to its simplicity of administration.

How the Evidian SafeKit software simply implements Apache load balancing and failover without network load balancers or dedicated proxy servers

How the Evidian SafeKit farm cluster implements Apache load balancing and failover?

On the previous figure, Apache services are running of the 3 servers. Users are connected to a virtual IP address which is configured locally on each server. The input traffic for the virtual IP address is received by all the servers and split among them by a filter inside each server's kernel.

The network load balancing algorithm inside the filter is based on the identity of the client packets (client IP address, client TCP port). Depending on the identity of the client packet input, only one filter in a server accepts the packet; the other filters in other servers reject it.

Once a packet is accepted by the filter on a server, only the CPU and memory of this server are used by Apache services that responds to the request of the client. The output messages are sent directly from the Apache server to the client.

If a server fails, the SafeKit membership protocol reconfigures the filters in the network load balancing cluster to re-balance the traffic on the remaining available servers.

Demonstration of a farm module

This demonstration shows the configuration of a generic farm.safe module with Apache but the configuration is the same with other applications. Note that with the apache_farm.safe module instead of the generic one, restart scripts are preconfigured for Apache.

Customers

Key differentiators of Apache load balancing and failover with the Evidian SafeKit farm cluster

Evidian SafeKit farm cluster with load balancing and failover

No load balancer or dedicated proxy servers or special multicast Ethernet address No load balancer or dedicated proxy servers

Like   The solution does not require load balancers or dedicated proxy servers above the farm for imlementing load balancing. SafeKit is installed directly on the Apache servers in the farm. The load balancing is based on a standard virtual IP address/Ethernet MAC address and is working with physical servers or virtual machines on Windows and Linux without special network configuration

Dislike  This is not the case with network load balancers

Dislike  This is not the case with dedicated proxies on Linux

Dislike  This is not the case with a specific multicast Ethernet address on Windows

All clustering features All clustering features

Like  The solution includes all clustering features: virtual IP address, load balancing on client IP address or on sessions, server failure monitoring, network failure monitoring, software failure monitoring, automatic Apache restart with a quick revovery time and a replication option with a mirror module

Dislike  This is not the case with other load balancing solutions. They are able to make load balancing but they do not include a full clustering solution with restart scripts and automatic Apache restart in case of failure. They do not offer a replication option

Like   The cluster configuration is very simple and made by means of a high availability application module. There is no domain controller or active directory to configure on Windows. The solution works on Windows and Linux

Remote sites Remote sites

Like   If servers are connected to the same IP network through an extended LAN between remote sites, the virtual IP address of SafeKit is working with load balancing at level 2

Like   If servers are connected to different IP networks between remote sites, the virtual IP address can be configured at the level of a load balancer. SafeKit offers a health check: the load balancer is configured with a URL managed by SafeKit which returns OK on the UP servers and NOT FOUND else. This solution is implemented for SafeKit in the Cloud but it can be also implemented with a load balancer on premise. Thus you can implement load balancing but also all the clustering features of SafeKit including an easy administration of the cluster through the SafeKit web console

Uniform high availability solution Uniform high availability solution

Like  SafeKit imlements a farm cluster with load balancing and failover. But it implements also a mirror cluster with replication and failover. Thus a N-tiers architecture can me made highly available and load balanced with the same solution on Windows and Linux (same installation, configuration, administration with the SafeKit console or with the command line interface). This is unique on the market

Dislike  This is not the case with an architecture mixing different technologies for load balancing, replication and failover

FAQ on Evidian SafeKit [+]

Installation of SafeKit for Apache load balancing and failover

Package installation on Windows

Package installation on Linux

Configuration instructions

The configuration is presented with the web console connected to 2 Windows servers but it is the same thing with 2 Linux servers.

Important: all the configuration must be done from a single browser.

Launch the web console in a browser by connecting to http://localhost:9010 (next image)

Start the SafeKit web console for configuring Apache

Enter IP address of the first node and click on Confirm (next image)

SafeKit web console - first Apache node in the cluster

Click on New node and enter IP address of the second node (next image)

SafeKit web console - second Apache node in the cluster

Click on the red floppy disk to save the configuration (previous image)

In the Configuration tab, click on apache_farm.safe (xxx.safe in the next image) then  enter apache_farm as the module name and Confirm (the console finds apache_farm.safe in the demo/ directory that you previously filled)

SafeKit web console - start configuration of Apache SafeKit web console - enter Apache module name

Click on Validate (next image)

SafeKit web console - enter Apache module nodes

Set the virtual IP address of the cluster, the TCP port of the service to load balance and the criterion on source IP address or source TCP port (next image).

A virtual IP address is a new unused IP address in the same IP network as the IP addresses of the two nodes. The load balancing will be made on the virtual IP address/TCP port according the criterion. This article explains difference between source IP address and source TCP port.

SafeKit web console - enter Apache parameters

For information:

Click on Validate (previous image)

SafeKit web console - stop the Apache module before configuration

Click on Configure (previous image)

SafeKit web console - check the success green messages of the Apache configuration

Check the success green message on both servers and click on Next (previous image)

SafeKit web console - start the Apache cluster on both nodes

Start the cluster on both nodes (previous image). Check that the status becomes UP (green) - UP (green) (next image).

SafeKit web console - Apache cluster started

The cluster is operational with Apache services running on both UP nodes (previous image).

Be careful, components which are clients of the Apache services must be configured with the virtual IP address. The configuration can be made with a DNS name (if a DNS name has been created and associated with the virtual IP address).

Tests

Check with Windows Microsoft Management Console (MMC) or with Linux command lines that the Apache services have been started by start_both on both UP nodes. Put Apache services with Boot Startup Type = Manual (SafeKit controls start of Apache services).

Stop one UP node by scrolling down the menu of the node and by clicking on Stop. Check that the load balancing is reconfigured with only the other node taking all TCP connections. And check with Windows Microsoft Management Console (MMC) or with Linux command lines that the Apache services have been stopped by stop_both on the STOP node.

Event logs of module and application

To see the module log of node 1 which contains events in the cluster (next image):

SafeKit web console - Module Log of the Apache node 1

To see the application log of node 1 which contains output messages from restart scripts (start_both and stop_both) (next image):

SafeKit web console - Application Log of Apache node 1

To see the logs of node 2 (previous image), click on W12R2server75/UP (it will become blue) on the left side and repeat the same operations.

Advanced configuration

In Advanced Configuration tab (next image), you can edit internal files of the module: bin/start_both and bin/stop_both and conf/userconfig.xml (next image on the left side). If you make change in the internal files here, you must apply the new configuration by a right click on the blue icon/xxx on the left side (next image): the interface will allow you to redeploy the modified files on both servers.

SafeKit web console - Advanced configuration of the Apache module

Configure boot start (next image on the right side) configures the automatic boot of the module when the server boots. Do this configuration on both nodes once the Apache load balancing and failover solution is correctly running.

SafeKit web console - automatic boot of the Apache module

Support

For getting support on the call desk of https://support.evidian.com, get 2 Snaphots (2 .zip files), one for each server and upload them in the call desk tool (next image).

SafeKit web console - Apache snaphots for support

Internal files of the Windows apache_farm.safe module

userconfig.xml

<!DOCTYPE safe>
<safe>
<macro name="VIRTUAL_IP" value="VIRTUAL_IP_TO_BE_DEFINED" />
<macro name="APACHE_PORT" value="TCP_PORT_TO_BE_DEFINED" />
<service mode="farm" maxloop="3" loop_interval="24">
  <!-- Farm topology configuration for the membership protocol -->
  <!-- Names or IP addresses on the default network are set during initialization in the console -->
  <farm>
    <lan name="default" />
  </farm>
  <!-- Virtual IP Configuration -->
  <!-- Use VIRTUAL_IP defined in macro above -->
  <vip>
    <interface_list>
      <interface check="on" arpreroute="on">
        <virtual_interface type="vmac_directed">
          <virtual_addr addr="%VIRTUAL_IP%" where="alias"/>
        </virtual_interface>
      </interface>
    </interface_list>
    <loadbalancing_list>
      <group name="APACHE">
        <!-- Set load-balancing rule on APACHE_PORT defined in macro above -->
        <rule port="%APACHE_PORT%" proto="tcp" filter="on_addr"/>
      </group>
    </loadbalancing_list>
  </vip>
  <!-- TCP Checker Configuration -->
  <!-- Use VIRTUAL_IP and APACHE_PORT defined in macros above -->
  <check>
    <tcp ident="HTTP_APACHE" when="both">
      <to
         addr="%VIRTUAL_IP%"
         port="%APACHE_PORT%"
         interval="10"
         timeout="5"
       />
    </tcp>
  </check>
  <!-- User scripts activation -->
  <user nicestoptimeout="300" forcestoptimeout="300" logging="userlog" />
</service>
</safe>

start_both.cmd

@echo off

rem Script called on all servers for starting applications

rem For logging into SafeKit log use:
rem "%SAFE%\safekit" printi | printe "message"

rem stdout goes into Application log
echo "Running start_both %*" 

set res=0

net start Apache2
if not %errorlevel% == 0 (
  %SAFE%\safekit printi "Apache start failed"
) else (
  %SAFE%\safekit printi "Apache started"
)

set res=%errorlevel%
if %res% == 0 goto end

:stop
set res=%errorlevel%
"%SAFE%\safekit" printe "start_both failed"

rem uncomment to stop SafeKit when critical
rem "%SAFE%\safekit" stop -i "start_both"

:end

stop_both.cmd

@echo off

rem Script called on all servers for stopping application

rem For logging into SafeKit log use:
rem "%SAFE%\safekit" printi | printe "message"

rem ----------------------------------------------------------
rem
rem 2 stop modes:
rem
rem - graceful stop
rem   call standard application stop with net stop
rem
rem - force stop (%1=force)
rem   kill application's processes
rem
rem ----------------------------------------------------------

rem stdout goes into Application log
echo "Running stop_both %*" 

set res=0

rem default: no action on forcestop
if "%1" == "force" goto end

%SAFE%\safekit printi "Stopping Apache..."
net stop Apache2

rem If necessary, uncomment to wait for the real stop of services
rem "%SAFEBIN%\sleep" 10

if %res% == 0 goto end

"%SAFE%\safekit" printe "stop_both failed"

:end

Internal files of the Linux apache_farm.safe module

userconfig.xml

<!DOCTYPE safe>
<safe>
<macro name="VIRTUAL_IP" value="VIRTUAL_IP_TO_BE_DEFINED" />
<macro name="APACHE_PORT" value="TCP_PORT_TO_BE_DEFINED" />
<service mode="farm" maxloop="3" loop_interval="24">
  <!-- Farm topology configuration for the membership protocol -->
  <!-- Names or IP addresses on the default network are set during initialization in the console -->
  <farm>
    <lan name="default" />
  </farm>
  <!-- Virtual IP Configuration -->
  <!-- Use VIRTUAL_IP defined in macro above -->
  <vip>
    <interface_list>
      <interface check="on" arpreroute="on">
        <virtual_interface type="vmac_directed">
          <virtual_addr addr="%VIRTUAL_IP%" where="alias"/>
        </virtual_interface>
      </interface>
    </interface_list>
    <loadbalancing_list>
      <group name="APACHE">
        <!-- Set load-balancing rule on APACHE_PORT defined in macro above -->
        <rule port="%APACHE_PORT%" proto="tcp" filter="on_addr"/>
      </group>
    </loadbalancing_list>
  </vip>
  <!-- TCP Checker Configuration -->
  <!-- Use VIRTUAL_IP and APACHE_PORT defined in macros above -->
  <check>
    <tcp ident="HTTP_APACHE" when="both">
      <to
         addr="%VIRTUAL_IP%"
         port="%APACHE_PORT%"
         interval="10"
         timeout="5"
       />
    </tcp>
  </check>
  <!-- User scripts activation -->
  <user nicestoptimeout="300" forcestoptimeout="300" logging="userlog" />
</service>
</safe>

start_both

#!/bin/sh 
# Script called on the primary server for starting applications 

# For logging into SafeKit log use:
# $SAFE/safekit printi | printe "message" 

#---------- Clean Apache residual processes 
# Call this function before starting Apache 
# to clean eventual resual Apache processes
clean_Apache()
{
  retval=0

  # $SAFE/safekit printw "Cleaning Apache processes"

  # example of a kill of started Apache process
  # warning: this command also kills the httpd process which managed the SafeKit web console
  # ps -e -o pid,comm | grep httpd | $AWK '{print "kill " $1}'| sh >/dev/null 2>&1

  return $retval
}

#---------- Apache
# Call this function for starting Apache Server
start_Apache()
{
  retval=0

  $SAFE/safekit printw "Starting Apache Server"

  # Apache - Starting 
  service httpd start   
  if [ $? -ne 0 ] ; then 
    $SAFE/safekit printw "Apache server start failed"
  else
    $SAFE/safekit printw "Apache server started"
  fi

  return $retval
}

# stdout goes into Application log
echo "Running start_both $*" 

res=0

[ -z "$OSNAME" ] && OSNAME=`uname -s`
OSNAME=`uname -s`
case "$OSNAME" in
    Linux)
	AWK=/bin/awk
	;;
    *)
	AWK=/usr/bin/awk
	;;
esac

# TODO
# remove Apache boot start 

# Clean Apache residual processes 
clean_Apache || res=$?

# Start Apache
start_Apache || res=$?

if [ $res -ne 0 ] ; then
  $SAFE/safekit printi "start_both failed"

  # uncomment to stop SafeKit when critical
  # $SAFE/safekit stop -i "start_both"
fi

exit 0

stop_both

#!/bin/sh
# Script called on the primary server for stopping application services

# For logging into SafeKit log use:
# $SAFE/safekit printi | printe "message" 

#----------------------------------------------------------
#
# 2 stop modes:
#
# - graceful stop
#   call standard application stop
#
# - force stop ($1=force)
#   kill application's processes
#
#----------------------------------------------------------

#---------- Clean Apache residual processes
# Call this function on force stop 
# to clean eventual resual Apache processes 
clean_Apache()
{
  retval=0

  # $SAFE/safekit printw "Cleaning Apache processes "

  # example of a kill of started Apache process
  # warning: this command also kills the httpd process which managed the SafeKit web console
  # ps -e -o pid,comm | grep httpd | $AWK '{print "kill " $1}'| sh >/dev/null 2>&1

  return $retval
}

#---------- Apache
# Call this function for stopping Apache
stop_Apache()
{
  retval=0

  if [ "$1" = "force" ] ; then
    # Apache force stop
    clean_Apache
    return $retval
  fi

  # Apache graceful stop
  $SAFE/safekit printw "Stopping Apache server"  
  service httpd stop
  if [ $? -ne 0 ] ; then 
    $SAFE/safekit printw "Apache server stop failed"
  else
    $SAFE/safekit printw "Apache server stopped"  
  fi

  return $retval
}

# stdout goes into Application log
echo "Running stop_both $*" 

res=0

[ -z "$OSNAME" ] && OSNAME=`uname -s`
case "$OSNAME" in
    Linux)
	AWK=/bin/awk
	;;
    *)
	AWK=/usr/bin/awk
	;;
esac

mode=
if [ "$1" = "force" ] ; then
  mode=force
  shift
fi

# Stop Apache  server 
stop_Apache $mode || res=$?

[ $res -ne 0 ] && $SAFE/safekit printi "stop_both failed"

exit 0