KVM: the simplest high availability cluster with real-time replication, automatic failover and load balancing of VMs between two redundant servers without shared disk

Installation of a SafeKit / KVM high availability cluster with real-time synchronous replication and failover between two redundant servers

Prerequisites

• you need KVM installed on 2 Linux servers

Package installation on Linux

• Install the free version of SafeKit on 2 Linux nodes with KVM services installed on both nodes
  
  SafeKit free trial >
• After the download of safekit_xx.bin package, execute it to extract the rpm and the safekitinstall script and then execute the safekitinstall script
• Answer yes to firewall automatic configuration
• Download the Linux module
  
  Module kvm.safe for Linux >
  
  The module contains the files userconfig.xml, start_prim and stop_prim described in the internals tab
• Put kvm.safe under /opt/safekit/Application_Modules/demo/ (create the demo directory if it does not exist)

   User's guide available here

Go to the Configuration tab (above)

Configuration of a SafeKit / KVM high availability cluster with real-time synchronous replication and failover between two redundant servers

Prerequisites

The KVM configuration is presented with a virtual machine named vm1. You will have to repeat this configuration for all VMs that you want to replicate. SafeKit supports up to 25 virtual machines.

At this step, we suppose that a virtual machine named vm1 is existing and configured on node 1. The path of the vm1 image is /var/lib/libvirt/images. Before the SafeKit configuration, you have to make the following configuration

On node 1:

• Stop vm1: virsh shutdown vm1
• Create a vm1/ directory: mkdir -p /var/lib/libvirt/images/vm1/
• Copy the vm1 image to the new location: cp -a /var/lib/libvirt/images/vm1.qcow2 /var/lib/libvirt/images/vm1/. The original VM image can be deleted as soon as tests with the new location are successfull.
• Edit the vm1 configuration file: EDITOR=vi virsh edit vm1 and change the line:
  <source file='/var/lib/libvirt/images/vm1.qcow2'> by <source file='/var/lib/libvirt/images/vm1/vm1.qcow2'>
  
• Set the cache option to 'none' in the same file, for data integraty in case of crash:
  <disk type='file' device='disk'> <driver name='qemu' type=’qcow2’ cache='none'/>
  
• Close the vm1 configuration file
• Disable vm1 automatic start: virsh autostart vm1 --disable
• Create a configuration xml dump for vm1: virsh dumpxml vm1 > vm1.xml

On node 2:

• Copy the xml configuration vm1.xml from node 1. Note: whenever vm1 configuration is changed on node 1, you must reapply the new configuration on node 2.
• Create vm1 but do not start it: virsh define vm1.xml
• Disable vm1 automatic start: virsh autostart vm1 --disable
• Create the directory for the image location: mkdir -p /var/lib/libvirt/images/vm1/

Check cache='none' in virt-manager for vm1

Autostart must be disabled in virt-manager for vm1

This operation must be repeated if you have another vm2 virtual machine. You will have also to repeat the SafeKit configuration below for vm2. At the end you will have two SafeKit KVM modules for vm1 and vm2. Each module are independant and thus the load can be shared between both KVM servers.

Snapshots

By default, snapshots of a virtual machine are not replicated. For replicating snapshots (after the configuration below):

• Replicate the directory that contains the snapshot xml files: /var/lib/libvirt/qemu/snapshot/%VM_NAME% where VM_NAME is the name of the virtual machine (vm1)
  Edit userconfig.xml file in the "Advanced configuration" tab and insert the lines below into the <rfs> section:
  <replicated dir="/var/lib/libvirt/qemu/snapshot/%VM_NAME%" mode="read_only">
  </replicated>
  Save changes, then apply the new configuration on both nodes
• Create snapshots only on the PRIM node either through virt-manager or a command line: virsh snapshot-create-as vm1 snapshot-name
  Note: when creating snapshot with a command line, you have to refresh the snapshot view into virt-manager.
• Snapshots created on the PRIM node are operationnal on failover, but not listed on the other node.
  The command line for listing snapshot is: virsh snapshot-list vm1
  For importing a snapshot on the other node, you have to run the command: virsh snapshot-create --redefine vm1 /var/lib/libvirt/qemu/snapshot/vm1/snapshot-name

Configuration instructions

  Important: all the SafeKit configuration is made from a single browser.

Launch the web console in a browser on one node by connecting to http://localhost:9010 (next image). Note that you can run the console in a browser on an external workstation.

Enter IP address of the first node and click on Confirm (next image)

Click on New node and enter IP address of the second node (next image)
Then click on the red floppy disk to save the configuration (next image)

   In the previous image, the save operation put the IP addresses in the file /var/safekit/cluster/cluster.xml and apply the distributed command /opt/safekit/safekit –H "*" -G which configures node 1 and node 2. So do not redo this operation on the other node. Check the connectivity and firewall between both nodes for troubleshooting.

In the Configuration tab, click on kvm.safe (xxx.safe in the next image) then enter vm1 as the module name and Confirm (the console finds kvm.safe in the demo/ directory that you previously filled)

Click on Validate (next image)

Give the path of the images directory (/var/lib/libvirt/images in next image) where vm1/ has been created and the name of VM (vm1 in next image). /var/lib/libvirt/images/vm1/ will be replicated by SafeKit and the VM vm1 will be restarted in the SafeKit scripts.

Click on Validate (previous image)

Click on Configure (previous image)

Check the success green message on both servers and click on Next (previous image). On Linux, you may have an error at this step if replicated directories are mount points. See this article to solve the problem.

We assume that node 1 is the referencial with the up-to-date data (otherwise do the following operation on node 2).
Select node 1 and click on start it (next image).
When you will start node 2, the first resynchronization will be made from node1 to node 2.
If in doubt, before this operation, we suggest you to make a copy of replicated directories before starting the cluster to avoid any errors.

node 1 must reach the ALONE (green) state (next image) meaning that the start_prim script has been executed on node 1.
When node 1 reaches the ALONE (green) state, you can check that the KVM application is started on node 1.
If the state is ALONE (green) and if the application is not started, check the start_prim script: you can find output messages of the script in the Application Log available in the Control tab.
If node 1 does not reach the ALONE (green) state, analyze why with the Module Log available in the Control tab.
Note: in the next image, if the cluster is in the state [WAIT (red) not uptodate - STOP (red) not uptodate], stop the WAIT server and force its start as primary (click here to see the operation).

We assume that the services started automatically by SafeKit in the start_prim script are stopped on node 2.
Start node 2 (previous image) which becomes SECOND green (next image) after resynchronisation of all replicated directories (binary copy from node 1 to node 2).

The replication and failover is operational for vm1. vm1 is running on the PRIM node (previous image). Only modifications inside the KVM image defining vm1 are replicated in real-time in this state.

Test

Check with KVM virt-manager that vm1 is started on the primary server and stopped on the secondary server.

Stop the PRIM node by scrolling down the menu of the primary node and by clicking on Stop. Check that there is a failover on the SECOND node. And check the failover of vm1 with KVM virt-manager.

More information on tests in the User's Guide

Automatic start of the module at boot

Configure boot start (next image on the right side) configures the automatic boot of the module when the server boots. Do this configuration on node1 and node2 once the high availability solution is correctly running.

For reading the SafeKit logs, go to the Troubleshooting tab

For editing userconfig.xml, start_prim and stop_prim, go to the Advanced Configuration tab

Troubleshooting of a SafeKit / KVM high availability cluster with real-time synchronous replication and failover between two redundant servers

Module log

Read the module log to understand the reasons of a failover, of a waiting state on the availability of a resource etc...
To see the module log of the primary server (next image):

• click on the Control tab
• click on node 1/PRIM (it becomes blue) on the left side to select the server
• click on Module Log
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the module log in a .txt file and to analyze in a text editor

Repeat the same operation to see the module log of the secondary server.

Application log

Read the application log to see the output messages of the stat_prim and stop_prim restart scripts.
To see the application log of the primary server (next image):

• click on the Control tab
• click on node 1/PRIM (it becomes blue) on the left side to select the server
• click on Application Log to see messages when starting and stopping VM1
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the application log in a .txt file and to analyze in a text editor

Repeat the same operation to see the application log of the secondary server.

More information on troubleshooting in the User's Guide

For support, go to the Support tab

Advanced configuration of a SafeKit / KVM high availability cluster with real-time synchronous replication and failover between two redundant servers

In Advanced Configuration tab (next image), you can edit internal files of the module: bin/start_prim and bin/stop_prim and conf/userconfig.xml (next image on the left side). If you make change in the internal files here, you must apply the new configuration by a right click on the icon/xxx on the left side (next image): the interface will allow you to redeploy the modified files on both servers.

More information on userconfig.xml in the User's Guide

Support of a SafeKit / KVM high availability cluster with real-time synchronous replication and failover between two redundant servers

For getting support on the call desk of https://support.evidian.com, get 2 Snaphots (2 .zip files), one for each server and upload them in the call desk tool (next image).

More information on support in the User's Guide

Internals of a SafeKit / KVM high availability cluster with synchronous replication and failover

Go to the Advanced Configuration tab, for editing these files

Internal files of the Linux kvm.safe module

userconfig.xml (description in the User's Guide)

<!-- Mirror Architecture with Real Time File Replication and Failover for KVM -->

<!DOCTYPE safe>
<safe>
  <!-- Set value to the path of the virtual machines repository -->
  <macro name="VM_PATH" value="/var/lib/libvirt/images" />
  <!-- Set value to the name of the virtual machine -->
  <macro name="VM_NAME" value="vm1" />

 <service mode="mirror" defaultprim="alone" maxloop="3" loop_interval="24" failover="on">
  <!-- Heartbeat Configuration --> 
  <heart>
    <heartbeat name="">
    </heartbeat>
  </heart>

  <!-- File Mirroring Configuration -->
  <rfs mountover="off" async="second" locktimeout="200" nbrei="3">
	<replicated dir="%VM_PATH%/%VM_NAME%" mode="read_only">
	</replicated>
	<!-- Uncomment for replicating the directory that contains snapshot xml files of the virtual machine
	<replicated dir="/var/lib/libvirt/qemu/snapshot/%VM_NAME%" mode="read_only">
	</replicated>
	-->	
  </rfs>

  <!-- User scripts Configuration -->
  <user>
    <var name="VM_PATH" value="%VM_PATH%/%VM_NAME%" />
    <var name="VM_NAME" value="%VM_NAME%" />
  </user>

 </service>
</safe>

start_prim

#!/bin/sh
# Script called on the primary server for starting application

# For logging into SafeKit log use:
# $SAFE/safekit printi | printe "message" 

# stdout goes into Application log
echo "Running start_prim $*" 

res=0

# Start VM_NAME
virsh start $VM_NAME
state=$(virsh list --all | grep " $VM_NAME " | awk '{ print $3}')
if ([ "x$state" == "x" ]) ; then
	res=1
	$SAFE/safekit printe "$VM_NAME not found"
else
	let i=1
	while ( [ $i -le 5 ] && [ "x$state" != "xrunning" ]); do
      sleep 5
      state=$(virsh list --all | grep " $VM_NAME " | awk '{ print $3}')
      let i=i+1
	done
	if ([ "x$state" != "xrunning" ]) ; then
		res=1
		$SAFE/safekit printe "$VM_NAME start failed"
	fi
fi

if [ $res -ne 0 ] ; then
  $SAFE/safekit printe "start_prim failed"
  
  # uncomment to stop SafeKit when critical
  $SAFE/safekit stop -i "start_prim"
fi

stop_prim

#!/bin/sh
# Script called on the primary server for stopping application

# For logging into SafeKit log use:
# $SAFE/safekit printi | printe "message" 

#----------------------------------------------------------
#
# 2 stop modes:
#
# - graceful stop
#   call standard application stop
#
# - force stop ($1=force)
#   kill application's processes
#
#----------------------------------------------------------

# stdout goes into Application log
echo "Running stop_prim $*" 

# Stop VM_NAME
virsh shutdown $VM_NAME
state=$(virsh list --all | grep " $VM_NAME " | awk '{ print $3}')
if ([ "x$state" == "x" ]) ; then
	res=1
	$SAFE/safekit printe "$VM_NAME not found"
else
	let i=1
	while ( [ $i -le 5 ] && [ "x$state" == "xrunning" ]); do
		# Stop VM_NAME
		virsh shutdown $VM_NAME
		sleep 5
      		state=$(virsh list --all | grep " $VM_NAME " | awk '{ print $3}')
      		let i=i+1
	done
	if ([ "x$state" == "xrunning" ]) ; then
		res=1
		$SAFE/safekit printe "$VM_NAME stop failed"
	fi
fi

res=0

# default: no action on forcestop
[ "$1" = "force" ] && exit 0

# Fill with your application stop call

[ $res -ne 0 ] && $SAFE/safekit printe "stop_prim failed"