Bosch AMS cluster without shared storage on a SAN
[SafeKit] Synchronous real-time replication, high availability and migration of virtual machines between two servers
The solution for Bosch AMS
Evidian SafeKit brings high availability to Bosch AMS between two servers of any brand.
This article explains how to implement quickly a Bosch AMS cluster without shared storage on a SAN and without specific skills.
The principle of the solution is to put Bosch AMS in a virtual machine under Hyper-V. SafeKit implements real-time replication and automatic failover of the virtual machine.
Note that Hyper-V is the free hypervisor included in all Windows versions (even Windows for PC).
A solution open to several applications
Several applications can be put in several virtual machines replicated and restarted by SafeKit. You have the possibility to migrate each virtual machine between both servers with the SafeKit console and thus balance the load in an active-active cluster.
![[SafeKit] A Bosch AMS cluster without shared storage on a SAN](/wp-content/uploads/2024/02/safekit-hyper-v.png)
Save costs with this solution
There is no need for complex VMware-type solution with three servers and shared storage on a SAN or vSAN. With SafeKit, you will have instead synchronous real-time replication and failover of several virtual machines between two servers.
And with the standard Hyper-V manager GUI, you will be able to manage very simply your virtual machines.
Note that you can implement with the SafeKit product real-time replication and failover of any file directory and service, database, complete Hyper-V or KVM virtual machines, Docker, Podman, K3S, Cloud applications (see the module list).
Partners, the success with SafeKit
This platform agnostic solution is ideal for a partner reselling a critical application and who wants to provide a redundancy and high availability option easy to deploy to many customers.
With many references in many countries won by partners, SafeKit has proven to be the easiest solution to implement for redundancy and high availability of building management, video management, access control, SCADA software...
How the SafeKit Hyper-V cluster works with replication and failover of Bosch AMS?
The following steps are described for one virtual machine containing Bosch AMS inside one mirror module. Each replicated virtual machine runs in an independent mirror module (with a maximum of 32 virtual machines) with a primary server that can be either the Hyper-V server 1 or the Hyper-V server 2.
Step 1. Real-time replication
Server 1 (PRIM) runs the VM (virtual machine) containing Bosch AMS. SafeKit replicates in real time the VM files (virtual hard disk, VM configuration). Only changes made in the files are replicated across the network.
The replication is synchronous with no data loss on failure contrary to asynchronous replication.
You just have to configure the VM directory name in SafeKit. There are no pre-requisites on disk organization. The directory may be located in the system disk.
Step 2. Automatic failover
When Server 1 fails, Server 2 takes over. SafeKit restarts the VM containing Bosch AMS on Server 2. Hyper-V finds the files replicated by SafeKit uptodate on Server 2.
The VM continues to run on Server 2 by locally modifying its files that are no longer replicated to Server 1.
The failover time is equal to the fault-detection time (set to 30 seconds by default) plus the VM reboot time.
Step 4. Back to normal
After reintegration, the VM files are once again in mirror mode, as in step 1. The system is back in high-availability mode, with the VM containing Bosch AMS running on Server 2 and SafeKit replicating file updates to Server 1.
If the administrator wishes the VM to run on Server 1, he/she can execute a "swap" command either manually at an appropriate time, or automatically through configuration.
More information on power outage and network isolation in a cluster.
Redundancy at the application level
In this type of solution, only application data are replicated. And only the application is restared in case of failure.
With this solution, restart scripts must be written to restart the application.
We deliver application modules to implement redundancy at the application level. They are preconfigured for well known applications and databases. You can customize them with your own services, data to replicate, application checkers. And you can combine application modules to build advanced multi-level architectures.
This solution is platform agnostic and works with applications inside physical machines, virtual machines, in the Cloud. Any hypervisor is supported (VMware, Hyper-V...).
Redundancy at the virtual machine level
In this type of solution, the full Virtual Machine (VM) is replicated (Application + OS). And the full VM is restarted in case of failure.
The advantage is that there is no restart scripts to write per application and no virtual IP address to define. If you do not know how the application works, this is the best solution.
This solution works with Windows/Hyper-V and Linux/KVM but not with VMware. This is an active/active solution with several virtual machines replicated and restarted between two nodes.
- Solution for a new application (no restart script to write): Windows/Hyper-V, Linux/KVM
More comparison between VM HA vs Application HA
Why a replication of a few Tera-bytes?
Resynchronization time after a failure (step 3)
- 1 Gb/s network ≈ 3 Hours for 1 Tera-bytes.
- 10 Gb/s network ≈ 1 Hour for 1 Tera-bytes or less depending on disk write performances.
Alternative
- For a large volume of data, use external shared storage.
- More expensive, more complex.
Why a replication < 1,000,000 files?
- Resynchronization time performance after a failure (step 3).
- Time to check each file between both nodes.
Alternative
- Put the many files to replicate in a virtual hard disk / virtual machine.
- Only the files representing the virtual hard disk / virtual machine will be replicated and resynchronized in this case.
Why a failover ≤ 32 replicated VMs?
- Each VM runs in an independent mirror module.
- Maximum of 32 mirror modules running on the same cluster.
Alternative
- Use an external shared storage and another VM clustering solution.
- More expensive, more complex.
Why a LAN/VLAN network between remote sites?
- Automatic failover of the virtual IP address with 2 nodes in the same subnet.
- Good bandwidth for resynchronization (step 3) and good latency for synchronous replication (typically a round-trip of less than 2ms).
Alternative
- Use a load balancer for the virtual IP address if the 2 nodes are in 2 subnets (supported by SafeKit, especially in the cloud).
- Use backup solutions with asynchronous replication for high latency network.
Prerequisites
- You need the Hyper-V role installed on 2 Windows nodes (embedded for free in all Windows versions including Windows for PC).
- You need your critical applications installed inside one or more virtual machines.
- As the failover script imports a virtual machine on the secondary Hyper-V, be careful on Hyper-V settings and processor compatibility between both nodes.
Package installation on Windows
-
Download the free version of SafeKit on 2 Windows nodes.
Note: the free version includes all SafeKit features. At the end of the trial, you can activate permanent license keys without uninstalling the package.
-
To open the Windows firewall, on both nodes start a powershell as administrator, and type
c:/safekit/private/bin/firewallcfg add -
To initialize the password for the default admin user of the web console, on both nodes start a powershell as administrator, and type
c:/safekit/private/bin/webservercfg.ps1 -passwd pwd
- Use aphanumeric characters for the password (no special characters).
- pwd must be the same on both nodes.
-
Exclude from antivirus scans C:\safekit\ (the default installation directory) and all replicated folders that you are going to define.
Antiviruses may face detection challenges with SafeKit due to its close integration with the OS, virtual IP mechanisms, real-time replication and restart of critical services.
Module installation on Windows
-
Download the hyperv.safe module.
The module is free. It contains the files userconfig.xml and the restart scripts.
- Put hyperv.safe under C:\safekit\Application_Modules\generic\.
The Hyper-V configuration is presented with a virtual machine named VM1 and containing the application to restart in case of failure.
You will have to repeat this configuration for all VMs that you want to replicate and to restart. SafeKit supports up to 32 virtual machines.
1. Prerequisites
The VM1 virtual machine files (VM1 configuration file, virtual hard disk...) must be put in a same folder: this folder will be replicated by SafeKit.
Make sure the virtual switch name(s) referenced by the virtual machine exist on both Hyper-V servers and corresponds to the same physical network.
It is important that all the files representing a VM (Virtual Machines/, Virtual Hard Disks/, .vmcx, .vmgs, .vmrs, .vhdx, ...) are in the same directory replicated by SafeKit. Else your VM will not reboot after replication. If you are not sure, take a snapshot or your VM before starting the procedure.
If all the files of VM1 are not in the same folder, use Hyper-V manager:
- Export VM1 in a folder, for example in D:\Repli-Hyper-V
- This export will create a folder D:\Repli-Hyper-V\VM1\ containing all VM1 files
- Remove VM1 from the inventory of Hyper-V manager
- Import VM1, previously exported, into Hyper-V manager
VM1 should only be created on a single node. The only thing to create on the other node is the VM1 directory (D:\Repli-Hyper-V\VM1\).
2. Launch the SafeKit console
- Launch the web console in a browser on one cluster node by connecting to
http://localhost:9010. - Enter
adminas user name and the password defined during installation.
You can also run the console in a browser on a workstation external to the cluster.
The configuration of SafeKit is done on both nodes from a single browser.
To secure the web console, see 11. Securing the SafeKit web service in the User's Guide.
3. Configure node addresses
- Enter the node IP addresses, press the Tab key to check connectivity and fill node names.
- Then, click on
Save and applyto save the configuration.
If either node1 or node2 has a red color, check connectivity of the browser to both nodes and check firewall on both nodes for troubleshooting.
If you want, you can add a new LAN for a second heartbeat and for a dedicated replication network.
This operation will place the IP addresses in the cluster.xml file on both nodes (more information in the training with the command line).
4. Select a module
- In
New module, click on thehyperv.safemodule.
The console finds xxx.safe in the Application_Modules/generic/ directory on the server side if you dropped a module there during installation.
5. Configure the module
- Choose an
Automaticstart of the module at boot without delay. - Normally, you have a single
Heartbeatnetwork on which the replication is made. But, you can define a private network if necessary (by adding a LAN at step 3). - Put in
VM_PATH, the root path of the replicated directory (D:\Repli-Hyper-V). - Enter in
VM_NAME, the name of the virtual machine (VM1).
We assume that all VM1 files are in D:\Repli-Hyper-V\VM1\ (see prerequisites). This directory will be replicated in real-time by SafeKit.
The NORMAL_STOP and FORCE_STOP values can be "stop", "save" or "off":
- "
stop" shutdowns the VM when the module is stopped. - "
save" saves the current state of the VM (suspend) when the module is stopped. - "
off" turns off the VM (power off) when the module is stopped.
"stop" is recommended because it causes the shutdown then the reboot of the VM when the module is stopped and restarted. Thus, if the application inside the VM fails, it is restarted.
This will be the case, for example, when switching between primary and secondary roles.
You do not need to configure a virtual IP address. VM1 will be rebooted on the secondary Hyper-V with its physical IP address, and this IP address will be rerouted.
6. Custom checker to detect VM malfunction
The custom checker sends heartbeat messages from the host to the VM at regular intervals. It is then the job of the Hyper-V Heartbeat Service installed in the VM to send a response to each of these heartbeat messages.
If the Hyper-V Heartbeat Service does not respond to the message (VM locked up, crashed or ceased to function), then the custom checker executes an action to restart the VM on the same Hyper-V node or on the other.
- Click on
Checkers / Custom(see image). - Just set a name of your choice in
Resource name(example VM1).Resource nameidentifies the virtual machine with a resource name in SafeKit:custom.VM1. - With
restartinAction, the VM is restarted on the same Hyper-V node. After 3 unsuccessful restarts in 24 hours, the SafeKit hyperv module stops on the primary node and there is a failover of the VM on the secondary node. - If you set
stopstartinAction, there is a direct failover on the other Hyper-V node as soon as the VM does not respond to heartbeats.
For maintenance, if you want to stop the virtual machine, the custom checker will restart it automatically. To avoid that, you can temporarly suspend the checker. Or you can remove it by clicking on Advanced configuration (toggle button at the top of the configuration).
7. Edit scripts (optional)
- Do not edit scripts.
8. Communication encryption (optional)
- Keep encryption of communication between nodes.
9. Save and apply
Save and applythe configuration and scripts on both nodes.
10. Verify successful configuration
- Check the
Successmessage (green) on both nodes and click onMonitor modules.
11. Start the node with up-to-date data
- If node 1 has the up-to-date replicated directory for
VM1/, select it and start itAs primary.
When node 2 will be started, all data will be copied from node 1 to node 2.
If you make the wrong choice, you run the risk of synchronizing outdated data on both nodes.
It is also assumed that VM1 is stopped on node 1 so that SafeKit installs the replication mechanisms and then starts VM1 in the start_prim script.
Use Start for subsequent starts: SafeKit retains the most up-to-date server. Starting As primary is a special start-up the first time or during exceptional operations.
12. Wait for the transition to ALONE (green)
- Node 1 should reach the ALONE (green) state, which means that the
start_primscript has been executed on node 1.
If ALONE (green) is not reached or if VM1 is not started, analyze why with the module log of node 1.
- click the "log" icon of
node1to open the module log and look for error messages such as a checker detecting an error and stopping the module. - click on
start_primin the log: output messages of the script are displayed on the right and errors can be detected such as VM1 incorrectly started.
If the cluster is in WAIT (red) not uptodate, STOP (red) not uptodate state, stop the WAIT node and force its start as primary.
13. Start node 2
- Start node 2 with its contextual menu.
- Wait for the SECOND (green) state.
Node 2 stays in the SECOND (orange) state while resynchronizing the replicated directories (copy from node 1 to node 2).
This may take a while depending on the size of files to resynchronize in replicated directories and the network bandwidth.
To see the progress of the copy, see the module log and the replication resources of node 2.
When using Hyper-V differencing disks, only the differencing disk needs resynchronization after the initial sync, saving time for large virtual hard disks.
14. Verify that the cluster is operational
- Check that the cluster is green/green with
VM1running on the PRIM node and not running on the SECOND node.
Only changes inside files are replicated in real time in this state.
15. Testing
- Stop the PRIM node by scrolling down its contextual menu and clicking
Stop. - Verify that there is a failover on the SECOND node which should become ALONE (green).
- Check with Hyper-V manager that
VM1is running on node 2.
If ALONE (green) is not reached on node2 or if VM1 is not started, analyze why with the module log of node 2.
- click the "log" icon of
node2to open the module log and look for error messages such as a checker detecting an error and stopping the module. - click on
start_primin the log: output messages of the script are displayed on the right and errors can be detected such as VM1 incorrectly started.
As the start_prim script imports the virtual machine on node 2, the failover can fail because of Hyper-V settings (see this article).
If everything is okay, initiate a start on node1, which will resynchronize the replicated directories from node2.
If things go wrong, stop node2 and force the start as primary of node1, which will restart with its locally healthy data at the time of the stop.
16. Automatic restart if a critical service fails inside the VM
If you want an automatic restart or failover when a critical service inside the VM fails, you can configure the Recovery properties of the service (see image).
First you must configure the custom checker previously described.
And then in Microsoft Service Manager inside the VM, select your critical service and in the recovery property of the service, you just have to configure shutdown of the VM when the critical service fails.
For that, in the Recovery options of your critical service, choose "Run a Program" on failures and in Run program options, set"C:\Windows\System32\shutdown.exe" and in "Command line parameters", set /s /c "service fails".
Of course, you can implement more subtle recovery with your own scripts. But just be aware that shutting down the VM will enable the custom checker in the host. The custom checker will detect that the Hyper-V heartbeat is no more responding and will restart the VM on the same Hyper-V server or will make a failover on the other Hyper-V server (depending on its configuration).
To test the feature, use Task Manager and kill the process (End task) associated to the critical service. A clean stop of the service through Service Manager or the "net stop" command does not trigger the recovery action inside Windows Service Manager.
17. Remove the start of VM1 in Hyper-V at boot
- In Hyper-V manager, configure VM1 with Automatic Start Action=nothing.
SafeKit controls the start and the stop of the VM on one node or the other.
18. Support
- For getting support, take 2 SafeKit
Snapshots(2 .zip files), one for each node. - If you have an account on https://support.evidian.com, upload them in the call desk tool.
19. If necessary, configure a splitbrain checker
- See below "What are the different scenarios in case of network isolation in a cluster?" to know if you need to configure a splitbrain checker.
- Go to the module configuration and click on
Checkers / Splitbrain(see image) to edit the splitbrain parameters. Save and applythe new configuration to redeploy it on both nodes (module must be stopped on both nodes to save and apply).
Parameters:
Resource nameidentifies the witness with a resource name:splitbrain.witness. You can change this value to identify the witness.Witness addressis the argument for a ping when a node goes from PRIM to ALONE or from SECOND to ALONE. Change this value with the IP of the witness (a robust element, typically a router).- Note: you can set several IPs separated by white spaces. Pay attention that the IP addresses must be accessible from one node but not from the other in the event of network isolation.
A single network
When there is a network isolation, the default behavior is:
- as heartbeats are lost for each node, each node goes to ALONE and runs the application with its virtual IP address (double execution of the application modifying its local data),
- when the isolation is repaired, one ALONE node is forced to stop and to resynchronize its data from the other node,
- at the end the cluster is PRIM-SECOND (or SECOND-PRIM according the duplicate virtual IP address detection made by Windows).
Two networks with a dedicated replication network
When there is a network isolation, the behavior with a dedicated replication network is:
- a dedicated replication network is implemented on a private network,
- heartbeats on the production network are lost (isolated network),
- heartbeats on the replication network are working (not isolated network),
- the cluster stays in PRIM/SECOND state.
A single network and a splitbrain checker
When there is a network isolation, the behavior with a split-brain checker is:
- a split-brain checker has been configured with the IP address of a witness (typically a router),
- the split-brain checker operates when a server goes from PRIM to ALONE or from SECOND to ALONE,
- in case of network isolation, before going to ALONE, both nodes test the IP address,
- the node which can access the IP address goes to ALONE, the other one goes to WAIT,
- when the isolation is repaired, the WAIT node resynchronizes its data and becomes SECOND.
Note: If the witness is down or disconnected, both nodes go to WAIT and the application is no more running. That's why you must choose a robust witness like a router.
Network load balancing and failover |
|
| Windows farm | Linux farm |
| Generic Windows farm > | Generic Linux farm > |
| Microsoft IIS > | - |
| NGINX > | |
| Apache > | |
| Amazon AWS farm > | |
| Microsoft Azure farm > | |
| Google GCP farm > | |
| Other cloud > | |
Advanced clustering architectures
Several modules can be deployed on the same cluster. Thus, advanced clustering architectures can be implemented:
- the farm+mirror cluster built by deploying a farm module and a mirror module on the same cluster,
- the active/active cluster with replication built by deploying several mirror modules on 2 servers,
- the Hyper-V cluster or KVM cluster with real-time replication and failover of full virtual machines between 2 active hypervisors,
- the N-1 cluster built by deploying N mirror modules on N+1 servers.
| SafeKit with the Hyper-V module or the KVM module | Microsoft Hyper-V Cluster & VMware HA |
|
|
 No shared disk - synchronous real-time replication instead with no data loss |
 Shared disk and specific extenal bay of disk |
| Remote sites = no SAN for replication |
Remote sites = replicated bays of disk across a SAN |
| No specific IT skill to configure the system |
Specific IT skills to configure the system |
Note that the Hyper-V/SafeKit and KVM/SafeKit solutions are limited to replication and failover of 25 VMs.
| VM HA with the SafeKit Hyper-V or KVM module | Application HA with SafeKit application modules |
|
SafeKit inside 2 hypervisors
|
SafeKit inside 2 virtual or physical machines
|
| Replicates more data (App+OS) | Replicates only application data |
|
Reboot of VM on hypervisor 2 if hypervisor 1 crashes Recovery time depending on the OS reboot |
Quick recovery time with restart of App on OS2 if crash of server 1 Around 1 mn or less (see RTO/RPO here) Application checker and software failover |
| Generic solution for any application / OS | Restart scripts to be written in application modules |
Software clustering vs hardware clustering More info > |
|
|
|
Shared nothing vs a shared disk cluster More info > |
|
|
|
Application High Availability vs Full Virtual Machine High Availability More info > |
|
|
|
High availability vs fault tolerance More info > |
|
|
|
Synchronous replication vs asynchronous replication More info > |
|
|
|
Byte-level file replication vs block-level disk replication More info > |
|
|
|
Heartbeat, failover and quorum to avoid 2 master nodes More info > |
|
|
|
Virtual IP address primary/secondary, network load balancing, failover More info > |
|
|
|
Evidian SafeKit 8.2
All new features compared to SafeKit 7.5 described in the release notes
Packages
- Windows (with Microsoft Visual C++ Redistributable)
- Windows (without Microsoft Visual C++ Redistributable)
- Linux
- Supported OS and last fixes
One-month license key
Technical documentation
Training
Product information
New application (real-time replication and failover)
Database (real-time replication and failover)
Full VM or container real-time replication and failover
New application (network load balancing and failover)
Web (network load balancing and failover)
Cloud (network load balancing and failover)
Physical security (real-time replication and failover)
- Installation guide with Milestone XProtect Management Server (milestone.safe)
- Installation guide with Microsoft SQL Server (slqserver.safe) for Genetec
- Installation guide with Nedap (nedap.safe)
- Installation guide with Windows Hyper-V (hyperv.safe) for Bosch AMS
- Installation guide with Windows Hyper-V (hyperv.safe) for Bosch BIS
- Installation guide with Windows Hyper-V (hyperv.safe) for Bosch BVMS
- Installation guide with Windows Hyper-V (hyperv.safe) for Hanwha Vision
- Installation guide with Windows Hyper-V (hyperv.safe) for Hanwha Wisenet
Siemens (real-time replication and failover)
- Installation guide with Windows Hyper-V (hyperv.safe) for Siemens Siveillance suite
- Installation guide with Windows Hyper-V (hyperv.safe) for Siemens Desigo CC
- Installation guide with Windows Hyper-V (hyperv.safe) for Siemens SiPass
- Installation guide with Windows Hyper-V (hyperv.safe) for Siemens SIPORT
- Installation guide with Siemens Siveillance VMS (SiveillanceVMS.safe)
- Installation guide with Windows Hyper-V (hyperv.safe) for Siemens SIMATIC WinCC
- Installation guide with Windows Hyper-V (hyperv.safe) for Siemens SIMATIC PCS 7
Cloud (real-time replication and failover)
Introduction
-
- Demonstration
- Examples of redundancy and high availability solution
- Evidian SafeKit sold in many different countries with Milestone
- 2 solutions: virtual machine or application cluster
- Distinctive advantages
- More information on the web site
-
- Cluster of virtual machines
- Mirror cluster
- Farm cluster
Installation, Console, CLI
- Install and setup / pptx
- Package installation
- Nodes setup
- Upgrade
- Web console / pptx
- Configuration of the cluster
- Configuration of a new module
- Advanced usage
- Securing the web console
- Command line / pptx
- Configure the SafeKit cluster
- Configure a SafeKit module
- Control and monitor
Advanced configuration
- Mirror module / pptx
- start_prim / stop_prim scripts
- userconfig.xml
- Heartbeat (<hearbeat>)
- Virtual IP address (<vip>)
- Real-time file replication (<rfs>)
- How real-time file replication works?
- Mirror's states in action
- Farm module / pptx
- start_both / stop_both scripts
- userconfig.xml
- Farm heartbeats (<farm>)
- Virtual IP address (<vip>)
- Farm's states in action
Troubleshooting
- Troubleshooting / pptx
- Analyze yourself the logs
- Take snapshots for support
- Boot / shutdown
- Web console / Command lines
- Mirror / Farm / Checkers
- Running an application without SafeKit
Support
- Evidian support / pptx
- Get permanent license key
- Register on support.evidian.com
- Call desk