Evidian SafeKit brings high availability to Milestone XProtect, the CCTV video surveillance system between two redundant servers. This article explains how to implement quickly a Milestone cluster with real-time replication and automatic failover of the management services and the SQL database, and without the complexity of Microsoft failover clustering (without shared disk and without domain controller configuration). The milestone.safe high availability module and a free trial are offered in the Installation tab below.
This clustering solution is recognized as the simplest to implement by our Milestone partners. It is also a complete solution that solves hardware failures (20% of problems) including the complete failure of a computer room, software failures (40% of problems) including smooth upgrade server by server and human errors (40% of problems) thanks to its simplicity of administration.
In the Milestone system architecture, SafeKit clusters can be installed for high availability of Milestone XProtect (any edition) with:
All the services (SQL+Management+Log+Event) can be installed on a single cluster and we consider this use case in this article. The configuration can also be easily split on several clusters.
The SafeKit solution has been verified by Milestone. Note that high availability of recording servers is already managed by the Milestone built-in solution.
Note that SafeKit is a generic product. You can implement with the same product real-time replication and failover
of new directories and services, databases, docker containers, full Hyper-V or KVM virtual machines, Cloud applications. See other examples of mirror modules here.
On the figure with two redundant servers, the server 1/PRIM runs Milestone XProtect and SQL services (any edition). Users are connected to the virtual IP address of the mirror cluster. SafeKit replicates files opened by Milestone XProtect and SQL services in real time. Only changes in the files are replicated across the network, thus limiting traffic (byte-level file replication). Names of file directories containing Milestone XProtect and SQL data are simply configured in SafeKit. There are no pre-requisites on disk organization for the two servers. Directories to replicate may be located in the system disk. SafeKit implements synchronous replication with no data loss on failure contrary to asynchronous replication.
In case of server 1 failure, there is an automatic failover on server 2 with virtual IP failover and restart of Milestone XProtect and SQL services. Then, when server 1 is restarted, SafeKit implements automatic failback with reintegration of data without stopping Milestone XProtect and SQL services on server 2. Finally, the system returns to synchronous replication between server 2 and server 1. The administrator can decide to swap the role of primary and secondary and to run Milestone XProtect and SQL services on server 1. The swap can also be done automatically by configuration.
Note that there is another SafeKit architecture very easy to deploy and based on Hyper-V to replicate Milestone XProtect server. The SafeKit/Hyper-V architecture replicates the full Hyper-V virtual machine containing Milestone XProtect and restart it in case of failure. Hyper-V manager is freely included in all versions of Windows including Windows 10 Pro. High availability at the application level is better than high availability at the virtual machine level as explained here.
Important. A new milestone.safe module is delivered since Milestone 2019 R2 with its identity server: follow the special configuration instructions in the Configuration tab.
Evidian SafeKit mirror cluster with real-time file replication and failover
|All clustering features|
A SafeKit cluster runs on Windows and Linux without the need for expensive shared or replicated disk bays
SafeKit includes all clustering features: synchronous real-time file replication, monitoring of server/network/software failures, automatic Milestone XProtect and SQL restart, virtual IP address switched in case of failure to reroute clients
This is not the case with replication-only solutions like replication at the database level which implements only replication
The cluster configuration is very simple and made with the milestone.safe module. There is no domain controller or active directory to configure as with Microsoft cluster
Other services and other replicated directories can be added to the milestone.safe module to complement the SafeKit / Milestone XProtect and SQL high availability solution
SafeKit implements quick Milestone XProtect and SQL restart in case of failure: around 1 mn or less (see RTO/RPO here)
Quick Milestone XProtect and SQL restart is not ensured with full virtual machines replication. In case of hypervisor failure, a full VM must be rebooted on a new hypervisor with a recovery time depending on the OS reboot as with VMware HA or Hyper-V cluster
The real-time replication is synchronous with no data loss on failure
This is not the case with asynchronous replication
|Fully automated failback procedure|
After a failure when a server reboots, the replication failback procedure is fully automatic and the failed server reintegrates the cluster without stopping Milestone XProtect and SQL on the only remaining server
This is not the case with most replication solutions particularly with replication at the database level. Manual operations are required for resynchronizing a failed server. Milestone XProtect and SQL may even be stopped on the only remaining server during the resynchonization of the failed server
|Replication of any type of data|
The replication is working for Milestone XProtect and SQL but also for any files which shall be replicated
This not the case for replication at the database level
|File replication vs disk replication|
The replication is based on file directories that can be located anywhere (even in the system disk)
This is not the case with disk replication where special Milestone XProtect and SQL configuration must be made to put the application data in a special disk
|File replication vs shared disk|
The servers can be put in two remote sites
This is not the case with shared disk solutions
|Remote sites and virtual IP address|
All SafeKit clustering features are working for 2 servers in remote sites. Performances of replication depends on the interconnect latency for real-time synchronous replication and on the bandwidth for resynchronizing data on a failed server
If both servers are connected to the same IP network through an extended LAN between two remote sites, the virtual IP address of SafeKit is working with rerouting at level 2
If both servers are connected to two different IP networks between two remote sites, the virtual IP address can be configured at the level of a load balancer. SafeKit offers a health check: the load balancer is configured with a URL managed by SafeKit which returns OK on the primary server and NOT FOUND else. This solution is implemented for SafeKit in the Cloud but it can be also implemented with a load balancer on premise
With remote sites, the solution works with only 2 servers and for the quorum (network isolation), a simple split brain checker to a router is offered to support a single execution
This is not the case for most clustering solutions where a 3rd server is required for the quorum
The secondary server is not dedicated to the restart of the primary server. The cluster can be active-active by running 2 different mirror modules
This is not the case with a fault-tolerant system where the secondary is dedicated to the execution of the same application synchronized at the instruction level
|Uniform high availability solution|
SafeKit implements a mirror cluster with replication and failover. But it imlements also a farm cluster with load balancing and failover. Thus a N-tiers architecture can be made highly available and load balanced with the same solution on Windows and Linux (same installation, configuration, administration with the SafeKit console or with the command line interface). This is unique on the market
This is not the case with an architecture mixing different technologies for load balancing, replication and failover
High availability architectures comparison
|Software clustering vs hardware clustering|| |
A simple software cluster with the SafeKit package just installed on two servers
Complex hardware clustering with external storage or network load balancers
|Shared nothing vs a shared disk cluster|| |
SafeKit is a shared-nothing cluster: easy to deploy even in remote sites
A shared disk cluster is complex to deploy
|Application High Availability vs Full Virtual Machine High Availability|| |
Application HA supports hardware failure and software failure with a quick recovery time (RTO around 1 mn or less).
Smooth upgrade of application and OS possible server by server (version N and N+1 can coexist)
Full virtual machines HA supports only hardware failure with a VM reboot and a recovery time depending on the OS reboot.
Smooth upgrade not possible
|High availability vs fault tolerance|| |
No dedicated server with SafeKit. Each server can be the failover server of the other one.
Software failure with restart in another OS environment.
Smooth upgrade of application and OS possible server by server (version N and N+1 can coexist)
Secondary server dedicated to the execution of the same application synchronized at the instruction level.
Software exception on both servers at the same time.
Smooth upgrade not possible
|Synchronous replication vs asynchronous replication|| |
SafeKit implements real-time synchronous replication with no data loss in case of failure
With asynchronous replication, there is data loss on failure
|Byte-level file replication vs block-level disk replication|| |
SafeKit implements real-time byte-level file replication and is simply configured with application directories to replicate even in the system disk
Block-level disk replication is complex to configure and requires to put application data in a special disk
|Heartbeat, failover and quorum to avoid 2 master nodes|| |
To avoid 2 masters, SafeKit proposes a simple split brain checker configured on a router
To avoid 2 masters, other clusters require a complex configuration with a third machine, a special quorum disk, a special interconnect
| Virtual IP address|
primary/secondary, network load balancing, failover
No dedicated proxy servers and no special network configuration are required in a SafeKit cluster for virtual IP addresses
Special network configuration is required in other clusters for virtual IP addresses. Note that SafeKit offers a health check adapted to load balancers
|OEM Software||Distributed Enterprise||Remote Sites|
|A software publisher uses SafeKit as an OEM software for high availability of its application||A distributed enterprise deploys SafeKit in many branches without specific IT skills||SafeKit is deployed in two remote sites without the need for replicated bays of disks through a SAN|
The ideal product for a software publisher
“SafeKit is the ideal application clustering solution for a software publisher. We currently have deployed more than 80 SafeKit clusters worldwide with our critical TV broadcasting application.”
The product very easy to deploy for a reseller
“WithNCompany has deployed in South Korea many SafeKit high availability solutions with the Hanwha Video Surveillance Platform. SafeKit is appreciated because the product is easy to install and very quickly deployed.”
The product to gain time for a system integrator
“Thanks to a simple and powerful product, we gained time in the integration and validation of our critical projects like the supervision of Paris and Marseille metro lines (the control rooms).”
In video surveillance systems and access control, Evidian SafeKit implements high availability with synchronous replication and failover of
Sebastien Temoin, Technical and Innovation Director, NOEMIS, value added distributor of Milestone solutions:
"SafeKit by Evidian is a professional solution making easy the redundancy of Milestone Management Server, Event Server, Log Server. The solution is easy to deploy, easy to maintain and can be added on existing installation. We have assisted integrators to deploy the solution on many projects such as city surveillance, datacenters, stadiums and other critical infrastructures. SafeKit is a great product, and Evidian provides great support. Happy to help if you have any questions."
Harmonic is using SafeKit as a software OEM high availability solution and deploys it with its TV broadcasting solutions over satellites, terrestrials, cable, IPTV.
Over 80 SafeKit clusters are deployed on Windows for replication of Harmonic database and automatic failover of the critical application.
Philippe Vidal, Product Manager, Harmonic says:
“SafeKit is the ideal application clustering solution for a software publisher looking for a simple and economical high availability software. We are deploying SafeKit worldwide and we currently have more than 80 SafeKit clusters on Windows with our critical TV broadcasting application through terrestrial, satellite, cable and IP-TV. SafeKit implements the continuous and real-time replication of our database as well as the automatic failover of our application for software and hardware failures. Without modifying our application, it was possible for us to customize the installation of SafeKit. Since then, the time of preparation and implementation has been significantly reduced.”
The European Society of Warranties and Guarantees in Natixis uses SafeKit as a high availability solution for its applications.
Over 30 SafeKit clusters are deployed on Unix and Windows in Natixis.
Fives Syleps implements high availability of its ERP with SafeKit and deploys the solution in the food industry.
Over 20 SafeKit clusters are deployed on Linux and Windows with Oracle.
Testimonial of Fives Syleps:
"The automated factories that we equip rely on our ERP. It is not possible that our ERP is out of service due to a computer failure. Otherwise, the whole activity of the factory stops.
We chose the Evidian SafeKit high availability product because it is an easy to use solution. It is implemented on standard servers and does not require the use of shared disks on a SAN and load balancing network boxes.
It allows servers to be put in remote computer rooms. In addition, the solution is homogeneous for Linux and Windows platforms. And it provides 3 functionalities: load balancing between servers, automatic failover and real-time data replication.”
Over 20 SafeKit clusters are deployed on Windows.
Tony Myers, Director of Business Development says:
"By developing applications for air traffic control, Copperchase is in one of the most critical business activities. We absolutely need our applications to be available all the time. We have found with SafeKit a simple and complete clustering solution for our needs. This software combines in a single product load balancing, real time data replication with no data loss and automatic failover. This is why, Copperchase deploys SafeKit for air traffic control in airports in the UK and the 30 countries where we are present."
Software vendor Wellington IT deploys SafeKit high availability with its banking application for Credit Unions in Ireland and UK.
Peter Knight, Sales Manager says:
"Business continuity and disaster recovery are a major concern for our Locus banking application deployed in numerous Credit Unions around Ireland and the UK. We have found with SafeKit a simple and robust solution for high availability and synchronous replication between two servers with no data loss. With this software solution, we are not dependent on a specific and costly hardware clustering solution. It is a perfect tool to provide a software high availability option to an application of a software vendor."
Paris transport company (RATP) chose the SafeKit high availability and load balancing solution for the centralized control room of line 1 of the Paris subway.
20 SafeKit clusters are deployed on Windows and Linux.
Stéphane Guilmin, RATP, Project manager says:
"Automation of line 1 of the Paris subway is a major project for RATP, requiring a centralized command room (CCR) designed to resist IT failures. With SafeKit, we have three distinct advantages to meet this need. Firstly, SafeKit is a purely software solution that does not demand the use of shared disks on a SAN and network boxes for load balancing. It is very simple to separate our servers into separate machine rooms. Moreover, this clustering solution is homogeneous for our Windows and Linuxplatforms. SafeKit provides the three functions that we needed: load balancing between servers, automatic failover after an incident and real time data replication."
And also, Philippe Marsol, Atos BU Transport, Integration Manager says:
“SafeKit is a simple and powerful product for application high availability. We have integrated SafeKit in our critical projects like the supervision of Paris metro Line 4 (the control room) or Marseille Line 1 and Line 2 (the operations center). Thanks to the simplicity of the product, we gained time for the integration and validation of the solution and we had also quick answers to our questions with a responsive Evidian team.”
The software integrator Systel deploys SafeKit high-availability solution in firefighter and emergency medical call centers.
Over 30 SafeKit clusters are deployed on Windows with SQL Server.
Marc Pellas, CEO says:
"SafeKit perfectly meets the needs of a software vendor. Its main advantage is that it brings in high availability through a software option that is added to our own multi-platform software suite. This way, we are not dependent on a specific and costly hardware clustering solution that is not only difficult to install and maintain, but also differs according to client environments. With SafeKit, our firefighter call centers are run with an integrated software clustering solution, which is the same for all our customers, is user friendly and for which we master the installation up to after-sales support."
ERP high availability and load balancing of the French army (DGA) are made with SafeKit.
Alexandre Barth, Systems administrator says:
"Our production team implemented the SafeKit solution without any difficulty on 14 Windows and Linux clusters. Our critical activity is thus secure, with high-availability and load balancing functions. The advantages of this product are easy deployment and administration of clusters, on the one hand, and uniformity of the solution in the face of heterogeneous operating systems, on the other hand."
This video shows the configuration of a Milestone XProtect mirror module with synchronous real-time replication and failover.
After a failover, automatic restart and reconnection of a Recording Server is also demonstrated.
This video shows a farm module configuration with load balancing and failover.
The load balancing and the failover are configured for Apache but it works in the same manner for other web services.
Real-time file replication and failover: click on the blue buttons
|Microsoft SQL Server||-|
|Genetec SQL Server||-|
|Hanwha Wisenet SSM||-|
|Amazon AWS mirror|
|Microsoft Azure mirror|
|Google GCP mirror|
|Cloud generic mirror|