Hanwha Wisenet SSM (Smart Security Manager): The Simplest High Availability Cluster with Synchronous Replication and Failover

Installation of a SafeKit / Wisenet SSM high availability cluster with real-time synchronous replication and failover between two redundant servers

Prerequisites

• you need Wisenet SSM installed on 2 nodes (virtual machines or physical servers)
• on Windows, with Windows services manager, put Wisenet SSM services with Boot Startup Type = Manual on both nodes. SafeKit controls start of Wisenet SSM services in start_prim. Edit start_prim during the configuration to check if you have put all services in Manual boot including the new ones that you can add.
• on Windows 10, enable execution of powershell scripts on both nodes

Package installation on Windows

• Install the free version of SafeKit on 2 Windows nodes with Wisenet SSM services installed on both nodes
  
  Free trial of SafeKit on Windows >
• Download the Windows module
  
  Module hanwha_ssm.safe for Windows >
  
  The module contains the files userconfig.xml, start_prim and stop_prim described in the internals tab
• Put hanwha_ssm.safe under C:\safekit\Application_Modules\demo\ (create the demo directory if it does not exist)
• To open the Windows firewall, on both nodes start a command line as administrator, goto C:\safekit\private\bin and type  .\firewallcfg.cmd add
• For synchronizing SafeKit at boot and at shutdown, on both nodes start a command line as administrator, goto C:\safekit\private\bin and type  .\addStartupShutdown.cmd only once

   User's guide available here

Go to the Configuration tab (above)

Configuration of a SafeKit / Wisenet SSM high availability cluster with real-time synchronous replication and failover between two redundant servers

Configuration instructions

  Important: all the SafeKit configuration is made from a single browser.

Launch the web console in a browser on one node by connecting to http://localhost:9010 (next image). Note that you can run the console in a browser on an external workstation.

Enter IP address of the first node and click on Confirm (next image)

Click on New node and enter IP address of the second node (next image)
Then click on the red floppy disk to save the configuration (next image)

   In the previous image, the save operation put the IP addresses in the file c:\safekit\var\cluster\cluster.xml and apply the distributed command c:\safekit\safekit –H "*" -G which configures node 1 and node 2. So do not redo this operation on the other node. Check the connectivity and firewall between both nodes for troubleshooting.

In the Configuration tab (next image), click on hanwha_ssm.safe (xxx.safe in the next image) then enter hanwha_ssm as the module name and Confirm (the console finds hanwha_ssm.safe in the 'demo/' directory that you previously filled during the installation)

Click on Validate (next image)

After the click on Validate, this new window appears (next image)

On the previous image:

• Check that replicated directories are installed on both nodes and contain application data.
  Replication of data and also logs is required for a database.
  You can add new directories to replicate if necessary.
• Enter a virtual IP address. A virtual IP address is a standard IP address in the same IP network (same subnet) as the IP addresses of both nodes.
  Clients of the application must be configured with the virtual IP address (or the DNS name associated to the virtual IP address).
  The virtual IP address is automatically switched in case of failover.
• start_prim and stop_prim must contain the start and the stop of the Wisenet SSM application.
  Edit the scripts and check that the names of services in these scripts are those installed on your both machines (else modify them in the scripts).
  You can add new services in the scripts if necessary.
• Stop the services configured in start_prim on both nodes.
• On Windows and on both nodes, with Windows services manager, put Wisenet SSM services set inside start_prim with Boot Startup Type = Manual on both servers (SafeKit controls start of Wisenet SSM services in start_prim).
• Note that if a process name is displayed in Process checker, it will be monitored with the action restart in case of failure. The services will be stopped an restarted locally on the primary server if this process disappears from the list of running processes. After 3 unsuccessful local restarts, the module is stopped on the primary server and the services are restarted on the secondary server.

Click on Validate (previous image)

Click on Configure (previous image)

Check the success green message on both servers and click on Next (previous image).

We assume that node 1 is the referencial with the up-to-date data (otherwise do the following operation on node 2).
We assume that the services started automatically by SafeKit in the start_prim script are stopped on node 1.
Select node 1 and click on start it (next image).
When you will start node 2, the first resynchronization will be made from node1 to node 2.
If in doubt, before this operation, we suggest you to make a copy of replicated directories before starting the cluster to avoid any errors.

node 1 must reach the ALONE (green) state (next image) meaning that the start_prim script has been executed on node 1.
When node 1 reaches the ALONE (green) state, you can check that the Wisenet SSM application is started on node 1.
If the state is ALONE (green) and if the application is not started, check the start_prim script: you can find output messages of the script in the Application Log available in the Control tab.
If node 1 does not reach the ALONE (green) state, analyze why with the Module Log available in the Control tab.
Note: in the next image, if the cluster is in the state [WAIT (red) not uptodate - STOP (red) not uptodate], stop the WAIT server and force its start as primary (click here to see the operation).

We assume that the services started automatically by SafeKit in the start_prim script are stopped on node 2.
Start node 2 (previous image) which becomes SECOND green (next image) after resynchronisation of all replicated directories (binary copy from node 1 to node 2).

The cluster is operational with Wisenet SSM services running on the PRIM node and nothing running on the SECOND node (previous image). Only modifications inside files are replicated in real-time in this state.

Be careful, components which are clients of the Wisenet SSM services must be configured with the virtual IP address. The configuration can be made with a DNS name (if a DNS name has been created and associated with the virtual IP address).

Test

Check with Windows Microsoft Management Console (MMC) that the Wisenet SSM services have been started on the primary server by the start_prim script and stopped on the secondary server by the stop_prim script.

Stop the PRIM node by scrolling down the menu of the primary node and by clicking on Stop. Check that there is a failover on the SECOND node. And with Windows Microsoft Management Console (MMC), check the failover of Wisenet SSM services (stopped and started in the stop_prim and start_prim scripts).

More information on tests in the User's Guide

Automatic start of the module at boot

Configure boot start (next image on the right side) configures the automatic boot of the module when the server boots. Do this configuration on node1 and node2 once the high availability solution is correctly running.

Note that on Windows, with Windows services manager, we assume that Wisenet SSM services are with Boot Startup Type = Manual on both nodes. SafeKit controls start of Wisenet SSM services when starting the module in start_prim.

Note that for synchronizing SafeKit at boot and at shutdown, on both nodes, we assume that a command line has been run during installation as administrator: .\addStartupShutdown.cmd in C:\safekit\private\bin (otherwise dot it now only once).

For reading the SafeKit logs, go to the Troubleshooting tab

For editing userconfig.xml, start_prim and stop_prim, go to the Advanced Configuration tab

Troubleshooting of a SafeKit / Wisenet SSM high availability cluster with real-time synchronous replication and failover between two redundant servers

Module log

Read the module log to understand the reasons of a failover, of a waiting state on the availability of a resource etc...
To see the module log of the primary server (next image):

• click on the Control tab
• click on node 1/PRIM (it becomes blue) on the left side to select the server
• click on Module Log
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the module log in a .txt file and to analyze in a text editor

Repeat the same operation to see the module log of the secondary server.

Application log

Read the application log to see the output messages of the stat_prim and stop_prim restart scripts.
To see the application log of the primary server (next image):

• click on the Control tab
• click on node 1/PRIM (it becomes blue) on the left side to select the server
• click on Application Log to see messages when starting and stopping Wisenet SSM services
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the application log in a .txt file and to analyze in a text editor

Repeat the same operation to see the application log of the secondary server.

More information on troubleshooting in the User's Guide

For support, go to the Support tab

Advanced configuration of a SafeKit / Wisenet SSM high availability cluster with real-time synchronous replication and failover between two redundant servers

In Advanced Configuration tab (next image), you can edit internal files of the module: bin/start_prim and bin/stop_prim and conf/userconfig.xml (next image on the left side). If you make change in the internal files here, you must apply the new configuration by a right click on the icon/xxx on the left side (next image): the interface will allow you to redeploy the modified files on both servers.

More information on userconfig.xml in the User's Guide

For an example of userconfig.xml, start_prim and stop_prim, go to the Internals tab

Support of a SafeKit / Wisenet SSM high availability cluster with real-time synchronous replication and failover between two redundant servers

For getting support on the call desk of https://support.evidian.com, get 2 Snaphots (2 .zip files), one for each server and upload them in the call desk tool (next image).

More information on support in the User's Guide

Internals of a SafeKit / Hanwha SSM high availability cluster with synchronous replication and failover

Go to the Advanced Configuration tab, for editing these files

Internal files of the Windows hanwha_ssm.safe module

userconfig.xml (description in the User's Guide)

<!DOCTYPE safe>
<safe>
<service mode="mirror" defaultprim="alone" maxloop="3" loop_interval="24" failover="on">
  <!-- Heartbeat Configuration -->
  <!-- Names or IP addresses on the default network are set during initialization in the console -->
  <heart pulse="700" timeout="30000">
    <heartbeat name="default" ident="flow">
    </heartbeat>
  </heart>
  <!-- Virtual IP Configuration -->
  <!-- Replace
     * VIRTUAL_TO_BE_DEFINED by the name of your virtual server 
  -->
  <vip>
    <interface_list>
        <interface check="on" arpreroute="on"> 
	      <real_interface>
               <virtual_addr addr="VIRTUAL_TO_BE_DEFINED" where="one_side_alias" />
          </real_interface>
        </interface>
    </interface_list>
  </vip>
  <!-- Software Error Detection Configuration -->
  <errd polltimer="10">
    <!-- Samsung SSM process -->
         <proc name="pg_ctl.exe" atleast="1" action="restart" class="prim" />
         <proc name="ServiceManager.exe" atleast="1" action="restart" class="prim" />
		<!--         
			<proc name="HAServerService.exe" atleast="1" action="restart" class="prim"/>
		-->
  </errd>
  <!-- File Replication Configuration -->
  <!-- Adapt with the directory of your PostgreSQL database and logs
  -->
  <rfs async="second" acl="off" nbrei="3">
     <replicated dir="C:\PostgreSQL\9.1\data" mode="read_only">
        <notreplicated path="pg_log"/>
        <notreplicated path="postmaster.pid"/>
     </replicated>
     <replicated dir="C:\Program Files (x86)\Samsung\SSM\SystemManager\MapFile" mode="read_only"/>
  </rfs>
  <!-- User scripts activation -->
  <user nicestoptimeout="300" forcestoptimeout="300" logging="userlog" />
</service>
</safe>

start_prim.cmd

@echo off

rem Script called on the primary server for starting application services 

rem For logging into SafeKit log use:
rem "%SAFE%\safekit" printi | printe "message"

rem stdout goes into Application log
echo "Running start_prim %*" 

set res=0

rem TODO: set to manual the start of services when the system boots

net start "postgresql-9.1 - PostgreSQL Server 9.1"
if not %errorlevel% == 0 (
  %SAFE%\safekit printi "PostgreSQL (postgresql-9.1 - PostgreSQL Server 9.1) start failed"
  goto stop
) else (
  %SAFE%\safekit printi "PostgreSQL (postgresql-9.1 - PostgreSQL Server 9.1) started"
)

net start "SSM System Manager"
if not %errorlevel% == 0 (
  %SAFE%\safekit printi "SSM System Manager start failed") 
  goto stop
) else (
  %SAFE%\safekit printi "SSM System Manager started"
)

net start "SSM Watch Services Manager"
if not %errorlevel% == 0 (
  %SAFE%\safekit printi "SSM Watch Services Manager start failed"
  goto stop
) else (
  %SAFE%\safekit printi "SSM Watch Services Manager started"
)

rem net start "HA Server Service"
rem if not %errorlevel% == 0 (
rem  %SAFE%\safekit printi "HA Server Service start failed"
rem  goto stop
rem ) else (
rem   %SAFE%\safekit printi "HA Server Service started"
rem )

if %res% == 0 goto end

:stop
set res=%errorlevel%
"%SAFE%\safekit" printe "start_prim failed"

rem uncomment to stop SafeKit when critical
rem "%SAFE%\safekit" stop -i "start_prim"

:end

stop_prim.cmd

@echo off

rem Script called on the primary server for stopping application services 

rem For logging into SafeKit log use:
rem "%SAFE%\safekit" printi | printe "message"

rem ----------------------------------------------------------
rem
rem 2 stop modes:
rem
rem - graceful stop
rem   call standard application stop with net stop
rem
rem - force stop (%1=force)
rem   kill application's processes
rem
rem ----------------------------------------------------------

rem stdout goes into Application log
echo "Running stop_prim %*" 

set res=0

rem action on force stop
if "%1" == "force" (
  %SAFE%\safekit printi "Force stop: kill processes of Samsung SSM application" 
  %SAFE%\safekit kill -name="watchservices.exe" -level="terminate"
  %SAFE%\safekit kill -name="java.exe" -argregex=".*systemmanager.*" -level="terminate"
  %SAFE%\safekit kill -name="pg_ctl.exe" -level="terminate"
  %SAFE%\safekit kill -name="postgres.exe" -level="terminate"
  %SAFE%\safekit kill -name="HAServerService.exe" -level="terminate"
  %SAFE%\safekit kill -name="systemanager.exe" -level="terminate"
  goto end
)

rem %SAFE%\safekit printi "Stopping HA Server Service"
rem net stop "HA Server Service"

%SAFE%\safekit printi "Stopping SSM Watch Services Manager"
net stop "SSM Watch Services Manager"

%SAFE%\safekit printi "Stopping SSM System Manager"
net stop "SSM System Manager"

%SAFE%\safekit printi "Stopping PostgreSQL (postgresql-9.1 - PostgreSQL Server 9.1)"
net stop "postgresql-9.1 - PostgreSQL Server 9.1"
del C:\PostgreSQL\9.1\data\postmaster.pid

rem Wait a little for the real stop of services
"%SAFEBIN%\sleep" 10

if %res% == 0 goto end

"%SAFE%\safekit" printe "stop_prim failed"

:end