Siemens SIMATIC WinCC: redundancy & high availability with a very simple active active solution

1. Prerequisites

The VM1 virtual machine files (VM1 configuration file, virtual hard disk...) must be put in a same folder: this folder will be replicated by SafeKit.

Make sure the virtual switch name(s) referenced by the virtual machine exist on both Hyper-V servers and corresponds to the same physical network.

If all the files of VM1 are not in the same folder, use Hyper-V manager:

• Export VM1 in a folder, for example in D:\Repli-Hyper-V
• This export will create a folder D:\Repli-Hyper-V\VM1\ containing all VM1 files
• Remove VM1 from the inventory of Hyper-V manager
• Import VM1, previously exported, into Hyper-V manager

2. Launch the SafeKit console

• Launch the web console in a browser on one node by connecting to http://localhost:9010.
• Enter admin as user name and the password defined during installation.

You can also run the console in a browser on a workstation external to the cluster.

3. Configure node addresses

• Enter the node IP addresses.
• Then, click on Apply to save the configuration.

4. Choose the module

• In the Configuration tab, click on the hyperv.safe module.

5. Configure the module

• Choose an automatic start of the module at boot without delay.
• Normally, you have a single heartbeat network on which the replication is made. But, you can define a private network if necessary.
• Put in VM_PATH, the root path of the replicated directory (D:\Repli-Hyper-V).
• Enter in VM_NAME, the name of the virtual machine (VM1).

We assume that all VM1 files are in D:\Repli-Hyper-V\VM1\ (see prerequisites). This directory will be replicated in real-time by SafeKit.

The NORMAL_STOP and FORCE_STOP values can be "stop", "save" or "off":

• "stop" shutdowns the VM when the module is stopped.
• "save" saves the current state of the VM (suspend) when the module is stopped.
• "off" turns off the VM (power off) when the module is stopped.

"stop" is recommended because it causes the shutdown then the reboot of the VM when the module is stopped and restarted. Thus the application inside the VM is restarted.

This will be the case, for example, when switching between primary and secondary roles.

6. Verify successful configuration

• Check the success message (green) on both nodes and click Next.

7. Start the node with up-to-date data

• If node 1 has the up-to-date VM1\ replicated directory, select it and start it.

When node 2 will be started, all data in VM1\ will be copied from node 1 to node 2.

8. Wait for the transition to ALONE (green)

• Node 1 should reach the ALONE (green) state, which means that the start_prim script has been executed on node 1.

9. Start node 2

• Start node 2 with its contextual menu.
• Wait for the SECOND (green) state.

10. Verify that the cluster is operational

• Check that the cluster is green/green with VM1 running on the PRIM node and not running on the SECOND node.

Only changes inside files are replicated in real time in this state.

11. Testing

• Stop the PRIM node by scrolling down its contextual menu and clicking Stop.
• Verify that there is a failover on the SECOND node which should become ALONE (green).
• Check the restart of VM1 with Hyper-V manager.

12. Custom checker to detect VM malfunction

A custom checker is included in the hyperv.safe module to run in the host and to detect VM malfunction using Hyper-V heartbeat feature: VM locked up, crashed or ceased to function.

The custom checker sends heartbeat messages from the host to the VM at regular intervals. It is then the job of the Hyper-V Heartbeat Service installed on the VM to send a response to each of these heartbeat messages. If the Hyper-V Heartbeat Service does not respond to the message, then the custom checker restarts the VM on the same Hyper-V. If after 3 unsuccessful restarts (maxloop) in 24 hours (loop_interval), then the SafeKit hyperv module stops on the primary server and there is a failover of the VM on the secondary server. There is also a configuration with a direct failover on the other Hyper-V server as soon as the VM does not respond to heartbeats.

How to change conf/userconfig.xml to configure the custom checker

In Advanced Configuration tab, you can edit conf/userconfig.xml .
1) Declare the custom checker and its associated resource custom.VM

<service mode="mirror" maxloop="3" loop_interval="24" ...>
  ...
  <check>
	<custom ident="VM" exec="vmcheck.ps1" arg="%VM_NAME%" when="prim"></custom>
  </check>
  ...

2) Add a failover rule to react on the resource change in conf/userconfig.xml

<failover>
    <![CDATA[
	vm_failure:
		if( custom.VM == down ) then restart();
    ]]>
 </failover>

With stopstart() instead of restart(), there is an immediate failover on the secondary server.
You must apply the new configuration by a right click on the icon/xxx on the left side (see image): the interface will allow you to redeploy the modified userconfig.xml file on both servers.

13. Automatic restart if a critical service fails inside the VM

If you want an automatic restart or failover when a critical service inside the VM fails, you can configure the Recovery properties of the service (see image).

First you must configure the custom checker previously described.

And then in Microsoft Service Manager inside the VM, select your critical service and in the recovery property of the service, you just have to configure shutdown of the VM when the critical service fails.

For that, in the Recovery options of your critical service, choose "Run a Program" on failures and in Run program options, set"C:\Windows\System32\shutdown.exe" and in "Command line parameters", set /s /c "service fails".

Of course, you can implement more subtle recovery with your own scripts. But just be aware that shutting down the VM will enable the custom checker in the host. The custom checker will detect that the Hyper-V heartbeat is no more responding and will restart the VM on the same Hyper-V server or will make a failover on the other Hyper-V server (depending on its configuration).

To test the feature, use Task Manager and kill the process (End task) associated to the critical service. A clean stop of the service through Service Manager or the "net stop" command does not trigger the recovery action inside Windows Service Manager.

14. Remove the start of VM1 in Hyper-V at boot

• In Hyper-V manager, configure VM1 with Automatic Start Action=nothing.

SafeKit controls the start and the stop of the VM on one node or the other.

Module log

• Read the module log to understand the reasons of a failover, of a waiting state etc...

To see the module log of node 1 (image):

• click on the Control tab
• click on node 1/PRIM on the left side to select the server (it becomes blue)
• click on Module Log
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the module log in a .txt file and to analyze in a text editor

Click on node2 to see the module log of the secondary server.

Application log

• Read the application log to see the output messages of the start_prim and stop_prim restart scripts.

To see the application log of node1 (image):

• click on the Control tab
• click on node 1/PRIM on the left side to select the server (it becomes blue)
• click on Application Log to see messages when starting and stopping services
• click on the Refresh icon (green arrows) to update the console
• click on the floppy disk to save the application log in a .txt file and to analyze in a text editor

Click on node 2 to see the application log of the secondary server.

Advanced configuration

• In Advanced Configuration tab, you can edit internal files of the module: bin/start_prim and bin/stop_prim and conf/userconfig.xml .

If you make change in the internal files here, you must apply the new configuration by a right click on the icon/xxx on the left side (see image): the interface will allow you to redeploy the modified files on both servers.

Support

• For getting support, take 2 SafeKit Snapshots (2 .zip files), one for each server.

If you have an account on https://support.evidian.com, upload them in the call desk tool.