The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This chapter provides detailed information about installing VTS in high availability (HA) mode. It details the procedure to enable VTS L2.
See Enabling VTS L2 High Availability for the detailed procedure to enable VTS L2 HA.
master_name and slave_name can not be the same
master_network_interface and slave_network_interface are interface names of VTC1 and VTC2 where the real IP resides. They should be the same.
If you are using VTF's, fill in vip_private and private_network_interface fields. Otherwise, leave these two fields blank.
Private_network_interface is the secondary interface names of VTC1 and VTC2 on the private network that VTF is also on.
vip_private is the vip for the VTS master's private interface.
private_gateway is the gateway for the private network of your vip_private.
This chapter has the following sections.
To enable VTC L2 HA, VTC1 and VTC2 must be on the same subnet.
Note | Cisco VTS supports dual stack clusters for L2 HA. Have both the VTCs (vts01 and vts02) installed and configured with IPv6 & IPv4 address for dual stack to be supported. Both of the VTCs should be reachable by any means with IPv6 address or IPv4 address. |
Note | Before enabling HA, make sure that both VTC 1 and VTC 2 have the same password. If not, go to the VTC GUI and do a change password on newly brought up VTC, to make the password identical with that of the other VTC . When you upgrade a VTC / bring up a new VTC / do a hardware upgrade of VTC host, you should make sure that password is the same. |
You need to set up the VTC environment before you run the high availability script.
You must run the cluster_install.sh script on both VTCs to enable high availability.
Step 1 | Run the cluster
installer script /opt/vts/bin/cluster_install.sh on both VTC1 and VTC2 . For
example:
admin@vts02:/opt/vts/etc$ sudo su - [sudo] password for admin: root@vts02:/opt/vts/etc$ cd ../bin root@vts02:/opt/vts/bin# ./cluster_install.sh 172.23.92.200 vts01 172.23.92.201 vts02 2001:420:10e:2015:c00::200 vts01 2001:420:10e:2015:c00::201 vts02 Change made to ncs.conf file. Need to restart ncs Created symlink from /etc/systemd/system/multi-user.target.wants/pacemaker.service to /lib/systemd/system/pacemaker.service. Created symlink from /etc/systemd/system/multi-user.target.wants/corosync.service to /lib/systemd/system/corosync.service. Both nodes are online. Configuring master Configuring Pacemaker resources Master node configuration finished HA cluster is installed |
Step 2 | Check the status on both the nodes to verify whether both nodes
online, and node which got installed first is the master, and the other, slave.
For example:
admin@vts02:/opt/vts/log/nso$ sudo crm status [sudo] password for admin: Last updated: Mon Apr 10 18:43:52 2017 Last change: Mon Apr 10 17:15:21 2017 by root via crm_attribute on vts01 Stack: corosync Current DC: vts01 (version 1.1.14-70404b0) - partition with quorum 2 nodes and 4 resources configured Online: [ vts01 vts02 ] Full list of resources: Master/Slave Set: ms_vtc_ha [vtc_ha] Masters: [ vts02 ] Slaves: [ vts01 ] ClusterIP (ocf::heartbeat:IPaddr2): Started vts02 ClusterIPV6 (ocf::heartbeat:IPaddr2): Started vts02 |
To do this:
There are two of ways to switch over from Master to Slave node.
Restart the nso service on the Master. The switchover happens automatically. For example:
admin@vts02:/opt/vts/log/nso$ sudo service nso restart admin@vts02:/opt/vts/log/nso$ sudo crm status [sudo] password for admin: Last updated: Mon Apr 10 18:43:52 2017 Last change: Mon Apr 10 17:15:21 2017 by root via crm_attribute on vts01 Stack: corosync Current DC: vts01 (version 1.1.14-70404b0) - partition with quorum 2 nodes and 4 resources configured Online: [ vts01 vts02 ] Full list of resources: Master/Slave Set: ms_vtc_ha [vtc_ha] Masters: [ vts01 ] Slaves: [ vts02 ] ClusterIP (ocf::heartbeat:IPaddr2): Started vts01 ClusterIPV6 (ocf::heartbeat:IPaddr2): Started vts01Or,
Set the Master node to standby, and then bring it online.
In the below example, vts02 is initially the Master, which is then switched over to the Slave role.
admin@vts01:~$ sudo crm node standby [sudo] password for admin: admin@vts01:/opt/vts/log/nso$ sudo crm status [sudo] password for admin: Last updated: Mon Apr 10 18:43:52 2017 Last change: Mon Apr 10 17:15:21 2017 by root via crm_attribute on vts01 Stack: corosync Current DC: vts01 (version 1.1.14-70404b0) - partition with quorum 2 nodes and 4 resources configured Node vts01 standby Online: [ vts02 ] Full list of resources: Master/Slave Set: ms_vtc_ha [vtc_ha] Masters: [ vts02 ] Stopped: [ vts01 ] ClusterIP (ocf::heartbeat:IPaddr2): Started vts02 ClusterIPV6 (ocf::heartbeat:IPaddr2): Started vts02 admin@vts01~$ sudo crm node online admin@vts02:/opt/vts/log/nso$ sudo crm status [sudo] password for admin: Last updated: Mon Apr 10 18:43:52 2017 Last change: Mon Apr 10 17:15:21 2017 by root via crm_attribute on vts01 Stack: corosync Current DC: vts01 (version 1.1.14-70404b0) - partition with quorum 2 nodes and 4 resources configured Online: [ vts01 vts02 ] Full list of resources: Master/Slave Set: ms_vtc_ha [vtc_ha] Masters: [ vts02 ] Slaves: [ vts01 ] ClusterIP (ocf::heartbeat:IPaddr2): Started vts02 ClusterIPV6 (ocf::heartbeat:IPaddr2): Started vts02
Note | Make sure the ncs server is active/running. Then run this script on both the active and standby nodes. |
root@vts02:/opt/vts/bin# ./cluster_uninstall.sh
This will move HA configuration on this system back to pre-installed state. Proceed?(y/n) y
If a password change is performed while the VTS Active and Standby were up, and the change does not get applied to the Standby, the changed password will not get updated in the /opt/vts/etc/credentials file on the Standby. Due to this, when VTS Standby VM is brought up, it cannot connect to NCS. CRM_MON shows the state as shutdown for Standby, and it does not come online.
To troubleshoot this:Step 1 | Copy the /opt/vts/etc/credentials file from the VTC Active to the same location ( /opt/vts/etc/credentials) on the VTC Standby node. |
Step 2 | Run the crm node
command on VTC Standby to bring it online.
crm node online VTC2 |
Step 3 | Run the command
crm status to show both VTC1 and VTC2 online.
crm status |
VTSR high availability mode needs to be enabled before you install VTF(s) in your set up. The second VTSR will not get registered to the VTC if it starts up after VTF installation .
Generating two ISOs for the Master and the Slave VMs. See Generating an ISO for VTSR for details.
Deploy the two VTSR VMs using the respective ISO files generated during the process. See Deploying VTSR on OpenStack or Deploying VTSR on VMWare, based on your VMM type.
The system automatically detects which VM is the Master and which is the slave, based on the information you provide while generating the ISO files.
root@vtsr01:/opt/cisco/package# crm_mon -Afr1 Last updated: Fri Apr 14 06:04:40 2017 Last change: Thu Apr 13 23:08:25 2017 by hacluster via crmd on vtsr01 Stack: corosync Current DC: vtsr02 (version 1.1.14-70404b0) - partition with quorum 2 nodes and 11 resources configured Online: [ vtsr01 vtsr02 ] Full list of resources: dl_server (ocf::heartbeat:anything): Started vtsr01 Clone Set: cfg_dl_clone [cfg_dl] Started: [ vtsr01 vtsr02 ] Clone Set: rc_clone [rc] Started: [ vtsr01 vtsr02 ] Clone Set: confd_clone [confd] Started: [ vtsr01 vtsr02 ] Clone Set: mping_clone [mgmt_ping] Started: [ vtsr01 vtsr02 ] Clone Set: uping_clone [underlay_ping] Started: [ vtsr01 vtsr02 ] Node Attributes: * Node vtsr01: + mping : 100 + uping : 100 * Node vtsr02: + mping : 100 + uping : 100 Migration Summary: * Node vtsr02: * Node vtsr01:
This section describes the various HA scenarios.
To do a manual failover:
When the VTC Active reboots, much like a manual failover, the other VTC takes over as the Active. After coming up out of the reboot, the old Active VTC will automatically come up as the Standby.
When there is a network break and both VTCs are still up, VTC HA attempts to ascertain where the network break lies. During the network failure, the Active and Standby will lose connectivity with each other. At this point, the Active will attempt to contact the external ip (a parameter set during the initial configuration) to see if it still has outside connectivity.
If it cannot reach the external ip, VTC cannot know if the Standby node is down or if it has promoted itself to Active. As a result, it will shut itself down to avoid having two Active nodes.
The Standby, upon sensing the loss of connectivity with the Active, tries to promote itself to the Active mode. But first, it will check if it has external connectivity. If it does, it will become the Active node. However, if it also cannot reach the external ip (for instance if a common router is down), it will shut down.
At this point, the VTC that had the network break cannot tell if the other VTC is Active and receiving transactions. When the network break is resolved, it will be able to do the comparison and the VTC with the latest database will become Active.
admin@vtc1:/home/admin# sudo /opt/vts/bin/force_master.py
When both VTC are down at the same time, a double failure scenario has occurred. After a VTC has come up, it does not immediately know the state of the other VTC's database. Consequently, before HA is resumed, an agent runs in the background trying to compare the two databases. When both systems have recovered, it will be able to do the comparison and the VTC with latest database will become the Active.
admin@vtc1:/home/admin# sudo /opt/vts/bin/force_master.py