Release 3.1: Cisco StadiumVision Director Server Administration Guide

Configuring Failover Between Redundant Cisco StadiumVision Director Servers

Cisco StadiumVision Director supports an environment best described as warm standby between two servers that run the Cisco StadiumVision Director software—one of the servers operates as the primary active server, and the other server operates as a secondary backup server. If a failure occurs, you can configure the backup server to become the active server, but the failover process is not automatic.

Contents

•Prerequisites for Configuring Failover Between Redundant Cisco StadiumVision Director Servers

•Restrictions for Configuring Failover Between Redundant Cisco StadiumVision Director Servers

•Information About Failover Between Redundant Cisco StadiumVision Director Servers

•How to Promote a Standby Secondary Server to the Active Server

•How to Restore the Primary Server to Active

Note Restoring the primary server after failover to secondary requires a service interruption and should only be conducted during scheduled downtime. Be aware that until you change the IP address of the primary server to remove the addressing conflict with the newly active secondary server, you will be unable to schedule backups between the two servers. You will also need to reconfigure the backup/restore environment using the Text Utility Interface (TUI) after the restore. For more information, see the Backing Up and Restoring Cisco StadiumVision Director module.

Prerequisites for Configuring Failover Between Redundant Cisco StadiumVision Director Servers

Before you promote a secondary server to become the primary active server, be sure that the following requirements are met:

•You have either physical console access or an SSH client such as PuTTY to log into both Cisco StadiumVision Director servers.

•You understand how to use the Text Utility Interface (TUI). For more information, see the "Cisco StadiumVision Director Server Text Utility Interface" module. For simplicity in these tasks, the instruction to "select" a particular menu item implies that you type the character that corresponds to the menu option and press Enter.

•Verify that you have a successful backup of the primary server on the secondary server. For more information, see the Backing Up and Restoring Cisco StadiumVision Director module.

•The Cisco StadiumVision Director backup server is on the same subnet as the primary server.

•The servers are using the Eth0 interface for connection to the network.

•You have the IP addresses of the primary and secondary servers.

Tip The IP address for the server is displayed on the screen when you log into the TUI. You also can view the /etc/hosts information on each server using the TUI System Settings > System Information menu option.

Restrictions for Configuring Failover Between Redundant Cisco StadiumVision Director Servers

The Cisco StadiumVision Director server redundancy architecture has the following restrictions:

•Redundancy, and therefore failover, is not supported for Cisco StadiumVision Director Remote servers.

•The Cisco StadiumVision Director server architecture does not support automatic failover when a failure occurs on the active server.

•Depending on your environment, 30 minutes or more is needed to complete the manual failover process.

•In addition, after the manual failover process is completed, a script push will be required if you are in an active event, which depending on your deployment and content size, can take anywhere from minutes to an hour. When pushing the script again, there will be a service interruption.

Information About Failover Between Redundant Cisco StadiumVision Director Servers

Figure 1 shows the architecture of Cisco StadiumVision Director server redundancy under normal network conditions and operation. The primary and secondary servers are addressed as independent hosts with two different IP addresses on the same subnet in the Cisco Connected Stadium network.

While the secondary server is still connected to the network, notice that communication and control only occurs between the primary Cisco StadiumVision Director server and the rest of the network, including the Digital Media Players (DMPs).

The secondary server is only connected to the network to be made available as a backup to the primary should a failure occur. In addition, the secondary server can (and should) be configured to be backed up with data from the primary server on a scheduled basis.

Figure 1 Cisco StadiumVision Director Server Redundancy Under Normal Operation

Figure 2 shows the redundancy environment when connectivity from the primary Cisco StadiumVision Director server fails. When the primary server fails, a manual process must take place to restore the secondary server from a backup, shut down the primary server, and activate the secondary server as the primary.

Notice that the secondary server must be reconfigured to use the same IP address the original primary server. In this example, the secondary server IP address is changed to 10.0.0.1 (from 10.0.0.2) to match the primary server address. When the process is complete, communication and control only occurs between the newly activated secondary server and the rest of the network.

Note The word "failover" does not mean automatic activation of a secondary server. The failover process is manual.

Figure 2 Cisco StadiumVision Director Server Redundancy Under Failover Operation

How to Promote a Standby Secondary Server to the Active Server

This section describes the related tasks to perform when a primary Cisco StadiumVision Director server fails in a redundant server environment. It includes tasks to activate the secondary server to replace the functionality of the primary server for Cisco StadiumVision operation.

Note For simplicity in these tasks, any instruction to go to or select a particular TUI menu item implies that you type the character that corresponds to the menu option and press Enter.

This section includes the following tasks:

•Starting and Configuring the Services on the Secondary Server (required)

•Restoring the Secondary Server with System Data From a Backup File (required)

•Stopping Services and Auto-Restart, and Shutting Down the Primary Server (required)

•Shutting Down Services on the Secondary Server (required)

•Changing the IP Address on the Secondary Server (required)

•Restarting the Network Service on the Secondary Server (required)

•Verifying Network Connectivity to the Secondary Server (required)

•Clearing the ARP Cache on the Switch (optional)

•Restarting Cisco StadiumVision Director on the Secondary Server (required)

•Verifying the Cisco StadiumVision Director Configuration on the Secondary Server (required)

Starting and Configuring the Services on the Secondary Server

To start and configure the services on the secondary server, complete the following steps:

Step 1 Log into the TUI as installer on the secondary server using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 2 Go to the StadiumVision Server Administration > Failover sub-menu.

Step 3 Select the Promote as Primary/Active option.

The Cisco StadiumVision Director services are started and also configured to start automatically when a reboot occurs.

Restoring the Secondary Server with System Data From a Backup File

To restore the secondary server with system data from a backup file, complete the following steps:

Step 1 Log into Cisco StadiumVision Director on the secondary server using an administrator account.

Step 2 From the Cisco StadiumVision Director main menu, click Management Dashboard.

Step 3 From the Dashboard Drawers, select Tools > Advanced > Restore system data from backup.

Step 4 Select the components that you want to restore, and select the date of the backup file to use for the restore.

Step 5 Click Apply.

The restore begins. A dialog box appears notifying you when the restore process has successfully completed.

Stopping Services and Auto-Restart, and Shutting Down the Primary Server

Note If the primary server has become unavailable for this task, be sure that you power down the server so that it will not conflict with the newly active secondary server.

To stop services and auto-restart, and shut down the primary server, complete the following steps:

Step 1 Log into the TUI as installer on the primary server using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 2 Go to the StadiumVision Server Administration > Failover sub-menu.

Step 3 Select the Configure as Secondary/Inactive option.

Step 4 Press any key to return to the Failover sub-menu.

Step 5 Return to the StadiumVision Server Administration menu by typing R and pressing Enter.

Step 6 Select the Power Off option.

The primary server is shut down.

Shutting Down Services on the Secondary Server

To shut down services on the secondary server, complete the following steps:

Step 1 Log into the TUI as installer on the secondary server using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 2 Go to the StadiumVision Server Administration sub-menu.

Step 3 Select the Shutdown StadiumVision Director software option.

All Cisco StadiumVision Director services are stopped.

Step 4 Return to the Main Menu by typing R and pressing Enter.

Changing the IP Address on the Secondary Server

Prerequisites

Before you change the IP address on the secondary server, be sure that the following requirements are met:

•You have the IP address of the primary server.

•You understand how to use the vi editor. For information about using the vi editor, see the "Cisco StadiumVision Director Server Text Utility Interface"

Note The system will run if the localhost entry exists but the hostname entry is missing in the /etc/hosts file. However, when the secondary hostname exists, the IP address of the secondary server must be changed to match the IP address of the primary server.

Procedure

To change the IP address on the secondary server, complete the following steps:

Step 1 From the TUI Main Menu on the secondary server, go to the System Settings menu.

Step 2 Select the Network Settings option.

The Network Settings sub-menu is displayed.

Step 3 Select the Setup Network Information option.

Step 4 At the Configure Network confirmation screen, press any key to continue.

The Select Action screen is displayed with the "Edit Devices" option highlighted.

Step 5 Press Enter to select.

The Select a Device screen is displayed with the "eth0" network interface highlighted.

Step 6 Press Enter to select.

The Ethernet Configuration screen is displayed.

Note The Linux screen is misspelled "Devernet Configuration."

Step 7 Press the tab key until the cursor is positioned on the Static IP address line.

Step 8 Press the backspace key to go to the beginning of the line and type in the IP address of the primary server.

In our example from Figure 2, this would be 10.0.0.1. Be sure to use the actual IP address of your primary server.

Step 9 Press the tab key until the Ok button is highlighted and press Enter.

You return to the Select a Device screen.

Step 10 Press the tab key until the Save button is highlighted and press Enter.

You return to the Select Action screen.

Step 11 Press the tab key until the Save&Quit button is highlighted and press Enter.

You return to the TUI Configure Network screen.

Step 12 Press any key to return to the Network Settings sub-menu.

Step 13 Select the Edit hosts file option.

Step 14 Press any key to enter edit mode.

Step 15 Replace this server's IP address with the IP address of the primary server.

Step 16 Save the configuration and exit vi using the following command:

:wq

Restarting the Network Service on the Secondary Server

To restart the network service on the secondary server, complete the following steps:

Step 1 From the TUI Main Menu on the secondary server, go to the Services Control sub-menu.

Step 2 Select the Networking option.

The Networking sub-menu is displayed.

Step 3 Select the Restart networking option.

The network daemon is restarted and the IP address change is put into effect on the secondary server.

Note If you are connected to the server through the network using SSH, your session is disconnected and you will need to reconnect using the IP address of the primary server.

Verifying Network Connectivity to the Secondary Server

To verify network connectivity to the secondary server, complete the following steps:

Step 1 From the TUI Main Menu on the secondary server, go to the Troubleshooting sub-menu.

Step 2 Select the Ping a host option.

Step 3 At the "Enter hostname" prompt, type the hostname or IP address of the secondary server and press Enter.

Step 4 Look for successful transmission and receipt of PING packets.

Note If you cannot reach the secondary server, go to the "Clearing the ARP Cache on the Switch" section.

Step 5 Press Ctrl-C to stop sending PING packets.

Step 6 Press any key to return to the Troubleshooting menu.

Clearing the ARP Cache on the Switch

This task is optional, as the ARP cache on the switch will refresh in 5-10 minutes. However, if you cannot access the secondary server after changing its IP address, you can clear the ARP cache for that IP address on the switch using the clear ip arp privileged EXEC command.

To clear the ARP cache on the switch, complete the following steps:

Step 1 Use a directly-connected console, or if you know the IP address of the switch, use Telnet to access the switch as shown in the following example, where ip-address is the address of your switch:

telnet ip-address

Step 2 At the corresponding prompts, enter your login information as shown in the following example, where yourname and yourpass are your username and password:

Username: yourname

Password: yourpass

switch>

Step 3 Enter privileged EXEC mode using the enable command and corresponding password:

switch> enable

Password: enablepassword

switch# 

Step 4 To clear the ARP cache of the newly-assigned IP address now used by the secondary server, use the clear ip arp command as shown in the following example:

clear ip arp 10.0.0.1

Restarting Cisco StadiumVision Director on the Secondary Server

To restart Cisco StadiumVision Director on the secondary server, complete the following steps:

Step 1 Do one of the following on the secondary server:

•If still logged into the TUI, go to the Main Menu.

•If not still logged into the TUI as installer on the secondary server, use the new IP address and log in again using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 2 Go to the StadiumVision Server Administration sub-menu.

Step 3 Select the Restart StadiumVision Director Software option.

All Cisco StadiumVision Director services are started.

Step 4 Press any key to return to the StadiumVision Server Administration sub-menu.

Step 5 Press R and Enter until you return to the Main menu.

Step 6 Press X to exit the TUI.

Verifying the Cisco StadiumVision Director Configuration on the Secondary Server

To verify the Cisco StadiumVision Director configuration on the secondary server, complete the following steps:

Step 1 Log in to Cisco StadiumVision Director on the secondary server using an administrator account.

Step 2 From the Cisco StadiumVision Director main menu, click Management Dashboard.

Step 3 From the Dashboard Drawers, select DMP and TV Controls > Monitoring > Get Status.

Confirm that you have successful communication between the DMPs and Cisco StadiumVision Director.

Step 4 Verify that all of the content is on this server.

Step 5 To establish control of the DMPs, start a script without any content and with the No Staging radio button selected. This should only require less than 10 minutes.

Note You can push a script with content, but this will result in a longer period of downtime.

How to Restore the Primary Server to Active

Note This task requires a service interruption.

At a scheduled downtime, you should restore the primary server as the active server to re-establish your normal operating environment and clean up the original primary server from the failure, make IP addressing changes, and have regularly scheduled backups again between the two servers.

Note For simplicity in these tasks, any instruction to go to or select a particular TUI menu item implies that you type the character that corresponds to the menu option and press Enter.

This section includes the following tasks:

•Prerequisites

•Stopping Services and Auto-Restart on the Secondary Server (required)

•Changing the IP Address on the Secondary Server (required)

•Restarting Cisco StadiumVision Director on the Secondary Server (required)

•Verifying Network Connectivity on the Secondary Server

•Starting and Configuring the Services on the Original Primary Server (required)

•Restoring the Original Primary Server with System Data From a Backup File (as required)

•Restarting the Local Control Service on the Primary Server (required after restore run)

•Verifying Network Connectivity to the Primary Server (required)

•Verifying the Cisco StadiumVision Director Configuration on the Original Primary Server (required)

Prerequisites

If you have made any administrative changes on the active secondary server, be sure that a successful backup has been run.

While the secondary server is still active, re-configure the backup environment and run a manual backup from the Management Dashboard. The latest backup will then be copied to the primary (inactive) server. For more information, see the "Backing Up and Restoring Cisco StadiumVision Director Servers".

Stopping Services and Auto-Restart on the Secondary Server

To stop services and auto-restart of them on the secondary server, complete the following steps:

Step 1 Log into the TUI as installer on the secondary server using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 2 Go to the StadiumVision Server Administration > Failover sub-menu.

Step 3 Select the Configure as Secondary/Inactive option.

Step 4 Press any key to return to the Failover sub-menu.

Step 5 Return to the StadiumVision Server Administration menu by typing R and pressing Enter.

Step 6 Select the Shutdown StadiumVision Director Software option.

Changing the IP Address on the Secondary Server

Prerequisites

Before you change the IP address on the secondary server, be sure that the following requirements are met:

•You have the IP address of the secondary server.

•You understand how to use the vi editor. For information about using the vi editor, see the "Cisco StadiumVision Director Server Text Utility Interface"

Procedure

To change the IP address on the secondary server, complete the following steps:

Step 1 From the TUI Main Menu on the secondary server, go to the System Settings menu.

Step 2 Select the Network Settings option.

The Network Settings sub-menu is displayed.

Step 3 Select the Setup Network Information option.

Step 4 At the Configure Network confirmation screen, press any key to continue.

The Select Action screen is displayed with the "Edit Devices" option highlighted.

Step 5 Press Enter to select.

The Select a Device screen is displayed with the "eth0" network interface highlighted.

Step 6 Press Enter to select.

The Ethernet Configuration screen is displayed.

Note The Linux screen is misspelled "Devernet Configuration."

Step 7 Press the tab key until the cursor is positioned on the Static IP address line.

Step 8 Press the backspace key to go to the beginning of the line and type in the IP address of the secondary server.

In our example from Figure 1, this would be 10.0.0.2. Be sure to use the actual IP address of your secondary server.

Step 9 Press the tab key until the Ok button is highlighted and press Enter.

You return to the Select a Device screen.

Step 10 Press the tab key until the Save button is highlighted and press Enter.

You return to the Select Action screen.

Step 11 Press the tab key until the Save&Quit button is highlighted and press Enter.

You return to the TUI Configure Network screen.

Step 12 Press any key to return to the Network Settings sub-menu.

Step 13 Select the Edit hosts file option.

Step 14 Press any key to enter edit mode.

Step 15 Replace this server's IP address with the IP address of the secondary server.

Step 16 Save the configuration and exit vi using the following command:

:wq

Restarting Cisco StadiumVision Director on the Secondary Server

To restart Cisco StadiumVision Director on the secondary server, complete the following steps:

Step 1 Do one of the following on the secondary server:

•If still logged into the TUI, go to the Main Menu.

•If not still logged into the TUI as installer on the secondary server, use the new IP address and log in again using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 2 Go to the StadiumVision Server Administration sub-menu.

Step 3 Select the Restart StadiumVision Director Software option.

All Cisco StadiumVision Director services are started.

Step 4 Press any key to return to the StadiumVision Server Administration sub-menu.

Step 5 Press R and Enter until you return to the Main menu.

Step 6 Press X to exit the TUI.

Verifying Network Connectivity on the Secondary Server

To verify network connectivity to the secondary server, complete the following steps:

Step 1 From the TUI Main Menu on the secondary server, go to the Troubleshooting sub-menu.

Step 2 Select the Ping a host option.

Step 3 At the "Enter hostname" prompt, type the hostname or IP address of the secondary server and press Enter.

Step 4 Look for successful transmission and receipt of PING packets.

Note If you cannot reach the secondary server, go to the "Clearing the ARP Cache on the Switch" section.

Step 5 Press Ctrl-C to stop sending PING packets.

Step 6 Press any key to return to the Troubleshooting menu.

Starting and Configuring the Services on the Original Primary Server

To start and configure the services on the original primary server, complete the following steps:

Step 1 Power on the original primary server.

Note It might take a few minutes for SSH to be available as the server boots.

Step 2 Log into the TUI as installer on the original primary server using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 3 Go to the StadiumVision Server Administration > Failover sub-menu.

Step 4 Select the Promote as Primary/Active option.

The Cisco StadiumVision Director services are started and also configured to start automatically when a reboot occurs.

Step 5 Press any key to return to the Failover sub-menu.

Step 6 Depending on the state of the server when it went down and what was done while the server was down, a script might be running on the original primary StadiumVision Director server. If a script is running, end the script in the Cisco StadiumVision Director software.

Restoring the Original Primary Server with System Data From a Backup File

If any administrative changes were made to the system while in failover to the other server, you should restore the backup from the secondary.

Note This step requires that a backup was run from the secondary server to the primary before reactivating the primary server.

To restore the original primary server with system data from a backup file, complete the following steps:

Step 1 Log in to Cisco StadiumVision Director on the original primary server using an administrator account.

Step 2 From the Cisco StadiumVision Director main menu, click Management Dashboard.

Step 3 From the Dashboard Drawers, select Tools > Advanced > Restore system data from backup.

Step 4 Select the components that you want to restore, and select the date of the backup file to use for the restore.

Step 5 Click Apply.

The restore begins.

Restarting the Local Control Service on the Primary Server

After you perform a restore, you must stop and start the local control service (svd-localctl) to resume normal operation of the local control application programming interface (API).

To restart the local control service on the primary server, complete the following steps:

Step 1 Log into the TUI as installer on the original primary server using a directly-connected console or SSH client.

The TUI Main Menu is displayed.

Step 2 Go to the Services Control > StadiumVision Director Services > SVD Local Control sub-menu.

Step 3 Select the Stop service option.

The svd-localctl service is stopped.

Step 4 When prompted, press any key to return to the SVD Local Control sub-menu.

Step 5 Select the Start service option.

The svd-localctl service is started. Look for the message that the command completed successfully.

Step 6 When prompted, press any key to return to the SVD Local Control sub-menu.

Step 7 Press R and Enter until you return to the Main menu.

Step 8 Press X to exit the TUI.

Verifying Network Connectivity to the Primary Server

To verify network connectivity to the primary server, complete the following steps:

Step 1 From the TUI Main Menu on the original primary server, go to the Troubleshooting sub-menu.

Step 2 Select the Ping a host option.

Step 3 At the "Enter hostname" prompt, type the hostname or IP address of the primary server and press Enter.

Step 4 Look for successful transmission and receipt of PING packets.

Note If you cannot reach the secondary server, go to the "Clearing the ARP Cache on the Switch" section.

Step 5 Press Ctrl-C to stop sending PING packets.

Step 6 Press any key to return to the Troubleshooting menu.

Step 7 Press R and Enter until you return to the Main menu.

Step 8 Press X to exit the TUI.

Verifying the Cisco StadiumVision Director Configuration on the Original Primary Server

To verify the Cisco StadiumVision Director configuration on the original primary server, complete the following steps:

Step 1 Log in to Cisco StadiumVision Director on the original primary server using an administrator account.

Step 2 From the Cisco StadiumVision Director main menu, click Management Dashboard.

Step 3 From the Dashboard Drawers, select DMP and TV Controls > Monitoring > Get Status.

Confirm that you have successful communication between the DMPs and Cisco StadiumVision Director.

Step 4 Verify that all of the content is on this server.

Step 5 Test the system by looking at the status in the Management Dashboard and by running test scripts to verify operation of the system.