EttF High Availability Testing
High availability is critical to maximize network and system uptime, thereby meeting predefined SLAs. This section outlines the validation methodology and the corresponding results of the testing.
HA Test Methodology
Each layer in the EttF solution is verified for HA functionality and recovery times. Simulated Layer 3 traffic is traversing end-to-end from the cell/area zone through the manufacturing zone, and from the cell/area zone through the DMZ to the outside world. Various failures are triggered at each layer, and the convergence time is measured to characterize and quantify the impact on availability.
HA Test Topology
Figure D-1 shows the test topology.
Figure D-1 HA Test Topology
Layer 3 traffic is flowing from Tx1 ‡ Tx2 and from Tx1 ‡ Tx3. Tx2 and Tx3 are injecting 1000 simulated OSPF routes into the network and Tx1 is sending to all the simulated routes. The idea is to simulate TCP-based traffic as if originating from an application server such as Historian.
HA Test Scenarios
Three test suites are explored to characterize different failure/recovery times at different layers in the EttF design. Various disruptions are initiated at the cell/area zone, manufacturing zone, and DMZ levels. With each failure, convergence time is measured using the following formula:
[(Tx - Rx) / packet rate] * 1000
Where:
Tx = Packets transmitted
Rx = Packets received
PPS = 10,000 pps
Test Suite 1—HA in the Cell/Area Zone (Tx1 ‡ Tx2)
•
Use Case 1—Fail master stack with stack-mac persistent enabled
•
Use Case 2—Fail slave stack with stack-mac persistent enabled
•
Use Case 3—Fail master stack with HSRP subsecond timers configured
•
Use Case 4—Fail slave stack with HSRP subsecond timers configured
Note
HSRP with only one logical router was selected so that the end nodes would have a VIP and Virtual MAC that remains persistent. It turns out that the better solution is to enable the stack-mac persistent 0 feature as indicated by the results.
Suite 1 Test Results
The Suite 1 test results (all in milliseconds) are provided in the following tables.
Table D-1 Test Case 1—Fail Master Stack with stack-mac persistent Enabled
|
|
1 |
1988.2 |
2 |
1998.6 |
3 |
2160.3 |
Avg |
2049.033 |
Table D-2 Test Case 2—Fail Slave Stack with stack-mac persistent Enabled
|
|
1 |
1201.4 |
2 |
1091.4 |
3 |
1183.6 |
Avg |
1158.8 |
Table D-3 Use Case 3—Fail Master Stack with HSRP Subsecond Timers Configured
|
|
1 |
14994.5 |
2 |
13726.1 |
3 |
14106.9 |
Avg |
14275.83333 |
Table D-4 Use Case 4—Fail Slave Stack with HSRP Subsecond Timers Configured
|
|
1 |
2136.9 |
2 |
2126.9 |
3 |
2086.1 |
Avg |
2116.633333 |
Test Suite 2—HA in the Manufacturing Zone (Tx1 ‡ Tx2)
•
Use Case 1—Fail physical link in EtherChannel to 4500-1
•
Use Case 2—Fail physical link in EtherChannel to 4500-2
•
Use Case 3—Supervisor failover on 4500-1
•
Use Case 4—Supervisor failover on 4500-2
Suite 2 Test Results
The Suite 2 test results (all in milliseconds) are provided in the following tables.
Table D-5 Test Case 1—Fail Physical Link in EtherChannel to 4500-1
|
|
1 |
1.4 |
2 |
1.4 |
3 |
1.6 |
Avg |
1.466667 |
Table D-6 Test Case 2—Fail Physical Link in EtherChannel to 4500-2
|
|
1 |
1.8 |
2 |
1.4 |
3 |
1.4 |
Avg |
1.533333 |
Table D-7 Use Case 3—Supervisor Failover on 4500-1
|
|
1 |
16.3 |
2 |
16.1 |
3 |
16 |
Avg |
16.13333 |
Table D-8 Test Case 4—Supervisor Failover on 4500-2
|
|
1 |
15.9 |
2 |
18.2 |
3 |
16.1 |
Avg |
16.73333 |
Test Suite 3—HA in the DMZ (Tx1 ‡ Tx3)
Test suite 3 removes the control network-facing interface on the active ASA, as follows:
1.
Active ASA link failure on control network-facing interface
2.
Active ASA link failure on DMZ-facing interface
3.
Standby ASA link failure on control network-facing interface
4.
Standby ASA link failure on DMZ-facing interface
5.
Reload of active ASA
6.
Reload of standby ASA
7.
Run "failover active" on the active ASA
8.
Run "failover active" on the standby ASA
9.
4500-1 switchover to standby supervisor (HSRP active)
10.
4500-1 chassis failure (HSRP active)
11.
4500-2 switchover to standby supervisor
12.
4500-2 chassis failure
Suite 3 Test Results
Table D-9 Suite 3 Test Results
|
|
1 |
5610.9 |
2 |
6646.3 |
3 |
0 |
4 |
0 |
5 |
7189.9 |
6 |
0 |
7 |
0 |
8 |
134.2 |
9 |
270.8 |
10 |
25759.6 |
11 |
0 |
12 |
0 |
Test Tools
The following equipment is needed for performing these tests:
•
16 Cisco Catalyst C2955T-12 industrial switches
•
2 Cisco Catalyst WS-C3750G-24PS (stacked)
•
2 fully-redundant Cisco Catalyst 4507R switches with Supervisor IV
•
1 Ixia traffic generator
•
Various Rockwell Automation (RA) equipment