- Preface
- Ethernet-to-the-Factory Solution Overview
- Solution Architecture
- Basic Network Design
- Implementation of the Cell/Area Zone
- Implementation of Security
- Implementation of High Availability
- Implementation of Network Management
- Characterization of the EttF Cell/Area Zone Design
- Configuration of the EttF Cell/Area Zone
- Configuration of the EttF Demilitarized Zone
- EttF High Availability Testing
- STP Testing
- 16-Switch Ring—STP Testing
- Redundant Star Topology—STP Testing
- Latency/Jitter Testing
- IGMP Testing
Characterization of the EttF Cell/Area Zone Design
All factory floor devices are connected at the cell/area zone layer of EttF 1.1. From a network design perspective, a solid spanning tree and multicast design are critical for reliability and to meet predefined service-level agreements (SLAs). As mentioned in earlier sections, Cisco recommends that Rapid Spanning Tree Protocol (RSTP 802.1w) and IGMP snooping with querier be deployed at this layer. This appendix outlines the validation methodology and the corresponding results of the testing.
STP Testing
STP Test Methodology
Eight Cisco Catalyst 2955 switches, each running Native IOS 12.1(22)EA9, plus one Catalyst 3750 running Native IOS 12.2(25)SEB4, are linked together in back-to-back fashion via 802.1q Gigabit Ethernet trunks to form a ring topology (See Figure A-1).
All nine devices are running RSTP 802.1w carrying only one VLAN (vlan 20). This topology creates a Layer 2 loop, and therefore one port must be blocked by STP. By configuring switch CZ-3750-1 with the lowest spanning tree priority, it is elected as the root bridge for this VLAN. The STP parameters on all the other switches are left at their default settings. Accordingly, switch Cell-2955-4 is now the furthest away from the root bridge in terms of path cost and must select a port to block (GigabitEthernet 0/2). The root port on devices Cell-2955-1 through Cell-2955-4 is GigabitEthernet 0/1, and the root port on devices Cell-2955-5 through Cell-2955-8 is GigabitEthernet 0/2. A traffic generator (Ixia 400 Tf) is attached to switches Cell-2955-7, Cell-2955-8, Cell-2955-4, and Cell-2955-5 via ports Tx1, Tx2, Tx3, Tx4 respectively. The traffic generator is used to measure the convergence time in various failure scenarios.
STP Test Topology
Figure A-1 shows the test topology.
Figure A-1 Test Topology
Multiple bidirectional traffic streams are configured on Ixia between Tx1 <-> Tx2 and Tx3 <-> Tx4. Each of these pairs is referred to as a traffic suite with their own test cases. Each stream for each test suite is designed to source 1, 50, 100, and 200 MAC addresses destined to 1, 50, 100, and 200 MAC addresses. Thus, twice the number of configured source MAC addresses are traversing the ring for each test case. Each packet is a mix of 500 and 64 bytes in length sent in a continuous fashion until the STP has re-converged and the network has stabilized. At steady state for Suite 1 (Tx1 <-> Tx2), the traffic flow is as follows:
1. 
Packet egresses Tx1 of Ixia and ingresses port fa0/1 of Cell-2955-7
2. Packet then egresses gi0/2 on Cell-2955-7 and ingresses port g0/1 on Cell-C2955-8
3. Packet then egresses fa0/2 on Cell-C2955-8 and finally ingresses Tx2 of Ixia
This flow is reversed for streams going in the opposite direction (Tx2 to Tx1).
At steady state for Suite 2 (Tx3 <-> Tx4), the traffic flow is as follows:
1. Packet egresses Tx3 of Ixia and ingresses port fa0/3 of Cell-C2955-4
2. Because STP is blocking port gi0/2, the packet egresses gi0/1 of Cell-C2955-4 and traverses the entire ring in the clockwise direction until it reaches Cell-C2955-5
3. Packet ingresses gi0/2 on Cell-C2955-5 and egresses fa0/4
4. Packet finally ingresses Tx4 of Ixia
This flow is reversed for streams going in the opposite direction (Tx4 to Tx3).
STP Test Scenarios
Two test suites are explored, each with a different traffic flow. Within each test suite, multiple failure scenarios are introduced to simulate various STP changes in the ring topology. With each failure, convergence time is measured using the following formula:
[(Tx - Rx) / packet rate] * 1000
Where:
Tx = Packets transmitted
Rx = Packets received
PPS = 10,000 pps
Test Suite 1—Bidirectional Traffic (Tx1 <-> Tx2)
At steady state, traffic is flowing bidirectionally point-to-point between Cell-C2955-7 and Cell-C2955-8 (essentially a best-case scenario). (See Figure A-2.)
Figure A-2 Test Suite 1—Bidirectional Traffic Flow
However, after simulating a failure between these two switches, the traffic must then traverse the entire ring (becoming the worst-case scenario) to reach its destination. (See Figure A-3.)
Figure A-3 Test Suite 1—Worse-Case Scenario
The eight failure scenarios in Suite 1 are as follows:
•Failure 1—Software shut link between Cell-C2955-7 and Cell-C2955-8
•Failure 2—Software unshut link between Cell-C2955-7 and Cell-C2955-8
•Failure 3—Physically remove link between Cell-C2955-7 and Cell-C2955-8
•Failure 4—Physically re-insert link between Cell-C2955-7 and Cell-C2955-8
•Failure 5—Root bridge down
•Failure 6—Root bridge up
•Failure 7—Stack master down on CZ-C3750
•Failure 8—Stack master re-established on CZ-C3750
For each failure scenario, the following is measured and verified:
•Convergence time
•Verify Rockwell Automation (RA) equipment functioning properly after disruption
•Measure CPU and memory on cell devices that are sending and receiving Ixia traffic
Test Suite 2—Bidirectional Traffic (Tx3 <-> Tx4)
At steady state, traffic is flowing bidirectionally between Cell-C2955-4 and Cell-C2955-5 across the entire ring. (See Figure A-4.)
Figure A-4 Test Suite 2—Bidirectional Traffic Flow
Because STP is blocking gi0/2 on Cell-C2955-4, this constitutes the worse-case scenario before any failures are introduced. However, after a network failure and a subsequent STP topology change, traffic flows to its directly-connected neighbor and becomes the best-case scenario. (See Figure A-5.)
Figure A-5 Test Suite 2—Best-Case Scenario
The eight failure scenarios in Suite 2 are as follows:
•Failure 1—Software shut link between Cell-C2955-1 and CZ-C3750
•Failure 2—Software unshut link between Cell-C2955-1 and CZ-C3750
•Failure 3—Physically remove link between Cell-C2955-1 and CZ-C3750
•Failure 4—Physically re-insert link between Cell-C2955-1 and CZ-C3750
•Failure 5—Root bridge down
•Failure 6—Root bridge up
•Failure 7—Stack master down on CZ-C3750
•Failure 8—Stack master re-established on CZ-C3750
For each failure scenario, the following is measured and verified:
•Convergence time
•Verify RA equipment functioning properly after disruption
•Measure CPU and memory on cell devices that are sending and receiving Ixia traffic
Test Tools
The following equipment is needed for performing these tests.
•16 Cisco Catalyst C2955T-12 industrial switches
•2 Cisco Catalyst WS-C3750G-24PS (stacked)
•1 Ixia traffic generator
•1 set of RA equipment
STP Test Results
Suite 1 Test Cases
Suite 2 Test Cases
Sample Trend Line for Link Failure Between Adjacent Switches
Figure A-6 shows the trend line for link failure between the C2955-7 and C2955-8 switches.
Figure A-6 Link Failure—C2955-7 to C2955-8
Key findings were as follows:
•~.5 seconds with one MAC address, and without CIP and producer/consumer traffic
•~2 seconds with between 50-200 MAC addresses with producer/consumer traffic
•Traffic load (CIP, producer/consumer) increases convergence time
•Number of MAC addresses not a large influence on convergence time
Sample Trend Line for Link Failure To Root Bridge
Figure A-7 shows the trend line for link failure to the root bridge.
Figure A-7 Link Failure to Root Bridge
Key findings were as follows:
•Convergence time is ~490-590 milliseconds, with a slight upward trend depending on number of MAC addresses.
16-Switch Ring—STP Testing
To verify some scaling parameters, testing was performed with double the amount of 2955s in the cell/area zone (16), and a spot check of certain test cases was performed to compare against the 8-switch ring STP tests. The same traffic flow and test methodology exists from the 8-switch tests except that there are more L2 hops from source to destination. Following are the tests and the corresponding results.
Test Suite 1—Bidirectional Traffic from (Tx1 <-> Tx2)
Key findings were as follows:
•Very similar results as 8 switch tests
–Slightly longer convergence times with 200*2 MAC Addresses from ~2.1 seconds to ~2.7 seconds upon link failure
•Other tests not performed because of the similarity of the worst case scenario test from above
Note No baseline measurements were gathered (background traffic was always running).
Test Suite 2—Bidirectional Traffic (Tx3 <-> Tx4)
Key findings were as follows:
•Very similar results as 8-switch tests
–Slightly longer convergence time on link failure test (all MAC addresses), which was expected
•Other tests not performed because of the similarity of the worst-case scenario test from above
Note No baseline measurements were gathered (background traffic was always running).
Redundant Star Topology—STP Testing
Although the majority of the testing was done with the ring topology, the redundant star topology was also tested for comparison purposes. (See Cell/Area Network—Star Topology, page 2-24.) Ixia connections were used between two adjacent 2955 switches. The following test cases were performed.
Key findings were as follows:
•Best convergence times compared to ring-8 or ring-16, averaging 500ms consistently
•More consistent numbers
•Worse case for traffic flow is only two L2 hops away
Latency/Jitter Testing
To characterize different network topologies under steady state, latency and jitter measurements were captured. Unlike spanning tree convergence testing, no failures were introduced. These tests assume that the network is functioning normally with typical control device traffic running in the background. Simulated source/destination patterns are worse-case scenarios (in the ring topologies) with traffic traversing the entire ring. This is done by having knowledge of the STP-blocked port before the testing begins.
The following test cases were completed for latency/jitter measurements:
•8-switch ring—Bidirectional traffic 2955-5 <> 2955-4
•16-switch ring—Bidirectional traffic 2955-8 <> 2955-9
•Hub/spoke—Bidirectional traffic 2955-12 <> 2955-13
The following results are in microseconds (µ):
|
|
|
|
|
|
1 |
43.068 |
30.94 |
42.822 |
33.8 |
2 |
65.902 |
36.92 |
65.606 |
36.94 |
3 |
27.022 |
36.34 |
25.447 |
34.7 |
Key findings were as follows:
•Consistent with results from disruptive tests from above
–Hub/spoke has the best latency, followed by 8-ring and 16-ring.
–Jitter was consistent across all tests.
IGMP Testing
IGMP Snooping Test Methodology
The same eight devices that were used for the STP tests were also used to verify IGMP snooping. The testing exercised various combinations of IGMP snooping with and without a querier on various switches in the network, with active producer-consumer traffic running between PACs and a variable frequency drive. However, as per the Cisco recommended deployment, IGMP snooping works properly only in the presence of a querier. (For more information, see the following URL: http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008059a9df.shtml).
Thus, the only relevant test results are IGMP snooping with querier, and no IGMP snooping with or without querier enabled. A protocol analyzer (http://www.wireshark.org) was used to verify the presence of multicast data traffic.
IGMP Snooping Test Topology
Figure A-8 shows the IGMP snooping test topology.
Figure A-8 IGMP Snooping Test Topology
In this topology, Controller A (Receiver/Consumer) is consuming traffic from Drive A (Producer/Source) across the ring topology. On the same switch that is producing multicast data traffic, a sniffer (Sniffer A) is connected on a different port that has IGMP snooping enabled to verify that this traffic is not seen on this port. The same verification is done on another port with IGMP snooping disabled. Finally, the test was repeated on a different switch in the ring with Sniffer C and Sniffer D, respectively.
IGMP Snooping Test Results
Table A-28 shows the IGMP snooping test results. Note the following:
•Yes = Receiving multicast data traffic (destined to 239.x.x.x)
•No = Not receiving multicast data traffic (destined to 239.x.x.x)
|
|
|
|
|
OFF/OFF |
Yes |
Yes |
Yes |
ON/ON |
Yes |
No |
No |
Feedback