Replace Nexus 9236C Spine Switch - CPS
Available Languages
Contents
Introduction
This document describes the steps that are required in order to replace a faulty Spine Switch (Nexus 9236C) in an Ultra-M setup that hosts Cisco Policy Suite (CPS) Virtual Network Functions (VNFs).
Background Information
Ultra-M is a pre-packaged and validated virtualized mobile packet core solution designed to simplify the deployment of VNFs. The servers that are part of the Ultra-M setup are connected to three different types of switches:
- Catalyst Switch
- Leaf Switch
- Spine Switch
The network topology of an Ultra-M setup is as shown in this image:
UltraM Network Topology
Abbreviations
| VNF | Virtual Network Function |
| SPINE | Nexus 9236C Switch as Spine |
| MOP | Method of Procedure |
| LAN | Local Area Network |
| FTP | File Transfer Protocol |
| TFTP | Trivial File Transfer Protocol |
| CIMC |
Cisco Integrated Management Controller |
| BGP |
Border Gateway Protocol |
| BFD |
Bidirectional Forwarding Detection |
Workflow of the MoP
Highlevel Workflow of the Replacement Procedure
Spine Switch in Ultra-M Setup
Prerequisite
Take a backup of the configuration file from the Spine switch with the use of ftp/tftp before you proceed with the switch replacement as shown here.
Nexus-POD1-spine2# copy running-config sftp:
Enter destination filename: [Nexus-POD1-spine2-running-config] backup-spine-cfg-2
Enter vrf (If no input, current vrf 'default' is considered): management
Enter hostname for the sftp server: 10.1.1.1
Enter username: admin
The authenticity of host '10.1.1.1 (10.1.1.1)' can't be established.
RSA key fingerprint is SHA256:fnbUmd2mL5yE94zxrRoKAlvYfQbheXJfQox7m3XfpIU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.1' (RSA) to the list of known hosts.
User Access Verification
Password:
Connected to 10.1.1.1.
sftp> put /var/tmp/vsh/backup-spine-cfg-2 backup-spine-cfg-2
Uploading /var/tmp/vsh/backup-spine-cfg-2 to /backup-spine-cfg-2
/var/tmp/vsh/backup-spine-cfg-2 100% 33KB 33.2KB/s 00:00
sftp> exit
Copy complete, now saving to disk (please wait)...
Copy complete.
Nexus-POD1-spine2#
Health Checks
1. Check the current software version that runs in the switch and make a note of it as shown here.
Nexus-POD1-spine2# show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2018, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source. This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0 or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
Software
BIOS: version 07.59
NXOS: version 7.0(3)I7(3)
BIOS compile time: 08/26/2016
NXOS image file is: bootflash:///nxos.7.0.3.I7.3.bin
NXOS compile time: 2/12/2018 13:00:00 [02/12/2018 19:13:48]
Hardware
cisco Nexus9000 C9236C chassis
Intel(R) Xeon(R) CPU @ 1.80GHz with 16400560 kB of memory.
Processor Board ID FDO21120SSN
Device name: Nexus-POD1-spine2
bootflash: 53298520 kB
Kernel uptime is 108 day(s), 13 hour(s), 15 minute(s), 12 second(s)
Last reset
Reason: Unknown
System version: 7.0(3)I7(3)
Service:
plugin
Core Plugin, Ethernet Plugin
Active Package(s):
2. Check the current license usage.
Nexus-POD1-spine2# show license usage
Feature Ins Lic Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
N9K_LIC_1G No - Unused -
VPN_FABRIC No - Unused -
FCOE_NPV_PKG No - Unused -
SECURITY_PKG No 0 Unused -
N9K_UPG_EX_10G No - Unused -
TP_SERVICES_PKG No - Unused -
NXOS_ADVANTAGE_GF No - Unused -
NXOS_ADVANTAGE_M4 No - Unused -
NXOS_ADVANTAGE_XF No - Unused -
NXOS_ESSENTIALS_GF No - Unused -
NXOS_ESSENTIALS_M4 No - Unused -
NXOS_ESSENTIALS_XF No - Unused -
SAN_ENTERPRISE_PKG No - Unused -
PORT_ACTIVATION_PKG No 0 Unused -
NETWORK_SERVICES_PKG No - Unused -
NXOS_ADVANTAGE_M8-16 No - Unused -
NXOS_ESSENTIALS_M8-16 No - Unused -
FC_PORT_ACTIVATION_PKG No 0 Unused -
LAN_ENTERPRISE_SERVICES_PKG Yes - In use Never -
--------------------------------------------------------------------------------
3. Ensure that the physical cables connected to the switch port-channel, VLAN and also the port status are good as expected.
Nexus-POD1-spine2# show interface status | grep connected
mgmt0 -- connected routed full 100 --
Eth1/1 -- connected trunk full 100G QSFP-100G-AOC5M
Eth1/2 -- connected trunk full 100G QSFP-100G-AOC5M
Eth1/5 -- connected trunk full 100G QSFP-100G-AOC5M
Eth1/6 -- connected trunk full 100G QSFP-100G-AOC5M
Eth1/11/1 Connected to NMNET connected 101 full 10G QSFP-40G-SR4
Eth1/21/1 -- connected routed full 10G QSFP-40G-SR4
Eth1/21/2 -- connected routed full 10G QSFP-40G-SR4
Eth1/21/3 -- connected routed full 10G QSFP-40G-SR4
Eth1/21/4 -- connected routed full 10G QSFP-40G-SR4
Eth1/22/1 -- connected routed full 10G QSFP-40G-SR4
Eth1/22/2 -- connected routed full 10G QSFP-40G-SR4
Eth1/22/3 -- connected routed full 10G QSFP-40G-SR4
Eth1/22/4 -- connected routed full 10G QSFP-40G-SR4
Eth1/23/1 -- connected routed full 10G QSFP-40G-SR4
Eth1/23/2 -- connected routed full 10G QSFP-40G-SR4
Eth1/23/3 -- connected routed full 10G QSFP-40G-SR4
Eth1/23/4 -- connected routed full 10G QSFP-40G-SR4
Eth1/29 "spine-spine link" connected trunk full 100G QSFP-100G-AOC5M
Eth1/30 "spine-spine link" connected trunk full 100G QSFP-100G-AOC5M
Eth1/31 "spine-spine link" connected trunk full 100G QSFP-100G-AOC5M
Po22 portchannel to Lea connected trunk full 100G --
Po24 portchannel to Lea connected trunk full 100G --
Po30 -- connected routed full 10G --
Po30.3201 -- connected routed full 10G --
Po30.3202 -- connected routed full 10G --
Po30.3203 -- connected routed full 10G --
Po30.3204 -- connected routed full 10G --
Po30.3205 -- connected routed full 10G --
Po30.3206 -- connected routed full 10G --
Po99 spine1-spine2 connected trunk full 100G --
Lo1 -- connected routed auto auto --
Lo2 -- connected routed auto auto --
Lo3 -- connected routed auto auto --
Lo4 -- connected routed auto auto --
Lo5 -- connected routed auto auto --
Vlan2601 Global VRF spine-2 connected routed auto auto --
Vlan2602 GI VRF spine-2 <-> connected routed auto auto --
Vlan2603 IPV6SUB VRF spine- connected routed auto auto --
Vlan2604 METROE-E VRF spine connected routed auto auto --
Vlan2605 NMNET VRF spine-2 connected routed auto auto --
Vlan2721 Global VRF spine-2 connected routed auto auto --
Vlan2722 GI VRF spine-2 <-> connected routed auto auto --
Vlan2723 IPV6SUB VRF spine- connected routed auto auto --
Vlan2724 METROE-E VRF spine connected routed auto auto --
Vlan2741 Global VRF spine-2 connected routed auto auto --
Vlan2742 GI VRF spine-2 <-> connected routed auto auto --
Vlan2743 IPV6SUB VRF spine- connected routed auto auto --
Vlan2744 METROE-E VRF spine connected routed auto auto --
4. Ensure that the Border Gateway Protocol (BGP) neighbors are up.
Nexus-POD1-spine2# show ip bgp summary vrf all
BGP summary information for VRF GI, address family IPv4 Unicast
BGP router identifier 10.0.202.2, local AS number 65145
BGP table version is 28, IPv4 Unicast config peers 4, capable peers 4
1 network entries and 1 paths using 236 bytes of memory
BGP attribute entries [1/160], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.92.1 4 65145 312621 312617 28 0 0 15w3d 1
10.0.122.2 4 65251 312624 312616 28 0 0 15w3d 0
10.0.142.2 4 65252 312623 312617 28 0 0 15w3d 0
10.178.240.185 4 65137 312618 312616 28 0 0 12w0d 0
BGP summary information for VRF IPV6SUB, address family IPv4 Unicast
BGP summary information for VRF METROE-E, address family IPv4 Unicast
BGP router identifier 10.0.202.4, local AS number 65145
BGP table version is 10, IPv4 Unicast config peers 4, capable peers 4
1 network entries and 1 paths using 236 bytes of memory
BGP attribute entries [1/160], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.94.1 4 65145 312619 312617 10 0 0 15w3d 1
10.0.124.2 4 65251 312623 312616 10 0 0 15w3d 0
10.0.144.2 4 65252 312618 312617 10 0 0 15w3d 0
10.178.240.181 4 65137 312618 312616 10 0 0 12w0d 0
BGP summary information for VRF NMNET, address family IPv4 Unicast
BGP router identifier 10.0.202.5, local AS number 65145
BGP table version is 3, IPv4 Unicast config peers 1, capable peers 1
1 network entries and 1 paths using 236 bytes of memory
BGP attribute entries [1/160], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.95.1 4 65145 312616 312616 3 0 0 15w3d 1
BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.0.202.1, local AS number 65145
BGP table version is 261, IPv4 Unicast config peers 4, capable peers 3
35 network entries and 49 paths using 10108 bytes of memory
BGP attribute entries [5/800], BGP AS path entries [3/22]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.91.1 4 65145 312681 312635 261 0 0 15w3d 35
10.0.121.2 4 65251 312629 312630 261 0 0 15w3d 0
10.0.141.2 4 65252 312683 312620 261 0 0 15w3d 14
10.178.240.177 4 65137 0 124577 0 0 0 15w3d 12
Nexus-POD1-spine2# show ipv6 bgp summary vrf all
BGP summary information for VRF GI, address family IPv6 Unicast
BGP router identifier 10.0.202.2, local AS number 65145
BGP table version is 64, IPv6 Unicast config peers 4, capable peers 4
1 network entries and 1 paths using 248 bytes of memory
BGP attribute entries [1/160], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
fc00:420:81:122::2
4 65251 312637 312629 64 0 0 15w3d 0
fc00:420:81:142::2
4 65252 312637 312630 64 0 0 15w3d 0
fc00:420:81:992::1
4 65145 312635 312630 64 0 0 15w3d 1
fd01:976a:c002:1:4d13:36f3:0:22
4 65137 312631 312627 64 0 0 12w0d 0
BGP summary information for VRF IPV6SUB, address family IPv6 Unicast
BGP router identifier 10.0.202.3, local AS number 65145
BGP table version is 44, IPv6 Unicast config peers 4, capable peers 4
0 network entries and 0 paths using 0 bytes of memory
BGP attribute entries [0/0], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
fc00:420:81:123::2
4 65251 312632 312629 44 0 0 15w3d 0
fc00:420:81:143::2
4 65252 312636 312629 44 0 0 15w3d 0
fc00:420:81:993::1
4 65145 312631 312629 44 0 0 15w3d 0
fd01:976a:c002:1:4d13:36f3:0:24
4 65137 312629 312628 44 0 0 12w0d 0
BGP summary information for VRF METROE-E, address family IPv6 Unicast
BGP summary information for VRF NMNET, address family IPv6 Unicast
BGP summary information for VRF default, address family IPv6 Unicast
5. Ensure that the Bidirectional Forwarding Detection (BFD) neighbors state is Up.
Nexus-POD1-spine2# show bfd neighbors vrf al
OurAddr NeighAddr LD/RD RH/RS Holdown(mult) State Int Vrf
10.0.124.1 10.0.124.2 1090519045/1090519205 Up 672(3) Up Vlan2724 METROE-E
10.0.122.1 10.0.122.2 1090519046/1090519206 Up 639(3) Up Vlan2722 GI
10.0.92.2 10.0.92.1 1090519050/1090519094 Up 585(3) Up Vlan2602 GI
10.0.95.2 10.0.95.1 1090519051/1090519095 Up 585(3) Up Vlan2605 NMNET
10.0.121.1 10.0.121.2 1090519052/1090519208 Up 639(3) Up Vlan2721 default
10.0.94.2 10.0.94.1 1090519054/1090519097 Up 585(3) Up Vlan2604 METROE-E
10.0.91.2 10.0.91.1 1090519055/0 Up N/A(3) Up Vlan2601 default
10.0.141.1 10.0.141.2 1090519056/1090519212 Up 610(3) Up Vlan2741 default
10.0.144.1 10.0.144.2 1090519057/1090519213 Up 610(3) Up Vlan2744 METROE-E
10.0.142.1 10.0.142.2 1090519059/1090519214 Up 610(3) Up Vlan2742 GI
10.178.240.182 10.178.240.181 1090519061/0 UP N/A(3) Up Po30.3202 METROE-E
10.178.240.186 10.178.240.185 1090519062/0 Up N/A(3) Up Po30.3203 GI
Nexus-POD1-spine2#
Nexus-POD1-spine2# show bfd ipv6 neighbor vrf all
OurAddr NeighAddr
LD/RD RH/RS Holdown(mult) State Int Vrf
fc00:420:81:123::1 fc00:420:81:123::2
1090519044/1090519204 Up 615(3) Up Vlan2723 IPV6SUB
fc00:420:81:122::1 fc00:420:81:122::2
1090519047/1090519207 Up 731(3) Up Vlan2722 GI
fc00:420:81:142::1 fc00:420:81:142::2
1090519048/1090519210 Up 696(3) Up Vlan2742 GI
fc00:420:81:143::1 fc00:420:81:143::2
1090519049/1090519211 Up 696(3) Up Vlan2743 IPV6SUB
fc00:420:81:993::2 fc00:420:81:993::1
1090519053/1090519096 Up 544(3) Up Vlan2603 IPV6SUB
fc00:420:81:992::2 fc00:420:81:992::1
1090519058/1090519099 Up 544(3) Up Vlan2602 GI
Switch Replacement Procedure
1. Install the new switch in the rack and connect the cables to the switch as noted. The steps for switch installation can be found here: Spine Nexus 236C NX-OS mode Switch Hardware Installation Guide
2. Check the software version of the Nexus Spine Switch and upgrade/downgrade to the software version as per this link: Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide, Release 7.x
3. Transfer the configuration backup to the new switch and copy that configuration to the startup configuration.
server-backup$ sftp admin@10.1.1.1
The authenticity of host '10.1.1.1 (10.1.1.1)' can't be established.
RSA key fingerprint is SHA256:fnbUmd2mL5yE94zxrRoKAlvYfQbheXJfQox7m3XfpIU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.1' (RSA) to the list of known hosts.
User Access Verification
Password:
Password:
Connected to 10.1.1.1.
sftp> ls
20170607_193430_poap_15489_1.log 20170607_193430_poap_15489_2.log 20170607_193430_poap_15489_init.log backup-spine-cfg-2 backup-spine.cfg
backup_run_02152018 bios_daemon.dbg cfg-july25th cfg-july25th-spine1 flash:
flash:cfg-aug8th-ybattina-afterNSO flash:cfg-jul28th home l3-cfg-aug10th license_FDO211406K1_16.lic
lost+found nxos.7.0.3.I4.4.bin nxos.7.0.3.I5.2.bin nxos.7.0.3.I7.3.bin platform-sdk.cmd
scripts spine-config-bkp starat virtual-instance vlan.dat 100% 33KB 21.7KB/s 00:01
sftp> put backup-spine-cfg-2
Uploading backup-spine-cfg-2 to /backup-spine-cfg-2
backup-spine-cfg-2 100% 33KB 23.5KB/s 00:01
sftp> bye
Nexus-spine1# copy bootflash:///backup-spine-cfg-2 startup-config
Copy progress 100% 33KB
Copy complete, now saving to disk (please wait)...
Copy complete.
4. Reload the switch after you load the backup configuration.
Nexus-POD1-spine2#reload
Verify Replaced Spine Switch
After the switch is available for access, perform the health checks with the use of these commands on the Spine Switch in order to verify that a replaced switch is available in the expected state.
Nexus-POD1-spine2# show port-channel summary
Nexus-POD1-spine2# show ip bgp summary vrf all
Nexus-POD1-spine2# show ipv6 bgp summary vrf all
Nexus-POD1-spine2# show bfd neighbor vrf all
Nexus-POD1-spine2# show bfd ipv6 neighbor vrf all
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)