- Preface
- Overview
- Initial Configuration
- Connecting to the Management Network and Securing Access
- Protecting the CPU
- Integrated Intrusion Detection Security
- Cisco NX-OS Software Upgrade or Downgrade
- EPLD Software Upgrade or Downgrade
- Enabling or Disabling Features
- IP Management
- Management Tools for Usability
- Verifying Hardware Diagnostics and Logging
- Managing Hardware Resource Utilization
- Collecting Data for the Cisco TAC
Managing Hardware Resource Utilization
This chapter contains Cisco NX-OS procedures recommended when managing hardware resources utilization such as the CPU, memory and I/O module TCAM table utilization.
This chapter includes the following sections:
•
Unicast or Multicast TCAM Tables
CPU Processes
This section contains information for verifying the CPU utilization for the supervisor module.
Utilization
Introduced: Cisco NX-OS Release 4.0(1)
The show system resources command displays the high level CPU utilization for the supervisor module. The show process cpu command with the sort option lists all of the processes sorted by the highest CPU utilization per process. The show process cpu history command displays the CPU history in three increments: 60 seconds, 60 minutes, 72 hours. Viewing the CPU history is valuable when correlating a network event with the past CPU utilization. The sort and history options for the show process cpu command were introduced in Cisco NX-OS Release 4.2(1).
It should be noted that Cisco NX-OS takes advantage of preemptive CPU multitasking, so processes can take advantage of an Idle CPU to complete tasks faster. Therefore, the history option may report CPU spikes that do not necessarily mean there is an issue. Additional investigation should take place if the average CPU remains close to 100%.
n7000# show system resources
Load average: 1 minute: 0.06 5 minutes: 0.04 15 minutes: 0.00
Processes : 310 total, 1 running
CPU states : 0.0% user, 0.5% kernel, 99.5% idle
Memory usage: 4135780K total, 1180900K used, 2954880K free
0K buffers, 759580K cache
n7000# show process cpu sort
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
3102 1692 371648 4 2.0% platform
1 162 49364 3 0.0% init
<Text Omitted>
n7000# show process cpu history
1 1 1 1
151 2 1 176 6112 2212 1 21 511 1 2 31 151 1 10
100
90
80
70
60
50
40
30
20 # #
10 # ## # # # # #
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
Restarting a Process
Introduced: Cisco NX-OS Release 4.0(1)
This section is included for reference and may not be required.
Some Cisco NX-OS processes can be restarted with the restart command. A process should not require a manual restart, but in the event it does a process can be restarted without re-configuring the protocol, or reloading the chassis. Restarting a process may be disruptive, so this feature should be used with caution.
n7000# restart ospf 10
Memory
This section contains information for verifying the supervisor module DRAM and Flash memory utilization.
DRAM Utilization
Introduced: Cisco NX-OS Release 4.0(1)
The supervisor module memory utilization for a chassis can be monitored with the following commands. The show system resources command displays the overall memory utilization for the supervisor module and the show process memory command displays memory utilization per process per VDC.
n7000# show process memory
PID MemAlloc MemLimit MemUsed StackBase/Ptr Process
----- -------- ---------- ---------- ----------------- ----------------
<Text Omitted>
11849 2994176 329981836 127692800 bffff5e0/bfffc820 nfm
12019 13029376 334518976 115449856 bfffe1c0/bfffde30 ospf
12266 155648 0 1712128 bfffe800/bfffe5cc more
12267 1118208 0 48463872 bffff670/bfff9c08 vsh
12268 0 0 0 bfffe410/bfffdd28 ps
<Text Omitted>
Flash Utilization
Introduced: Cisco NX-OS Release 4.0(1)
The flash file system capacity can be verified for each supervisor module. The following example has one supervisor module in slot 5. The bootflash: refers to the 2 GB onboard flash, and the logflash, and slot0 refers to the external compact flash slots on the supervisor module. The dir command displays the contents for each type of flash memory (output not displayed).
n7000# show hardware capacity | begin flash
5 bootflash 1767480 1055144 40
5 logflash 7997912 7555672 5
5 slot0 1996928 1652944 17
n7000# dir bootflash:
n7000# dir logflash:
n7000# dir slot0:
MAC Address TCAM Tables
This section contains information for verifying the MAC address TCAM table utilization and modifying the aging-time if necessary.
Utilization
Introduced: Cisco NX-OS Release 4.0(1)
The Cisco Nexus 7000 Series uses a distributed forwarding architecture in which each Ethernet M series module has a forwarding engine responsible for forwarding packets. A forwarding engine on an M series module is capable of storing 128,000 MAC Address entries. MAC address tables are synchronized between Ethernet M series modules that have ports configured in the same Virtual Device Context (VDC). The following command is useful for verifying the MAC address table capacity for all modules in a chassis.
n7000# show hardware capacity forwarding | begin L2
L2 Forwarding Resources
-----------------------
L2 entries: Module total used mcast ucast lines lines_full
1 131072 6 1 5 8192 0
2 131072 6 1 5 8192 0
<Text Omitted>
Aging Time
Introduced: Cisco NX-OS Release 4.0(1)
This section was included for reference and may not be required.
The default MAC-Address table aging time is 1,800 seconds (30 minutes). The aging time can be modified to a more or less aggressive timeout value. The MAC Address aging time should be consistent for all of the devices within a switched domain.
n7000(config)# mac address-table aging-time ?
<0-0> 0 disables aging
<120-918000> Aging time in seconds.
Unicast or Multicast TCAM Tables
This section contains information for verifying the unicast/multicast TCAM table utilization.
Utilization
Introduced: Cisco NX-OS Release 4.0(1)
The Cisco Nexus 7000 Series uses a distributed forwarding architecture in which each Ethernet M series module has a forwarding engine responsible for forwarding packets. A forwarding engine on an M series module is capable of storing 128,000 IPv4/IPv6 routing entries or 1,000,000 entries if it is an XL module with a Scalable-Feature license installed. IPv4/IPv6 unicast/multicast tables are synchronized between Ethernet M series modules that have ports configured in the same Virtual Device Context (VDC). The following example displays the default TCAM allocation for a non-XL module. Beginning in Cisco NX-OS Release 4.2(1), Cisco NX-OS supports dynamic TCAM allocation. This allows for better resource utilization in the event and address family (i.e. IPv6 unicast) requires additional entries.
n7000# show hardware capacity forwarding | begin TCAM
Key: Log/Phys = Logical entries / Physical entries
Note: IPv4 Multicast/IPv6 Unicast entries share one FIB TCAM entry pool
Module 1 usage:
Route Type Used %Used Free %Free Total
(Log/Phys) (Log/Phys) (Log/Phys)
-------------------------------------------------------------------------
IPv4 Unicast: 19/19 0 57325/57325 99 57344/57344
IPv4 Multicast: 4/8 0 16380/32760 99 16384/32768
IPv6 Unicast: 9/18 0 16375/32750 99 16384/32768
IPv6 Multicast: 5/20 0 2043/8172 99 2048/8192
NetFow TCAM Tables
This section contains information for verifying the NetFlow TCAM table utilization.
Utilization
Introduced: Cisco NX-OS Release 4.0(1)
The Cisco Nexus 7000 Series uses a distributed forwarding architecture in which each Ethernet M series module has a forwarding engine responsible for forwarding packets. A forwarding engine on an M series module is capable of storing 512,000 NetFlow entries. This value is the same for both non-XL and XL M series modules.
n7000# show hardware capacity forwarding | begin Netflow
n7000# show hardware capacity forwarding | begin Netflow
Netflow Resources
-----------------
Flow Table Usage: Module Util Used Free Fail
1 0.00% 0 515090 0
2 0.00% 0 515090 0
ICAM Usage: Module Util Used Free
1 0.00% 0 16
2 0.00% 0 16
IPv4 Mask Usage: Module Util Used Free
1 0.00% 0 32
2 0.00% 0 32
IPv6 Mask Usage: Module Util Used Free
1 0.00% 0 32
2 0.00% 0 32
ACL or QoS TCAM Tables
This section contains information for verifying the ACL or QoS TCAM table utilization and enabling ACL TCAM chaining if required.
Utilization
Introduced: Cisco NX-OS Release 4.0(1)
The Cisco Nexus 7000 Series uses a distributed forwarding architecture in which each Ethernet M series module has a forwarding engine responsible for forwarding packets. A forwarding engine on an M series module is capable of storing 64,000 (non-XL) or 128,000 ACL QoS entries if it is an XL module with the Scalable Feature license installed.
n7000# show hardware capacity | begin ACL
ACL Hardware Resource Utilization (Module 1)
--------------------------------------------
Used Free Percent
Utilization
-----------------------------------------------------
Tcam 0, Bank 0 1 16383 0.00
Tcam 0, Bank 1 2 16382 0.01
Tcam 1, Bank 0 1 16383 0.00
Tcam 1, Bank 1 2 16382 0.01
LOU 0 104 0.00
Both LOU Operands 0
Single LOU Operands 0
LOU L4 src port: 0
LOU L4 dst port: 0
LOU L3 packet len: 0
LOU IP tos: 0
LOU IP dscp: 0
LOU ip precedence: 0
TCP Flags 0 16 0.00
Protocol CAM 0 7 0.00
Mac Etype/Proto CAM 0 14 0.00
Non L4op labels, Tcam 0 0 6143 0.00
Non L4op labels, Tcam 1 0 6143 0.00
L4 op labels, Tcam 0 0 2047 0.00
L4 op labels, Tcam 1 0 2047 0.00
ACL Resource Polling
Introduced: Cisco NX-OS Release 4.2(1)
This section is included for reference and may not be required.
The ACL TCAM is divided into four banks (16K per bank for non-XL and 32K per bank for XL modules) on the current M series forwarding engines. Prior to Cisco NX-OS Release 4.2(1) an ACL could only contain 1 bank of entries (16K or 32K entries depending on the module type). Starting in Cisco NX-OS Release 4.2(1) a single ACL can be programmed across multiple banks allowing up to 64, 000 entries in a single ACL per non-XL and 132,000 entries in an XL module. This feature should only be enabled on systems that require ACLs that contain more than 16,000 entries. This feature is configured in the default VDC(1) for all VDCs.
n7000(config)# hardware access-list resource pooling module 1
n7000# show hardware access-list resource pooling
Module 1 enabled
Fabric Utilization
The fabric utilization can be monitored to verify the ingress and egress bandwidth utilization. The show hardware fabric-utilization commands are useful for verifying the high-level and detailed utilization. The show hardware capacity fabric-utilization is useful for verifying the peak utilization history.
n7000# show hardware fabric-utilization
------------------------------------------------
Slot Total Fabric Utilization
Bandwidth Ingress % Egress %
------------------------------------------------
1 138 Gbps 0.0 0.0
2 138 Gbps 0.0 0.0
4 138 Gbps 0.0 0.0
5 69 Gbps 0.0 0.0
7 138 Gbps 0.0 0.0
8 138 Gbps 0.0 0.0
9 138 Gbps 0.0 0.0
10 138 Gbps 0.0 0.0
n7000# show hardware fabric-utilization detail
------------------------------------------------------------------------
Fabric Planes:
A -- Unicast fabric interface
B -- Multicast/Multidestination fabric interface
------------------------------------------------------------------------
Unidirectional Fabric Bandwidth per Fab Link is 23 Ggps (A+B)
------------------------------------------------------------------------
I/O Fab Fab Fab Fab Fab Fabric Utilization
Slot Mod Ins Chnl Link Plane Ingress% Egress%
------------------------------------------------------------------------
1 1 1 5 0 A 0 0
1 1 1 5 0 B 0 0
1 1 1 3 1 A 0 0
1 1 1 3 1 B 0 0
1 2 1 5 2 A 0 0
1 2 1 5 2 B 0 0
1 2 1 3 3 A 0 0
1 2 1 3 3 B 0 0
1 3 1 5 4 A 0 0
1 3 1 5 4 B 0 0
1 3 1 3 5 A 0 0
1 3 1 3 5 B 0 0
<Text omitted>
n7000# show hardware capacity fabric-utilization
------------------------------------------------------------------------
Fabric Planes:
A -- Unicast fabric interface
B -- Multicast/Multidestination fabric interface
-------------------------PEAK FABRIC UTILIZATION------------------------
I/O |-----FABRIC----| Ingress | Egress
Slot |Mod Inst Plane| Util Time | Util Time
------------------------------------------------------------------------
1 1 1 A 0% 11-01@23:09:42 0% 11-01@23:09:42
1 1 1 B 0% 11-01@23:09:42 0% 11-01@23:09:42
1 1 1 A 0% 11-01@23:09:42 0% 11-01@23:09:42
1 1 1 B 0% 11-01@23:09:42 0% 11-01@23:09:42
1 2 1 A 0% 11-01@23:09:42 0% 11-01@23:09:42
1 2 1 B 0% 11-01@23:09:42 0% 11-01@23:09:42
1 2 1 A 0% 11-01@23:09:42 0% 11-01@23:09:42
1 2 1 B 0% 11-01@23:09:42 0% 11-01@23:09:42
1 3 1 A 0% 11-01@23:09:42 0% 11-01@23:09:42
VDC Resource Utilization
Introduced: Cisco NX-OS Release 4.0(1)
Global VDC resources can be verified with the show vdc resource command. This is useful to know, since VDCs can contend for common resources such as memory, SPAN sessions, etc.).
n7000# show vdc resource
vlan 16 used 48 unused 16368 free 16320 avail 16384 total
monitor-session 0 used 0 unused 2 free 2 avail 2 total
monitor-session-erspan-dst 0 used 0 unused 23 free 23 avail 23 total
vrf 8 used 0 unused 992 free 992 avail 1000 total
port-channel 0 used 0 unused 768 free 768 avail 768 total
u4route-mem 120 used 0 unused 396 free 396 avail 516 total
u6route-mem 36 used 0 unused 172 free 172 avail 208 total
m4route-mem 82 used 0 unused 118 free 118 avail 200 total
Feedback