Cisco MDS NX-OS Configuration Limits

This document discusses the configuration and scalability limits for various SAN switching parameters:

Change History

Date Description
August 18, 2023 Added scale values for Cisco MDS 9396V Multilayer Fabric switch.

Added Zone Fanout scale limits.

October 20, 2021

Updated zone member footnotes in the Fabric-Level Configurations Limits table.
August 24, 2021

Added scale values for Fabric Performance Monitor (FPM).
March 29, 2021

Added scale values for Cisco MDS 9220i switch.
July 1, 2020

Cisco MDS 9718 switches support 8,000 FLOGIs per switch.
May 1, 2020 The following scale values were updated:

  • Cisco MDS 9710 switches support 8,000 FLOGIs per switch.

  • Cisco MDS 9396T switches support 4,000 FLOGIs per switch.

  • Smart Zones support a maximum of 250 members per zone.

Switch-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches

System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.

The following table lists the switch-level configuration limits for Cisco MDS 9000 Series switches:

Table 1. Switch-Level Configuration Limits

Feature Name

Parameter

MDS 9700 Series

MDS 9250i

MDS 9148S

MDS 9396S

MDS 9132T

MDS 9148T

MDS 9396T

MDS 9220i

MDS 9124V

MDS 9148V

 MDS 9396V

Physical Fabric Logins

FLOGIs or FDISCs per port

2561

2561

2561

2561

2561

2561

2561

2561
2561

2561

 2561

FLOGIs or FDISCs per module

1,0002
2,0003

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

None (refer to chassis limits)

 None (refer to chassis limits)

FLOGIs or FDISCs per switch
4,0004
8,0005

400

1,000

1,000

2,000

2,000
2,0006
4,0007

2,000

2,000

2,000

 4,000

N-Port Virtualization (NPV)

NPV switches per NPIV core switch

105

8

8

8

8

8

8

8

8

8

 8

Inter-Switch Links (ISLs)

ISL instances per switch 8

Up to 200 ISLs, each ISL with 16 VSANs, for a total number of 3200-port VSAN instances

40 ISLs, each with 29 VSANs, for a total number of 1160 port VSAN instances

48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances

96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances

32 ISLs, each with 29 VSANs, for a total number of 928 port VSAN instances

48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances

96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances

12 ISLs, each with 29 VSANs, for a total number of 348 port VSAN instances

24 ISLs, each with 29 VSANs, for a total number of 696 port VSAN instances

48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances

 96 ISLs, each with 29 VSANs, for a total number of 2784 port VSAN instances
Port channels (PC)9

Fibre Channel port channels

256

40

48

96

32

48

96

12

24

48

 96

Members in each Fibre Channel port channel

16

16

16

16

16

16

16

12

16

16

 16

Ethernet port channels

128

8

None

None

None

None

None

None

None

None

 None

Members in each Ethernet port channel

16

8

None

None

None

None

None

None

None

None

 None
SSH 10

Simultaneous SSH sessions per switch

16

16

16

16

16

16

16

16

16

16

 16

SAN Analytics and SAN Telemetry Streaming

Active Initiator Target LUNs (ITLs) or Initiator Target Namespace ID (ITNs) per sampling interval per switch

20,00011

40,00012
100,00013

None

None

None

20,000

20,000

20,000

None

20,000

40,000

 None

Active ITLs or ITNs per sampling interval per module

DS-X9648-1536K9: 20,000

None

None

None

20,000

20,000

20,000

None

None

None

 None

Telemetry receivers per switch

2

None

None

None

2

2

2

None

2

2

 None

Fabric Performance Monitor (FPM)14

Slow devices15 per port

200

None

None

None

200

200

200

200

200

200

 200

Slow devices per module

200

None

None

None

None

None

None

None

None

None

 None

Slow devices per switch

800

None

None

None

200

200

200

200

200

200

 200

DIRL Dynamically rate limited ports per module

10

None

None

None

None

None

None

None

None

None

 None

DIRL Dynamically rate limited ports per switch

80

None

None

None

10

10

10

10

10

10

 10
1

This is the recommended limit. FLOGIs or FDISCs are not rejected until 512 logins have been accepted on the port.

2

Supported on DS-X9448-768K9.

3

Supported on DS-X9648-1536K9 and DS-X9748-3072K9.

4

Supported in Cisco MDS 9706 switches.

Supported in Cisco MDS 9710 switches in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.

5

Supported in Cisco MDS 9710 switches with Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2) and later releases.

Supported in Cisco MDS 9718 switches with Cisco MDS 9700 Series Supervisor-1 Module (DS-X97-SF1E-K9) or Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2a) and later releases.

6

Supported in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.

7

Supported from Cisco MDS NX-OS Release 8.4(2) and later releases.

8

ISLs can be an individual FC link, VFC link, FC port channel, or VFC port channel. One port channel interface with up to 16 member links is considered as one ISL instance.

9

A system message appears in the user’s session when the maximum supported limit of port channels and member ports is exceeded.

10

A system message appears on the user’s session when the maximum supported number of SSH sessions is exceeded.

11 Supported in Cisco MDS NX-OS Release 8.3(1) and earlier releases.
12

Supported from Cisco MDS NX-OS Release 8.3(2) and later releases.

When there are more than 40,000 ITLs or ITNs distributed across multiple modules in multilayer directors or 20,000 ITLs or ITNs in fabric switches, a random set of ITL or ITN data is reported. The ITL or ITN data may change frequently and becomes unpredictable. Hence, we recommend that you limit the number of ITLs or ITNs in a switch to less than the maximum limit or query only view types. For example, a single query configured can display up to the maximum active ITLs or ITNs. If there are two queries configured, the queries together can display up to the maximum active ITLs or ITNs.

Beginning with Cisco MDS NX-OS Release 9.3(2), the number of ITLs or ITNs distributed across multiple modules in multilayer directors is increased from 40,000 to 100,000

13

Supported from Cisco MDS NX-OS Release 8.3(2) and later releases.

When there are more than 40,000 ITLs or ITNs distributed across multiple modules in multilayer directors or 20,000 ITLs or ITNs in fabric switches, a random set of ITL or ITN data is reported. The ITL or ITN data may change frequently and becomes unpredictable. Hence, we recommend that you limit the number of ITLs or ITNs in a switch to less than the maximum limit or query only view types. For example, a single query configured can display up to the maximum active ITLs or ITNs. If there are two queries configured, the queries together can display up to the maximum active ITLs or ITNs.

Beginning with Cisco MDS NX-OS Release 9.3(2), the number of ITLs or ITNs distributed across multiple modules in multilayer directors is increased from 40,000 to 100,000

14 The scale values provided are the maximum tested number of devices and ports that can be monitored by FPM on supported platforms. However, FPM may support more devices and ports than the tested values.
15 The devices that have been detected slow and FPM is now handling them via congestion-isolation or via FPIN. For more information on slow devices see, Congestion Management

Physical Fabric Configuration Limits for Cisco MDS 9000 Series Switches

The following table lists the configuration limits for Cisco MDS 9000 fabrics.

Table 2. Physical Fabric Configuration Limits

Feature Name

Parameter

Fabric Composition

(lowest number is the limit for the entire fabric unless noted)

MDS 9700 (Supervisor 1, Supervisor 1E, Supervisor 4)

MDS 9396V, 9124V, MDS 9148V, MDS 9396T, MDS 9148T, MDS 9132T, MDS 9220i

MDS 9396S, MDS 9250i, MDS 9148S

VSANs

VSANs per physical fabric

80
3216

Logins and aliases

FCNS entries in physical fabric
20,00017
10,00018

Device alias entries in physical fabric

20,000

8,000

FC alias entries per VSAN

2,048

2,048

Domains and hops

Domains per physical fabric

80

80

Switch hops from server to storage

7

7

Zones and Smart Zones

Configured zone sets

1,000

1,000

Active zones
16,00019
8,00020

Active zone members
32,00021

16,00022
Active zone database size23
3.8 MB24

2 MB

Inter-VSAN Routing (IVR)

Active IVR zone sets

32

32
Active IVR zones25

2,000

2,000

Active IVR zone members

4,000

4,000

Active IVR service groups

16

16

CFS
CFS peers26

80

80
CFS static peers over IP27

100

100
16

In a physical fabric with mixed VSAN limits, all switches may be in the first 32 VSANs. The remaining VSANs may only exist on the switches that support more than 32 VSANs.

17

An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.

18

A warning message is logged to the configuration session when this limit is exceeded; however, further increases are not blocked.

19

An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.

20

A warning message is logged to the configuration session when this limit is exceeded; however, further increases are not blocked.

21

This limit has two components. The first limit is the FCNS maximum of 20,000 online entries in the fabric. The second limit is a maximum of 32,000 zone members in the zone database. Up to the FCNS limit of these may be online. The remaining may be unique offline members or reused online members. This limit was tested with 20,000 unique online devices and 12,000 reused online zone members.

22 This limit has two components. The first limit is the FCNS maximum of 10,000 online entries in the fabric. The second limit is a maximum of 16,000 zone members in the zone database. Up to the FCNS limit of these may be online. The remaining may be unique offline members or reused online members. This limit was tested with 10,000 unique online devices and 6,000 reused online zone members.
23

This limit is supported when all switches in the fabric are running Cisco MDS NX-OS Release 6.2(7) or later. If there are any switches in the fabric that are running Cisco MDS NX-OS Release 6.2(5) or earlier, then the lower 2 MB zone database limit prevails and must not be exceeded for reliable zone operation.

24

An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.

25

Total across all VSANs. Any single VSAN may not have more than 1,000 IVR zones.

26

An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.

27

An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.

Example: Checking the Size of the Zone Database

The following example shows how to check the size of the zone database in VSAN 310: 


(config)# show zone status vsan 310 | Inc "Db Size" P 1

Full Zoning Database :
    Db Size: 1040524 Bytes
--
Active Zoning Database :
    Db Size: 962156 Bytesc

Now, Add 962156 with 1040524 = 2002680

2002680 / (1024 * 1024) = 1.9 MB.


Note

This database size does not include the pending changes in a session.

Configuration Limits for Port Line Rate Throughput on Cisco MDS 9700 40 Gbps 24-Port FCoE Modules

For ports to operate at full-line rate on Cisco MDS 9706, MDS 9710, and MDS 9718 switches, the switch should have a specific configuration. The following table lists the maximum number of line rate ports for given Cisco MDS 9700 switch configurations.

Table 3. Line Rate Limitation for Cisco MDS 9700 Series Switches Using Cisco MDS 9700 40 Gbps 24-Port FCoE Module

No. of Fabric 1 Cards

Front Panel FCoE Bandwidth/Slot

Maximum No. of Ports/Speed

3

660 Gbps

16 ports/40 Gbps

4

880 Gbps

20 ports/40 Gbps

5

1100 Gbps

24 ports/40 Gbps

No. of Fabric 3 Cards

Front Panel FCoE Bandwidth/Slot

Maximum No. of Ports/Speed

3

1320 Gbps

33 ports/40 Gbps

4

1760 Gbps

44 ports/40 Gbps

5

2200 Gbps

55 ports/40 Gbps


Note

If the number of ports that you configure exceeds the maximum limit for the number of ports, it results in all ports operating in oversubscribed mode.

Fibre Channel Topology Combinations and Scaling

From Cisco MDS NX0-OS Release 6.2(7), the following configurable features that support scale enhancements are supported:

  • Fibre Channel Name Server (FCNS) bulk notification

  • Coalesce switch Registered State Change Notification (SW-RSCN)

For more information about the FCNS and SW-RSCN features, see the Cisco MDS 9000 Series NX-OS Fabric Configuration Guide and the Cisco MDS 9000 Family Command Reference documents.


Note
In multidimensional scale configurations, supervisors with 1-GB memory may experience the system manager hap-reset error.

The following table provides the configuration limits with and without the FCNS and SW-RSCN optimizations.

Table 4. Fibre Channel Topology Combination and Scaling

Scale Topology

FCNS Bulk Notification

Coalesce SW-RSCN

Configuration Limits

Scale Optimizations

Cisco MDS NX-OS Release 6.2(9) and later on Cisco MDS 9700 Series-only Fabric

On

On

MDS NX-OS Release 6.2(7) and later

Enabled

FLOGI per module: 1,000

FLOGI node: 4000

FCNS: 20,000

Zone: 16,000

Zone members: 32,000

Off

Off

MDS NX-OS Release 6.2(5) and earlier

Disabled (Disabled by default in Cisco MDS NX-OS Release 6.2(7))

FLOGI per module: 500

FLOGI node: 2500

FCNS: 10,000

Zone: 8,000

Zone members: 16,000

Cisco MDS NX-OS Release 6.2(9) and later on MDS Mixed Fabric28 (Cisco MDS 9700 Series, 9500, 9250i, 9222i, and 9148)

On or off

On or off

FLOGI per module: 400

FLOGI node: 2000

FCNS: 10,000, 13,000 29
Zone: 8,000, 10,400 30
Zone members: 16,000, 20,800 31

Enabled or disabled

28

The scale enhancements introduced in Cisco MDS NX-OS 6.2(7) and later are available on Cisco MDS 9700 Series Switches (only networks with Cisco MDS NX-OS Release 6.2(7) and later, running on all the switches and the configurable optimizations): FCNS bulk notification and RSCN coalesce enabled. These enhancements are not available in mixed fabrics regardless of whether FCNS and RSCN are enabled. In Cisco MDS NX-OS Release 6.2(9) and later, FCNS bulk notification is enabled by default. (To disable FCNS bulk notification, use the fcns no-bulk-notify command.) Coalesce SW-RSCN is disabled by default.

29

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

30

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

31

This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.

Fibre Channel Scale Limitations

The various Fibre Channel scale limitations are described in this section.

Limitation 1

From Cisco MDS NX-OS Release 6.2(7), the fcns bulk-notify and rscn coalesce swrscn vsan commands are available on all Cisco MDS switches. We recommend that you use these commands only on Cisco MDS 9700 Series switches with 48-port 16-Gbps Fibre Channel switching module because the higher configuration limits are currently not supported on any other MDS platform.


Note
The FCNS bulk notification and Coalesce SW-RSCN features are supported only in NX-OS Release 6.2(7) and later releases. Enable the Coalesce SW-RSCN only if all the switches in a physical fabric are Cisco MDS switches that are running Cisco MDS NX-OS Release 6.2(7) or later.

Limitation 2

The maximum zone database size is increased from 2 to 3.8 MB in Cisco MDS Release 6.2(7) and later for all MDS 9700 Series switches to enable zone scale enhancements. The new limit of 16000 zones is supported only on the Cisco MDS 9700 only fabric. Physical fabrics with Cisco MDS 9500, 9200, or 9100 Series switches continue to have the 2-MB zone database limit, supporting only up to 8000 zones.

Limitation 3

In a three-node serial topology, traffic imbalance may occur if the number of port channels that are configured between the switches are the same. We recommend that you have a single port channel between two switches with any number of member ports. If you configure more than one port channel, ensure that the count of port channels between the switches varies.

Limitation 4

The maximum latency (round-trip time) and the TCP retransmits that are supported on FCIP links are 250 ms and 0.05 percent receptively.


Note

The maximum limit is the same regardless whether latency and packet drop conditions exist together or only one of the conditions exists.

Syslog and Warnings for Network Scale

The following table lists the syslogs and warnings for a network.

Table 5. Syslogs and Warnings

Syslog and Warnings

Cisco MDS 9700 Series

Cisco MDS 9500, 9200, and 9100

Zone limit

16,000 zones – Hard limit: Syslog warning that states no more zones can be configured: 

“Maximum configurable zone 
limit of 16,000 reached.
Creation of any more zones 
is not supported.”

8,000 zones – Hard limit: Syslog warning that states no more zones can be configured: 


“Maximum configurable zone 
limit of 8,000 reached. 
Creation of any more zones 
is not supported.”

10,400 zones – Hard limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning that states no more zones can be configured: 


“Maximum configurable zone 
limit of 10,400 reached. 
Creation of any more zones 
is not supported.”

FCNS limit

20,000 FCNS entries – Hard limit: Syslog warning stating that no more name server entries are supported: 


“Maximum Name-Server entry 
limit of 20,000 reached. 
No more entries are supported.”

10,000 FCNS entries – Soft limit: Syslog warning about validated limit: 


“Number of Name-Server entries has 
reached the maximum validated limit 
of 10,000. Any more entries could 
potentially destabilize the fabric.”

13,000 FCNS entries – Soft limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning about validated limit: 


“Number of Name-Server entries has 
reached the maximum validated limit of 13,000. 
Any more entries could potentially 
destabilize the fabric.”

20,000 FCNS entries – Hard limit: Syslog warning: 


“Maximum Name-Server entry 
limit of 20,000 reached. 
No more entries are supported.”

Switch-Level Configuration Limits for Fibre Channel over Ethernet

System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.

The following table lists the switch-level configuration limits for Fibre Channel over Ethernet (FCoE) on Cisco MDS.

Table 6. Switch-Level Configuration Limits for FCoE

Feature Name

MDS 9710 with 48-Port 10-Gigabit FCoE

FLOGIs per Port

256

FLOGIs per Module

1,000

FLOGIs per Switch

4,000

VSAN

80

VSAN-VLAN Mapping

80

Virtual Fibre Channel Port Channel and Member Ports

128 virtual fibre channel port channel and a maximum number of 16 members in one virtual fibre channel port channel

Physical Fabric Configuration Limits for Fibre Channel over IP

The following table lists the fabric-level configuration limits for Fibre Channel over IP (FCIP):

Table 7. Physical Fabric Configuration Limits for FCIP on Cisco MDS 9250i and MDS 9220i Fabric Switches

Feature Name

Cisco MDS 9250i and MDS 9220i Fabric Switches

Maximum latency (round-trip time) and TCP retransmits supported on FCIP links

250-ms round trip and 0.05% packet drop
Table 8. Physical Fabric Configuration Limits for FCIP on Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module

Feature Name

Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module

Maximum latency (round-trip time) and TCP retransmits supported on FCIP links

250-ms round trip and 0.05% packet drop

Cisco MDS 9000 I/O Accelerator Configuration Limits

The following table lists the Cisco MDS 9000 I/O Accelerator configurations and the corresponding limits.

Table 9. Cisco I/O Accelerator Configuration Limits

Parameter

MSM-18/4 or SSN-16 Module on MDS 9222i and MDS 9500 Modular Chassis and MDS 9250i Fabric Switch, 24/10-Port SAN Extension Module on MDS 9710 Chassis

Number of switches in a cluster

4

Number of clusters per switch

16

Number of switches in a SAN fabric for FC-Redirect

34

Number of hosts per target

128

Number of concurrent flows per Cisco MDS 9000 I/O Accelerator service engine

128

Number of flows per Cisco MDS 9000 I/O Accelerator service engine (hard limit)

128 - Release 4.2(1) on MDS 9222i and MDS 9500

512 - Release 4.2(7) or later on MDS 9222i and MDS 9500

512 - Release 6.2(5) or later on MDS 9250i

512 - Release 8.2(1) or later on MDS 9710

Number of flows per Cisco MDS 9000 I/O Accelerator service engine (soft limit)

64 - Release 4.2(1) on MDS 9222i/MDS 9500

256 - Release 4.2(7) or later on MDS 9222i/MDS 9500

256 - Release 6.2(5) or later on MDS 9250i

Note

 

If initiators or targets participating in Cisco MDS 9000 I/O Accelerator are present on MDS 9250i switches, the limit is 203 for tape and 160 for disk.

256 - Release 8.2(1) or later on MDS 9710

Number of flows in a cluster

1,024 - Release 4.2(7d)

1,248 - Release 5.2(6b) or later

Fibre Channel Flow Configuration Limits

The following table lists the Fibre Channel flow configurations limits.

Table 10. Fibre Channel Flow Configuration Limits

Cisco MDS Device

Aggregate Flow and Flow Statistics Limit

Flow Statements per Module

Cisco MDS 24/10 Port SAN Extension Module

512

512

Cisco MDS 9700 48-Port 16-Gbps Fibre Channel Switching Module (DS-X9448-768K9)

512

512

Cisco MDS 9700 48-Port 32-Gbps Fibre Channel Switching Module (DS-X9648-1536K9)

512

512

Cisco MDS 9700 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9)

512

512

Cisco MDS 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9) line card SAN Analytics ITL limit

The SAN Analytics ITL limit for Cisco MDS 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9) line cards is 20,000. Beginning with Cisco MDS NX-OS Release 9.3(2), the limit is increased to 40,000.

Configuring ITL Scale Limits for Cisco MDS 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9) line cards

To configure the ITL scale limits, perform these steps:

Procedure

Step 1

Enter configuration mode:

switch# configure

Step 2

switch(config)# analytics scale version {default |1|2}

default

Specifies the default configuration limit as 40,000

1

Specifies the configuration limit as 20,000

2

Specifies the configuration limit as 40,000

Zone Fanout Scale Limits

Table 11. Zone Fanout Scale Limits
Fanout Ratio (Target:Initiator) Initiator Target Plaftforms Tested
1:200 20 port each with 10 logins per port 15 port each with 3 logins per port MDS 9700 for Line Cards DS-X9448-768K9, DS-X9648-1536K9, and DS-X9748-3072K9

For DS-X9448-768K9 it was tested on DS-X97-SF1-K9

For DS-X9648-1536K9 and DS-X9748-3072K9 it was tested on DS-X97-SF4-K9