Connecting the Switch to the ACI Fabric

ACI Fabric Topology

The ACI fabric topology includes the following major components:

  • Application Centric Infrastructure Controller (APIC) appliance (cluster of APICs)

  • Leaf switches (for switch compatibility, please see the data sheets).

  • Spine switches (for switch compatibility, please see the data sheets).

For additional information, please see the Cisco APIC Installation, Upgrade, and Downgrade Guide.


Note

To prevent sub-optimal forwarding between endpoints, connect every leaf switch in the fabric to every spine switch in the same fabric.

As shown in the following figure, each APIC is connected to one or two leaf switches and each leaf switch should be connected to every spine switch in the same fabric.

Figure 1. Connections Between APIC Clusters, Leaf Nodes, and Spine Nodes

Preparing to Connect to Other Devices

When preparing to connect the fabric devices, consider the following for each type of interface, and gather all of the required equipment before making the connections:

  • Cabling type required for each interface type

  • Distance limitations for each signal type

  • Additional interface equipment required


Note

When running power and data cables in overhead or subfloor cable trays, we strongly recommend that you locate power cables and other potential noise sources as far away as practical from network cabling that terminates on Cisco equipment. In situations where long parallel cable runs cannot be separated by at least 3.3 feet (1 meter), we recommend that you shield any potential noise sources by housing them in a grounded metallic conduit.

The optical transceivers that are not already assembled to their cables come separate from their cables. To prevent these transceivers and their cables from being damaged, we recommend that you keep the transceivers disconnected from their cables when installing them in ports and then insert the optical cable into the transceiver. When removing transceivers from ports, remove their cables before removing the transceivers.

To maximize the effectiveness and life of your transceivers and optical cables, do the following:

  • Wear an ESD-preventative wrist strap that is connected to an earth ground whenever handling transceivers. The switch is typically grounded when you install transceivers and provides an ESD port to which you can connect your wrist strap. If you cannot find an ESD port, connect the wrist strap to an earth ground (such as the grounding connection for the chassis).

  • Do not remove or insert a transceiver more often than necessary. Repeated removals and insertions can shorten its useful life.

  • Keep the transceivers and fiber-optic cables clean and dust free to maintain high signal accuracy and to prevent damage to the connectors. Attenuation (loss of light) increases with contamination and should be kept below 0.35 dB.

    • Clean these parts before installing them to prevent dust from scratching the fiber-optic cable ends.

    • Clean the connectors regularly; the required frequency of cleaning depends upon the environment. In addition, clean connectors if they are exposed to dust or accidentally touched. Both wet and dry cleaning techniques can be effective; refer to your site's fiber-optic connection cleaning procedures.

    • Do not touch the ends of connectors. Touching the ends can leave fingerprints and cause other contamination.

  • Inspect routinely for dust and damage. If you suspect damage, clean and then inspect fiber ends under a microscope to determine if damage has occurred.

Connecting Leaf Switches to APICs

You must downlink one or two (recommended for redundancy) Cisco Nexus 9300 platform ACI-mode leaf switches to each Application Policy Infrastructure Controller (APIC) in your ACI fabric. The type of virtual interface card (VIC) installed on the APIC determines the types of interface cables that you can use to connect the leaf switches to the APICs.

  • The VIC 1225T module supports copper connectors, copper cables, and switches with copper downlink ports (such as: Cisco Nexus 93108TC-FX switche).

  • The VIC 1225 module supports optical transceivers, optical cables, and switches with optical downlink ports (such as: Cisco Nexus 93600CD-GX switche).

  • The VIC 1455 module supports optical transceivers, optical cables, and switches with optical downlink ports (such as: Cisco Nexus 93600CD-GX switche).


Note

Breakout ports cannot be used for Cisco APIC connectivity.

Before you begin

The APIC and leaf switches in the fabric must be fully installed in their racks and grounded.

Procedure

Step 1

Connect an interface cable to one of the two to four ports on the virtual interface card (VIC) installed on the APIC. If the cable is not already assembled to its transceivers, insert the transceiver into the VIC port and then connect the optical interface cable to the transceiver.

  • For a VIC 1225T 10GBASE-T copper module, use 10GBASE-T cables with RJ-45 connectors.

  • For a VIC 1225 optical module, use one of the following sets of transceivers and cables:

    • Cisco 10GBASE-LR transceivers (SFP-10G-LR) supporting a link length of up to 6.1 miles (10 km)

    • Cisco 10GBASE-SR transceivers (SFP-10G-SR) supporting the following link lengths:

      • Using 2000 MHz MMF (OM3) for up to 984 feet (300 m)

      • Using 4700 MHz MMF (OM4) for up to 1312 feet (400 m)

    • Cisco SFP+ Active Optical Cables (SFP-10G-AOCxM [where x=1, 3, 5, 7 for lengths in meters])

    • Cisco SFP+ Twinax Cables (SFP-H10GB-CUxM [where x=7 for lengths in meters])

  • For a VIC 1455 SFP28 module, 10-Gigabit only, use one of the following sets of transceivers and cables:

    • Cisco 10GBASE-LR transceivers (SFP-10G-LR) supporting a link length of up to 6.1 miles (10 km)

    • Cisco 10GBASE-SR transceivers (SFP-10G-SR) supporting the following link lengths:

      • Using 2000 MHz MMF (OM3) for up to 984 feet (300 m)

      • Using 4700 MHz MMF (OM4) for up to 1312 feet (400 m)

    • Cisco SFP+ Active Optical Cables (SFP-10G-AOCxM [where x=1, 2, 3, 5, 7, or 10 for lengths in meters])

    • Cisco SFP+ Twinax Cables (SFP-H10GB-CUxM [where x=1, 2, 3, 5, 7, or 10 for lengths in meters])

    Note

     

    The VIC 1455 has 4 ports, port-1, port-2, port-3, and port-4 from left to right.

    • All ports must have the same speed, either 10-Gigabit or 25-Gigabit.

    • Port-1 and port-2 is one pair, corresponding to eth2-1 on APIC and port-3 and port-4 is another pair, corresponding to eth2-2 on APIC. Only one connection is allowed for each pair. For example, you can connect one cable to either port-1 or port-2, and connect another cable to either port-3 or port-4 (please do not connect two cables on any pair).

    To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

    To see the transceiver specifications and installation information, see Transceiver Module Installation Guides.

Step 2

Connect the other end of the interface cable to a downlink port on a leaf switch.

  • For a Cisco 10GBASE-LR or -SR transceiver and cable, insert the transceiver into a downlink optical port on a leaf switch before connecting the cable to the transceiver.

  • For Cisco SFP+ Active Optical Cables, insert the transceiver on the cable into a downlink optical port on a leaf switch.

  • For a 10GBASE-T copper cable, insert the RJ-45 connector on the cable into a downlink BASE-T port on a leaf switch.

Note

 

To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

Connecting Leaf Switches to Spine Switches

For optimal forwarding between endpoints, you must connect each leaf switch to every spine switch in the same ACI fabric.

To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

To see the transceiver specifications and installation information, see Transceiver Module Installation Guides.


Warning

Statement 1055—Class 1/1M Laser

Invisible laser radiation is present. Do not expose to users of telescopic optics. This applies to Class 1/1M laser products.


Warning

Statement 1056—Unterminated Fiber Cable

Invisible laser radiation may be emitted from the end of the unterminated fiber cable or connector. Do not view directly with optical instruments. Viewing the laser output with certain optical instruments, for example, eye loupes, magnifiers, and microscopes, within a distance of 100 mm, may pose an eye hazard.

Before you begin

  • The leaf and spine switches in the fabric (such as: N9k-C9364C, N9K-C9332C, N9K-C9316D-GX, and N9K-C9332D-GX2B) must be fully installed in their racks and grounded.

  • If there are modular switches in the fabric, their ACI-mode line cards must already be installed. The line cards can be of the following types:

    • 32-port 100-Gigabit (such as: N9K-X9732C-EX)

    • 36-port 100-Gigabit (such as: N9K-X9736C-FX)

    • 16-port 400-Gigabit (such as: N9K-X9716D-GX)


Note

You cannot include NX-OS line cards in the same chassis when running in ACI mode.

Note

Multiple uplinks from a leaf switch to a spine switch is supported. A symmetrical topology is recommended so that all devices have equal access to resources.

Procedure

Step 1

For the transceivers with removable cables, make sure that the transceivers are separated from their interface cables.

Step 2

Insert the appropriate transceiver into an active uplink port on the leaf switch.

Step 3

Insert the same type of transceiver in the spine switch port on the line card.

Step 4

For transceivers with removable cables, insert the interface cable into the open end of each of those transceivers.

Step 5

Repeat Steps 1 through 4 for each spine switch in the ACI fabric.

The leaf switch is connected to each spine switch in the ACI fabric.

Step 6

Repeat Steps 1 through 5 for each leaf switch in the ACI fabric.

Each leaf switch in the ACI fabric is connected to each spine switch in the network,

The fabric automatically implements Equal Cost Multi-Pathing (ECMP) and enables all links. You do not need to configure the links.

FEX Conversion and Support

By default, Cisco Nexus switches operate in the switch mode. However, Cisco Nexus FX3 platform switches also support the Fabric Extender (FEX) mode.

This FEX mode allows a switch to operate like a Cisco Nexus 2000 series Fabric Extender. As a result, the switch will not require any:

  • Independent software upgrades

  • Configuration backups

  • Other maintenance tasks

In addition to existing advantages of FEX topologies on ACI, FX3 switches in FEX mode support speeds up to 25G per Host Interface (HIF) connectivity to host.

The Nexus FX3 Switches supports FEX mode from ACI release, 6.0(2).

Leaf to FEX Conversion

In Leaf to FEX conversion, FEX5 binary present in the leaf image will be used, so device will come up in FEX mode with the same version as before the conversion.

  • To convert a leaf switch to a FEX mode switch:

Procedure

Step 1

Login to the leaf, to be converted

Step 2

Run boot-fex command on the ibash shell. In the backend FEX image will be verified and set as boot variable.

Step 3

Run reload command. Then the leaf will be converted to FEX.

FEX to Leaf Conversion

In FEX to leaf conversion, FEX can either use the existing leaf image which was used before converting to FEX or copy a new one after configuring management.

  • To convert a FEX mode switch to a leaf switch:

Procedure

Step 1

Use an ACI image, which can be an existing leaf image in bootflash or copy a different ACI image to the FEX bootflash via copy scp command.

Step 2

Run boot aci <image> CLI to set boot variable and convert to leaf.

New FEX Setup

Cisco Nexus FX3 platform switches will be shipped with leaf mode.

  • To use a Cisco Nexus FX3 platform switch as a FEX"

Procedure

Step 1

Run the boot-fex CLI via console to set the FEX boot variable and reload to convert to FEX.

Step 2

Connect the new FEX to a Leaf in the topology.

Change an Existing Leaf in the Topology to FEX

In an existing ACI topology, a Leaf switch will be connected to a Spine switch directly.

Procedure

Step 1

Leaf to FEX conversion can be triggered by running via boot-fex command

Step 2

The FEX (previously leaf) will have to be re-connected to another Leaf in the topology for it to come up.

Step 3

FEX policies can be configured on the APIC as required.

Change an Existing FEX in the Topology to Leaf

In an existing ACI topology, a FEX switch will be connected to a Leaf switch directly.

Procedure

Step 1

FEX to Leaf conversion can be triggered using boot aci <image> command on FEX.

Step 2

The Leaf (previously FEX) will have to be re-connected to a Spine switch in the topology for it to come up in APIC.

Step 3

Leaf policies can then be configured on the APIC as required.

Installing a Gigabit Ethernet module (GEM)

Procedure

Step 1

Clear the switch's current configuration by using the setup-clean-config command.

Step 2

Power off the switch by disconnecting the power.

Step 3

Replace the current GEM card with the new GEM card.

Step 4

Power on the switch.

Migration of Nodes From a First Generation Switch to a Second Generation Switch

You have first generation Cisco Nexus 9000 series switches that may or may not be comprising a virtual port channel (vPC). You are migrating to second generation Cisco Nexus 9000 series switches using the same cables.

First generation Cisco Nexus 9000 series switches include those switches that do not contain -EX, -FX, or -GX in the product ID.

Second generation Cisco Nexus 9000 series switches include those switches that have the -EX, -FX, -GX, or later suffix in the product ID.

To migrate the first generation switches to second generation switches, you must perform the steps in this procedure.

To determine which transceivers, adapters, and cables support this switch, see the Cisco Transceiver Modules Compatibility Information document.

To see the transceiver specifications and installation information, see Transceiver Module Installation Guides.

Before you begin

  • Move any Cisco Application Policy Infrastructure Controllers (APICs) that are connected to the first generation switches that you are migrating to any other switches in the fabric and wait for the Cisco APIC cluster to become "Fully Fit."

  • The following migration paths are supported:

    1. Migrating from first generation Cisco Application Centric Infrastructure (ACI) switches to second generation Cisco ACI switches that are running the same software release.

    2. Migrating from first generation Cisco ACI switches to second generation Cisco ACI switches that are running different software releases.

      The second migration path is required where the existing switches are not supported on the new release that is required for the new switches. For example, if you want to migrate from the first generation Cisco ACI switches, such as Cisco Nexus 9300 (with the -E suffix or without any suffixes in the product ID) that are no longer supported starting on Cisco ACI switch 15.0(1) or later releases, to some of the new switches that are supported only from 15.0(1) or later.

      When the first generation switches are comprising a vPC, complete the following mandatory prerequisite steps before you proceed with the second migration path:

      1. Due to potential traffic loss, it is recommended that that you perform the vPC migration during a maintenance window.

      2. Before you perform this procedure, the Auto Firmware Update policy must be disabled.

      3. Upgrade the Cisco APIC cluster to the 4.2(7v) release if the cluster is running an older release. Also upgrade the first generation switches to the 14.2(7v) release. Wait for the fabric to converge.

      4. Upgrade the Cisco APIC cluster to 5.2(7f) release and wait for the cluster to become "Fully Fit."

      5. Ensure that the new second generation switches are preloaded and running the equivalent release as the Cisco APICs, that is 15.2(7f) release. Other than source and target version software releases 4.2(7v)/14.2(7v) and 5.2(7f)/15.2(7f), no other software releases are supported for this migration procedure.


Note

  • The number of ports and port types of the second generation switches must match the first generation switch that you are replacing. If the number does not match, then you must change the configuration to accommodate the new ports or port types. This is also applicable if you migrate the hardware while retaining the same software version.

  • To migrate first generation non-vPC leaf switches or first generation spine switches to second generation switches, follow Step 1 through Step 6 in the procedure outlined below. vPC-related information is not applicable for this migration.

    If you must migrate a first generation non-vPC leaf switch or a first generation spine switch to a second generation switch, the requirement of the source and target software release 4.2(7v)/14.2(7v) and 5.2(7f)/15.2(7f) is not required. Ensure that the Cisco ACI fabric is running the required software release that supports the second generation switch PID.

Procedure

Step 1

From the Cisco APIC GUI, perform the Remove From Controller operation for the operational secondary vPC switch node.

The Cisco APIC clean reboots the switch. Wait for about 10 minutes for this operation to finish. This action prompts all traffic to use the other first generation switch for data traffic.

Note

 

There will be a loss of traffic for a few seconds for the operational secondary vPC when you perform the Remove From Controller operation.

Step 2

Disconnect the cabling from the first generation switch that you just removed.

Step 3

Uninstall the first generation switch by reversing the order of the steps in the "Installing the Switch Chassis" section of the switch-specific Hardware Installation Guide.

Step 4

Install the second generation switch by following the steps in the "Installing the Switch Chassis" section of the switch-specific Hardware Installation Guide.

Step 5

Connect the loose cabling that you removed from the first generation switch to the same ports on the second generation switch.

Step 6

Register the new second generation switch with the Cisco APIC.

Register the new node with the same node name and node ID. This switch becomes part of the fabric. The Cisco APIC pushes the policies to the new switch and keeps down the vPC legs because there is a mismatch of the generation of switches. At this point, the vPC primary continues to send the data traffic.

Step 7

Before you proceed to Step 8, wait for 10 to 15 minutes for the new switch to download the configurations.

Step 8

From the Cisco APIC GUI, perform the Remove From Controller operation for the vPC primary. The Cisco APIC clean reboots the switch.

Wait for about 10 minutes for this operation to finish. The vPC leg on the second generation switch, which the Cisco APIC kept down earlier, comes up. This action prompts all traffic to move to the new second generation switch. The vPC ports on the new second generation switch can take a few minutes to come up, during which time there will be traffic drops. The duration of traffic drops varies by the scale and flows in the fabric.

Step 9

Disconnect the cabling from the first generation switch.

Step 10

Uninstall the first generation switch as you did in Step 3.

Step 11

Install the second generation switch as you did in Step 4.

Step 12

Connect the loose cabling as you did in Step 5.

Step 13

Register the new second generation switch with the Cisco APIC.

Register the new node with the same node name and node ID. This switch becomes part of the fabric. The Cisco APIC pushes policies to the new switch and the vPC legs comes up and starts passing traffic.

Setting Up an Optional Console Interface

You can optionally set up a console interface for performing the initial configuration of the switch. To do this, use the interface cable provided in the accessory kit to connect the switch to your console device. You can connect the console port on the switch to a modem. If you do not connect it to a modem, make the connection either before powering up the switch or after completing the boot process for the switch.

Before you begin

The console device must support VT100 terminal emulations and asynchronous transmissions.

Procedure

Step 1

Configure the terminal emulator program to match each of the following default port characteristics:

  • 9600 baud

  • 8 data bits

  • 1 stop bit

  • No parity

Step 2

Insert the RJ-45 connector on the interface cable found in the accessory kit into the RS-232 port on the switch and insert the DB-9 connector on the other end of the cable to the serial port on the console device.

What to do next

You can now perform the initial configuration for the switch (see the Cisco ACI Getting Started Guide).

Setting Up an Optional Management Connection

You can optionally set up an out-of-band management connection for monitoring and troubleshooting purposes. To do this, depending on your switch, you connect either the RJ-45 management port or the SFP management port on the switch to an external hub, switch, or router.

Before you begin

To prevent an IP address conflict, you must complete the initial configuration for the switch and establish an IP address before you create the management connection.

Procedure

Step 1

Connect the interface cable to a management port on the switch.

Step 2

Connect the other end of the cable to an external hub, switch, or router.

Maintaining Transceivers and Optical Cables

Transceivers and fiber-optic cables must be kept clean and dust free to maintain high signal accuracy and prevent damage to the connectors. Contamination increases attenuation (loss of light) and should be below 0.35 dB.

Consider the following maintenance guidelines:

  • Transceivers are static sensitive. To prevent ESD damage, wear an ESD-preventative wrist strap that is connected to the grounded chassis.

  • Do not remove and insert a transceiver more often than is necessary. Repeated removals and insertions can shorten its useful life.

  • Keep all optical connections covered when not in use. Clean them before using to prevent dust from scratching the fiber-optic cable ends.

  • Do not touch the ends of connectors. Touching the ends can leave fingerprints and cause other contamination.

  • Clean the connectors regularly; the required frequency of cleaning depends upon the environment. In addition, clean connectors if they are exposed to dust or accidentally touched. Both wet and dry cleaning techniques can be effective; refer to the fiber-optic connection cleaning procedures for your site.

  • Inspect routinely for dust and damage. If you suspect damage, clean and then inspect fiber ends under a microscope to determine if damage has occurred.