Add Switches for LAN Operational Mode, Release 12.1.3

When a new Cisco NX-OS device is powered on, typically that device has no startup configuration or any configuration state for that matter. Consequently, it powers on with NX-OS and post initialization, goes into a POAP loop. The device starts sending out DHCP requests on all the interfaces that are up including the mgmt0 interface.

As long as there is IP reachability between the device and the Nexus Dashboard Fabric Controller, the DHCP request from the device, will be forwarded to the Nexus Dashboard Fabric Controller. For easy day-0 device bring-up, the bootstrap options should be enabled on the Fabric Settings as mentioned earlier.

With bootstrap enabled for the fabric, the DHCP request coming from the device will be serviced by the Nexus Dashboard Fabric Controller. The temporary IP address allocated to the device by the Nexus Dashboard Fabric Controller will be employed to learn basic information about the switch including the device model, device NX-OS version, etc.

In the Nexus Dashboard Fabric Controller UI, choose Switch > Actions > Add Switches.

The Add Switches window appears with default tabs.

Choose Bootstrap(POAP) radio button.

As mentioned earlier, Nexus Dashboard Fabric Controller retrieves the serial number, model number, and version from the device and displays them on the Inventory Management along window. Also, an option to add the IP address, hostname, and password are made available. If the switch information is not retrieved, refresh the window.

At the top left part of the window, export and import options are provided to export and import the .csv file that contains the switch information. You can pre-provision devices using the import option as well.

From Cisco NDFC Release 12.1.1e, for pre-provisioned and bootstrap switches dummy values can be added for the serial number. After configuring the network successfully, serial number can be changed with the appropriate number of the switch on the Switches tab.

You can change serial number only for Nexus 9000 Series switches.

Select the checkbox next to the switch and enter the switch credentials: IP address and host name.

Based on the IP address of your device, you can either add the IPv4 or IPv6 address in the IP Address field.

You can provision devices in advance. To pre-provision devices, refer to Pre-provisioning device section.

In the Admin Password and Confirm Admin Password fields, enter and confirm the admin password.

This admin password is applicable for all the switches displayed in the POAP window.

You can specify a new user. Choose radio button Specify a new user enter Username, Password and choose Authentication Protocol from drop-down list.

If you do not want to use admin credentials to discover switches, you can instead use the AAA authentication, that is, RADIUS or TACACS credentials for discovery only.

(Optional) Use discovery credentials for discovering switches.

Click Bootstrap at the top right part of the screen.

Nexus Dashboard Fabric Controller provisions the management IP address and other credentials to the switch. In this simplified POAP process, all ports are opened up.

Click Refresh Topology to get updated information. The added switch goes through the POAP cycle. Monitor and check the switch for POAP completion.

After the added switch completes POAP, the fabric builder topology page is refreshed with the added switch thereby depicting its discovered physical connections. Set the appropriate role for the switch followed by a Deploy Config operation at the fabric level. The Fabric Settings, switch role, the topology etc. are evaluated by the Fabric Builder and the appropriate intended configuration for the switch is generated as part of the Save operation. The pending configuration will provide a list of the configurations that need to be deployed to the new switch in order to bring it IN-SYNC with the intent.

For any changes on the fabric that results in the Out-of-Sync, then you must deploy the changes. The process is the same as explained in the Discovering Existing Switches section.

During fabric creation, if you have entered AAA server information (in the Manageability tab), you must update the AAA server password on each switch. Else, switch discovery fails.

When discovering devices using SNMP, if you have configured to use an AAA server for authentication, the command sync-snmp-password <password> <username> is run on the switch through NDFC to generate a cached user. The authentication uses MD5, by default. You must specify the SNMPv3 authentication and Privacy protocol attributes in the switch AV-pair as follows:

snmpv3:auth=SHA priv=AES-128

After the pending configurations are deployed, the Progress column displays 100% for all switches.

Click Close to return to the fabric builder topology.

Click Refresh Topology to view the update. All switches must be in green color indicating that they are functional.

The switch and the link are discovered in Nexus Dashboard Fabric Controller. Configurations are built based on various policies (such as fabric, topology, and switch generated policies). The switch image (and other required) configurations are enabled on the switch.

In the Nexus Dashboard Fabric Controller GUI, the discovered switches can be seen in the Standalone fabric topology. Up to this step, the POAP is completed with basic settings. You must set up interfaces through the LAN > Switches. Select a switch, then a slide-in pane appears. Click the Launch icon. On the Switches Overview tab, click the Interface tab for any additional configurations, but not limited to the following:

To resolve, go to the Interfaces > Actions > Deploy tab and deploy the Shutdown operation on the nve interface followed by a No Shutdown configuration. This is depicted in the figure below where the up arrow corresponds to a No Shutdown operation while a down arrow corresponds to a Shutdown operation.

After you click Add Switches, click Discover Switches to add one or more existing switches into the fabric.

In this case, a switch with known credentials and a pre-provisioned IP address, is added to the fabric.

The IP address (Seed IP), username, and password (Username and Password fields) of the switch are provided as the input by a user. The Preserve Config check box is chosen by default. This is the option that a user would select for a brownfield import of a device into the fabric. For a greenfield import where the device configuration will be cleaned up as part of the import process, the user should set the Preserve Config check box is not selected.

BGP Fabric does not support brownfield import of a device into the fabric.

Click Discover Switches.

The Add Switches window appears. Since the Max Hops field was populated with 2 (by default), the switch with the specified IP address (leaf-91) and switches two hops from that switch, are populated in the Add Switches result.

If the Cisco Nexus Dashboard Fabric Controller was able to perform a successful shallow discovery to a switch, the status column shows as Manageable. Choose the check box next to the appropriate switch(es) and click Add Switches.

Though this example describes the discovery of one switch, multiple switches can be discovered at once.

The switch discovery process is initiated. The Progress column displays progress for all the selected switches. It displays done for each switch on completion.

You must not close the screen (and try to add switches again) until all selected switches are imported or an error message comes up.

If an error message comes up, close the screen. The fabric topology screen comes up. The error messages are displayed at the top right part of the screen. Resolve the errors wherever applicable and initiate the import process again by clicking Add Switches in the Actions panel.

Cisco Nexus Dashboard Fabric Controller discovers all the switches, and the Progress column displays done for all switches, close the screen. The Standalone fabric topology screen comes up again. The switch icons of the added switches are displayed in it.

You will encounter the following errors during switch discovery sometimes.

Click Refresh topology to view the latest topology view.

When all switches are added and roles assigned to them, the fabric topology contains the switches and connections between them.

After discovering the devices, assign an appropriate role to each device. For more information on roles, see Assigning Switch Roles.

If you choose the Hierarchical layout for display (in the Actions panel), the topology automatically gets aligned as per role assignment, with the leaf devices at the bottom, the spine devices connected on top of them, and the border devices at the top.

Assign vPC switch role - To designate a pair of switches as a vPC switch pair, right-click the switch and choose the vPC peer switch from the list of switches.

AAA server password - During fabric creation, if you have entered AAA server information (in the Manageability tab), you must update the AAA server password on each switch. Else, switch discovery fails.

When a new vPC pair is created and deployed successfully using Cisco Nexus Dashboard Fabric Controller, one of the peers might be out-of-sync for the no ip redirects CLI even if the command exists on the switch. This out-of-sync is due to a delay on the switch to display the CLI in the running configuration, which causes a diff in the configuration compliance. Re-sync the switches in the Config Deployment window to resolve the diff.

Click Save.

The template and interface configurations form the underlay network configuration on the switches. Also, freeform CLIs that were entered as part of fabric settings (leaf and spine switch freeform configurations entered in the Advanced tab) are deployed.

Configuration Compliance: If the provisioned configurations and switch configurations do not match, the Status column displays out-of-sync. For example, if you enable a function on the switch manually through a CLI, then it results in a configuration mismatch.

To ensure configurations provisioned from Cisco Nexus Dashboard Fabric Controller to the fabric are accurate or to detect any deviations (such as out-of-band changes), Nexus Dashboard Fabric Controller’s Configuration Compliance engine reports and provides necessary remediation configurations.

When you click Deploy Config, the Config Deployment window appears.

If the status is out-of-sync, it suggests that there is inconsistency between the Nexus Dashboard Fabric Controller and configuration on the device.

The Re-sync button is displayed for each switch in the Re-sync column. Use this option to resynchronize Nexus Dashboard Fabric Controller state when there is a large scale out-of-band change, or if configuration changes do not register in the Nexus Dashboard Fabric Controller properly. The re-sync operation does a full CC run for the switch and recollects "show run" and "show run all" commands from the switch. When you initiate the re-sync process, a progress message is displayed on the screen. During the re-sync, the running configuration is taken from the switch. The Out-of-Sync/In-Sync status for the switch is recalculated based on the intent defined in Nexus Dashboard Fabric Controller.

Click the Preview Config column entry (updated with a specific number of lines). The Config Preview screen comes up.

The Pending Config tab displays the pending configurations for successful deployment.

The Side-by-side Comparison tab displays the current configurations and expected configurations together.

Multi-line banner motd configuration can be configured in Cisco Nexus Dashboard Fabric Controller with freeform configuration policy, either per switch using switch_freeform, or per fabric using leaf/spine freeform configuration. Note that after the multi-line banner motd is configured, deploy the policy by executing the Deploy Config option in the (top right part of the) fabric topology screen. Else, the policy may not be deployed properly on the switch. The banner policy is only to configure single-line banner configuration. Also, you can only create one banner related freeform configuration/policy. Multiple policies for configuring banner motd are not supported.

Close the screen.

After successful configuration provisioning (when all switches display a progress of 100%), close the screen.

The fabric topology is displayed. The switch icons turn green to indicate successful configuration.

If a switch icon is in red color, it indicates that the switch and Nexus Dashboard Fabric Controller configurations are not in sync. When deployment is pending on a switch, the switch is displayed in blue color. The pending state indicates that there is a pending deployment or pending recomputation. You can click on the switch and review the pending deployments using the Preview or Deploy Config options, or click Deploy Config to recompute the state of the switch.

If there are any warning or errors in the CLI execution, a notification will appear in the Fabric builder window. Warnings or errors that are auto-resolvable have the Resolve option.

An example of the Deploy Config option usage is for switch-level freeform configurations. Refer for details.

Choose LAN > Switches > Add Switches.

Choose Bootstrap(POAP) radio button.

Click Actions and add Switches.

You can add switches one at a time using the Add option or add multiple switches at the same time using the Import option.

If you use the Add option, ensure you enter all the required details.

It might take some time for the switches to appear.

Choose a required switch.

Click Edit.

The Edit bootstrap switch dialog appears.

Enter the required details.

Click Save.

Choose the switch.

Enter the admin password in the Admin password field.

Click Import Selected Switches.

Double-click on the fabric containing the pre-provisioned device from the LAN Fabrics window.

The Fabric Overview window appears.

On the Interfaces tab, click Actions > Create Interface.

The Create Interface window appears.

Enter all the required details in the Create Interface window.

Type: Select Ethernet from the drop-down list.

Select a device: Select the pre-provisioned device.

You cannot add an Ethernet interface to an already managed device .

Interface Name: Enter a valid interface name based on the module type. For example, Ethernet1/1, eth1/1, or e1/1. The interface with same name should be available on the device after it is added.

Policy: Select a policy that should be applied on the interface.

For more information, see Add Interfaces for LAN Operational Mode.

Click Save.

Click Preview to check the expected configuration that will be deployed to the switch after it is added.

The Deploy button is disabled for Ethernet interfaces since the devices are pre-provisioned.

Import both the devices into the fabric. For more information, refer Pre-provisioning a Device.

Two Cisco Nexus 9000 Series devices that are pre-provisioned and added to an existing Fabric. Choose Add Switches from the Actions drop-down list. On the Inventory Management screen, click PowerOn Auto Provisioning (POAP).

The devices will show up in the fabric as gray/undiscovered devices.

Right click and select appropriate roles for these devices similar to other reachable devices.

To create vPC pairing between the devices with physical peer-link or MCT, perform the following steps:

Create physical ethernet interfaces on the pre-provisioned devices. Add a vPC host interface similar to a regular vPC pair or switches. For more information, see Pre-provisioning an Ethernet Interface.

For example, leaf1-leaf2 represents the pre-provisioned vPC device pair, assuming that Ethernet interfaces 1/1 is already pre-provisioned on both devices leaf1 and leaf2.

Create a vPC host truck interface .

Preview and Deploy actions doesn’t yield any result, because both require the device to be present. The vPC host interface is created and displays status as Not discovered.

Choose required switch, choose Actions > Discovery > Update Credentials.

The Update Discovery Credentials window appears.

In the Update Discovery Credentials window, enter the discovery credentials such as discovery username and password.

Click Update to save the discovery credentials.

If the discovery credentials are not provided, Nexus Dashboard Fabric Controller uses the admin user and password to discover switches.

Choose required switch, choose Actions > Discovery > Update VRF.

The Update Discovery VRF window appears.

In the Update Discovery VRF window, choose New VRF and Interface from drop-down list.

Click OK to save new VRF details.

Choose LAN > Fabrics. Double-click on the required fabric.

The Fabric Overview window appears.

On the Switches*tab, click *Actions > Add Switches.

For more information, see the section "Adding Switches to a Fabric" in Add Switches for LAN Operational Mode.

Right-click on an existing switch or the newly added switch, and use the Set role option to set the appropriate super spine role.

If the Super Spine role exists in the fabric, you can assign border super spine and border gateway super spine roles for any new device.
If super spine or any of its variation role is not assigned, you may assign the role to any new device if it is connected to a non-border spine. After a Recalculate & Deploy, you will receive an error that can be resolved by clicking on the Resolve button as shown in the below steps.

On the Fabric Overview window, click Actions > Recalculate & Deploy.

The following error message is displayed:

Super Spine role cannot be allowed in the existing fabric as it is disruptive. Please go to 'Event Analytics' and click on the resolve button to proceed.

Choose Event Analytics > Alarms, click on the ID.

The Alarm ID slide-in pane appears.

Click Resolve.

The Confirm action dialog box appears.

Click Confirm.

On the Fabric Overview window, click Actions > Recalculate & Deploy.

Do not add a devices with super spine, border super spine, or border gateway super spine role if the device is connected to a border spine or border gateway spine. This action results in an error after you recalculate and deploy the configuration. To use existing devices with border spine roles, remove the device and add the device with appropriate roles.

Right-click a vPC switch and choose vPC Pairing.

The vPC peer screen comes up.

Click Unpair at the bottom right part of the screen.

The vPC pair is deleted and the fabric topology window appears.

Click Deploy Config.

(Optional) Click the value under the Recalculate Config column.

View the pending configuration in the Config Preview dialog box. The following configuration details are deleted on the switch when you unpair: vPC feature, vPC domain, vPC peerlink, vPC peerlink member ports, loopback secondary IPs, and host vPCs. However, the host vPCs and port channels are not removed. Delete these port channels from the Interfaces window if required.

Choose check box for required switch, choose Actions > More > Provision RMA.

The Provision RMA window appears.

The Provision RMA UI will show the replacement device 5-10 minutes after it is powered on.

Choose check box for required switch, choose Actions > More > Copy Run Start.

The Copy Running Config to Startup Config screen appears. In the Progress column shows the process in progress and status description shows Deployment in progress.

A confirmation window appears, click OK.

The status description column displays Deployment completed and progress column in green.

Click Close to close this window.

Choose Actions > More > Reload.

The Restore Switch window appears and you are in the Select a Backup tab. See Backing Up and Restoring LAN Operational Mode Setups for more information.

The Select a Backup tab displays the fabric backup details. It includes the following information:

Choose radio button for required backup to mark as golden, choose Actions > Mark as golden, a confirmation window appears, click Confirm.

Choose radio button for backup to delete from golden, choose Actions > Remove as golden, a confirmation window appears, click Confirm.

Most of this information is at the fabric level, and may or may not directly impact the proceedings of the switch-level restore.

Click Next to move to the Restore Preview step.

You can view information about the switch name, switch serial, IP address, status, restore supported, delta configuration and the VRF details.

(Optional) Click Get Config to preview device configuration details.

The Config Preview window appears, which has three tabs.

Click Restore Intent to proceed to the Restore Status step in restoring.

The restore status and description appears for the switch.

Click Finish after the restoring process is complete.

Choose Actions > More > Show commands.

The Switch Show Commands window appears.

Choose required commands from drop-down list and enter required information in text field.

Click Execute to view the CLI output and to clear the output, click Clear Output.

Choose Actions > More > Exec commands.

The Switch Show Commands window appears.

From the Template drop-down list, choose exec_freeform or exec_elam_capture.

Enter the commands in the Freeform CLI for exec_freeform and required IP addresses.

Click Deploy to run the EXEC commands.

In the CLI Execution Status window, you can check the status of the deployment. Click Detailed Status under the Command column to view details.

In the Command Execution Details window, click the info under the CLI Response column to view the output or response.

Log into Cisco Nexus Dashboard.

Choose Administrative > Security.

Navigate to Security Domains tab.

Click Create Security Domain.

Enter the required details and click Create.

Log into Cisco Nexus Dashboard.

Choose Administrative > Users.

Click Create Local User.

Enter the required details and click Add Security Domain.

Choose a domain from the drop-down list.

Assign a Cisco Nexus Dashboard Fabric Controller service read or write role by checking the appropriate check box.

Click Save.