Introducing Cisco Programmable Fabric Management, Operations, and Provisioning

Introduction to Programmable Fabric Management, Operations, and Provisioning

This chapter briefly describes POAP auto configuration profiles:

Power On Auto Provisioning in Programmable Fabric

A NX-OS device with Power On Auto Provisioning (POAP) feature enabled needs to be able to find its IP address and download image/configuration and successfully complete the POAP process via the DHCP server and a repository server or Cisco DCNM that is located across in a VXLAN/EVPN fabric.

In a VXLAN-EVPN based Programmable Fabric deployment, day-0 bring up of switches is supported in an automated way via POAP. Along with the traditional POAP option via the mgmt0 (out-of-band) interfaces, day-0 bring-up of the devices can also be done over the front-panel Ethernet (inband) interfaces.


Note

Inband POAP and management are available beginning with Cisco NX-OS Release 7.0(3)I5(2) for Cisco Nexus 9000 Series switches and Cisco NX-OS 7.1(2) on Cisco DCNM with 10.1(2)ST(1) Cisco DCNM templates.

In an IP fabric, the use of IP unnumbered interfaces is preferred to simplify IP address management on the underlay where only one unique IP address per device is required to bring up the routing table state between the various devices. All the core facing interfaces on a device that participate in the IGP (for example, IS-IS or OSPF) share this per device unique IP. While it greatly simplifies the IP address management on the underlay, it certainly complicates how DHCP relay functionality works on the inband interfaces as there are no longer regular Layer 3 interfaces under which a relay can be readily configured. Without DHCP relay functionality, inband POAP cannot work.

POAP over IP numbered interfaces works when the DHCP server is configured with unique subnet scopes for every Layer 3 core-facing interface pair and IP DHCP relay is enabled under every core-facing interface. The DHCP server can be connected to a leaf that is attached to the default VRF.

Prerequisites for Inband POAP over IP Unnumbered Links

See the following illustration for inband POAP via IP unnumbered fabric interfaces:

Figure 1. Inband POAP via IP Unnumbered Fabric Interfaces

See the following prerequisites for inband POAP over IP unnumbered links:

  • The seed/edge leaf leaf-1 is the leaf that is connected to the POAP network. In the topology, a router sits in between the edge leaf and the DCNM.

  • On the seed/edge leaf node, the IP address of the port connecting to the DCNM network is recommended to be a /30 IP. It means that the IP on the other side, the next hop, is the other IP in the /30 network.

  • On the router connecting the seed leaf to the DHCP network, DHCP relay configuration for reachability to the DHCP server IP must be made. An example CLI configuration on a Cisco Nexus switch that acts as a router is ip dhcp relay address <dhcp_server_ip> .

  • The DHCP server IP is the IP address of the DCNM if DCNM is used for POAP. Otherwise it is the IP address of the standalone DHCP server used. If DCNM is used for fabric deployment, the DCNM POAP template has the option to configure two DHCP Servers.

  • For inband POAP, the DHCP server’s dhcpd.conf must be configured manually to include new scope(s) for the networks in the fabric.

  • On the DCNM, vCenter, or on any other Network Management Station in the internal network, the route configuration must be done manually for network reachability (192.178.16.0 and 10.10.10.0 networks in the above sample topology).

  • There needs to be at least one seed/edge leaf node in the fabric with connectivity to the DCNM/DHCP server/vCenter/Management Station etc. for the other NX-OS nodes in the IP Fabric to successfully go through inband POAP.

With DCNM as the DHCP and Repository/Configuration Server:

  • When DCNM is used as the configuration server for the fabric, DCNM POAP templates are used for day-0 configuration on the devices. The DCNM POAP templates support the topology displayed in the illustration.

  • For the devices to be discovered in the inband IP fabric topology on the DCNM, the Management IP field in the POAP template's General tab should be the management IP of the device which is its loopback0 IP. This is not the mgmt0 interface IP as the vrf is default and not management.

  • The first time the devices are being brought up, write erase and reload must be issued on the device and not via the DCNM GUI. It is because the management IP on DCNM for inband case is that of the loopback0 IP and the fabric is not discovered yet before POAP for reachability from DCNM.

  • DHCP scope for the underlay network must be manually added either by configuring DHCP Scope via DCNM GUI or by editing the dhcpd.conf file in the DHCP server. Routes for reachability must also be manually added on other devices like the router in between, DCNM, and vCenter etc.

  • The seed leaf needs to be brought up first, following which the RR needs to be brought up. After the seedleaf and RR spine are up, any of the other nodes (non-RR spine, other Leaf nodes) are ready to successfully finish inband POAP. The devices loop in POAP phase until the seed leaf, RR are up and until there is a successful DHCP handshake and a successful TFTP operation.

Prerequisites for Inband POAP over IP Numbered Links

See the following illustration for inband POAP via IP numbered fabric interfaces:

Figure 2. Inband POAP via IP Numbered Fabric Interfaces

See the following prerequisites for inband POAP over IP numbered links:

  • The seed/edge leaf leaf-1 is the leaf that is connected to the POAP network. In the topology, a router sits in between the edge leaf and the DCNM.

  • On the seed/edge leaf node, the IP address of the port connecting to the DCNM network is recommended to be a /30 IP. It means that the IP on the other side, the next hop, is the other IP in the /30 network.

  • On the router connecting the seed leaf to the DHCP network, DHCP relay configuration for reachability to the DHCP server IP must be made. An example CLI configuration on a Cisco Nexus switch that acts as a router is ip dhcp relay address <dhcp_server_ip> .

  • The DHCP server IP is the IP address of the DCNM if DCNM is used for POAP. Otherwise it is the IP address of the standalone DHCP server used. If DCNM is used for fabric deployment, the DCNM POAP template has the option to configure two DHCP Servers.

  • For inband POAP, the DHCP server’s dhcpd.conf must be configured manually to include new scope(s) for the networks in the fabric. In the IP Numbered topology, the DHCP scope is required for every fabric link.

    Alternately, the following script is available as part of the DCNM OVA installation and it can be used to generate the DHCP scopes automatically on DCNM: /root/utils/inband_p2p_dhcp_scope.py. The script can be modified or customized if needed. The instructions are provided inside the script itself.

  • On the DCNM, vCenter, or on any other Network Management Station in the internal network, the route configuration must be done manually for network reachability (192.178.16.0 and 192.0.1.0 networks in the above sample topology).

  • There needs to be at least one seed/edge leaf node in the fabric with connectivity to the DCNM/DHCP server/vCenter/Management Station etc. for the other NX-OS nodes in the IP Fabric to successfully go through inband POAP.

With DCNM as the DHCP and Repository/Configuration Server:

  • When DCNM is used as the configuration server for the fabric, DCNM POAP templates are used for day-0 configuration on the devices. The DCNM POAP templates support the topology displayed in the illustration.

  • For the devices to be discovered in the inband IP fabric topology on the DCNM, the Management IP field in the POAP template's General tab should be the management IP of the device which is its loopback0 IP. This is not the mgmt0 interface IP as the vrf is default and not management.

  • The first time the devices are being brought up, write erase and reload must be issued on the device and not via the DCNM GUI. It is because the management IP on DCNM for inband case is that of the loopback0 IP and the fabric is not discovered yet before POAP for reachability from DCNM.

  • DHCP scope for the underlay network must be manually added either by configuring DHCP Scope via DCNM GUI or by editing the dhcpd.conf file in the DHCP server. Routes for reachability must also be manually added on other devices like the router in between, DCNM, and vCenter etc.

  • The seed leaf needs to be brought up first, following which the RR needs to be brought up. After the seedleaf and RR spine are up, any of the other nodes (non-RR spine, other Leaf nodes) are ready to successfully finish inband POAP. The devices loop in POAP phase until the seed leaf, RR are up and until there is a successful DHCP handshake and a successful TFTP operation.

Inband Management

Device management is done via vrf management in the out-of-band network or via vrf default in the inband network. This feature adds support to manage the network devices from DCNM via the front panel inband ports under vrf default for Cisco Nexus 9000 Series switches.


Note

Inband POAP and inband management is available beginning with Cisco NX-OS Release 7.0(3)I5(2) for Cisco Nexus 9000 Series switches and Cisco NX-OS 7.1(2) on Cisco DCNM with 10.1(2)ST(1) Cisco DCNM templates.

Inband POAP with inband management (Inband POAP and vrf = default for management) and Out-of-band POAP with Out-of-band management (POAP over mgmt0 and vrf = management for management) are currently supported via the DCNM POAP templates.

DCNM POAP Templates

See the following templates that are created to help generate the required configuration for VXLAN EVPN feature for Cisco Nexus switches:

  • Leaf

  • Spine

  • Border Leaf

  • Border Spine


Note

POAP is day-0 device automated provisioning. For information on the installation of DCNM POAP templates, see the Cisco DCNM Installation Guide. For POAP launchpad, see the Cisco DCNM Web Client Online Help.

POAP template parameters are divided into various groups and these parameters are used by the template to generate a set of CLI commands to be run on a switch.


Note

The parameters may vary based on your POAP template selection.

The VXLAN EVPN leaf template supports Dynamic Virtual Ports (DVP) and multi-mobility domain with DCNM version 10.0(1) or later, but the DVP remains disabled by default.

You can configure the encapsulation settings from Admin > Fabric > Fabric Encapsulation Settings to specify the following settings:

  • Enable VXLAN Encapsulation for Leaf Network

    • Specify multicast group subnet (default is 239.1.1.0/25)

    • Specify number of Rendezvous Points (RPs): 1, 2, or 4

      • One or multiple multicast groups based on the number of RPs specified are generated by Cisco DCNM automatically.

    • Specify RP Subnet (default is 10.254.254.0/24)

      • Phantom RP addresses are generated from the RP subnet

      • RP Redundancy is same as number of RPs

      • Corresponding loopback interfaces with IP addresses in the same subnet as the phantom RP addresses but with different masks are added to each RP.

Fabric Settings

The following global settings can be configured from the Cisco DCNM application.

You can specify the general settings for the Fabric. From the menu bar, select Admin > Fabric > General Settings to specify the following settings:

  • Configure LDAP Server and Segment/Partition ID ranges

    • Segment ID Range

    • Partition ID Range

  • Configure dynamic VLAN ranges used by auto configuration

    • System Dynamic VLANs

    • Core Dynamic VLANs

  • Configure global anycast gateway MAC

You can set common parameters, that are populated as default values in POAP templates. For a new POAP template, values defined in this global settings page, are automatically pre-populated. From the menu bar, select Admin > Fabric > POAP Settings to specify the following settings:

  • Configure global setting for the fabric backbone

    • BGP AS

    • BGP RR IP/IP

    • Backbone Prefix Length

    • Enable conversational learning

You can configure the encapsulation settings from Admin > Fabric > Fabric Encapsulation Settings to specify the following settings:

  • Enable VXLAN encapsulation for leaf network

  • Specify multicast group subnet (default is 239.1.1.0/25)

  • Specify number of rendezvous points (RPs): 1, 2, or 4

  • One or multiple multicast groups based on the number of RPs specified are generated by Cisco DCNM automatically

  • Specify RP Subnet (default is 10.254.254.0/24)

  • Phantom RP addresses are generated from the RP subnet

  • RP Redundancy is same as number of RPs

  • Corresponding loopback interfaces with IP addresses in the same subnet as the phantom RP addresses but with different masks are added to each RP


Note

The inband POAP and management support is available on Cisco DCNM POAP templates for Cisco Nexus 9000 Series switches since Cisco DCNM 10.1(x) release.

You can configure the following fields in the templates for inband POAP template configuration on leaf and spine nodes:

Cisco Nexus 9000 Series switches Leaf POAP Template:

For Seed/Edge Leaf configuration:

  • General->Management Vrf : Select default from the dropdown.

  • General->Seed device to enable inband POAP and management.

  • General->Inband Port

  • General->Inband Local Port IP

  • General->Inband Next-Hop Port IP

  • General->DHCP Server IP Address

  • General->Second DHCP Server IP Address

Spine POAP Template:

  • General->Management VRF: Select default from the dropdown.

  • Manageability->DHCP Server IP

  • Manageability->Second DHCP Server IP Address

Auto-Configuration in Programmable Fabric

Configuration Profile

A configuration profile in Cisco Programmable Fabric is a collection of commands used to instantiate a specific configuration. Based on appropriate end-host triggers (VDP or data plane trigger (any data frame)), the configuration profiles are grouped to allow flexible and extensible options to instantiate day-1 tenant-related configurations on a Cisco Programmable Fabric leaf on need basis.

The commands are entered using variables for certain parameters instead of entering the actual value. The switch then populates the actual values to derive the completed command. When the required parameters for a particular configuration profile are available, the profile can be instantiated to create a configuration set. The switch then applies this configuration set to complete the command execution belonging to the configuration set.

The commands which are supported under a configuration profile are called config-profile-aware commands. Most of the commands in the switch can be entered into the configuration profile.


Note
Various sets of configuration profiles can be created and stored in the network database, and each network can use a different configuration profile. The configuration profiles can be used from the network to set up on the leaf whenever required.

Profile Refresh

Profile refresh involves updating and/or removing profile parameters (arguments or variables) without disrupting the traffic while using universal profiles. After the changes are done, Cisco DCNM executes the fabric database refresh vni/vrf command.

LDAP Configuration

There are four different tables, which you can query from:

  • Network Table

  • Partition Table

  • Profile Table

  • Host Table

  • BL-DCI Table

Network Table

All the parameters for a network host are stored in this table in the LDAP. ToR will query this LDAP table for network parameters based on the following configuration: 
Switch# configure terminal
Switch(config)# fabric database type network
Switch(config)# server protocol ldap ip 10.1.1.2 vrf <default/management> 
Switch(config)# db-table ou=networks, dc=cisco, dc=com key-type 1

Partition Table

All the parameters that are required to provision a VRF on the ToR are stored in this table. When a network is using a Universal Profile, querying this table can identify corresponding VRF profile. ToR queries this table for every new network that is provisioned using Universal Profile. The following configuration helps ToR to query the partition table: 
Switch# configure terminal
Switch(config)# fabric database type partition
Switch(config)# server protocol ldap ip 10.1.1.2 vrf <default/management>
Switch(config)# db-table ou=partitions, dc=cisco, dc=com

Profile Table

Multi-tenancy lite version

This table is used to store configuration profiles, which are pre-packaged with DCNM and custom config profiles that are created. The Cisco Nexus 5000 Series Switches or Cisco Nexus 6000 Series Switches employ this tenancy model where a maximum of 4000 VLANs are supported on the ToR. So, if a profile is not pre-configured on the system then ToR will query profile table to download profile contents and cache it locally.

The following configuration helps ToR to query the profile table: 
Switch# configure terminal
Switch(config)# fabric database type profile
Switch(config)# server protocol ldap ip 10.1.1.2 vrf <default/management>
Switch(config)# db-table ou=profiles, dc=cisco, dc=com

Multi-tenancy full version

The Cisco Nexus 7000 Series Switches employ this tenancy model where, up to 4000 VLANs or dot1q tags can be supported on a per port basis.

The following configuration helps ToR to query the profile table: 
Switch# configure terminal
Switch(config)# fabric database type profile
Switch(config)# server protocol ldap ip 10.1.1.2 vrf <default/management>
Switch(config)# db-table ou=profilesBridgeDomain, dc=cisco, dc=com

Host Table

Host table contains the information about the host to network mappings. The following configuration helps ToR to query the host table: 
Switch(config)# fabric database type host
Switch(config)# server protocol ldap host ip [ip address] vrf [vrf name]
Switch(config)# db-table ou=hosts,dc=cisco,dc=com

Note

A host table is required for LLDP auto-config support.

BL-DCI Table

All the parameters for provision BL-DCI (See the DCI Auto-configuration section for details) are stored in this table in the LDAP. The following configuration helps the switches to query the BL-DCI table: 

switch(config)# fabric database type bl-dci
switch(config)# server protocol ldap ip [ip address] vrf [vrf name] 
switch(config)# db-table ou=bl-dcis,dc=cisco,dc=com

Note

A BL-DCI table is required for border leaf. DCI Auto-configuration is not applicable to Cisco Nexus 9000 Series switches.

Specifying profile Mapping for network instances

It is necessary to specify how a particular network instance will be provisioned. That is either using network database lookup or by using a static Profile on the switch. The following configuration describes how to configure ToR to fetch network parameters from Remote database (LDAP). 

Switch(config)# fabric database profile-map global
Switch(config-profile-map-global)# dot1q default dynamic
Switch(config-profile-map-global)# vni default dynamic
The following configuration is for DCI Profile. 

Switch(config)# fabric database profile-map global
Switch(config-profile-map-global)# vrf default dynamic

LDAP Preemptive


Note

LDAP Preemptive is currently not supported on Cisco Nexus 9000 Series switches.

Customer can deploy multiple LDAP servers. One server can be specified as primary server and the other servers become secondary. When the primary LDAP server is up, the device always queries the primary server. When the primary server is down, the device queries the secondary servers instead. However, when the primary server comes back, the device performs all new LDAP queries on the primary server again instead of sticking on the secondary. One of the reasons for this is for load-balancing, for example, half devices set LDAP-A as primary and half devices set LDAP-B as primary. Therefore, the load is evenly distributed between the two servers. Only if one server is down, all queries go to the server that is still up.

The following configuration specifies the primary LDAP server:

Switch(config)#fabric database server primary {ip <ipaddr> | ipv6 <ipv6addr> | host <hostname>} [port <portnum>] [vrf {<vrf-name>}] [enable-ssl]

Note that the primary LDAP server specified in this CLI must have the same format as it is specified in the DB table configuration, for example, if the DB table configuration uses host name for the server, this CLI must use the host name. Similarly, if the DB table configuration uses the port/vrf/enable-ssl options, this CLI must use those options.

Routable Loopback Auto-Configuration


Note

Routable loopback auto-configuration is not supported on Cisco Nexus 9000 Series switches.

In multi-tenant environments, every leaf switch needs to have a unique routable IP address per VRF for reachability. The user can use a knob to turn ON for an on demand for a per VRF basis. By turning on this knob, on every leaf switch where the VRF exists, there will be unique routable IP address allocated per leaf switch on that VRF (typically via a loopback interface). The reachability of this loopback address will be advertised via BGP-EVPN using route-type 5, thereby preventing any additional sources of floods or ARPs.

In many scenarios, there is a requirement of having a per leaf switch, per VRF, unique interface. Example scenarios that require this include DHCPv4 relay support when the DHCP server and client are in non-default VRFs behind a vPC pair of switches. In addition, this is a pre-requisite for a VXLAN OAM functionality. To automatically configure a loopback interface per leaf switch per VRF, the partition profile vrf-common-evpn-loopback can be used. During profile application, the leaf switches will automatically pick a free loopback interface ID, configure it with the respective VRF/partition, and associate the loopback interface with the IP address used for the VTEP associated loopback interface.

The following configuration results in the VRF profile being updated on the leaf switch resulting in the loopback routable IP address being auto-configured under that VRF as well as advertised via EVPN to all leaf switches. 


configure profile vrf-common-evpn-loopback 
interface loopback $system_auto_loopbackId
vrf member $vrfName
ip address $system_auto_backboneIpAddress/32 tag 12345 
vrf context $vrfName

vni $include_vrfSegmentId rd auto
ip route 0.0.0.0/0 $include_serviceNodeIpAddress
 address-family ipv4 unicast
 route-target both auto
 route-target both auto evpn
 address-family ipv6 unicast
 route-target both auto
 route-target both auto evpn
router bgp $asn
vrf $vrfName
 address-family ipv4 unicast
  advertise l2vpn evpn
  redistribute direct route-map FABRIC-RMAP-REDIST-
  SUBNET maximum-paths ibgp 2
 address-family ipv6 unicast
  advertise l2vpn evpn
  redistribute direct route-map FABRIC-RMAP-REDIST-
  SUBNET maximum-paths ibgp 2
interface nve $nveId
   member vni $include_vrfSegmentId associate-vrf
!

DCI Auto-Configuration

Following are the DCI auto configuration supported for Cisco Programmable Fabric.

LISP

Locator/ID Separation Protocol (LISP) is routing architecture that provides new semantics for IP addressing. Following is the day zero configuration done on a switch. 

feature lisp

vrf context <COMMON-RLOC-VRF>
	ip lisp itr-etr
	ip lisp itr map-resolver <MR-address>
	ip lisp etr map-server <MS-address> key <registration-key>
Following is the DCI auto configuration for a per partition configuration on a DC edge device. 

configure profile lispDCEdgeNodeProfile
vrf context $vrfName
	ip lisp itr-etr
	ip lisp locator-vrf $COMMON-RLOC-VRF-NAME
	lisp instance-id $include_dciId
register-route-notifications tag $asn

	lisp dynamic-eid $vrfName
	database-mapping 0.0.0.0/0 $include_RLOC1-address, $include_RLOC1-priority, $include_RLOC1-weight
 database-mapping 0.0.0.0/0 $include_RLOC2-address, $include_RLOC2-priority, $include_RLOC2-weight

Layer-3 DCI for VXLAN EVPN

Layer-3 DCI supports both single box and two box solutions with EVPN to VRF Lite handoff at border leaf.

Following is a DCI one box configuration. 

config profile vrf-common-universal-evpn-bl-dc-edge
vrf context $vrfName
  vni $include_l3SegmentId
  rd auto
  address-family ipv4 unicast
    route-target both $rsvdGlobalAsn:$dciId
    route-target both auto evpn
  address-family ipv6 unicast
    route-target both $rsvdGlobalAsn:$dciId
    route-target both auto evpn

router bgp $asn
vrf $vrfName
   address-family ipv4 unicast
                allow vpn default-originate
   address-family ipv6 unicast
                allow vpn default-originate

interface nve $nveId
      member vni $include_l3SegmentId associate-vrf
end

configure profile defaultCombinedEvpnBorderLeafDcEdgeProfile
  router bgp $asn
  include profile any
Following is a DCI two box configuration. 

config profile vrf-common-universal-evpn-bl
vrf context $vrfName
  vni $include_l3SegmentId
  rd auto
  address-family ipv4 unicast
    route-target both auto evpn
  address-family ipv6 unicast
    route-target both auto evpn

router bgp $asn
   vrf $vrfName 
       address-family ipv4 unicast
          maximum-paths 2
          maximum-paths ibgp 2
       allow vpn default-originate

     address-family ipv6 unicast  
       maximum-paths 2
       maximum-paths ibgp 2
       allow vpn default-originate

   neighbor $include_peerIpAddress1 remote-as $include_peerAsn
       address-family ipv4 unicast
         send-community both
    
     neighbor $include_peerIpv6Address1 remote-as $include_peerAsn
       address-family ipv6 unicast 
         send-community both

     neighbor $include_peerIpAddress2 remote-as $include_peerAsn
              address-family ipv4 unicast   
         send-community both
   
     neighbor $include_peerIpv6Address2 remote-as $include_peerAsn
       address-family ipv6 unicast     
         send-community both
end

Auto-Configuration Triggers

The Cisco Programmable Fabric supports the following triggers for VLAN, VRF, and SVI instantiation. The leaf switch will query LDAP database for configuration information specific to this VRF and/or VLAN/segment. Based on the response received the leaf switch will auto configure server facing SVI. If this is the first SVI for the VRF it belongs to, VRF configuration is also applied.

Following triggers are supported:

CLI (Static-host) Trigger

Beginning with Cisco NX-OS Release 7.0(3)I7(1), CLI dot1q based auto-configuration (auto-pull), CLI VNI with overwrite-vlan based auto-configuration (auto-pull), and CLI VNI based auto-configuration (auto-pull) is not supported. Instead, a new configuration command is introduced to support the persistence for the CLI triggered auto-configuration. Now you can use static host VNI configuration with an optional overwrite-vlan.

Use the following CLI configuration command to add the static host:


fabric database static-host
vni <vni-id>
    overwrite-vlan <ow-vlan-id>
    interface <interface-id>

Note

If the overwrite-vlan option is used to specify the overwrite-vlan, the overwrite-vlan command should precede the interface command, for the overwrite-vlan to take effect.

Use the following CLI commands to add the static host configuration for a particular VNI: 


fabric database static-host
  vni <vni-id>
    interface <interface-id>

For example, the following CLI command adds the static host for VNI 8000 at interface e1/2 and triggers a host add:


fabric database static-host
  vni 8000
    interface e1/2

You can specify multiple interface IDs under the same VNI, if they share the same VNI.

Use the following commands to remove the static host for a particular VNI:


fabric database static-host
  no vni <vni-id>

Note

The CLI command no fabric database static-host is not allowed. Delete the individual entries to delete the static hosts.

Note that the above command removes all hosts under multiple interfaces if more than one interface is added under the same VNI. Alternatively, one can also remove a specific interface under the VNI (if multiple interfaces are under the same VNI) using the following command: 


fabric database static-host
     vni <vni-id>
           no interface <interface-id>

You can check the current static host running configuration by executing the following command:


# show run fabric forwarding 

!Command: show running-config fabric forwarding
!Time: Wed Jul  5 20:42:17 2017

version 7.0(3)I7(1)
feature fabric forwarding

fabric forwarding anycast-gateway-mac 0000.abcd.dcba
fabric database mobility-domain md0
fabric database profile-map global
  ethernet-tag encapsulation dot1q default dynamic
  ethernet-tag encapsulation vni default dynamic
  vdp vni default dynamic
  vdp dot1q default dynamic
  vrf default dynamic
fabric database static-host
  vni 11700
    interface Ethernet1/1
fabric database timer cleanup 5


interface Vlan1800
  fabric forwarding mode anycast-gateway

Use the show fabric database static-host command to display the status of the configured static hosts as displayed in the following example: 


show fabric database static-host

                                            Retry
VLAN/VNI Interface       State      Delay (sec) Count   
-------- --------------- ---------- ----------- --------
   11700 Ethernet2/3     Retry      10          [#00001]
   11701 Ethernet2/3     Active

Note

After ISSU from an older release to Cisco NX-OS Release 7.0(3)I7.(1), older VDP and VMT host profiles are migrated seamlessly to the new software release. However, the older auto-pull exec commands have been deprecated in the new release and they are replaced by the new static-host command. The auto-pull profiles from the older release are recovered after an ISSU to Cisco NX-OS Release 7.0(3)I7(1). Note that you have to issue the clear fabric database host command to remove the older auto-pull entries after ISSU, replace them with equivalent static host configuration commands, and save the configuration.

Data Frame Snooping (DFS)/Dot1q Trigger

Auto-configuration is triggered when tagged tenant data traffic packets ingress the leaf switches. The packet VLAN is used to trigger the auto-configuration.


Note

This functionality is not supported on Cisco Nexus 9000 Series switches.

The DFS auto-configuration is enabled on the switch by default. The following global configuration CLI can be used to control the state: 


switch(config)# platform fabric database dot1q ?
  disable  Disable dot1q auto detection
  enable   Enable dot1q auto detection

LLDP Trigger


Note

This functionality is currently not supported on Cisco Nexus 9000 Series switches.

Link Layer Discovery Protocol (LLDP) is used by network devices for advertising their identity, capabilities and neighbors on IEEE 802 local area network, principally wired Ethernet. LLDP information is sent by devices from each of their interfaces at fixed time interval in the form of Ethernet frame. The frame ends with special TLV, named end of LLDPDU. LLDP only interacts with HMM in triggering auto-configuration. Based on the response from HMM, LLDP manipulates VLAN on the interface.

In a multi-tenancy network, tenants can be dynamically added or removed from the network. Auto-configuration provides the ability to dynamically allocate or deallocate resources for every tenant. Auto-configuration helps with generating and executing the necessary configuration CLI for enabling a tenant to be connected to the network. Switch can support only a subset of tenants at a time, depending on the necessity of network connectivity.

For example, creating the VNI for the tenant, allocating the resources (such as a bridge-domain), hardware resources (such as PVTCAM), and so on.

Auto-Configuration for Bare-Metal Severs

Auto-Configuration for Bare-Metal Severs provides a touchless orchestration scheme with the repository hosted on Cisco DCNM that has the MAC to segmentId mapping. LLDP running either on the server or on the CNA attached to the server, the MAC address of the server will be notified to the leaf or ToR, which in turn should trigger auto-configuration for the same via this notification.

The following command helps to enable or disable the LLDP auto-configuration feature in the config mode: 

[no] lldp fabric auto-config
The following show command displays all the ports for the triggered auto-configuration feature. 

Switch(config)# show lldp fabric auto-config

Interface        PORT-CHANNEL  Mac-Address      VNI     VLAN    Port-Mode   Status
--------------------------------------------------------------------------------------
Ethernet1/30     Po100         8478.ac1b.70c4   40000   200     native      ADD SUCCESS
Ethernet1/29     NA            8478.ac1b.70c1   -       100     native      ADD SUCCESS

Virtual Machine Tracker Auto-Configuration

Virtual Machine Tracker auto-config is a feature that automatically configures a tenant for provisioning where the leaf switches listen to VMWare vCenter events and perform required auto-configuration.

Virtual Machines (VMs) on a single physical server can belong to different Layer-2 domains and can require the Cisco Nexus switches to provision multiple VLANs on the ports that connect to the physical servers. To provide seamless movement of VMs from one physical server to another, the servers must be reachable through the same Layer-2 domain so that the VMs can retain their IP addresses and network connectivity even after moving. A static predefined configuration requires provisioning of all possible VLANs that can be used by VMs in a server management domain on each port of the switch. This process can result in more logical port-based VLANs than the switch can support. Alternatively, you can dynamically provision VLANs on a switch, Layer-2 Ethernet or Layer-2 port-channel interface (attached to a server) based on the tracking of the VMs that are connected to the port and the VLAN requirements of these VMs.

Virtual Machine Tracker (VM Tracker) enables you to do the following:

  • Identify the switch port that is used for each VM

  • Identify the VLAN requirements of each VM

  • Track the movement of VMs from one host (ESXi) to another

  • Track VM configuration changes such as additions, deletions, or modifications of VLANs, and configure VLANs on switch ports accordingly

  • Track whether the VMs are in the power on or power off state, and configure VLANs on local switch ports for a VM only if the VM is in the power on state


Note

For all ports on which VM Tracker is enabled, you must not perform any Layer 2 or Layer 3 configuration that is related to switchports and VLANs.

Virtual Machine Tracker and VMware vCenter

VM Tracker connects with VMware vCenter and collects information about the VMs that are connected to each host. This information includes the number of VMs that are connected to the host and the switch port through which the VM receives network traffic.

After you enable VM Tracker on the switch, it automatically extracts the following information about VMs from vCenter:

  • The host on which the VMs exist

  • The switch ports through which the VM traffic flows

  • The virtual network interface card (vNIC) that connects the VM to a virtual switch

  • The power state of the VM

  • The VLAN information of port groups or distributed virtual switch (DVS) port groups

  • The port groups or DVS port groups that are required for the VM

Virtual Machine Tracker Auto-Configuration Workflow

The following is a typical workflow to provision a tenant and provide an EVPN capable fabric connection to the tenant's workloads.


Note

All leaf switches in the network fabric are provisioned with the VM Tracker connections based on the set of hosts that are connected to the switch.

  • Identify the VLAN that a tenant’s virtual machine needs to utilize.

  • Provision the required port-groups on the VMware vSphere Distributed Virtual Switch (DVS) or virtual switch.

  • Determine the configuration profile that needs to be associated with the tenant for provisioning the VRF. For example, associating the vrf-common-universal-evpn profile.

  • Collect the tenant's network information and provision the network database with an application that uses the documented DCNM REST APIs. The network information includes:

    • Required configuration profile associated with the server-facing network that is being created.

    • All the network parameters required to provision the required configuration profile.

    • The mobility domain that is associated with the network. At this point, the fabric is ready for auto provisioning the workloads at any switch which is configured with the suitable mobility domain and the VM Tracker connection.


      Note

      Only one mobility domain can be supported.

  • When the workloads are powered on, VM Tracker detects the workload and pulls the appropriate network information from the LDAP database and provisions the network on the switch where the workload is connected.

  • When a workload is no longer necessary, powering the workload off ensures that the previously configured provisioning is removed.

  • VM Tracker on Cisco Nexus 5000 and 6000 Series switches do not update the interface trunk allowed VLANs.

VM Tracker in VPC

You must configure VM Tracker on both the vPC switches.

In a multi-tenancy full VPC setup, VPC secondary will not trigger auto-configuration. The auto-configuration profile is synced from primary to standby.

If primary vPC interface is down, there will be no actions from primary vPC side. But when VM Tracker retry timer expires on secondary, the skipped profile is inspected and VM Tracker triggers auto-configuration from standby if it detects that the primary vPC interface is down. When both vPC pair interfaces are shut, the profile should be removed.

In a multi-tenancy full vPC setup, VM Tracker sends triggers on both sides. The host trigger is sent along with the vdc_mac to avoid race conditions in HMM where the same host sends triggers on both the peers.

Guidelines and Limitations for Virtual Machine Tracker Auto-Configuration

Virtual Machine Tracker has the following guidelines and limitations:

  • The switch must be able to reach VMware vCenter over IPv4. See the Deploying Cisco Programmable Fabric section for details. IPv6 cannot be used for connectivity from switch to vCenter. This is a limitation for Cisco Nexus 9000 Series switches only.)

  • Both hosts that are directly connected and connected through UCS-FI are supported. Only LLDP protocol is supported if the hosts are connected through UCS-FI and FEX cannot be in between the switch and FI.

  • LLDP based VM Tracker is not supported on Cisco Nexus 7000 Series Switches and Cisco Nexus 7700 Switches (for Multi-tenancy full setup) for 7.3 release.

  • Current version of VM Tracker:

    • Supports ESXi 5.1 and 5.5. For Cisco Nexus 9000 Series switches, ESXi 6.0 is also supported.

    • Supports only VMware orchestration, other orchestrations like SCVM is not supported

  • CDP/LLDP enable/disable, interface shut/no shut:

    • When CDP/LLDP is disabled or interface shut, VM Tracker triggers profile unapply when sync-full-timer expires. When CDP/LLDP is enabled or interface no shut, VM Tracker triggers profile apply when find-new-host timer expires. Both sync-full-timer and find-new-host timer have default expiration interval of 3600 seconds.

  • Enabling the VM Tracker auto-config feature is disruptive. A best practice is to disconnect all VM Tracker connections before enabling VM Tracker auto-config.

  • When VM Tracker auto-config is enabled after VM Tracker is already connected to VMware vCenter and has configured the switch, the existing VM Tracker configuration is removed and then VM Tracker auto-config is triggered.

  • When VM Tracker auto-config is disabled, the auto-config triggered configuration is removed and the VM Tracker reverts back to the configured VLAN that was initially created by VM Tracker.

  • The VLAN is always created when the VM Tracker auto-config triggers the configuration. The auto-vlan enable command is ineffective and is not supported when VM Tracker auto-config is enabled.

  • Allowed-vlan takes effect when VM Tracker auto-config is enabled.

  • When migrating from VM Tracker to VM Tracker auto-config in a vPC setup, the following procedure is a best practice:

    • Disconnect all VM Tracker connections on the vPC primary and the vPC secondary switches.

    • Enable VM Tracker auto-config using the vmtracker fabric auto-config command on the vPC primary and the vPC secondary switches.

    • Connect the VM Tracker connection on the vPC primary and vPC secondary switches.

Enabling Virtual Machine Tracker Auto-Configuration

Procedure
Step 1

Enters global configuration mode:

switch# configure terminal

Step 2

Enable VM Tracker feature.

switch(config)# feature vmtracker

Step 3

Enables VM Tracker auto-config trigger. The no form of the command disables the auto-config trigger:

switch(config)# [no] vmtracker fabric auto-config

Step 4

Enters VM Tracker connection configuration mode for the connection name specified. The no form of the command disables the connection:

switch(config)# [no] vmtracker connection connection-name

Step 5

Configures remote IP parameters:

switch(config-vmt-conn)# [no] remote {ip address ip_address | port port_number | vrf}

Step 6

Verifies the username and password to connect to vCenter:

switch(config-vmt-conn)# username username password password

Step 7

Connects to vCenter. The no form of the command disconnects VM Tracker from vCenter:

switch(config-vmt-conn)# [no] connect

Step 8

Displays the VM Tracker auto-configuration information:

switch(config-vmt-conn)# show vmtracker fabric auto-config

Example

The following example shows how to enable the VM Tracker auto-configuration trigger, followed by a verification command that displays the VM Tracker auto-configuration information: 


configure terminal
	feature vmtracker
	vmtracker fabric auto-config
	vmtracker connection v229
		remote ip address  172.29.21.29 port 80 vrf management
		username user1 password abc1234
		connect


switch(config-vmt-conn)# show vmtracker fabric auto-config
Fabric Auto Configuration is enabled
Auto Configure Retry Time left: 107 seconds
Switch Device: SAL1833YM0V
------------------------------------------------------
Port 								Port-Channel 			Vlan 			Status
------------------------------------------------------
Ethernet1/3 	port-channel13 		50 				Pending
Ethernet1/3 	port-channel13 		56 				Pending

VDP Trigger

Dynamic provisioning simplifies the management of the VRF and VLAN/BD configurations. Dynamic provisioning can be triggered by:

  • VDP signaling from the server

Per-port configuration can be used to enable or disable dynamic provisioning. When the VM comes up in a leaf switch for the first time, the port receives an ARP or ND packet and for cases where VDP is enabled, the hypervisor may issue a VDP packet. The platform is expected to punt these packets to the HMM component, which will then install a tenant profile based on the information in this packet. In the simplest case, a (Port, VLAN) information from the incoming packet will be used to derive the tenant profile that needs to be installed. The following needs to be implemented when the Vinci functionality is enabled on the switch to support this feature:

  • The server-facing port needs to be configured with 'default VSI' and 'auto-config' must be enabled. The Ingress CBL state must be set to allow packets in all the VLANs. Note that for DHCP based snooping, 'feature dhcp' should be enabled.

  • Global ACLs must be set up to punt the packets of interest such as ARP and ND to the HMM component. All the remaining packets must be dropped to retain the CBL behavior. The global ACLs must be used only for the (Port, VLAN) that has not been configured on the system.

Once the tenant profile is installed, the VLAN/BD configuration along with the associated port membership will be configured.

The following is an example for VN-Segment configuration: 

system bridge-domain 2-3967

feature vni
	vni 5000, 10010-10011, 20000

	system fabric bridge-domain 3001-3967	
	bridge-domain 2,10-11

vrf context management

	encapsulation profile vni all_tenants
	dot1q 6-204 vni 7002-7200
	encapsulation profile vni cisco_all
	dot1q 10-1010 vni 10010-11010
	bridge-domain 2,10-11
	member vni 5000,10010-10011

interface Ethernet3/6
	no shutdown
	service instance 3 vni
	no shutdown
	encapsulation profile cisco_all default
The following is an example for VN-Segment dynamic auto-configuration: 

interface ethernet 2/2
 	service instance vni default
    		encapsulation dynamic vdp


interface ethernet 2/2
 	service instance vni default
    		encapsulation dynamic frame-snoop profile cisco

Auto-configuration Coexistence

Network auto-configuration occurs from multiple triggers. Supported co-existence defines the set of compatible triggers. Co-existence requires both the VLAN and MD and VNI based LDAP entries with identical profile name and contents.

Per-Port Auto-Configuration Trigger

For auto-configuration to occur, interfaces connecting to the host or server workloads must be configured to specify the desired auto-configuration trigger.


Note

This section is applicable only for Cisco Nexus 5600 Switches. Per-port auto-configuration is not applicable for Cisco Nexus 9000 Series switches.

Auto-configuration trigger is not enabled on the interface by default. The auto-configuration trigger must be explicitly configured on the interface and only one auto-configuration trigger can be configured per interface. Configuration is supported on CE Trunk (switchport mode trunk) interfaces only. VPC interfaces must be configured consistently. You must ensure the configuration consistency.

The following example shows how to enable the interface for dot1q, vdp, lldp or vmtracker. To change the trigger from one interface to another, ensure to disable the previous interface configuration. 

[no] encapsulation dynamic {dot1q | vdp | lldp | vmtracker}

VXLAN OAM

VXLAN operations, administration, and maintenance (OAM) is a protocol for installing, monitoring, and troubleshooting Ethernet networks to enhance management in VXLAN based overlay networks.

Similar to utilities such as ping and traceroute or pathtrace that allow quick determination of problems in IP networks, equivalent troubleshooting tools have also been introduced to diagnose problems in VXLAN networks. VXLAN OAM tools like ping, pathtrace and traceroute provide the reachability information to VMs and VTEPs in a VXLAN network. There is a notion of OAM channel is used to identify type of VXLAN payload present in these OAM packets.

There are two types of payloads supported:

  • Conventional ICMP packet to the destination to be tracked

  • Special NVO3 OAM header that carries useful information

ICMP channel helps to reach the traditional hosts or switches that do not support these new OAM packet formats. Nvo3 channels helps to reach the supported hosts or switches and carries important diagnostic information as well. VXLAN NVO3 OAM messages may be identified via the reserved OAM ethertype or by using a well-known reserved source MAC address in the OAM packets depending on the implementation on different platforms. This constitutes a signature for recognition of VXLAN OAM packets. The VXLAN OAM tools are categorized as shown in table below.

Table 1. VXLAN OAM Tools

Category

Tools

Fault Verification Loopback Message
Fault Isolation Path Trace Message
Performance Delay Measurement, Loss Measurement
Auxiliary

Address Binding Verification, IP End Station Locator, Error Notification, OAM Command Messages, and Diagnostic Payload Discovery for ECMP Coverage

Loopback (Ping) Message

Loopback message (Ping and Loopback messages are the same and used interchangeably in this document) is used for fault verification. The loopback message utility is used to detect various errors and path failures. Consider the topology in figure below where there are three core (spine) switches labeled Spine 1, Spine 2, and Spine 3 and five leaf switches connected in a Clos topology. The path of an example loopback message initiated from Leaf 1 for Leaf 5 is shown when it traverses via Spine 3. When the loopback message initiated by Leaf 1 reaches Spine 3, it forwards it as VXLAN encapsulated data packet based on the outer header. The packet is not sent to software on Spine 3. On Leaf 3, based on the appropriate loopback message signature, the packet will be sent to software VXLAN OAM module, which, in turn, will generate a loopback response that will be sent back to the originator Leaf 1.

Here loopback (ping) message can be destined to VM or to the (VTEP on) leaf switch. This ping message can use different OAM channels. If the ICMP channel is used, the loopback message can reach all the way to the VM if the VMs IP address is specified. If NVO3 channel is used, this loopback message is terminated on the leaf switch attached to the VM, as the VMs do not support NVO3 headers in general. In that case leaf switch replies back to this message indicating the reachability of the VM. Ping message supports the following reachability options.

Ping

Check the network reachability (Ping command):

  • From Leaf1 (VTEP1) to VM1 (attached host) (Only ICMP channel)

  • From Leaf1 (VTEP1) to Leaf2 (VTEP2) (ICMP or NVO3 channel)

  • From Leaf1 (VTEP1) to VM2 (host attached to another VTEP) (ICMP or NVO3 channel)

Figure 3. Loopback Message

Traceroute or Pathtrace Message

Traceroute or Pathtrace message is used for fault isolation. In a VXLAN network, it may be desirable to find the list of switches that are traversed by a frame to reach the destination. When the loopback test from a source switch to a destination switch fails, the next step is to find out the offending switch in the path. The operation of the path trace message begins with the source switch transmitting a VXLAN OAM frame with a TTL value of 1. The next hop switch receives this frame, decrements the TTL, and on finding that TTL is 0, it transmits a TTL expiry message to the sender switch. The sender switch records this message as an indication of success from the first hop switch. Then the source switch increases the TTL value by one in the next path trace message to find the second hop. At each new transmission, the sequence number in the message will be incremented. Each intermediate switch along the path decrements the TTL value by 1 as is the case with regular VXLAN forwarding.

This process continues until a response is received from the destination switch, or path trace process timeout occurs, or the hop count reaches a maximum configured value. The payload in the VXLAN OAM frames is referred to as the flow entropy. The flow entropy can be populated so as to choose a particular path among multiple ECMP paths between a source and destination switch The TTL expiry message may also be generated by intermediate switches for actual data frames. The same payload of the original path trace request is preserved for the payload of the response.

Traceroute and pathtrace are similar, except that traceroute uses ICMP channel, whereas pathtrace use NVO3 channel. Pathtrace uses NVO3 channel there by carries additional diagnostic information like interface load, statistics of the hops taken by these messages. If an intermediate device do not support NVO3 channel, this pathtrace behaves as simple traceroute and provides only the hop information.

Traceroute

Trace the path the packet traverses in the VXLAN overlay (Traceroute command):

  • Traceroute uses icmp packets (channel-1), encapsulated in the VXLAN encapsulation to reach the host

Pathtrace

Trace the path the packet traverses in the VXLAN overlay using the nvo3 channel (Pathtrace command):

  • Pathtrace uses special control packets like NVO3 or TISSA (channel-2) to provide additional information regarding the path (for example, ingress interface and egress interface). But these packets will terminate at VTEP and does not reach the host, so only the VTEP will respond.

Figure 4. Traceroute Message

Configuring VXLAN OAM

As a prerequisite, ensure that the VXLAN configuration is complete. Create a loopback interface, assign an IP address with a /32 subnet mask, and associate the loopback interface with a customer VRF before verifying VXLAN OAM on the IP address. A sample configuration on two leaf switches is given below:

Leaf switch Leaf1

(config) #


interface loopback50
  vrf member VRF1
  ip address 209.165.201.2/32 tag 12345
Leaf switch Leaf2

(config) #


interface loopback50
  vrf member VRF1
  ip address 209.165.201.4/32 tag 12345

Procedure

Step 1

Enters the NGOAM feature:

switch(config)# feature ngoam

Step 2

Install NGOAM access control list (ACL).

switch(config)# ngoam install acl

Example

The following example shows how to create ngoam profile from the configuration mode. You can create a profile to define various options like the dot1q, interface, source port, ip address, oam-channel, payload information and so on. After you define them, you can enter these pre-defined profiles names to execute certain operations instead of typing the respective commands.


Note
  • Channel 1 is enabled by default. So, no ngoam profile configuration is required for this channel.
  • Channel 2 (NVO3) is not enabled by default, so oam-channel 2 configuration is required under the ngoam profile command to use this channel.

(config) #


ngoam profile 1
  oam-channel 2
  sport 12345, 54321
  payload pad 0x2
  flow forward
				ip source 209.165.201.1
    ip destination 209.165.201.11

ngoam profile 2
  payload pad 0x3
  flow reverse
				ip source 209.165.201.2
    ip destination 209.165.201.12
Figure 5. VXLAN Network

VXLAN OAM provides the visibility of the host at the switch level, which allows a leaf to ping the host using the ping nve command.

The following example shows how to ping from Leaf 1 to the VM2, via Spine 1. 


Switch# ping nve ip 209.165.201.5 vrf vni-31000 verbose

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request(parameter problem), 
'c' - Corrupted Data/Test

Sender handle: 34 
! sport 40673 size 56,Reply from 209.165.201.5,time = 18 ms
! sport 40673 size 56,Reply from 209.165.201.5,time = 1 ms 
! sport 40673 size 56,Reply from 209.165.201.5,time = 1 ms 
! sport 40673 size 56,Reply from 209.165.201.5,time = 1 ms 
! sport 40673 size 56,Reply from 209.165.201.5,time = 1 ms 

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/18 ms
Total time elapsed 49 ms

The following example shows how to traceroute from Leaf 1 to the VM2 via Spine 1. 


Switch# traceroute nve ip 209.165.201.5 vrf vni-31000 verbose

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request (parameter problem),
'c' - Corrupted Data/Test

Traceroute request to peer ip 209.165.201.4 source ip 209.165.201.2
Sender handle: 36
  1 !Reply from 209.165.201.3,time = 1 ms		
  2 !Reply from 209.165.201.4,time = 2 ms		
  3 !Reply from 209.165.201.5,time = 1 ms

The following example shows how to pathtrace from Leaf 2 to Leaf 1. 


Switch# pathtrace nve ip 209.165.201.4 vni 31000 verbose

Path trace Request to peer ip 209.165.201.4 source ip 209.165.201.2

Sender handle: 42
TTL			Code		Reply																	IngressI/f				EgressI/f				State
======================================================================
1     !Reply from 209.165.201.3,  Eth5/5/1      Eth5/5/2     UP/UP
2     !Reply from 209.165.201.4,  Eth1/3	    			Unknown     	UP/DOWN