Auto-Configuration of Layer-3 DCI

Border Leaf and DC Edge Router Auto-Configuration

This feature works in partnership with Cisco Prime DCNM (starting with version 7.1.1 release) to enable auto-configuration of Layer-3 fabric External connectivity on a per tenant basis. Enhancements and modifications have been made to the following list of products and components in order to simplify and automate the extension of the tenant networks:

  • Nexus 5600/6000/7000 as Border Leaf (Cisco NX-OS)

    • New POAP template for border leaf

    • Auto-configuration for VRF extension

  • Cisco Prime DCNM 7.1

    • GUI enhancement for extending connectivity

    • Modification of LDAP Schema

  • Cisco UCS Director 5.2

    • Workflow extension

  • OpenStack (Juno)

    • Addition of border leaf extension capability

This functionality can be used towards the Data Center Edge router for WAN connectivity beyond the fabric or to introduce Layer-3 Data Center Interconnect (DCI) capabilities.

In this release we introduce automation of border leaf by using auto-configuration for the most common topologies that Customers use to connect to the DC edge router and connectivity beyond the fabric. In subsequent releases we will enhance this functionality with deep integration of MPLS Layer-3 VPN. This guide will use a sample topology to illustrate and guide on how this functionality works.

Summary of Steps

Figure 1. Provisioning Steps

Detailed Steps

Resource Planning

The two most common topologies for the border leaf to DC Edge connectivity are described as follows. Each pair of border leaf can be repeated multiple times in order to scale-out and achieve the desired scale and redundancy for you specific tenant extension needs:
  • Direct Connect

    • A border leaf is only connected to the neighboring DC edge router. This topology can also be achieved with a pair of border leaf to achieve a certain level of redundancy.

  • Full mesh

    • A pair of border leaf is connected to a pair of DC edge routers. This is the recommended topology as it offers the highest level of redundancy for failure of every node.

Assign the desired IP addresses to the sub interfaces that will be automatically created. The same address pairs can be reused on a per-VRF base as long as these are not redistributed into a routing protocol and stay unique within the VRF.

Note

The maximum number of DC edge neighbors to a border leaf can have is two, similarly the maximum number of border leaf neighbors a DC edge router can have is also two. In future releases, will increase the amount of pairing neighbors.

Direct Connect Topology

A border leaf is connected to a DC edge device. This pairing can be repeated any number of times as per requirements. This does not offer redundancy on DC edge node failure. Multiple such directly connected pairs can be present in the topology.

Figure 2. Direct Connect Topology

Full Mesh Topology

This topology provides full redundancy and it is the most recommended way to set up the border leaf to DC edge connectivity. The below figure is referred to a Topology with a redundancy factor of 2 but can be replicated any number of times as per requirements to satisfy the scale and redundancy.

Figure 3. Full Mesh Topology

Note

The above topology examples show port-channel as the connected interface, as this is the best practice for redundancy. Ethernet interfaces are also supported, for this release the user will have to configure those on the box and not through POAP.

Provision Devices

POAP can be used to bring up the devices and the respective port-channels for the external connectivity between the devices. The port-channels between the border leaf and DC edge, 'can be a single member port-channel' and will be brought up with 'no switchport' mode so that a Layer-3 sub interfaces can be formed. The interfaces, port-channels and members, should be brought up with 'no shutdown'.

Note

In the first release, we only support Layer-3 port-channel for neighbor connectivity and the POAP Template is only supported for the Cisco Nexus 5600 Platform Switches and Cisco Nexus 6000 Series Switches.

The below figure shows the POAP configuration for the border leaf to DC edge interface connection, which is port-channel only; single member port-channel is supported.

Figure 4. POAP Interface Settings for Border Leaf
Figure 5. Interface Settings for Border Leaf (detail)

Manually Configure Interfaces between Border Leaf and DC Edge router

The links between border leaf and DC edge will use sub interfaces. The belonging parent interfaces have to be "no switchport" mode in order to provide Layer-3 sub interface functionality. Logical port-channel interfaces provide link redundancy and module redundancy if members from different modules. They also provide more bandwidth by bundling several interfaces together.

Following are the interface configuration examples:

Example using logical port-channel interfaces: 

interface Ethernet1/3
  no switchport
  channel-group 11 mode active

interface Ethernet1/4
  no switchport
  channel-group 11 mode active

interface port-channel11
  no switchport
Example using physical ethernet interface: 

interface Ethernet1/4
  no switchport

Mapping offline topology into DCNM


Note

Disable border leaf and DC edge router auto-configuration globally till topology mapping is done. This process will avoid partial configuration during setup phase. The default state is disabled.

  1. In DCNM, choose Admin and select Border Leaf Settings option from the Fabric area.

  2. Uncheck Enable Border Leaf/Edge Autoconfiguration check box to globally disable. By default it is unchecked.

  3. Click Apply.

Figure 6. Border Leaf Settings - Disable auto-configuration population

For more information on parameter description, seeGlobal Enable section.

Border Leaf to DC Edge router pairing

The following steps shows an example of using Cisco Prime DCNM 7.1, with Cisco Nexus 7000/7700 Series Switches, running Cisco NX-OS 6.2, and Cisco Nexus 5600/6000 Border Leaf Series Switches running Cisco NX-OS 7.1. With this combination, end-to-end auto-configuration for border leaf is supported. Cisco Nexus 7000/7700 configuration is generated within Cisco Prime DCNM and can be transported via copy/pasted into the Cisco Nexus 7000 command prompt. Future releases of Cisco Nexus 7000/7700, the admin can choose to deliver the Cisco Nexus 7000 DC edge configuration automatically. The option to send configuration to the DC edge device or not is controlled at Cisco Prime DCNM GUI when importing the DC edge device into Cisco Prime DCNM for device pairing.

Figure 7. Example Topology

Note

The neighboring DC edge router does not need to be imported or managed by Cisco Prime DCNM. For the following examples and figures the value is assumed to be 500 VRF as maximum.

Pair Border Leaf # 1 and DC Edge router # 1

To add DC edge device:

  1. On Cisco Prime DCNM, click Config tab and select Border Leaf Device Pairing option.

    Figure 8. Menu: Border Leaf Device Pairing

  2. Click Add.

    Figure 9. Add Edge Router

  3. In Add Edge Router dialog box, enter the following details:

    • Device Name: (Optional) Enter the device name as DCI-B-1. It is not mandatory to enter the device name to import into Cisco Prime DCNM.

    • IP Address: Enter the management IP address.

    • Maximum Number of Partitions: Enter the maximum number of partitions on this device as 500.

    • Notify Edge Router when relevant partitions are changed: Check this check box only when you have Cisco Nexus 7000 Series Switches that supports this feature. We recommend you to uncheck this check box.

  4. Click OK.

  5. Select the DCI-B-1 device.

  6. Click Add and select Border Leaf.

  7. Select BL-B-1 from the drop-down list of border leafs imported into Cisco Prime DCNM.

    If BL-B-1 does not show up in the drop-down list, then it has not been imported into Cisco Prime DCNM properly with the role of border leaf.

    The Connect New Border Leaf (BL) to Edge Router shows the pairing values and meaning:

    Figure 10. Add Border Leaf
    Table 1. Figure Legend

    1 - Border Leaf Configuration

    3 - Edge Router Configuration

    2 - Profile Parameters Border Leaf

    4 - Profile Parameters Edge Router

    1 - Border Leaf Configuration area:

    • Port-channel or Interface Name: The interface name on border leaf. This can be a different name than what is on the DC edge box.

    • Max Number of Partitions: The maximum number of partitions on the border leaf.

    • Default Profile Name: The border leaf profile for extension from the drop-down list.

    • Check the Notify Border Leaf when relevant partitions are changed check box so that Cisco Prime DCNM will trigger auto-configuration on border leaf. Uncheck, if auto-configuration is not required and manual copy and paste is preferred.

    2 - Profile Parameters area:

    • ipAddress and ipv6Address: The interface address and the mask to be used on sub interfaces created on border leaf. This address will be repeated on all sub interfaces for all tenants. Add IPv6 address if IPv6 address family is needed.

    • ifNamePc: Leave this field empty, it will be filled by Cisco Prime DCNM.

    • ifNameEth: Leave this field empty, it will be filled by Cisco Prime DCNM.

    • asn: ASN for border leaf will be obtained by FABRIC ASN setting in POAP.

    3 - Edge Router Configuration area:

    • Port-channel or Interface Name: Interface name on DC edge router. Leave this field empty, it will be filled by Cisco Prime DCNM.

    • VLAN Range: DOT1q range for the interface. A contiguous range is supported.

    • Default Profile Name: Choose this configuration profile n7KDcEdgeManualProfile from drop-down list.

    4 - Profile Parameters area:

    • ifNamePc: Leave this field empty, it will be filled by Cisco Prime DCNM.

    • ipAddress and ipv6Address: The interface address and the mask to be used on sub interfaces created on edge router. This address will be repeated on all sub interfaces for all tenants. Add IPv6 address if IPv6 address family is needed. DC edge end of IP address. Border leaf will automatically use this as peering address for BGP session.

    • asn: Set the ASN number for DC edge router here. It will be used to derive the peer ASN on border leaf also.

    • peerIpAddress1: BGP neighbor address, same as interface address on border leaf side.

    • peerAsn: The Border leaf ASN.

    • peerIpAddress2: If building full mesh, then put the interface address of DC-B-BL2. If building a direct topology, leave it blank.

  8. If direct pairing is required, then pairing of these two nodes is complete. To Pair more direct paring nodes, nodes if any in your topology, repeat the above steps. To accomplish full mesh topology, go to next step.

  9. Click Add and select Border Leaf to pair DCI-B-1 with DC-BL-2.

    Figure 11. Add 2nd Border Leaf
    Table 2. Figure Legend

    1 - Border Leaf Configuration

    2 - Profile Parameters Edge Router

  10. Repeat the above steps to add the peering of DCI-B-2 with DC-B-BL1 and DCI-B-2 with DC-B-BL2. The full mesh topology is now complete for one pod. If there are more such pods, repeat the above steps to pair them and iterate till topology is complete.

Enable Border Leaf Auto-Configuration

Testing the topology set up by extending from DCNM

Once topology has been set up and feature is globally enabled, auto-configuration is ready. Partition can be extended from Cisco Prime DCNM, UCSD5.2+ and OpenStack 2.0 for Cisco Nexus Fabric. Given below is an example using Cisco Prime DCNM.

Create a test partition and enable it for extension using the following figure at Cisco Prime DCNM. If there are no errors on Cisco Prime DCNM, ssh to the devices where partition is extended and verify that the configuration is fine.

Extend partition at DCNM

Figure 12. Extended DCNM partition

  • Select a profile, value that should be same as what it is in interior leaf nodes. Border leaf uses override command to use a border leaf specific profile.

  • Set DCI ID. This has to be same in every fabric for the same tenant. User has responsibility to ensure that its unique and matches on other fabric. Check the Extend the Partition across the Fabric check box and click OK.

Check which nodes the tenant is extended on

Use the following button to get the screen, which nodes the partition just extended 'my-test-org:TEST1' is deployed.

Figure 13. Border Leaf Extend Partitions

Manually configurations for DC Edge devices

Border leaf nodes are fully auto-configured. Verify the configuration on the border leaf by connecting to device and displaying the VRF configuration. DCNM generates the configuration at DCNM for all the extended partitions and all the nodes. For DCI-B-1 and DCI-B-2 by clicking the profile name next to the device pairing. For example, for the DCI-B-1 and DC-B-BL1 pairing, click the n7KDcEdgeManualProfile for DCI-B-1 configuration and then save the generated configuration to a file or directly copy from the pop-up window and paste to configuration shell of the device. Repeat this for every Cisco Nexus 7000 Series Switches.

Figure 14. Configuration Detail Extended Partitions

Example configuration generated by DCNM for N7K DC edge router

For DCI-B1 router, for the DC-B-BL1 pairing for partition my-test-org: TEST1 

interface port-channel 11.5 
    encapsulation dot1Q 5
    vrf member my-test-org:TEST1
    ip address 10.4.4.2/24 
    ipv6 address 10:4:4:4::2/64

!  interface ethernet $ifNameEth.5
    encapsulation dot1Q 5
    vrf member my-test-org:TEST1
    ip address 10.4.4.2/24
    ipv6 address 10:4:4:4::2/64
    no shutdown 

vrf context my-test-org:TEST1
    address-family ipv4 unicast
      route-target both 65500:555
    address-family ipv6 unicast
      route-target both 65500:555
    rd 65500:555

router bgp 100

    vrf my-test-org:TEST1
address-family ipv4 unicast
      maximum-paths 2
      maximum-paths ibgp 2
      additional-paths send 
      additional-paths receive
      additional-paths selection route-map ALL-PATHS
address-family ipv6 unicast
      maximum-paths 2
      maximum-paths ibgp 2
      additional-paths send
      additional-paths receive
      additional-paths selection route-map ALL-PATHS
      !keeping both neighbors as it is not predictable which neighbor will be populated
      !per each link
      neighbor 10.4.4.1 remote-as 65002
        address-family ipv4 unicast
          default-originate
          send-community both
      neighbor 10.5.5.1 remote-as 65002
        address-family ipv4 unicast
          default-originate
          send-community both 
      neighbor 10:4:4:4::1 remote-as 65002
        address-family ipv6 unicast
          default-originate
          send-community both
      neighbor 10:5:5:5::1 remote-as 65002
        address-family ipv6 unicast
          default-originate
          send-community both

Border Leaf required Configurations specific to this feature

If the device is loaded by POAP, these commands are already present.
  • VRF override profile command: This command over rides the partition profile with the one configured below. 
    

fabric database override-vrf-profile vrf-common-universal-bl
  • LDAP configuration for bl-dci table, rest of LDAP is same as interior leaf nodes. Configuration below shows optional redundant LDAP. 
    

fabric database type bl-dci
    server protocol ldap host ldap-server1.cisco.com vrf management enable-ssl
    db-security user cn=reader,dc=cisco,dc=com password1
    
  server protocol ldap host ldap-server2.cisco.com vrf management enable-ssl
    db-table ou=bl-dcis,dc=cisco,dc=com
   db-security user cn=reader,dc=cisco,dc=com password1

Limitations

The following restrictions are specific to Cisco Prime DCNM 7.1(1) and Cisco NX-OS 7.1.

  • Maximum number of neighbors is two

  • Profiles limited to Layer-3 unicast scenarios in this release

Global Enable

Set the following values in the Global Parameters.

BGP Route-Target AS #:

If it is intended to interconnect multiple fabrics together, this value has to be same on all DCNMs in all fabrics. It is a 2-byte unsigned integer value (1 – 65535) used to construct the RT used between DC edge devices.

The route-target is constructed by appending the DCI to this value in the following way:

RT = BGP Route-Target AS # : DCI ID

The DCI ID (4-byte unsigned integer) is a user entered value when creating/modifying a partition. It is your responsibility to ensure that same DCI ID is same in different fabrics when extending a partition across fabrics.

Note

This value is only used on DC edge device. If you are not using auto-configuration feature including the copy and paste on DC edge device then this value is not relevant but must be provided for this feature to work.

For example, if BGP Route-Target AS #: 65500 and DCI ID is 555 then the following RT is generated for DC edge router: 

vrf context my-test-org:TEST1
    address-family ipv4 unicast
      route-target both 65500:555
    address-family ipv6 unicast
      route-target both 65500:555
    rd 65500:555

Redundancy factor

The number of border leafs one tenant is provisioned on. If a border leaf is chosen then,

  1. If it is in a vPC pair with another border leaf then its vPC neighbor is also chosen, even if redundancy factor is 1. In presence of vPC pairs, number of border leafs a partition is deployed is always an even number. Without vPC pairs the number of border leaf a tenant is deployed is exactly equal to redundancy factor as long as capacity permits.

  2. Once a border leaf is chosen then all its DC edge neighbors are also chosen. Maximum of two neighbors per node are allowed.


Note

For vPC pair in vinci leaf, We recommend to configure vpc with dual-active exclude interface-vlan-bridge-domain allow-vlans and make sure SVI/BDI do not go down when vPC peer link goes down. This will prevent duplicate packet forwarding in TF mode.

Load Sharing Algorithm

Currently the only algorithm supported is round robin. In future, users may be able to provide their own algorithm. As the name suggests, this algorithm tries to round robin partition deployment over the available border leaf resources, maintaining the redundancy factor.

Figure 15. Border Leaf Settings

  • Click Admin and select Border Leaf Settings from Fabric.

  • After setting the BGP route-target AS and redundancy factor, check the Enable Border Leaf/Edge Router Autoconfiguration check box to enable the feature globally.

  • Click Apply. The system is ready for auto-configuration.