Configuring BGP-EVPN Intersite Connectivity

Configuring Infra: Orchestrator General Settings

This section describes how to configure general Infra settings for all the sites.


Note

Some of the following settings apply to all sites, while others are required for specific type of sites (for example, Cloud Network Controller sites). Ensure that you complete all the required configurations in infra general settings before proceeding to the site-local settings specific to each site.

Procedure

Step 1

Log in to the Cisco Nexus Dashboard Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Site Connectivity.

Step 3

In the main pane, click Configure.

Step 4

In the left sidebar, select General Settings.

Step 5

Provide Control Plane Configuration.

  1. Select the Control Plane Configuration tab.

  2. Choose BGP Peering Type.

    • full-mesh—All border gateway switches in each site will establish peer connectivity with remote sites' border gateway switches.

      In full-mesh configuration, Nexus Dashboard Orchestrator uses the spine switches for ACI managed fabrics and border gateways for NDFC managed fabrics.

    • route-reflector—The route-reflector option allows you to specify one or more control-plane nodes to which each site establishes MP-BGP EVPN sessions. The use of route-reflector nodes avoids creating MP-BGP EVPN full mesh adjacencies between all the sites managed by NDO.

      For ACI fabrics, the route-reflector option is effective only for fabrics that are part of the same BGP ASN.

  3. In the Keepalive Interval (Seconds) field, enter the keep alive interval seconds.

    We recommend keeping the default value.

  4. In the Hold Interval (Seconds) field, enter the hold interval seconds.

    We recommend keeping the default value.

  5. In the Stale Interval (Seconds) field, enter stale interval seconds.

    We recommend keeping the default value.

  6. Choose whether you want to turn on the Graceful Helper option.

  7. Provide the Maximum AS Limit.

    We recommend keeping the default value.

  8. Provide the BGP TTL Between Peers.

    We recommend keeping the default value.

  9. Provide the OSPF Area ID.

    If you do not have any Cloud Network Controller sites, this field will not be present in the UI.

    This is OSPF area ID used by cloud sites for on-premises IPN peering.

  10. (Optional) Enable IANA Assigned Port for CloudSec encryption.

    By default, CloudSec uses a proprietary UDP port. This option allows you to configure CloudSec to use the official IANA-reserved port 8017 for CloudSec encryption between sites.

    Note 

    The IANA-reserved port is supported for Cisco APIC sites running release 5.2(4) or later.

    To change this setting, CloudSec must be disabled on all sites. If you want to enable IANA reserved port, but already have CloudSec encryption enabled for one or more of your sites, disable CloudSec for all sites, enable IANA Reserve UDP Port option, then re-enable CloudSec for the required sites.

    For detailed information and steps for configuring CloudSec, see the "CloudSec Encryption" chapter of the Nexus Dashboard Orchestrator Configuration Guide for ACI Fabrics.

Step 6

Provide the IPN Devices information.

If you do not plan to configure inter-site connectivity between on-premises and cloud sites, you can skip this step.

When you configure inter-site underlay connectivity between on-premises and cloud sites as described in later sections, you will need to select an on-premises IPN device which will establish connectivity to the cloud CSRs. These IPN devices must first be defined here before they are available in the on-premises site configuration screen.

  1. Select the On Premises IPSec Devices tab.

  2. Click +Add On-Premises IPSec Device.

  3. Choose whether the device is Unmanaged or Managed and provide the device information.

    This defines whether or not the device is directly managed by NDFC:

    • For Unmanaged IPN devices, simply provide the Name and the IP Address of the device.

      The IP address you provide will be used as the tunnel peer address from the cloud CSRs, not the IPN device's management IP address.

    • For Managed IPN devices, choose the NDFC Site that contains the device and then the Device from that site.

      Then choose the Interface on the device that is facing the Internet and provide the Next Hop IP address, which is the IP address of the gateway that is connecting to the Internet.

  4. Click the check mark icon to save the device information.

  5. Repeat this step for any additional IPN devices you want to add.
Step 7

Provide the External Devices information.

If you do not have any Cloud Network Controller sites, this tab will not be present in the UI.

If you do not have any Cloud Network Controller sites in your Multi-Site domain or you do not plan to configure connectivity between cloud sites and branch routers or other external devices, you can skip this step.

The following steps describe how to provide information about any branch routers or external devices to which you want to configure connectivity from your cloud sites.

  1. Select the External Devices tab.

    This tab will only be available if you have at least one cloud site in your Multi-Site domain.

  2. Click Add External Device.

    The Add External Device dialogue will open.

  3. Provide the Name, IP Address, and BGP Autonomous System Number for the device.

    The IP address you provide will be used as the tunnel peer address from the Cloud Network Controller's CSRs, not the device's management IP address. The connectivity will be established over public Internet using IPSec.

  4. Click the check mark icon to save the device information.

  5. Repeat this step for any additional IPN devices you want to add.

After you have added all the external devices, ensure to complete the next step to provide the IPSec tunnel subnet pools from with the internal IP addresses will be allocated for these tunnels.

Step 8

Provide the IPSec Tunnel Subnet Pools information.

If you do not have any Cloud Network Controller sites, this tab will not be present in the UI.

There are two types of subnet pools that you can provide here:

  • External Subnet Pool—used for connectivity between cloud site CSRs and other sites (cloud or on-premises).

    These are large global subnet pools that are managed by Nexus Dashboard Orchestrator. The Orchestrator, creates smaller subnets from these pools and allocates them to sites to be used for inter-site IPsec tunnels and external connectivity IPsec tunnels.

    You must provide at least one external subnet pool if you want to enable external connectivity from one or more of your cloud sites.

  • Site-Specific Subnet Pool—used for connectivity between cloud site CSRs and external devices.

    These subnets can be defined when the external connectivity IPsec tunnels must be in a specific range. For example, where a specific subnet is already being used to allocate IP addresses to the external router and you want to continue using those subnets for IPsec tunnels for NDO and cloud sites. These subnets are not managed by the Orchestrator and each subnet is assigned to a site in its entirety to be used locally for external connectivity IPsec tunnels.

    If you do not provide any named subnet pools but still configure connectivity between cloud site's CSRs and external devices, the external subnet pool will be used for IP allocation. .

Note 

The minimum mask length for both subnet pools is /24.

To add one or more External Subnet Pools:

  1. Select the IPSec Tunnel Subnet Pools tab.

  2. In the External Subnet Pool area, click +Add IP Address to add one or more external subnet pools.

    This subnet will be used to address the IPsec tunnel interfaces and loopbacks of the Cloud Routers used for on-premises connectivity, which you previously configured in the Cloud Network Controller for inter-site connectivity in earlier Nexus Dashboard Orchestrator releases.

    The subnets must not overlap with other on-premises TEP pools, should not begin with 0.x.x.x or 0.0.x.x, and should have a network mask between /16 and /24, for example 30.29.0.0/16.

  3. Click the check mark icon to save the subnet information.

  4. Repeat these substeps for any additional subnet pools you want to add.

To add one or more Site-Specific Subnet Pools:

  1. Select the IPSec Tunnel Subnet Pools tab.

  2. In the Site-Specific Subnet Pools area, click +Add IP Address to add one or more external subnet pools.

    The Add Named Subnet Pool dialogue will open.

  3. Provide the subnet Name.

    You will be able to use the subnet pool's name to choose the pool from which to allocate the IP addresses later on.

  4. Click +Add IP Address to add one or more subnet pools.

    The subnets must have a network mask between /16 and /24and not begin with 0.x.x.x or 0.0.x.x, for example 30.29.0.0/16.

  5. Click the check mark icon to save the subnet information.

    Repeat the steps if you want to add multiple subnets to the same named subnet pool.

  6. Click Save to save the named subnet pool.

  7. Repeat these substeps for any additional named subnet pools you want to add.

What to do next

After you have configured general infra settings, you must still provide additional information for site-specific configurations based on the type of sites (ACI, Cloud Network Controller, or NDFC) you are managing. Follow the instructions described in the following sections to provide site-specific infra configurations.

Refreshing Cloud Site Connectivity Information

Any infrastructure changes, such as CSR and Region addition or removal, require a Multi-Site fabric connectivity site refresh. This section describes how to pull up-to-date connectivity information directly from each site's controller.

Procedure

Step 1

Log in to the Cisco Nexus Dashboard Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Site Connectivity.

Step 3

In the top right of the main pane, click Configure.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, click the Refresh button to discover any new or changed CSRs and regions.

Step 6

Finally, click Yes to confirm and load the connectivity information.

This will discover any new or removed CSRs and regions.
Step 7

Click Deploy to propagate the cloud site changes to other sites that have connectivity to it.

After you refresh a cloud site's connectivity and CSRs or regions are added or removed, you need to deploy infra configuration so other sites that have underlay connectivity to that cloud site get updated configuration.

Configuring Infra: Google Cloud Site Settings

This section describes how to configure site-specific Infra settings for Cloud Network Controller sites.

Procedure

Step 1

Log in to the Cisco Nexus Dashboard Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Site Connectivity.

Step 3

In the top right of the main pane, click Configure.

Step 4

In the left pane, under Sites, select a specific cloud site.

Step 5

Provide the general Inter-Site Connectivity information.

  1. In the right <Site> Settings pane, select the Inter-Site Connectivity tab.

  2. Enable the Multi-Site knob.

    This defines whether the overlay connectivity is established between this site and other sites.

    Note that the overlay configuration will not be pushed to sites which do not have the underlay intersite connectivity established as described in the next step.

Step 6

Provide site-specific Inter-Site Connectivity information.

  1. If using the BGP-EVPN protocol for site connectivity, enable Contract Based Routing option.

  2. In the right properties sidebar for the cloud site, click Add Site.

    The Add Site window opens.

  3. Under Connected to Site, click Select a Site and select the site (for example, Site2) to which you want to establish connectivity from the site you are configuring (for example, Site1) .

    Once you select the remote site, the Add Site window will update to reflect both directions of connectivity: Site1 > Site2 and Site2 > Site1.

  4. In the Site1 > Site2 area, from the Connection Type dropdown, choose the type of connection between the sites.

    The following options are available:

    • Public Internet—connectivity between the two sites is established via the Internet.

      This type is supported between any two cloud sites or between a cloud site and an on-premises site.

    • Private Connection—connectivity is established using a private connection between the two sites.

      This type is supported between a cloud site and an on-premises site.

    • Cloud Backbone—connectivity is established using cloud backbone.

      This type is supported between two cloud sites of the same type, such as Azure-to-Azure, AWS-to-AWS, or GCP-to-GCP.

    If you have multiple types of sites (on-premises, AWS, Azure, and GCP), different pairs of site can use different connection type.

  5. Choose the Protocol that you want to use for connectivity between these two sites.

    For this use case, we will use BGP-EVPN. You can optionally enable IPSec and choose which version of the Internet Key Exchange (IKE) protocol to use: IKEv1 (Version 1) or IKEv2 (Version 1) depending on your configuration.

    • For Public Internet connectivity, IPsec is always enabled.

    • For Cloud Backbone connectivity, IPsec is always disabled.

    • For Private Connection, you can choose to enable or disable IPsec.

    If using BGP-IPv4 connectivity instead, you must provide an external VRF which will be used for route leaking configuration from the cloud site you are configuring.

    After Site1 > Site2 connectivity information is provided, the Site2 > Site1 area will reflect the connectivity information in the opposite direction.

  6. Click Save to save the inter-site connectivity configuration.

    When you save connectivity information from Site1 to Site2, the reverse connectivity is automatically created from Site2 to Site1, which you can see by selecting the other site and checking the Inter-site Connectivity information in the right sidebar.

  7. Repeat this step to add inter-site connectivity for other sites.

    When you establish underlay connectivity from Site1 to Site2, the reverse connectivity is done automatically for you.

    However, if you also want to establish inter-site connectivity from Site1 to Site3, you must repeat this step for that site as well.

Step 7

Provide External Connectivity information.

If you do not plan to configure connectivity to external sites or devices that are not managed by NDO, you can skip this step.

Detailed description of an external connectivity use case is available in the Configuring External Connectivity from Cloud CSRs Using Nexus Dashboard Orchestrator document.

  1. In the right <Site> Settings pane, select the External Connectivity tab.

  2. Click Add External Connection.

    The Add External Connectivity dialog will open.

  3. From the VRF dropdown, select the VRF you want to use for external connectivity.

    This is the VRF which will be used to leak the cloud routes. The Regions section will display the cloud regions that contain the CSRs to which this configuration be applied.

  4. From the Name dropdown in the External Devices section, select the external device.

    This is the external device you added in the General Settings > External Devices list during general infra configuration and must already be defined as described in Configuring Infra: Orchestrator General Settings.

  5. From the Tunnel IKE Version dropdown, pick the IKE version that will be used to establish the IPSec tunnel between the cloud site's CSRs and the external device.

  6. (Optional) From the Tunnel Subnet Pool dropdown, choose one of the named subnet pools.

    Named subnet pool are used to allocate IP addresses for IPSec tunnels between cloud site CSRs and external devices. If you do not provide any named subnet pools here, the external subnet pool will be used for IP allocation.

    Providing a dedicated subnet pool for external device connectivity is useful for cases where a specific subnet is already being used to allocate IP addresses to the external router and you want to continue to use those subnets for IPSec tunnels for NDO and cloud sites.

    If you want to provide a specific subnet pool for this connectivity, it must already be created as described in Configuring Infra: Orchestrator General Settings.

  7. (Optional) In the Pre-Shared Key field, provide the custom keys you want to use to establish the tunnel.

  8. If necessary, repeat the previous substeps for any additional external devices you want to add for the same external connection (same VRF).

  9. If necessary, repeat this step for any additional external connections (different VRFs).

    Note that there's a one-to-one relationship for tunnel endpoints between CSRs and external devices, so while you can create additional external connectivity using different VRFs, you cannot create additional connectivity to the same external devices.