Step 1

On the menu bar, click Fabric > Inventory.

Step 2

In the Navigation pane, expand Quick Start and click Add Remote Leaf.

Step 3

In the Remote Leaf pane of the working pane, click Add Remote Leaf.

Step 4

In the Add Remote Leaf wizard, review the information in the Overview page.

Step 5

When you have finished reviewing the information in the Overview panel, click Get Started at the bottom right corner of the page.

Step 6

In the External TEP page, configure the necessary parameters.

1. Leave the Use Defaults checkbox checked, or uncheck it if necessary.

   When checked, the wizard automatically allocates data plane and unicast TEP addresses. Those fields are not displayed when the Use Defaults box is checked. Uncheck the Use Defaults box to view or modify the proposed addresses, if necessary.

2. In the External TEP Pool field, enter the external TEP for the physical pod.

   The external TEP pool must not overlap the internal TEP pool.

3. In the Unicast TEP IP field, change the value that is automatically populated in this field, if necessary.

   This address is automatically allocated by Cisco APIC from the External TEP Pool, and will be used for sending traffic from the remote leaf switch to the local leaf switches on that pod.

   Cisco APIC automatically configures the unicast TEP IP address when you enter the External TEP Pool address.

4. Repeat these steps for each pod, if you have a multi-pod configuration.

5. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

   The Pod Selection page appears.

Step 7

In the Pod Selection page, configure the necessary parameters.

1. In the Pod ID field, select the pod ID of the pod where the remote leaf switches will be associated.

2. In the Remote Leaf TEP Pool field, select an existing remote leaf TEP pool or enter a remote leaf TEP pool to allocate IP addresses to the remote leaf switches.

   Click the View existing TEP Pools link underneath the Remote Leaf TEP Pool field to see the existing TEP pools (internal TEP pools, remote leaf TEP pools, and external TEP pools). Use this information to avoid creating duplicate or overlapping pools.

3. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

   The Routing Protocol page appears.

Step 8

In the Routing Protocol page, select and configure the necessary parameters for the underlay protocol to peer between the remote leaf switches and the upstream router. Follow these substeps.

1. Under the L3 Outside Configuration section, in the L3 Outside field, create or select an existing L3Out to represent the connection between the remote leaf switches and the upstream router. Multiple remote leaf pairs can use the same L3 Outside to represent their upstream connection.

   For the remote leaf switch configuration, we recommend that you use or create an L3Out that is different from the L3Out used in the multi-pod configuration.

2. In Cisco APIC Release 5.2(3) and later releases, set the Underlay control to either OSPF or BGP

   In releases before Cisco APIC Release 5.2(3), no selection is necessary because OSPF is the only supported underlay protocol.

   Note	When both OSPF and BGP are used in the underlay for Multi-Pod, Multi-Site, or Remote Leaf, do not redistribute router-ids into BGP from OSPF on IPN routers. Doing so may cause a routing loop and bring down OSPF and BGP sessions between the spine switch and IPN routers.

3. Choose the appropriate next configuration step.

• For an OSPF underlay, configure the OSPF parameters in Step Step 9, then skip Step Step 10.
• For a BGP underlay, skip Step Step 9 and configure the BGP parameters in Step Step 10.

Step 9

(For an OSPF underlay only) To configure an OSPF underlay, follow these substeps in the Routing Protocol page.

1. Under the OSPF section, leave the Use Defaults checkbox checked, or uncheck it if necessary.

   The checkbox is checked by default. Uncheck it to reveal the optional fields, such as area cost and authentication settings.

2. Gather the configuration information from the IPN, if necessary.

   For example, from the IPN, you might enter the following command to gather certain configuration information:

   IPN# show running-config interface ethernet slot/chassis-number

   For example:

   IPN# show running-config interface ethernet 1/5.11
   ...
   ip router ospf infra area 0.0.0.59
   ...

3. In the Area ID field, enter the OSPF area ID.

   Looking at the OSPF area 59 information shown in the output in the previous step, you could enter a different area in the Area ID field (for example, 0) and have a different L3Out. If you are using a different area for the remote leaf switch, you must create a different L3Out. You can also create a different L3Out, even if you are using the same OSPF area ID.

4. In the Area Type field, select the OSPF area type.

   You can choose one of the following OSPF types:

   • NSSA area

   • Regular area

   Note	You might see Stub area as an option in the Area Type field; however, stub area will not advertise the routes to the IPN, so stub area is not a supported option for infra L3Outs.

   Regular area is the default.

5. In the Interface Policy field, enter or select the OSPF interface policy.

   You can choose an existing policy or create a new one using the Create OSPF Interface Policy dialog box. The OSPF interface policy contains OSPF-specific settings such as OSPF network type, interface cost, and timers.

6. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

   The Remote Leafs page appears.

Step 10

(For a BGP underlay only) If the following BGP fields appear in the Routing Protocol page, follow these substeps. Otherwise, click Next to continue.

1. Under the BGP section, leave the Use Defaults checkbox checked, or uncheck it if necessary.

   The checkbox is checked by default. Uncheck it to reveal the optional fields, such as peering type, peer password, and route reflector nodes.

2. Note the nonconfigurable values in the Spine ID, Interface, and IPv4 Address fields.

3. In the Peer Address field, enter the IP address of the BGP neighbor.

4. In the Remote AS field, enter the Autonomous System (AS) number of the BGP neighbor.

5. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

   The Remote Leafs page appears.

Step 11

In the Remote Leafs page, configure the necessary parameters.

1. In the Serial field, enter the serial number for the remote leaf switch or select a discovered remote leaf switch from the dropdown menu.

2. In the Node ID field, assign a node ID to the remote leaf switch.

3. In the Name field, assign a name to the remote leaf switch.

4. In the Router ID field, enter a router ID that will be used to establish the control-plane communication with the upstream router and the rest of the Cisco ACI fabric.

5. In the Loopback Address field, enter the IPN router loopback IP address, if necessary.

   Leave this field blank if you use a router ID address.

6. Under the Interfaces section, in the Interface field, enter interface information for this remote leaf switch.

7. Under the Interfaces section, in the IPv4 Address field, enter the IPv4 IP address for the interface.

8. Under the Interfaces section, in the MTU field, assign a value for the maximum transmit unit of the external network.

   The range is 1500 to 9216.

9. If you selected a BGP underlay, enter the IP address of the BGP neighbor in the Peer Address field, and enter the Autonomous System (AS) number of the BGP neighbor in the Remote AS field.

10. Enter information on additional interfaces, if necessary.

    Click + within the Interfaces box to enter information for multiple interfaces.

11. When you have entered all of the necessary information for this remote leaf switch, enter information for additional remote leaf switches, if necessary.

    Click + to the right of the Interfaces box to enter information for multiple remote leaf switches.

12. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

    The Confirmation page appears.

Step 12

In the Confirmation page, review the list of policies that the wizard will create and change the names of any of the policies, if necessary, then click Finish at the bottom right corner of the page.

Step 13

In the Remote Leaf Summary page, click the appropriate button.

Step 14

In the Navigation pane, click Fabric Membership, then click the Nodes Pending Registration tab to view the status of the remote leaf switch configuration.

Step 15

Log into the spine switch connected to the IPN and enter the following command:

switch# show nattable

––––––– NAT TABLE ––––––––--
Private Ip   Routeable Ip
10.0.0.1     192.0.2.100
10.0.0.2     192.0.2.101
10.0.0.3     192.0.2.102

Step 16

On the IPN sub-interfaces connecting the remote leaf switches, configure the DHCP relays for each interface.

switch# configure terminal
switch(config)# interface ethernet 1/5.11
switch(config-subif)# ip dhcp relay address 192.0.2.100
switch(config-subif)# ip dhcp relay address 192.0.2.101
switch(config-subif)# ip dhcp relay address 192.0.2.102
switch(config-subif)# exit
switch(config)# interface ethernet 1/7.11
switch(config-subif)# ip dhcp relay address 192.0.2.100
switch(config-subif)# ip dhcp relay address 192.0.2.101
switch(config-subif)# ip dhcp relay address 192.0.2.102
switch(config-subif)# exit
switch(config)# exit
switch#

Step 17

In the Navigation pane, click Fabric Membership, then click the Registered Nodes tab to view the status of the remote leaf switch configuration.

Step 18

On the menu bar click System > System Settings.

Step 19

In the Navigation pane, choose System Global GIPo.

Step 20

For Use Infra GIPo as System GIPo, choose Enabled.

Step 1

Configure the TEP pool for the remote leaf switches, with the following steps:

1. On the menu bar, click Fabric > Inventory.

2. In the Navigation pane, click Pod Fabric Setup Policy.

3. On the Fabric Setup Policy panel, double-click the pod where you want to add the pair of remote leaf switches.

4. Click the + on the Remote Pools table.

5. Enter the remote ID and a subnet for the remote TEP pool and click Submit.

6. On the Fabric Setup Policy panel, click Submit.

Step 2

Configure the L3Out for the spine switch connected to the IPN router, with the following steps:

1. On the menu bar, click Tenants > infra.

2. In the Navigation pane, expand Networking, right-click L3Outs, and choose Create L3Out.

3. In the Name field, enter a name for the L3Out.

4. From the VRF drop-down list, choose overlay-1.

5. From the L3 Domain drop-down list, choose the external routed domain that you previously created.

6. In the Use for control, select Remote Leaf.

7. To use BGP as the IPN underlay protocol, uncheck the OSPF checkbox.

   Beginning with Cisco APIC Release 5.2(3), the IPN underlay protocol can be either OSPF or BGP.

8. To use OSPF as the IPN underlay protocol, in the OSPF area, where OSPF is selected by default, check the box next to Enable Remote Leaf with Multipod, if the pod where you are adding the remote leaf switches is part of a multipod fabric.

   This option enables a second OSPF instance using VLAN-5 for multipod, which ensures that routes for remote leaf switches are only advertised within the pod they belong to.

9. Click Next to move to the Nodes and Interfaces window.

Step 3

Configure the details for the spine and the interfaces used in the L3Out, with the following steps:

1. Determine if you want to use the default naming convention.

   In the Use Defaults field, check if you want to use the default node profile name and interface profile names:

   • The default node profile name is L3Out-name_nodeProfile, where L3Out-name is the name that you entered in the Name field in the Identity page.

   • The default interface profile name is L3Out-name_interfaceProfile, where L3Out-name is the name that you entered in the Name field in the Identity page.

2. Enter the following details.

   • Node ID—ID for the spine switch that is connected to the IPN router.

   • Router ID—IP address for the IPN router

   • External Control Peering—disable if the pod where you are adding the remote leaf switches is in a single-pod fabric

3. Enter necessary additional information in the Nodes and Interfaces window.

4. When you have entered the remaining additional information in the Nodes and Interfaces window, click Next.

   The Protocols window appears.

Step 4

Enter the necessary information in the Protocols window of the Create L3Out wizard.

1. If you chose BGP as the IPN underlay protocol, enter the Peer Address and the Remote AS of the BGP peer.

2. If you chose OSPF as the IPN underlay protocol, select an OSPF policy in the Policy field.

3. Click Next.

   The External EPG window appears.

Step 5

Enter the necessary information in the External EPG window of the Create L3Out wizard, then click Finish to complete the necessary configurations in the Create L3Out wizard.

Step 6

Navigate to Tenants > infra > Networking > L3Outs > L3Out_name > Logical Node Profiles > bLeaf > Logical Interface Profiles > portIf > OSPF Interface Profile.

Step 7

Enter the name of the interface profile.

Step 8

In the Associated OSPF Interface Policy Name field, choose a previously created policy or click Create OSPF Interface Policy.

Step 9

1. Under OSPF Profile, click OSPF Policy and choose a previously created policy or click Create OSPF Interface Policy.

2. Click Next.

3. Click Routed Sub-Interface, click the + on the Routed Sub-Interfaces table, and enter the following details:

   • Node—Spine switch where the interface is located.

   • Path—Interface connected to the IPN router

   • Encap—Enter 4 for the VLAN

4. Click OK and click Next.

5. Click the + on the External EPG Networks table.

6. Enter the name of the external network, and click OK.

7. Click Finish.

Step 10

To complete the fabric membership configuration for the remote leaf switches, perform the following steps:

1. Navigate to Fabric > Inventory > Fabric Membership.

   At this point, the new remote leaf switches should appear in the list of switches registered in the fabric. However, they are not recognized as remote leaf switches until you configure the Node Identity Policy, with the following steps.

2. For each remote leaf switch, double-click on the node in the list, configure the following details, and click Update:

   • Node ID—Remote leaf switch ID

   • RL TEP Pool—Identifier for the remote leaf TEP pool, that you previously configured

   • Node Name—Name of the remote leaf switch

   After you configure the Node Identity Policy for each remote leaf switch, it is listed in the Fabric Membership table with the role remote leaf.

Step 11

Configure the L3Out for the remote leaf location, with the following steps:

1. Navigate to Tenants > infra > Networking.

2. Right-click L3Outs, and choose Create L3Out.

3. Enter a name for the L3Out.

4. Click the OSPF checkbox to enable OSPF, and configure the OSPF details the same as on the IPN and WAN router.

   Note

   Do not check the Enable Remote Leaf with Multipod check box if you are deploying new remote leaf switches running Release 4.1(2) or later and you are enabling direct traffic forwarding on those remote leaf switches. This option enables an OSPF instance using VLAN-5 for multipod, which is not needed in this case. See About Direct Traffic Forwarding for more information.

5. Choose the overlay-1 VRF.

Step 12

Configure the nodes and interfaces leading from the remote leaf switches to the WAN router, with the following steps:

1. In the Nodes and Interfaces window in the Create L3Out wizard, enter the following details:.

   • Node ID—ID for the remote leaf that is connected to the WAN router

   • Router ID—IP address for the WAN router

   • External Control Peering—only enable if the remote leaf switches are being added to a pod in a multipod fabric

Step 13

Navigate to Tenants > infra > Networking > L3Outs > L3Out_name > Logical Node Profiles > bLeaf > Logical Interface Profiles > portIf > OSPF Interface Profile.

Step 14

In OSPF Interface Profiles, configure the following details for the routed sub-interface used to connect a remote leaf switch with the WAN router.

Step 15

Configure the Fabric External Connection Profile, with the following steps:

1. Navigate to Tenants > infra > Policies > Protocol.

2. Right-click Fabric Ext Connection Policies and choose Create Intrasite/Intersite Profile.

3. Enter the mandatory Community value in the format provided in the example.

4. Click the + on Fabric External Routing Profile.

5. Enter the name of the profile and add uplink interface subnets for all of the remote leaf switches.

6. Click Update and click Submit.

Step 16

On the menu bar click System > System Settings.

Step 17

In the Navigation pane, choose System Global GIPo.

Step 18

For Use Infra GIPo as System GIPo, choose Enabled.

Step 19

To verify that the remote leaf switches are discovered by the APIC, navigate to Fabric > Inventory > Fabric Membership, or Fabric > Inventory > Pod > Topology.

Step 20

To view the status of the links between the fabric and the remote leaf switches, enter the show ip ospf neighbors vrf overlay-1 command on the spine switch that is connected to the IPN router.

Step 21

To view the status of the remote leaf switches in the fabric, enter the acidiag fnvread NX-OS style command on the APIC using the CLI.

Step 1

Upgrade Cisco APIC and all the nodes in the fabric to Release 4.1(2) or later.

Step 2

Verify that the routes for the Routable Subnet that you wish to configure will be reachable in the Inter-Pod Network (IPN), and that the subnet is reachable from the remote leaf switches.

Step 3

Configure Routable Subnets in all the pods in the fabric:

1. On the menu bar, click Fabric > Inventory.

2. In the Navigation pane, click Pod Fabric Setup Policy.

3. On the Fabric Setup Policy panel, double-click the pod where you want to configure routable subnets.

4. Access the information in the subnets or TEP table, depending on the release of your APIC software:

   • For releases prior to 4.2(3), click the + on the Routable Subnets table.

   • For 4.2(3) only, click the + on the External Subnets table.

   • For 4.2(4) and later, click the + on the External TEP table.

5. Enter the IP address and Reserve Address, if necessary, and set the state to active or inactive.

   • The IP address is the subnet prefix that you wish to configure as the routeable IP space.

   • The Reserve Address is a count of addresses within the subnet that must not be allocated dynamically to the spine switches and remote leaf switches. The count always begins with the first IP in the subnet and increments sequentially. If you wish to allocate the Unicast TEP (covered later in these procedures) from this pool, then it must be reserved.

6. Click Update to add the external routable subnet to the subnets or TEP table.

7. On the Fabric Setup Policy panel, click Submit.

Step 4

Add Routable Ucast for each pod:

1. On the menu bar, click Tenants > infra > Policies > Protocol > Fabric Ext Connection Policies > intrasite-intersite_profile_name.

2. In the properties page for this intrasite/intersite profile, click + in the Pod Connection Profile area.

   The Create Pod Connection Profile window appears.

3. Select a pod and enter the necessary information in the Create Pod Connection Profile window.

   In the Unicast TEP field, enter a routable TEP IP address, including the bit-length of the prefix, to be used for unicast traffic over the IPN. This IP address is used by the spine switches in their respective pod for unicast traffic in certain scenarios. For example, a unicast TEP is required for remote leaf switch direct deployments.

   Note

   Beginning with Release 4.2(5), the APIC software will automatically raise the appropriate faults after an upgrade from a pre-4.2(5) release to 4.2(5) or later if you have any of the following incorrect configurations: Unicast TEP IP address of one pod configured with one of the IP addresses on the non-reserved portion of the external TEP pool of any pod with a 0 or non-zero reserve address count Unicast TEP IP address on one pod matches the Unicast TEP IP address of other pods in the fabric Unicast TEP IP address overlaps with the remote leaf TEP pool in all pods in the fabric You must make the appropriate configuration changes after upgrading to Release 4.2(5) or later in this case to clear the faults. You must make these configuration changes before attempting any kind of configuration export, otherwise a failure will occur on a configuration import, configuration rollback, or ID recovery going from Release 4.2(5) and later.

Step 5

Click Submit.

Step 6

Verify that COOP is configured correctly.

# show coop internal info global
# netstat -anp | grep 5000

Step 7

Verify that the BGP route reflector session in the remote leaf switch is configured correctly.

 
remote-leaf# show bgp vpnv4 unicast summary vrf all | grep 14.0.0
 14.0.0.227 4 100 1292 1164 395 0 0 19:00:13 52 
 14.0.0.228 4 100 1296 1164 395 0 0 19:00:10 52 

Step 8

Enable direct traffic forwarding on the remote leaf switches.

1. On the menu bar, click System > System Settings.

2. Click Fabric Wide Setting.

3. Click the check box on Enable Remote Leaf Direct Traffic Forwarding.

   When this is enabled, the spine switches will install Access Control Lists (ACLs) to prevent traffic coming from remote leaf switches from being sent back, since the remote leaf switches will now send directly between each remote leaf switches' TEPs. There may be a brief disruption in service while the tunnels are built between the remote leaf switches.

4. Click Submit.

5. To verify that the configuration was set correctly, on the spine switch, enter the following command:

   spine# cat /mit/sys/summary

   You should see the following highlighted line in the output, which is verification that the configuration was set correctly (full output truncated):

   
   ...
   podId : 1
   remoteNetworkId : 0
   remoteNode : no
   rldirectMode : yes
   rn : sys
   role : spine
   ...
   

Step 9

Add the Routable IP address of Cisco APIC as DHCP relay on the IPN interfaces connecting the remote leaf switches.

remote-leaf# moquery -c infraWiNode | grep routable

Step 10

Decommission and recommission each remote leaf switch one at a time to get it discovered on the routable IP address for the Cisco APIC.

Step 1

For a multipod configuration, configure a multipod-internal L3Out using VLAN-5.

Step 2

Provision back private network reachability if it was removed when you enabled the direct traffic forwarding feature on the remote leaf switches.

Step 3

Disable remote leaf switch direct traffic forwarding for all remote leaf switches by posting the following policy:

POST URL : https://<ip address>/api/node/mo/uni/infra/settings.xml
<imdata>
    <infraSetPol dn="uni/infra/settings" enableRemoteLeafDirect="no" />
</imdata>

Step 4

Remove the Routable Subnets and Routable Ucast from the pods in the fabric.

Step 5

Decommission and recommission each remote leaf switch to get it discovered on the non-routable internal IP address of the Cisco APIC.

Step 6

Downgrade the Cisco APIC and all the nodes in the fabric to a release prior to 4.1(2).

Step 1

Log in to Cisco APIC.

Step 2

Go to System > System Settings.

Step 3

In the System Settings navigation pane, choose Remote Leaf POD Redundancy Policy.

Step 4

In the Remote Leaf POD Redundancy Policy work pane, check the Enable Remote Leaf Pod Redundancy Policy check box.

Step 5

(Optional) Check the Enable Remote Leaf Pod Redundancy pre-emption check box.