Forwarding Configurations for Cisco Nexus 5600, 7000 and 9000 Series Switches in the Programmable Fabric
 Note |
For ease of use, the configuration mode from which you need to start configuring a task is mentioned at the beginning of each configuration. |
Cisco Nexus 5600 Series switch configuration
The following configurations are required for the Cisco Nexus 5600 Series switch for supporting BGP-EVPN with VXLAN overlay. Note that most of the configurations required for enabling VXLAN remain the same, EVPN configurations are what will be the emphasis here:
-
Initial configuration - Install the network virtualization overlay, BGP, and EVPN features on the VTEPs.
-
Implement Layer 2 VNI configurations for tenant networks within a tenant.
-
Implement Layer 3 VNI configurations for the tenant.
Note |
Though configuration examples are mainly IPv4, IPv6 addresses are also supported in the VXLAN EVPN fabric. |
Note |
A Cisco Nexus 5600 Series switch enabled for switching-mode store-and-forward may experience a egress ASIC buffer stuck under the following conditions:
To avoid this problem do not use random write-erase commands. If such a command was issued in error, immediately run the copy running-configuration startup-configuration command. |
Initial configuration
(config) #
install feature-set fabric
feature-set fabric
feature fabric forwarding
feature interface-vlan
feature ospf
OR
feature isis
Attention |
You can use either OSPF or IS-IS as the routing protocol. |
(config) #
feature nv overlay
feature bgp
feature vn-segment-vlan-based
nv overlay evpn
Configure the anycast gateway MAC address
(config) #
fabric forwarding anycast-gateway-mac 2020.0000.00aa
Configure BGP L2VPN EVPN address family
(config) #
router bgp 100
neighbor 10.1.1.53 remote-as 100
update-source loopback0
address-family l2vpn evpn
send-community both
Layer 2 VNI configurations for a tenant network
Associate a VLAN to the Layer 2 VNI
(config) #
vlan 200
vn-segment 30000
Create a loopback interface for BGP and assign an IP address to it
(config) #
interface loopback 0
ip address 10.1.1.54/32
Create a loopback interface for NVE and assign an IP address to it
(config) #
interface loopback 1
ip address 10.1.2.54/32
Associate the Layer 2 VNI to the overlay and configure multicast group membership
(config) #
interface nve 1
no shutdown
source-interface loopback1
host-reachability protocol bgp
member vni 30000
suppress-arp
mcast-group 239.1.1.0
Associate the Layer 2 VNI to the EVPN address family, and enable route distinguisher and route target functions for the VNI
(config) #
evpn
vni 30000 l2
rd auto
route-target import auto
route-target export auto
Note |
Alternatively, the following config can also be used: |
evpn
vni 30000 l2
rd auto
route-target both auto
The combination of the router BGP command (configured earlier) and the evpn command ensures that BGP EVPN is configured to advertise ‘MAC route’ or ‘MAC + associated host routes’ of servers attached to the VTEP, for the specified Layer 2 VNI (Route type 2 [Refer to the EVPN RFC document for more details]). By default, the MAC route will be advertised, and the associated host route will be advertised if either there is an SVI configured for that VLAN in anycast-gateway mode or if suppress-arp option is enabled for that L2 VNI (See ARP Suppression section).
In the above NVE example, the MAC+IP routes for the hosts are advertised into BGP-EVPN for hosts belonging to layer 2 VNI 30000.
Layer 3 VNI configurations for a tenant
Associate the VRF VNI (Layer 3 VNI) to the customer VRF.
Enable VRF route distinguisher and route target functions.
(config) #
vrf context coke
vni 50000
rd auto
address-family ipv4 unicast
route-target both auto evpn
In the above example, the option both is used to import and export routes associated with the Layer 3 VNI 50000.
Associate the VRF VNI to a VLAN and associate an SVI to the customer VRF
(config) #
vlan 2200
vn-segment 50000
(config) #
interface vlan 2200
vrf member coke
ip forward
ipv6 forward
no ip redirects
no ipv6 redirects
no shutdown
In order to avoid the overhead of creating a core facing vlan and corresponding SVI on a per vrf basis, we also provide an option of using a vrf-tenant-profile that automatically takes care of this. Note that if there is a vrf-tenant-profile configured, then the user must ensure the following CLIs related to dynamic and core-VLANs are also enabled.
(config) #
system fabric dynamic-vlans 100-2400
system fabric core-vlans 100-300
switch #
configure profile vrf-tenant-profile
vlan $vrfVlanId
vn-segment $vrfSegmentId
interface vlan $vrfVlanId
vrf member $vrfName
ip forward
ipv6 forward
no ip redirects
no ipv6 redirects
no shutdown
end
Add the Layer 3 VRF VNI to the overlay network
(config) #
interface nve 1
host-reachability protocol bgp
member vni 50000 associate-vrf
Associate the customer VRF to BGP and enable L2VPN EVPN route distribution
(config) #
router bgp 100
vrf coke
address-family ipv4 unicast
advertise l2vpn evpn
Enable host/server facing SVI (and associate it to a VRF) for Layer 3 connectivity on the distributed anycast gateway
(config) #
interface vlan 200
vrf member coke
ip address 209.165.202.129/27
fabric forwarding mode anycast-gateway
Cisco Nexus 5600 Series switches verification
For verification of MAC routes, refer these commands:
switch# show mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 200 2010.0000.0010 dynamic 270 F F Eth100/1/1
* 200 2010.0000.0011 dynamic 0 F F nve1/10.1.1.56
* 200 2010.0000.0012 dynamic 0 F F nve1/10.1.1.74
* 200 2010.0000.0013 dynamic 0 F F nve1/10.1.1.56
* 200 8080.c800.0038 dynamic 0 F F nve1/10.1.1.74
* 1 24e9.b392.316b dynamic 1190 F F Eth100/1/1
switch# show l2route evpn mac all
Topology Mac Address Prod Next Hop (s)
----------- -------------- ------ ---------------
200 2010.0000.0010 Local Eth100/1/1
200 2010.0000.0011 BGP 10.1.1.56
200 2010.0000.0012 BGP 10.1.1.74
200 2010.0000.0013 BGP 10.1.1.56
200 8080.c800.0038 BGP 10.1.1.74
2200 002a.6ab2.0181 VXLAN 10.1.1.56
2200 8c60.4f14.2efc VXLAN 10.1.1.74
Command output description
Prod (producer) column displays the source of origination of the MAC address.
Local means a MAC address learnt locally via a server facing or edge port, BGP means the remote end host MAC was learnt from a remote VTEP via BGP-EVPN and VXLAN indicates the router MAC of the remote VTEP as carried in the extended community in the BGP advertisement.
switch# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 198, local router ID is 10.1.1.54
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.1.1.54:32967 (L2VNI 30000)
*>l[2]:[0]:[0]:[48]:[2010.0000.0010]:[0]:[0.0.0.0]/216
10.1.1.54 100 32768 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>l[2]:[0]:[0]:[48]:[2010.0000.0010]:[32]:[200.0.0.10]/272
10.1.1.54 100 32768 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[32]:[200.0.0.11]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[32]:[200.0.0.12]/272
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[32]:[200.0.0.13]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[32]:[200.0.0.56]/272
10.1.1.74 100 0 i
Route Distinguisher: 10.1.1.56:3
*>i[5]:[0]:[0]:[24]:[209.165.202.130]:[0.0.0.0]/224
10.1.1.56 0 100 0 ?
Route Distinguisher: 10.1.1.56:32967
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[32]:[209.165.202.140]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[32]:[209.165.202.142]/272
10.1.1.56 100 0 i
Route Distinguisher: 10.1.1.74:32967
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[32]:[209.165.202.141]/272
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[32]:[209.165.202.143]/272
10.1.1.74 100 0 i
switch# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.1.1.56 Up CP 1d12h 002a.6ab2.0181
nve1 10.1.1.74 Up CP 1d12h 8c60.4f14.2efc
For verification of IP host and prefix routes, refer these commands:
switch# show ip arp vrf coke
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface
IP ARP Table for context coke
Total number of entries: 1
Address Age MAC Address Interface
209.165.202.13 00:18:23 2010.0000.0010 Vlan200
switch# show ip route vrf coke
IP Route Table for VRF "coke"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.1.1.0/24, ubest/mbest: 1/0, attached
*via 10.1.1.1, Vlan10, [0/0], 1d12h, direct
10.1.1.1/32, ubest/mbest: 1/0, attached
*via 10.1.1.1, Vlan10, [0/0], 1d12h, local
209.165.202.130/27, ubest/mbest: 1/0, attached
*via 209.165.202.129, Vlan200, [0/0], 1d12h, direct, tag 12345,
209.165.202.129/32, ubest/mbest: 1/0, attached
*via 209.165.202.129, Vlan200, [0/0], 1d12h, local, tag 12345,
209.165.202.139/32, ubest/mbest: 1/0, attached
*via 209.165.202.139, Vlan200, [190/0], 1d12h, hmm
209.165.202.140 /32, ubest/mbest: 1/0
*via 10.1.1.56%default, [200/0], 1d12h, bgp-100, internal, tag 100, (mpls-vpn)segid 50000 tunnel: 16843064 encap: 1
Command output description
Direct means that the subnet prefix is configured locally under a Layer-3 interface on this switch. Local means the IP address belongs to the switch aka locally configured under a Layer-3 interface on that switch (10.1.1.254/24).
switch# show l2route evpn mac-ip all
Topology ID Mac Address Prod Host IP Next Hop(s)
----------- -------------- ---- --------------------------------------- --------
-------
200 2010.0000.0010 HMM 209.165.202.139 N/A
200 2010.0000.0011 BGP 209.165.202.140 10.1.1.56
200 2010.0000.0012 BGP 209.165.202.141 10.1.1.74
200 2010.0000.0013 BGP 209.165.202.142 10.1.1.56
200 8080.c800.0038 BGP 209.165.202.143 10.1.1.74
switch# show bgp l2vpn evpn
Route Distinguisher: 10.1.1.54:3 (L3VNI 50000)
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[32]:[209.165.202.144]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[32]:[209.165.202.141]/272
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[32]:[209.165.202.143]/272
10.1.1.56 100 0 i
*>l[5]:[0]:[0]:[24]:[209.165.202.130]:[0.0.0.0]/224
10.1.1.54 0 100 32768 ?
* i 10.1.1.56 0 100 0 ?
Cisco Nexus 7000 Series switch configuration
The following BGP, EVPN and overlay configurations are required for the Cisco Nexus 7000 Series and 7700 Series switches with F3 or M3 cards:
-
Initial configuration - Install the network virtualization overlay, BGP, and EVPN features on the VTEPs.
-
Layer 2 VNI configurations for tenant networks within a tenant.
-
Layer 3 VNI configurations for a tenant.
Note |
Though configuration examples are mainly IPv4, IPv6 addresses are also supported in the VXLAN EVPN fabric. |
VXLAN BGP EVPN configuration for the Cisco Nexus 7000 Series switches is also available here. While the 7.2 release only supported the border leaf and border spine functionality, the 7.3 version in addition also supports the leaf functionality.
A switch VDC with M3 modules cannot perform the role of a VXLAN BGP EVPN leaf switch.
Initial configurations
(config) #
install feature-set fabric
feature-set fabric
feature fabric forwarding
feature interface-vlan
feature ospf
OR
feature isis
Attention |
You can use either OSPF or IS-IS as the underlay routing protocol. |
Note |
The install feature-set fabric command should only be used in the admin VDC. When using a VDC, ensure the VDC is of type F3 or M3, for EVPN. A sample configuration is given below: (config) # |
(config) #
feature nv overlay
feature bgp
feature vni
nv overlay evpn
Configure the anycast gateway MAC address
(config) #
fabric forwarding anycast-gateway-mac 2020.0000.00aa
Configure BGP L2VPN EVPN address family
(config) #
router bgp 100
neighbor 10.1.1.53 remote-as 100
update-source loopback0
address-family l2vpn evpn
send-community both
Layer 2 VNI configurations for a tenant network
Create a bridge domain and associate the Layer 2 VNI with it
(config) #
vni 30000
system bridge-domain 200-210
bridge-domain 200
member vni 30000
While the system bridge-domain command identifies the bridge domain IDs, the bridge-domain command configures the specified bridge domain(s).
Associate a VLAN (or dot1q tag) with the Layer 2 VNI:
(config) #
encapsulation profile vni cisco
dot1q 50 vni 30000
Note |
For an access port, you should use the untagged keyword, as shown below. |
Associate the encapsulation profile with the server facing interface
(config) #
interface Ethernet 1/12
no shutdown
no switchport
service instance 1 vni
encapsulation profile cisco default
no shutdown
Create a loopback interface for BGP and assign an IP address to it
(config) #
interface loopback 0
ip address 10.1.1.54/32
Create a loopback interface for NVE and assign an IP address to it
(config) #
interface loopback 1
ip address 10.1.2.54/32
Associate the Layer 2 VNI to the overlay and configure multicast group membership
(config) #
interface nve 1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 30000
suppress-arp
mcast-group 239.1.1.0
Enable EVPN and associate the Layer 2 VNI to it
Enable route distinguisher and route target functions for the Layer 2 VNI
(config) #
evpn
vni 30000 l2
rd auto
route-target import auto
route-target export auto
Note that with the Cisco Nexus 7000 Series switches, a VNI is associated with a bridge-domain (1:1). Refer to the respective configuration guide for more information on bridge-domains. The combination of the router BGP command (configured earlier) and the evpn command ensures that BGP EVPN is configured to advertise ‘MAC route’ or ‘MAC + associated host routes’ of servers attached to the VTEP, for the specified Layer 2 VNI.
In the above NVE example, MAC+ IP routes are advertised into BGP-EVPN for hosts belonging to layer 2 VNI 30000.
Layer 3 VNI configurations for a tenant
Associate the VRF VNI to the customer VRF
Enable VRF route distinguisher and VRF route target functions for the Layer 3 VNI
(config) #
vrf context coke
vni 50000
rd auto
address-family ipv4 unicast
route-target both auto evpn
In the above example, the option both is used to import and export routes associated with the Layer 3 VNI 50000. Specifically, the layer-3 routes will be advertised with route-target 100:50000 where 100 is the BGP Autonomous system number and 50000 is the layer-3 VNI.
Associate the VRF VNI to a bridge-domain and associate a BDI to the customer VRF
(config) #
system bridge-domain add 2200
vni 50000
bridge-domain 2200
member vni 50000
interface bdi2200
vrf member coke
ip forward
no ip redirects
no shutdown
While the system bridge-domain command identifies the bridge domain IDs, the bridge-domain command configures the specified bridge domain(s).
Add the Layer 3 VRF VNI to the overlay network and enable BGP reachability
(config) #
interface nve 1
host-reachability protocol bgp
member vni 50000 associate-vrf
Configure BGP, associate the customer VRF to BGP and enable L2VPN EVPN route distribution
(config) #
router bgp 100
vrf coke
address-family ipv4 unicast
advertise l2vpn evpn
Enable host/server facing BDI (and associate it to a VRF) for Layer 3 connectivity on the distributed anycast gateway
(config) #
interface bdi200
vrf member coke
ip address 10.1.1.1/24
fabric forwarding mode anycast-gateway
no shutdown
Cisco Nexus 7000 Series switches verification
For verification of MAC routes, refer these commands:
The following is sample output to verify that end host MAC addresses (local and remote) are added to the MAC address table:
switch# show mac address-table dynamic
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link, E -
EVPN entry
(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve
age info
VLAN/BD MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 200 2010.0000.0010 dynamic 270 F F Eth100/1/1
* 200 2010.0000.0011 dynamic 0 F F nve1/10.1.1.56
* 200 2010.0000.0012 dynamic 0 F F nve1/10.1.1.74
* 200 2010.0000.0013 dynamic 0 F F nve1/10.1.1.56
* 200 8080.c800.0038 dynamic 0 F F nve1/10.1.1.74
* 1 24e9.b392.316b dynamic 1190 F F Eth100/1/1
The following is sample output for viewing MAC addresses of end hosts across all EVPN instances (EVIs) pertaining to the switch:
switch# show l2route evpn mac all
Topology Mac Address Prod Next Hop (s)
----------- -------------- ------ ---------------
200 2010.0000.0010 Local Eth100/1/1
200 2010.0000.0011 BGP 10.1.1.56
200 2010.0000.0012 BGP 10.1.1.74
200 2010.0000.0013 BGP 10.1.1.56
200 8080.c800.0038 BGP 10.1.1.74
2200 002a.6ab2.0181 VXLAN 10.1.1.56
2200 8c60.4f14.2efc VXLAN 10.1.1.74
The following sample output displays BGP routing table information for the L2VPN EVPN address family. It includes route distinguisher and next hop information.
switch # show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 198, local router ID is 10.1.1.54
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.1.1.54:32967 (L2VNI 30000)
*>l[2]:[0]:[0]:[48]:[2010.0000.0010]:[0]:[0.0.0.0]/216
10.1.1.54 100 32768 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>l[2]:[0]:[0]:[48]:[2010.0000.0010]:[32]:[209.165.202.139]/272
10.1.1.54 100 32768 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[32]:[209.165.202.140]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[32]:[209.165.202.141]/272
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[32]:[209.165.202.142]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[32]:[209.165.202.143]/272
10.1.1.74 100 0 i
Route Distinguisher: 10.1.1.56:3
*>i[5]:[0]:[0]:[24]:[209.165.202.130]:[0.0.0.0]/224
10.1.1.56 0 100 0 ?
Route Distinguisher: 10.1.1.56:32967
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[0]:[0.0.0.0]/216
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[32]:[209.165.202.140]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[32]:[209.165.202.142]/272
10.1.1.56 100 0 i
Route Distinguisher: 10.1.1.74:32967
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[0]:[0.0.0.0]/216
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[32]:[209.165.202.141]/272
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[8080.c800.0038]:[32]:[209.165.202.143]/272
10.1.1.74 100 0 i
The following sample output displays peer VTEP device information.
switch # show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.1.1.56 Up CP 1d12h 002a.6ab2.0181
nve1 10.1.1.74 Up CP 1d12h 8c60.4f14.2efc
For IP host and prefix routes verification, refer these commands:
The following sample output displays tenant (VRF) information
switch # show ip arp vrf coke
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface
IP ARP Table for context coke
Total number of entries: 1
Address Age MAC Address Interface
209.165.202.144 00:18:23 2010.0000.0010 Bdi200
The following sample output displays tenant (VRF) information
switch # show ip route vrf coke
IP Route Table for VRF "coke"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.1.1.0/24, ubest/mbest: 1/0, attached
*via 10.1.1.1, Bdi10, [0/0], 1d12h, direct
10.1.1.1/32, ubest/mbest: 1/0, attached
*via 10.1.1.1, Bdi10, [0/0], 1d12h, local
209.165.202.130/27, ubest/mbest: 1/0, attached
*via 209.165.202.129, Bdi200, [0/0], 1d12h, direct, tag 12345,
209.165.202.129/32, ubest/mbest: 1/0, attached
*via 209.165.202.129, Bdi200, [0/0], 1d12h, local, tag 12345,
209.165.202.139/32, ubest/mbest: 1/0, attached
*via 209.165.202.139, Bdi200, [190/0], 1d12h, hmm
209.165.202.140 /32, ubest/mbest: 1/0
*via 10.1.1.56%default, [200/0], 1d12h, bgp-100, internal, tag 100, (mpls-vpn)segid 50000 tunnel: 16843064 encap: 1
The following sample output displays MAC - IP address binding for all attached and remote end hosts (learned through the BGP EVPN control plane).
switch # show l2route evpn mac-ip all
Topology ID Mac Address Prod Host IP Next Hop(s)
----------- -------------- ---- --------------------------------------- --------
200 2010.0000.0010 HMM 209.165.202.139 N/A
200 2010.0000.0011 BGP 209.165.202.140 10.1.1.56
200 2010.0000.0012 BGP 209.165.202.141 10.1.1.74
200 2010.0000.0013 BGP 209.165.202.142 10.1.1.56
200 8080.c800.0038 BGP 209.165.202.143 10.1.1.74
The following sample output displays BGP routing table information for Layer-3 VNIs.
switch # show bgp l2vpn evpn
Route Distinguisher: 10.1.1.54:3 (L3VNI 50000)
*>i[2]:[0]:[0]:[48]:[2010.0000.0011]:[32]:[209.165.202.144]/272
10.1.1.56 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0012]:[32]:[209.165.202.141]/272
10.1.1.74 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0013]:[32]:[209.165.202.143]/272
10.1.1.56 100 0 i
*>l[5]:[0]:[0]:[24]:[209.165.202.130]:[0.0.0.0]/224
10.1.1.54 0 100 32768 ?
* i 10.1.1.56 0 100 0 ?
Cisco Nexus 9000 Series switch configuration
The following configurations are required for a Cisco Nexus 9000 Series switch for the VXLAN BGP EVPN fabric.
-
Initial configuration - Install the network virtualization overlay, BGP, and EVPN features on the VTEPs.
-
Layer 2 VNI configurations for tenant networks within a tenant.
-
Layer 3 VNI configurations for a tenant.
Note |
Though configuration examples are mainly IPv4, IPv6 addresses are also supported in the VXLAN BGP EVPN fabric. |
Initial configuration
(config) #
nv overlay evpn
feature bgp
feature ospf
OR
feature isis
Attention |
You can use either OSPF or IS-IS as the underlay routing protocol. |
(config) #
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
Configure the anycast gateway MAC address
(config) #
fabric forwarding anycast-gateway-mac 2020.0000.00aa
Configure BGP L2VPN EVPN address family
(config) #
router bgp 100
neighbor 192.0.2.1
remote-as 100
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
Layer 2 VNI configurations for a tenant network
Associate a VLAN to the Layer 2 VNI
(config) #
vlan 200
vn-segment 30000
Create a loopback interface for BGP and assign an IP address to it
(config) #
interface loopback 0
ip address 192.0.2.10/32
Create a loopback interface for NVE and assign an IP address to it
(config) #
interface loopback 1
ip address 198.51.100.1/32
Associate the Layer 2 VNI to the overlay and configure multicast group membership
(config) #
interface nve 1
no shutdown
source-interface loopback1
host-reachability protocol bgp
member vni 30000
suppress-arp
mcast-group 239.1.1.0
(config)#
interface nve 1
source-interface loopback 1
host-reachability protocol bgp
global suppress-arp
global mcast-group 224.1.1.1 L2
global mcast-gropu 225.1.1.1 L3
member vni 10000
suppress-arp disable
member 20000 associate-vrf
mcast-group 225.1.1.10Associate the Layer 2 VNI to the EVPN address family, and enable route distinguisher and route target functions for the VNI
(config) #
evpn
vni 30000 l2
rd auto
route-target import auto
route-target export auto
Note |
Alternatively, the following configurations can also be used: |
evpn
vni 30000 l2
rd auto
route-target both auto
The combination of the router BGP command (configured earlier) and the evpn command ensures that BGP EVPN is configured to advertise ‘MAC route’ or ‘MAC + associated host routes’ of servers attached to the VTEP, for the specified Layer 2 VNI. (Route type 2 [Refer to the EVPN RFC document for more details]). By default, the MAC route will be advertised, and the associated host route will be advertised if there is an SVI configured for that VLAN in the anycast-gateway mode or if suppress-arp option is enabled for that L2 VNI (see ARP Suppression section).
In the above NVE example, MAC and IP routes are advertised into BGP-EVPN for end hosts belonging to layer 2 VNI 30000.
Layer 3 VNI configurations for a tenant
Associate the VRF VNI (Layer 3 VNI) to the customer VRF
Enable VRF route distinguisher and route target functions
(config) #
vrf context coke:vrf1
vni 50000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
In the above example, the option both is used to import and export routes associated with the Layer 3 VNI 50000.
Associate the VRF VNI to a VLAN and associate an SVI to the customer VRF
(config) #
vlan 2500
vn-segment 50000
(config) #
interface vlan 2500
vrf member coke:vrf1
ip forward
ipv6 forward
no ip redirects
no ipv6 redirects
no shutdown
In order to avoid the overhead of creating a core facing VLAN and corresponding SVI on a per VRF basis, the vrf-tenant-profile (that automatically takes care of this) is provided. If you configure a vrf-tenant-profile, you should enable the following CLIs related to dynamic and core VLANs.
(config) #
system fabric dynamic-vlans 2500-3500
system fabric core-vlans 2500-2999
configure profile vrf-tenant-profile
vlan $vrfVlanId
vn-segment $vrfSegmentId
interface vlan $vrfVlanId
vrf member $vrfName
ip forward
ipv6 forward
no ip redirects
no ipv6 redirects
no shutdown
end
Add the Layer 3 VRF VNI to the overlay network
(config) #
interface nve 1
host-reachability protocol bgp
member vni 50000 associate-vrf
Associate the customer VRF to BGP and enable L2VPN EVPN route distribution
(config) #
router bgp 100
vrf coke:vrf1
address-family ipv4 unicast
advertise l2vpn evpn
Enable host/server facing SVI (and associate it to a VRF) for Layer 3 connectivity on the distributed anycast gateway
(config) #
interface vlan 200
vrf member coke:vrf1
ip address 203.0.113.3/24 tag 12345
fabric forwarding mode anycast-gateway
Cisco Nexus 9000 Series switches verification
For verification of MAC routes, refer these commands
The following is sample output to verify that end host MAC addresses (local and remote) are added to the MAC address table:
switch# show mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 1 a036.9f22.a277 dynamic 0 F F Eth1/7
C 200 002a.6a85.a67c dynamic 0 F F nve1(198.51.100.10)
* 200 2010.0000.0012 dynamic 0 F F Eth1/7
C 200 2010.0000.0015 dynamic 0 F F nve1(198.51.100.10)
The following is sample output for viewing MAC addresses of end hosts across all EVPN instances (EVIs) pertaining to the switch:
switch# show l2route evpn mac all
Topology Mac Address Prod Flags Seq No Next-Hops
----------- -------------- ------ ------------- ---------- ----------------
200 002a.6a85.a67c BGP SplRcv 0 198.51.100.10
200 2010.0000.0012 Local L, 0 Eth1/7
200 2010.0000.0015 BGP SplRcv 0 198.51.100.10
2500 7c0e.ceca.f2ff VXLAN Rmac 0 198.51.100.10
Command output description
Prod (producer) column displays the source of origination of the MAC address.
Local means a MAC address learnt locally via a server facing or edge port, BGP means the remote end host MAC was learnt from a remote VTEP via BGP-EVPN and VXLAN indicates the router MAC of the remote VTEP as carried in the extended community in the BGP advertisement.
The following sample output displays BGP routing table information for the L2VPN EVPN address family. It includes route distinguisher and next hop information:
switch # show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 26, local router ID is 192.0.2.10
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.0.2.20:3
*>i[5]:[0]:[0]:[24]:[203.0.113.6]:[0.0.0.0]/224
198.51.100.10 0 100 0 ?
Route Distinguisher: 192.0.2.20:32967
*>i[2]:[0]:[0]:[48]:[002a.6a85.a67c]:[0]:[0.0.0.0]/216
198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0015]:[0]:[0.0.0.0]/216
198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[002a.6a85.a67c]:[32]:[200.0.0.52]/272
198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0015]:[32]:[200.0.0.15]/272
198.51.100.10 100 0 i
Route Distinguisher: 192.0.2.30:3
*>i[5]:[0]:[0]:[24]:[200.0.0.0]:[0.0.0.0]/224
198.51.100.10 0 100 0 ?
Route Distinguisher: 192.0.2.30:32967
*>i[2]:[0]:[0]:[48]:[002a.6a85.a67c]:[0]:[0.0.0.0]/216
198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0015]:[0]:[0.0.0.0]/216
198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[002a.6a85.a67c]:[32]:[200.0.0.52]/272
198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0015]:[32]:[200.0.0.15]/272
198.51.100.10 100 0 i
Route Distinguisher: 192.0.2.10:32967 (L2VNI 30000)
* i[2]:[0]:[0]:[48]:[002a.6a85.a67c]:[0]:[0.0.0.0]/216
198.51.100.10 100 0 i
*>i 198.51.100.10 100 0 i
*>l[2]:[0]:[0]:[48]:[2010.0000.0012]:[0]:[0.0.0.0]/216
192.0.2.10 100 32768 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0015]:[0]:[0.0.0.0]/216
198.51.100.10 100 0 i
* i 198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[002a.6a85.a67c]:[32]:[203.0.113.12]/272
198.51.100.10 100 0 i
* i 198.51.100.10 100 0 i
*>l[2]:[0]:[0]:[48]:[2010.0000.0012]:[32]:[203.0.113.5]/272
192.0.2.10 100 32768 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0015]:[32]:[203.0.113.8]/272
198.51.100.10 100 0 i
* i 198.51.100.10 100 0 i
Route Distinguisher: 192.0.2.10:3 (L3VNI 50000)
*>i[2]:[0]:[0]:[48]:[002a.6a85.a67c]:[32]:[203.0.113.12]/272
198.51.100.10 100 0 i
* i 198.51.100.10 100 0 i
*>i[2]:[0]:[0]:[48]:[2010.0000.0015]:[32]:[203.0.113.8]/272
198.51.100.10 100 0 i
* i 198.51.100.10 100 0 i
* i[5]:[0]:[0]:[24]:[200.0.0.0]:[0.0.0.0]/224
198.51.100.10 0 100 0 ?
* i 198.51.100.10 0 100 0 ?
*>l 198.51.100.1 0 100 32768 ?
The following sample output displays peer VTEP device information:
switch # show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 198.51.100.10 Up CP 3d00h 7c0e.ceca.f2ff
For verification of IP host and prefix routes, refer these commands
The following sample output displays tenant (VRF) information:
switch # show ip arp vrf coke:vrf1
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Dervied from L2RIB Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context coke:vrf1
Total number of entries: 1
Address Age MAC Address Interface Flags
203.0.113.5 00:12:46 2010.0000.0012 Vlan200
The following sample output displays tenant (VRF) information:
switch # show ip route vrf coke:vrf1
IP Route Table for VRF "coke:vrf1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
203.0.113.6/24, ubest/mbest: 1/0, attached
*via 203.0.113.3, Vlan200, [0/0], 3d00h, direct, tag 12345
203.0.113.3/32, ubest/mbest: 1/0, attached
*via 203.0.113.3, Vlan200, [0/0], 3d00h, local, tag 12345
203.0.113.5/32, ubest/mbest: 1/0, attached
*via 203.0.113.5, Vlan200, [190/0], 00:14:04, hmm
203.0.113.8/32, ubest/mbest: 1/0
*via 198.51.100.10%default, [200/0], 3d00h, bgp-100, internal, tag 100 (evpn) segid: 50000 tunnelid: 0x16020202 encap: VXLAN
203.0.113.12/32, ubest/mbest: 1/0
*via 198.51.100.10%default, [200/0], 00:13:46, bgp-100, internal, tag 100 (evpn) segid: 50000 tunnelid: 0x16020202 encap: VXLAN
Command output description
Direct means that the subnet prefix is configured locally under a Layer-3 interface on this switch. Local means the IP address belongs to the switch aka locally configured under a Layer-3 interface on that switch (200.0.0.1/24).
The following sample output displays MAC - IP address binding for all attached and remote end hosts (learned through the BGP EVPN control plane):
switch # show l2route evpn mac-ip all
Topology Mac Address Prod Flags Seq No Host IP Next-Hops
----------- -------------- ------ ---------- --------------- ---------------
200 2010.0000.0012 HMM -- 0 203.0.113.5 Local
200 2010.0000.0015 BGP -- 0 203.0.113.8 198.51.100.10
200 002a.6a85.a67c BGP -- 0 203.0.113.12 198.51.100.10
Feedback