Configure VXLAN

Introduction

This document describes a high level overview of Virtual Extensible LAN (VXLAN) and configuration examples with verification commands and output.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Multicast routing concepts such as Rendezvous Point (RP) and Platform Independent Multicast (PIM).
• Virtual Port Channel (vPC) concepts.

This document assumes that the IP routing and multicast routing has been established prior to VXLAN configuration.

Components Used

The information in this document is based on these software and hardware versions:

• Nexus 9396s as vPC Virtual Tunnel Endpoints (VTEPs) that run Version 7.0(3)I1(1b)
• Nexus 3172 that runs Version 6.0(2)U5(1)
• LAN_ENTERPRISE_SERVICES_PKG license installed

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Terminology

VXLAN (Virtual Extensible LAN) - The technology that provides the same Ethernet Layer 2 network services as VLAN does today, but with greater extensibility and flexibility.

VNID (Vxlan Network Identifier) - 24 bit segment ID that defines the broadcast domain. Interchangeable with "VXLAN Segment ID".

VTEP (Virtual Tunnel Endpoint) - This is the device that does the encapsulation and de-encapsulation.

NVE (Network Virtual Interface) - Logical interface where the encapsulation and de-encapsulation occur.

What is VXLAN

• VXLAN is a technology that allows overlaying a Layer 2 (L2) network over a Layer 3 (L3) underlay with use of any IP routing protocol.
• It uses MAC-in-UDP Encapsulation.

VXLAN solves three main problems:

1. 16M VNIs (broadcast domains) versus the 4K offered by traditional VLANs.
2. Allows L2 to be extended anywhere in an IP network.
3. Optimized flooding.

Why VXLAN

• VLAN Scalability - VXLAN extends the L2 Segment ID field to 24-bits, which potentially allows up to 16 million unique L2 segments over the same network.
• L2 Segment Elasticity over L3 Boundary - VXLAN encapsulates an L2 frame in an IP-UDP header, which allows L2 adjacency across router boundaries.
• Leverages multicast in the transport network in order to simulate flooding behavior for broadcast, unknown unicast, and multicast in the L2 segment.
• Leverage Equal Cost Multi-pathing (ECMP) in order to achieve optimal path usage over the transport network.

Configure

Network Diagram

Configurations

These configurations are specific to the VXLAN portion of configuration. Notice that 9396-A and B are in a vPC domain while 3172-A is not. These configurations assume full reachability to all L3 interfaces in the topology with the routing protocol of your choice. Open Shortest Path First (OSPF) was used in this example. It also assumes the multicast routing has been established over these same L3 interfaces.

3172-A

feature ospf
feature pim
feature vn-segment-vlan-based
feature nv overlay

vlan 10
  vn-segment 160010
vlan 20
  vn-segment 160020

interface nve1
  source-interface loopback1
  member vni 160010 mcast-group 203.0.113.1
  member vni 160020 mcast-group 203.0.113.1
  no shutdown
  
interface Ethernet1/3
  no switchport
  ip address 192.168.1.10/30
  ip router ospf 2 area 0.0.0.0
  ip pim sparse-mode

interface loopback1
  ip address 192.168.2.5/32
  ip router ospf 2 area 0.0.0.0
  ip pim sparse-mode

9396-A

Note: When vPCs are used as VTEPs, the secondary IP of the loopback interface is used and shared between the two peers. This is how both peers represent themselves as a single VTEP to the remote NVE peers.

feature ospf
feature pim
feature vn-segment-vlan-based
feature nv overlay

ip pim rp-address 192.168.1.100 group-list 224.0.0.0/4

vlan 1,10,20
vlan 10
  vn-segment 160010
vlan 20
  vn-segment 160020

vpc domain 1
  peer-switch
  peer-keepalive destination 10.122.140.99
  peer-gateway

interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel48
  switchport mode trunk
  vpc 48  

interface nve1
  mtu 9216
  no shutdown
  source-interface loopback1
  member vni 160010 mcast-group 203.0.113.1
  member vni 160020 mcast-group 203.0.113.1
interface Ethernet1/7
  no switchport
  ip address 192.168.1.2/30
  ip router ospf 1 area 0.0.0.0
  ip pim sparse-mode
  no shutdown
  
interface loopback1
  ip address 192.168.2.2/32
  ip address 192.168.2.1/32 secondary
  ip router ospf 1 area 0.0.0.0
  ip pim sparse-mode

9396-B

Note: When vPCs are used as VTEPs, the secondary IP of the loopback interface is used and shared between the two peers. This is how both peers represent themselves as a single VTEP to the remote NVE peers.

feature ospf
feature pim
feature vn-segment-vlan-based
feature nv overlay

ip pim rp-address 192.168.1.100 group-list 224.0.0.0/4

vlan 1,10,20
vlan 10
  vn-segment 160010
vlan 20
  vn-segment 160020
  
vpc domain 1
  peer-switch
  peer-keepalive destination 10.122.140.98
  peer-gateway

interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel48
  switchport mode trunk
  vpc 48
 
interface nve1
  mtu 9216
  no shutdown
  source-interface loopback1
  member vni 160010 mcast-group 203.0.113.1
  member vni 160020 mcast-group 203.0.113.1
    
interface Ethernet1/7
  no switchport
  ip address 192.168.1.6/30
  ip router ospf 1 area 0.0.0.0
  ip pim sparse-mode
  no shutdown
  
interface loopback1
  ip address 192.168.2.3/32
  ip address 192.168.2.1/32 secondary
  ip router ospf 1 area 0.0.0.0
  ip pim sparse-mode

Verify

Use this section to confirm that your configuration works properly.

The Cisco CLI Analyzer (registered customers only) supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.

• show nve peers < --- you can not see any output for this until traffic is initiated from both sides of the overlay
• show nve vni
• show run interface nve1
• show nve internal platform interface detail (9K only)
• show mac address-table
• show ip mroute detail

Example Outputs

These outputs are in a steady state. The VTEP peers have discovered each other and traffic has passed between both in the encap and decap directions.

3172-A

3172-A# show nve peers
Interface          Peer-IP           Peer-State
----------------   ---------------   -------------
nve1               192.168.2.1        Up         

3712-A# show nve vni
Interface          VNI        Multicast-group   VNI State
----------------   --------   ---------------   ---------
nve1               160010     203.0.113.1         Up  
nve1               160020     203.0.113.1         Up            

3172-A# show run interface nve1

!Command: show running-config interface nve1
!Time: Sat Apr 25 15:09:13 2015

version 6.0(2)U5(1)

interface nve1
  source-interface loopback1
  member vni 160010 mcast-group 203.0.113.1
  member vni 160020 mcast-group 203.0.113.1
  no shutdown

3172-A# show nve internal platform interface detail

3172-A# show mac address-table vlan 10
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since first seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10       0000.1111.1111    dynamic   5030       F    F  Eth1/48
* 10       0000.2222.2222    dynamic   5010       F    F  nve1(192.168.2.1)

3172-A# show ip mroute detail 
IP Multicast Routing Table for VRF "default"

Total number of routes: 3
Total number of (*,G) routes: 1
Total number of (S,G) routes: 1
Total number of (*,G-prefix) routes: 1

(*, 231.1.1.1/32), uptime: 3w3d, static(1) pim(0) ip(0) 
  Stats: 15/1539 [Packets/Bytes], 0.000   bps
  Incoming interface: Ethernet1/3, RPF nbr: 192.168.1.9, uptime: 1w0d
  Outgoing interface list: (count: 1)
    loopback1, uptime: 3w3d, static

(192.168.2.5/32, 231.1.1.1/32), uptime: 3w3d, ip(0) mrib(1) pim(1) 
  Stats: 142751/9136064 [Packets/Bytes], 34.133  bps
  Incoming interface: loopback1, RPF nbr: 192.168.2.5, uptime: 3w3d
  Outgoing interface list: (count: 2)
    Ethernet1/3, uptime: 1w0d, pim
    loopback1, uptime: 3w3d, mrib, (RPF)

(*, 232.0.0.0/8), uptime: 3w3d, pim(0) ip(0) 
  Stats: 0/0 [Packets/Bytes], 0.000   bps
  Incoming interface: Null, RPF nbr: 0.0.0.0, uptime: 3w3d
  Outgoing interface list: (count: 0)

9396-A

9396-A# show nve peers
Interface Peer-IP          State LearnType Uptime   Router-Mac       
--------- ---------------  ----- --------- -------- -----------------
nve1      192.168.2.5      Up    DP        2d20h    n/a              

9396-A# show nve vni
Codes: CP - Control Plane        DP - Data Plane          
       UC - Unconfigured         SA - Suppress ARP
       
Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1      160010   203.0.113.1         Up    DP   L2 [10]    
nve1      160020   203.0.113.1         Up    DP   L2 [20]                              

9396-A# show run interface nve1

!Command: show running-config interface nve1
!Time: Sat Apr 25 15:20:45 2015

version 7.0(3)I1(1a)

interface nve1
  mtu 9216
  no shutdown
  source-interface loopback1
  member vni 160010 mcast-group 203.0.113.1
  member vni 160020 mcast-group 203.0.113.1

9396-A# show nve internal platform interface detail 
Printing details of all NVE Interfaces
|======|=========================|===============|===============|=====|=====|
|Intf  |State                    |PriIP          |SecIP          |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1  |UP                       |192.168.2.2    |192.168.2.1    |2    |1    |
|======|=========================|===============|===============|=====|=====|

SW_BD/VNIs of interface nve1:
================================================
|======|======|=========================|======|====|======|
|Sw BD |Vni   |State                    |Intf  |Type|Vrf-ID|
|======|======|=========================|======|====|======|
|10    |160010|UP                       |nve1  |DP  |0     
|20    |160020|UP                       |nve1  |DP  |0     
|======|======|=========================|======|====|======|
Peers of interface nve1:
============================================

peer_ip: 192.168.2.5, peer_id: 1, state: UP MAC-learning: Enabled
active_swbds: 
add_pending_swbds: 
rem_pending_swbds:

9396-A# show mac address-table vlan 10
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+   10     0000.1111.1111   dynamic  0         F      F    nve1(192.168.2.5)
*   10     0000.2222.2222   dynamic  0         F      F    Po48
G    -     7c0e.ceca.f177   static   -         F      F    sup-eth1(R)

9396-A# show ip mroute detail 
IP Multicast Routing Table for VRF "default"

Total number of routes: 4
Total number of (*,G) routes: 1
Total number of (S,G) routes: 2
Total number of (*,G-prefix) routes: 1

(*, 231.1.1.1/32), uptime: 2d21h, nve(1) ip(0) pim(0) 
  Data Created: No
  Stats: 1/64 [Packets/Bytes], 0.000   bps
  Stats: Inactive Flow
  Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.1
  Outgoing interface list: (count: 1)
    nve1, uptime: 2d21h, nve

(192.168.2.1/32, 203.0.113.1/32), uptime: 2d21h, nve(0) ip(0) mrib(0) pim(0) 
  Data Created: Yes
  VXLAN Flags
    VXLAN Encap
  Stats: 1/51 [Packets/Bytes], 0.000   bps
  Stats: Inactive Flow
  Incoming interface: loopback1, RPF nbr: 192.168.2.1
  Outgoing interface list: (count: 0)

(192.168.2.5/32, 203.0.113.1/32), uptime: 2d21h, ip(0) mrib(0) nve(1) pim(0) 
  Data Created: Yes
  Stats: 16474/1370086 [Packets/Bytes], 13.600  bps
  Stats: Active Flow
  Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.1
  Outgoing interface list: (count: 1)
    nve1, uptime: 2d21h, nve

(*, 232.0.0.0/8), uptime: 2d21h, pim(0) ip(0) 
  Data Created: No
  Stats: 0/0 [Packets/Bytes], 0.000   bps
  Stats: Inactive Flow
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 0)

9396-A# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1   
Peer status                       : peer adjacency formed ok      
vPC keep-alive status             : peer is alive                 
Configuration consistency status  : success 
Per-vlan consistency status       : success                       
Type-2 consistency status         : success 
vPC role                          : secondary                     
Number of vPCs configured         : 1   
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans    
--   ----   ------ --------------------------------------------------
1    Po1    up     1,10,20                                                

vPC status
----------------------------------------------------------------------
id   Port   Status Consistency Reason                     Active vlans
--   ----   ------ ----------- ------                     ------------
48   Po48   up     success     success                    1,10

9396-B

9396-B# show nve peers 
Interface Peer-IP          State LearnType Uptime   Router-Mac       
--------- ---------------  ----- --------- -------- -----------------
nve1      192.168.2.5      Up    DP        1w0d     n/a              

9396-B# show nve vni
Codes: CP - Control Plane        DP - Data Plane          
       UC - Unconfigured         SA - Suppress ARP
       
Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1      160010   203.0.113.1         Up    DP   L2 [10]  
nve1      160020   203.0.113.1         Up    DP   L2 [20]                                

9396-B# show run interface nve1

!Command: show running-config interface nve1
!Time: Sat Apr 25 15:23:25 2015

version 7.0(3)I1(1b)

interface nve1
  mtu 9216
  no shutdown
  source-interface loopback1
  member vni 160010 mcast-group 203.0.113.1
  member vni 160020 mcast-group 203.0.113.1
  
9396-B# show nve internal platform interface detail
Printing details of all NVE Interfaces
|======|=========================|===============|===============|=====|=====|
|Intf  |State                    |PriIP          |SecIP          |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1  |UP                       |192.168.2.3    |192.168.2.1    |2    |1    |
|======|=========================|===============|===============|=====|=====|

SW_BD/VNIs of interface nve1:
================================================
|======|======|=========================|======|====|======|
|Sw BD |Vni   |State                    |Intf  |Type|Vrf-ID|
|======|======|=========================|======|====|======|
|10    |160010|UP                       |nve1  |DP  |0     
|20    |160020|UP                       |nve1  |DP  |0     
|======|======|=========================|======|====|======|
Peers of interface nve1:
============================================

peer_ip: 192.168.2.5, peer_id: 1, state: UP MAC-learning: Enabled
active_swbds: 
add_pending_swbds: 
rem_pending_swbds: 

9396-B# show mac address-table vlan 10
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   10     0000.1111.1111   dynamic  0         F      F    nve1(192.168.2.5)
+   10     0000.2222.2222   dynamic  0         F      F    Po48
G    -     58f3.9ca3.64dd   static   -         F      F    sup-eth1(R)

9396-B# show ip mroute detail 
IP Multicast Routing Table for VRF "default"

Total number of routes: 4
Total number of (*,G) routes: 1
Total number of (S,G) routes: 2
Total number of (*,G-prefix) routes: 1

(*, 231.1.1.1/32), uptime: 2w1d, nve(1) ip(0) pim(0) 
  Data Created: No
  VXLAN Flags
    VXLAN Decap
  VPC Flags
    RPF-Source Forwarder
  Stats: 1/64 [Packets/Bytes], 0.000   bps
  Stats: Inactive Flow
  Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.5
  Outgoing interface list: (count: 1)
    nve1, uptime: 2w1d, nve

(192.168.2.1/32, 203.0.113.1/32), uptime: 2w1d, nve(0) ip(0) mrib(0) pim(1) 
  Data Created: Yes
  VXLAN Flags
    VXLAN Encap
  VPC Flags
    RPF-Source Forwarder
  Stats: 5/511 [Packets/Bytes], 0.000   bps
  Stats: Inactive Flow
  Incoming interface: loopback1, RPF nbr: 192.168.2.1
  Outgoing interface list: (count: 1)
    Ethernet1/7, uptime: 1w0d, pim

(192.168.2.5/32, 203.0.113.1/32), uptime: 2w1d, ip(0) mrib(0) pim(0) nve(1) 
  Data Created: Yes
  VXLAN Flags
    VXLAN Decap
  VPC Flags
    RPF-Source Forwarder
  Stats: 86621/7241564 [Packets/Bytes], 13.600  bps
  Stats: Active Flow
  Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.5
  Outgoing interface list: (count: 1)
    nve1, uptime: 2w1d, nve

(*, 232.0.0.0/8), uptime: 2w1d, pim(0) ip(0) 
  Data Created: No
  Stats: 0/0 [Packets/Bytes], 0.000   bps
  Stats: Inactive Flow
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 0)

9396-B# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1   
Peer status                       : peer adjacency formed ok      
vPC keep-alive status             : peer is alive                 
Configuration consistency status  : success 
Per-vlan consistency status       : success                       
Type-2 consistency status         : success 
vPC role                          : primary                       
Number of vPCs configured         : 1   
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans    
--   ----   ------ --------------------------------------------------
1    Po1    up     1,10,20                                                

vPC status
----------------------------------------------------------------------
id   Port   Status Consistency Reason                     Active vlans
--   ----   ------ ----------- ------                     ------------
48   Po48   up     success     success                    1,10

VXLAN Packet Capture

The packet capture (PCAP) is from the previous topology and contains the OSPF hellos, the PIM Joins/Registers, and the VXLAN encapsulated traffic for the toplogy shown in the network diagram. You can notice some Internet Control Message Protocol (ICMP) flags such as 'no response'. This is due to the nature of the monitor session completed on the RP.

The monitor session included interfaces Eth4/17-18 and Eth4/20, so it throws off Wireshark some. The important information is the format and flags.

Note: All encapsulated packets (BUM, or known unicast) are sourced from the VTEP loopback IP destined to the remote VTEP loopback IP. This is the secondary loopback IP on any vPC VTEPs.

BUM (Broadcast, Unknown unicast, Multicast) traffic can be destined to the mcast-group.

Unicast traffic is then destined to the remote VTEP loopback IP.

Troubleshoot

There is currently no specific information available to troubleshoot this configuration.

Related Information

• VXLAN Overview: Cisco Nexus 9000 Series Switches
• Technical Support & Documentation - Cisco Systems