IP tunnels can encapsulate a same-layer or higher layer protocol and transport the result over IP through a tunnel created between two devices.

IP tunnels can encapsulate a same-layer or higher layer protocol and transport the result over IP through a tunnel created between two devices.

IP tunnels can encapsulate a same-layer or higher layer protocol and transport the result over IP through a tunnel created between two devices.

IP tunnels have the following prerequisites:

• You must be familiar with TCP/IP fundamentals to configure IP tunnels.

• You are logged on to the switch.

• You must enable the tunneling feature in a device before you can configure and enable any IP tunnels.

IP tunnels have the following configuration guidelines and limitations:

• Beginning with Cisco NX-OS Release 9.3(3):

  • Total number of 16 GRE/IPIP tunnels are supported on Cisco Nexus 9200, 9300-EX/FX/FX2 switches, and 9500 switches with 9700-EX/FX line cards.

  • Multiple IP-in-IP/GRE tunnel interfaces on a same Cisco Nexus Device can be sourced from or destined to the same IP Address across different VRFs. This is supported on Cisco Nexus 9200 and 9300-EX/FX/FX2 platforms. This is not supported on Cisco Nexus 9500 switches with 9300-GX line cards.

  • More than 1 and up to 16 IPIP Decap-any tunnels are supported - 1 decap-any tunnel per VRF. This is supported on Cisco Nexus 9200, and 9300-EX/FX/FX2 platforms. This is not supported on Cisco Nexus 9300-GX and 9500 platforms where only 1 decap-any tunnel is supported per device.

  • VRF membership of the interface, where IPIP/GRE encapsulated packets are ingressing on the terminating node, should match with the tunnel transport VRF for tunnel to correctly terminate the packets.

  • The IPIP/GRE packet coming on a non default VRF may get terminated by a tunnel in default VRF if the packet outer header matches with the tunnel source and the tunnel destination.

• Beginning with Cisco NX-OS Release 9.3(5), the following features are supported on N9K-C9316D-GX, N9K-C93600CD-GX and N9K-C9364C-GX switches:

  • A total number of 16 GRE/IPIP tunnels.

  • Multiple IP-in-IP/GRE tunnel interfaces on a same Cisco Nexus device can be sourced from or destined to the same IP address across different VRFs

  • More than 1 and upto 16 IPIP Decap-any tunnels are supported -- 1 decap-any tunnel per VRF.

• GRE Stripping is not supported on N9K-C9364C and N9K-C9332C family switches and in N9K-C9508-FM-E2, N9K-C9516-FM-E2 modules.

• Guidelines for source-direct and ipv6ipv6-decapsulate-any options for tunnels:

  • The source-direct command is supported on the Cisco Nexus 9500 platform switches with the Application Spine Engine (ASE) and the Leaf Spine Engine (LSE).

    Cisco Nexus 9500 platform switches with the Network Forwarding Engine (NFE) do not support the tunnel source direct command.

    The tunnel source direct command with the tunnel mode ipv6ipv6 decapsulate-any command on the Cisco Nexus 9500 platform switches is only supported in the MPLS heavy routing template.

  • The IP tunnel supports the tunnel source CLI command with interface, IPv4 address, IPv6 address, or IPv4 prefix. You can configure IP-in-IP tunnel decapsulation on directly connected IP addresses (for example, physical interface, port-channel, loopback, and SVI) using the new tunnel source direct CLI command. You can select the IP ECMP links when there are multiple IP links between the two switches. A single tunnel interface can decapsulate the tunneled packets whose outer destination IP is any of the IPv4 or IPv6 address that is locally configured and it is operationally Up in the switch.

  • Currently, tunnel mode ipip decapsulate-any is supported for decapsulating IPv4 payload over IPv4 transport (IPv4inIPv4 packets). tunnel mode ipv6ipv6 decapsulate-any command is introduced to support IPv6 payload over IPv6 transport (IPv6inIPv6 packets).

  • The tunnel source direct and tunnel mode ipv6ipv6 decapsulate-any CLI commands are not supported on Cisco Nexus 9500 platform switches with the Network Formation Engine (NFE).

  • The tunnel source direct CLI command is supported only when an administrator uses the IP-in-IP decapsulation to source route the packets through the network. The source-direct tunnel is always operationally Up unless it is administratively shut down. The directly connected interfaces are identified using the show ip route direct CLI command.

  • The tunnel source direct CLI command is supported only on decapsulate-any tunnel modes, for example, tunnel mode ipip decapsulate-any and tunnel mode ipv6ipv6 decapsulate-any .

  • Auto-recovery for source-direct is not supported.

  • For ipv6ipv6 decapsulate-any, inter-VRF is not supported. The tunnel interface VRF (iVRF) and tunnel transport or forwarding VRF (fVRF) must be the same. Only one decapsulate-any tunnel (irrespective of VRF) can be present in Cisco Nexus 9200, 9300-EX, and 9300-FX platform switches; Cisco Nexus 9500 platform modular switches with EX and FX line cards.

  • To enable IPv6 on ipv6ipv6 decap-any tunnel interface, configure a valid IPv6 address or configure ipv6 address use-link-local-only under the tunnel interface.

• From Cisco NX-OS Release 10.4(1)F, you can configure loopback IP address as tunnel source IP address using the tunnel source CLI command with loopback interface.

• See the following hardware limitations on the maximum sources that can be accommodated on a source direct tunnel and the related behavior:

  • Source direct tunnel is now supported for Cisco Nexus 9000 Series switches with Network Forwarding Engine (NFE), Application Spine Engine (ASE), and Leaf Spine Engine (LSE). Most of the limitations are only in case of scaled SIP (number of total IP/IPv6 addresses on the interfaces (L3, sub-interface, PC, PC-sub interfaces, loopback, SVI, and any secondary IP/IPv6 addresses.)

    See the following sample use cases.

    • Use Case 1: Non-deterministic behavior of which SIP gets installed if the number of IP/IPv6 interface scale is more.

      Both the switches have 512 entries for tunnel SIP. With tunnel source, direct any IP or IPv6 address w.r.t ipip or ipv6ipv6 decap any with tunnel source gets installed in the above table.

      The insertion of these entries is on a first come first serve basis without any CLI command to control which interface IP addresses get installed. If the system has more number of IP/IPv6 interfaces to be installed, the behavior is non-deterministic (The behavior can change across reload with interface flaps.)

    • Use Case 2: The scale numbers are different in both switches and each has its own advantages and disadvantages.

      IPv4 individual scale can be more (up to 512) in case of switches with NFE but it is shared with IPv6. In the switches with ASE and LSE, the IPv4 individual scale can be 256 but it is not shared with IPv6.

      Whenever the tunnel decap table gets filled, the TABLE_FULL error is displayed. If some entry gets deleted after the table gets full, the table full error is cleared.

      Table 1. Scale Numbers

      Commands	Switches with NFE: Table size 512, v4 takes 1 entry, v6 takes 4 entries	Switches with ASE and LSE: Table size 512, v4 takes 1 entry, v6 takes 2 entries (paired index)
      IPIP decap any with tunnel source direct	Shared between v4 and v6, v6 takes 4 entries v4 + 4 *v6 =512 Maximum entries can be 512 with no v6 entries	Dedicated 256
      IPv6IPv6 decap any with tunnel source direct	Shared between v4 and v6, v6 takes 4 entries v4 + 4 *v6 =512 Maximum entries can be 128 with no v4 entries	Dedicated 128

    • Use Case 3: Auto-recovery is not supported.

      If any entry does not get installed in the hardware due to exhaustion of above table, removal of an already installed IP/IPv6 from interfaces does not automatically trigger the addition of the failed SIP in the table though the table has space now. You need to flap the tunnel interface or IP interface to get them installed.

      However, if an entry does not get installed in the hardware due to a duplicate entry (if there was already a decap-any with one source present and now the source direct tunnel CLI command is configured, there is a duplicate entry for the prior source configured) that was taken care of by removing the entry only when both the tunnels get deleted.

  • For Cisco Nexus 9000 Series switches with Network Forwarding Engine (NFE) and Application Spine Engine (ASE), the syslog is different as the dedicated IPv4 and IPv6 decap entries are carved in the syslog. If the tunnel-decap-table is full, the user gets a syslog as follows:

    
    2017 Apr 6 12:18:04 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV4_TUNNEL_DECAP_TABLE_FULL: IPv4 tunnel decap hardware table full. 
    IP tunnel decapsulation may not work for some GRE/IPinIP traffic
    
    2017 Apr 6 12:18:11 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV6_TUNNEL_DECAP_TABLE_FULL: IPv6 tunnel decap hardware table full. 
    IP tunnel decapsulation may not work for some GRE/IPinIP traffic

    If the table is full and once some entry gets deleted from the table (due to an interface being operationally down or removal of IP address), the clear syslog for the table is displayed. Note that the deletion of the tunnel removes all the entries that are added as part of that tunnel.

    
    2017 Apr 5 13:29:25 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV4_TUNNEL_DECAP_TABLE_FULL_CLRD: IPv4 tunnel decap hardware table full exception cleared
    
    2017 Apr 4 19:41:22 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV6_TUNNEL_DECAP_TABLE_FULL_CLRD: IPv6 tunnel decap hardware table full exception cleared

• IP-in-IP tunnel decapsulation is supported on IPv6 enabled networks.

  
  !
  interface tunnel 1
    ipv6 address use-link-local-only             <<< enable IPv6
    tunnel mode ipv6ipv6 decapsulate-any
    tunnel source direct
    description IPinIP Decapsulation Interface
    mtu 1476
    no shutdown

• The show commands with the internal keyword are not supported.

• Cisco NX-OS supports only the following protocols:

  • IPv4 passenger protocol.

  • GRE carrier protocol.

• Cisco NX-OS supports the following maximum number of tunnels prior to Cisco NX-OS Release 9.3(3):

  • IP tunnels - 8 tunnels.

  • GRE and IP-in-IP regular tunnels - 8 tunnels. (6.1(2)I3(4) and later)

    GRE and IP-in-IP regular tunnels - 8 tunnels.

• Beginning with Cisco NX-OS Release 9.3(3), the maximum number of supported GRE and IP-in-IP regular tunnels is 16.

• IP tunnels do not support access control lists (ACLs) or QoS policies.

• Cisco NX-OS supports the GRE header defined in IETF RFC 2784. Cisco NX-OS does not support tunnel keys and other options from IETF RFC 1701.

• Cisco NX-OS does not support GRE tunnel keepalives.

• All unicast routing protocols are supported by IP tunnels.

• The IP tunnel interface cannot be configured to be a span source or destination.

• IP tunnels do not support PIM or other Multicast features and protocols. (6.1(2)I3(4) and later)

  IP tunnels do not support PIM or other Multicast features and protocols.

• The selection of IP-in-IP tunnel based on the PBR policy is not supported. (6.1(2)I3(4) and later)

  The selection of IP-in-IP tunnel based on the PBR policy is not supported.

• Beginning with Cisco NX-OS Release 10.3(3)F, the selection of GRE or IP-in-IP tunnel destination based on the PBR policy is supported.

• IP tunnels are supported only in the default system routing mode and not in other modes. (6.1(2)I3(4) and later)

  IP tunnels are supported only in the default system routing mode and not in other modes.

• GRE and IP-in-IP tunnels do not support Path MTU Discovery (PMTUD).

• When configuring a tunnel interface to ipip mode , the maximum mtu value is9196.

  When downgrading from NX-OS 9.2(1)to an earlier release, with a tunnel interface in ipip mode that has an mtu value of 9196, the mtu configuration is lost as a result of the downgrade operation. As a best practice, adjust the mtu value to 9192 before commencing the downgrade to avoid losing the mtu configuration.

• When configuring a tunnel interface to ipip mode , the default mtu value is1480.

  When downgrading from NX-OS 9.2(1) or later release to an earlier release, with a tunnel interface in ipip mode with no explicit mtu configuration, the mtu value changes as a result of the downgrade operation from 1480 to 1476. As a best practice, adjust the mtu value to 1476 before commencing the downgrade to avoid any changes to the mtu value.

  When upgrading to NX-OS 9.2(1) or later release,, with a tunnel interface in ipip mode with no explicit mtu configuration, the mtu value changes as a result of the upgrade operation from 1476 to 1480. As a best practice, adjust the mtu value to 1480 before commencing the upgrade to avoid any changes to the mtu value.

• On Cisco Nexus 9200 Series switches, GRE packets that are received on an IP-in-IP tunnel are not dropped as expected and are instead forwarded to the packet destination.

• Tx packets originating from the switch, such as control pkts, are not included in Tx statistics.

• Tunnel destinations that are reachable over another tunnel are not supported.

• The consistency checker is not supported for routes over a tunnel.

• Non-IP routing protocols, such as isis, are not supported over IP-in-IP tunnels.

• RFC5549 is not be supported over tunnels.

• BGP adjacency over tunnel is not supported in a scenario where the tunnel interface and tunnel source are in same VRF (example: VRF-A) and tunnel destination is reachable with route-leak from opposite end (example: via VRF-B)

• You can configure only 8 GRE tunnels per device.

• GRE tunnels does not support RACLs.

• GRE tunnel must belong to the same VRF as the underlying routing infrastructure. In other words, the tunnel use-vrf and vrf member values must always match on the same GRE tunnel.

• GRE tunnels supports only limited traffic (ingress or egress) counters.

• Layer 3 FEX interfaces not are allowed as tunnel source and/or destination

• Double encapsulation is not allowed on GRE tunnels.

• BFD is not supported on GRE tunnels.

• On Cisco Nexus N9K-C9300-GX platforms, GRE/IPinIP tunnel interfaces cannot co-exist with Dot1Q tagged L2 bcast or 1Q tagged L2/L3 mcast transit traffic. When you configure feature tunnel on Cisco Nexus N9300-GX platform, the following warning is displayed and you get a syslog message warning you. You should not configure feature tunnel if you have Dot1Q tagged L2 bcast or 1Q tagged L2/L3 mcast transit traffic on the device.

  N9300-GX(config)# feature tunnel
  WARN:GRE/IPinIP cannot coexist with 1Q tagged L2 bcast or 1Q tagged L2/L3 mcast transit packets on this
  platform
  N9300-GX(config)#
  N9300-GX(config)# show logging logfile
  2019 Dec 12 00:41:08 N9300-GX %TUNNEL-2-TRAFFIC_WARNING: GRE/IPinIP cannot coexist with 1Q
  tagged L2 bcast or 1Q tagged L2/L3 mcast transit packets on this platform
  N9300-GX(config)#

• The feature feature tunnel on the Cisco Nexus 9000 switches cannot co-exist with the VXLAN feature, feature nv overlay.

• Cisco Nexus 9200, 9300-EX, 9300-FX, 9300-FX2 series switches and Cisco Nexus 9500 platform switches with 9700-EX/FX line cards may not have multiple tunnel interfaces in a single VRF that are sourced from or destined to the same IP address. For example, a device may not have tunnel 0 and tunnel 1 interfaces in the default VRF that are sourced from the same IP address or interface.

• Cisco Nexus 9300-EX, 9300-FX, 9300-GX and Nexus 9500 platform switches in vPC can act as GRE Tunnel endpoints for their respective tunnels. However, the tunnel destination can not be through a vPC.

• Beginning with Cisco NX-OS Release 10.3(3)F, the PBR policy on a tunnel interface is supported only for gre ip, ipip ip, and ipip decapsulate-any ip modes on Cisco Nexus 9300-FX2/FX3/GX/GX2 platform switches .

• Beginning with Cisco NX-OS Release 10.4(1)F, GRE tunnel is supported on Cisco Nexus 9332D-H2R switch.

• Beginning with Cisco NX-OS Release 10.4(2)F, GRE tunnel is supported on Cisco Nexus 93400LD-H1 switch.

The following table lists the default settings for IP tunnel parameters.

To verify the IP tunnel configuration information, perform one of the following tasks:

The following example shows a simple GRE tunnel. Ethernet 1/2 is the tunnel source for router A and the tunnel destination for router B. Ethernet interface 2/1 is the tunnel source for router B and the tunnel destination for router A.

Router A:

feature tunnel
interface tunnel 0
ip address 209.165.20.2/8
tunnel source ethernet 1/2
tunnel destination 192.0.2.2
tunnel mode gre ip
tunnel path-mtu-discovery 25 1500

interface ethernet 1/2
ip address 192.0.2.55/8

Router B:

feature tunnel
interface tunnel 0
ip address 209.165.20.1/8
tunnel source ethernet 2/1
tunnel destination 192.0.2.55
tunnel mode gre ip

interface ethernet 2/1
ip address 192.0.2.2/8