Introduction

This document describes how to configure and troubleshoot QoS over tunnel GRE in Nexus 9300 (EX-FX-GX) model.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• QoS
• Tunnel GRE
• Nexus 9000

Components Used

The information in this document is based on these software and hardware versions:

• Hardware: N9K-C9336C-FX2
• Version: 9.3(8)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

You can use generic routing encapsulation (GRE) as the carrier protocol for a variety of passenger protocols.

You see in the image that the IP tunnel components for a GRE tunnel. The original passenger protocol packet becomes the GRE payload and the device adds a GRE header to the packet.

The device then adds the transport protocol header to the packet and transmits it.

Traffic is processed based on how you classify it and the policies that you create and apply to traffic classes.

To configure QoS features, use these steps:

1. Classes are created that classify ingress packets to the nexus that match criteria such as IP address or QoS fields.

2. Creates policies that specify the actions to be taken on traffic classes, such as watch, mark, or discard packets.

3. Apply policies to a port, port channel, VLAN, or subinterface.

Commonly Used DSCP Values

Network Diagram

Configure

The goal of the configuration of QoS over tunnel GRE is to set a DSCP for traffic of a certain VLAN to pass through the GRE Tunnel between N9K-EX1 and N9K-EX2.

The Nexus encapsulates the traffic and sends it on the Tunnel GRE without loss of QoS marking as you did previously in the VLAN for the DSCP value, for this case the value of DSCP AF-11 is used for VLAN 9.

Host-A

interface Ethernet1/3
  switchport
  switchport access vlan 9
  no shutdown

interface Vlan9
  no shutdown
  ip address 192.168.9.9/24

Host-B

interface Ethernet1/3
switchport
switchport access vlan 18
no shutdown

interface Vlan18
no shutdown
ip address 192.168.18.18/24

N9K-EX1 interfaces configuration

interface Ethernet1/1
ip address 10.10.10.1/30
no shutdown

interface Ethernet1/3
switchport
switchport access vlan 9
no shutdown

interface Tunnel1
ip address 172.16.1.1/30
tunnel source Ethernet1/1
tunnel destination 10.10.10.2
no shutdown

interface Vlan9
no shutdown
ip address 192.168.9.1/24

N9K-EX1 Routing configuration

ip route 0.0.0.0/0 Tunnel

N9K-EX1 QoS configuration

Since QoS is not supported on the GRE tunnel interface in NXOS, it is necessary to configure and apply the service policy in the VLAN configuration. As you can see, first create the ACL to match the source and destination, then set the QoS configuration with the desired DSCP, finally use the service policy the VLAN configuration.

ip access-list TAC-QoS-GRE
10 permit ip any 192.168.18.0/24
class-map type qos match-all CM-TAC-QoS-GRE
match access-group name TAC-QoS-GRE
policy-map type qos PM-TAC-QoS-GRE
class CM-TAC-QoS-GRE
set dscp 10

vlan configuration 9
service-policy type qos input PM-TAC-QoS-GRE

N9K-EX2 Interfaces configuration

interface Ethernet1/1
ip address 10.10.10.2/30
no shutdown

interface Ethernet1/3
switchport
switchport access vlan 18
no shutdown

interface Tunnel1
ip address 172.16.1.2/30
tunnel source Ethernet1/1
tunnel destination 10.10.10.1
no shutdown

interface Vlan18
no shutdown
ip address 192.168.18.1/24

N9K-EX2 Routing configuration

ip route 0.0.0.0/0 Tunnel

Troubleshoot

Tunnel Verification

Both commands:

• show ip interface brief
• show interface tunnel 1 brief

Displays if the tunnel is Up.

N9K-EX1# show ip interface brief

IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Vlan9 192.168.9.1 protocol-up/link-up/admin-up 
Tunnel1 172.16.1.1 protocol-up/link-up/admin-up 
Eth1/1 10.10.10.1 protocol-up/link-up/admin-up 


N9K-EX1# show interface tunnel 1 brief

--------------------------------------------------------------------------------
---------------------------
Interface Status IP Address 
Encap type MTU 
--------------------------------------------------------------------------------
---------------------------
Tunnel1 up 172.16.1.1/30 
GRE/IP 1476 

Both commands

• show interface tunnel 1
• show interface tunnel 1 counters

Displays similar information such as as received and transmitted packets.

N9K-EX1# show interface tunnel 1
Tunnel1 is up
Admin State: up 
Internet address is 172.16.1.1/30 
MTU 1476 bytes, BW 9 Kbit
Tunnel protocol/transport GRE/IP
Tunnel source 10.10.10.1 (Ethernet1/1), destination 10.10.10.2
Transport protocol is in VRF "default"
Tunnel interface is in VRF "default"
Last clearing of "show interface" counters never
Tx
3647 packets output, 459522 bytes
Rx
3647 packets input, 459522 bytes

N9K-EX1# show interface tunnel 1 counters

--------------------------------------------------------------------------------
--
Port InOctets InUcastPk
ts
--------------------------------------------------------------------------------
--
Tunnel1 459522 36
47

--------------------------------------------------------------------------------
--
Port InMcastPkts InBcastPk
ts
--------------------------------------------------------------------------------
--
Tunnel1 -- 
--

--------------------------------------------------------------------------------
--
Port OutOctets OutUcastPk
ts
--------------------------------------------------------------------------------
--
Tunnel1 459522 36
47

--------------------------------------------------------------------------------
--
Port OutMcastPkts OutBcastPk
ts
--------------------------------------------------------------------------------
--
Tunnel1 -- 
--
N9K-EX1#

Traffic Captures

SPAN Captures

This image shows the capture of the ARP request at the entry of the Interface Ethernet 1/3 on the N9K-EX1 switch. You can see that the traffic is not marked with the DSCP (AF11) you want to use yet since the capture is at the input of the switch.

The image shows the capture of the ARP request at the entry of the Interface Ethernet 1/1 on the N9K-EX2 switch. You can see that the traffic already has the DSCP AF11 value that you need to use. You also notice that the packet is encapsulated by the tunnel that is configured between the two Nexus.

The image shows the capture of the ARP reply at the output of the interface Ethernet 1/3 on the N9K-EX1 switch. You can see that the traffic still has the DSCP AF11 value that you need to use. You also notice that the packet is not encapsulated by the tunnel that is configured between the two Nexus.

This image shows the capture of the ARP reply at the output of the Interface Ethernet 1/1 on the N9K-EX2 switch. You can see that the traffic still has the DSCP AF11 value that you need to use. You also notice that the packet is encapsulated by the tunnel that is configured between the two Nexus.

It is important to note that the packet captures do not show the tunnel IP for encapsulation since the Nexus uses the physical ones. This is the natural behavior of the Nexus when using GRE tunneling since they use the physical ips to route the packages.

ELAM Capture

You use the ELAM capture on N9KEX-2 with in-select 9 to see the outer l3 and inner l3 header. You must filter by the source and target IP.

debug platform internal tah elam 
trigger init in-select 9
reset 
set inner ipv4 src_ip 192.168.9.9 dst_ip 192.168.18.18 
start 
report 

You can verify that the Nexus receives the packet via interface 1/1. Also, you see the outer l3 header is the physical IP address of the interfaces that are directly connected and the l3 inner header has the IPs of the host A and host B.

SUGARBOWL ELAM REPORT SUMMARY
slot - 3, asic - 1, slice - 0
============================

Incoming Interface: Eth1/1
Src Idx : 0x41, Src BD : 4433
Outgoing Interface Info: dmod 2, dpid 10
Dst Idx : 0x3, Dst BD : 18

Packet Type: IPv4

Outer Dst IPv4 address: 10.10.10.2
Outer Src IPv4 address: 10.10.10.1
Ver = 4, DSCP = 10, Don't Fragment = 0
Proto = 47, TTL = 255, More Fragments = 0
Hdr len = 20, Pkt len = 108, Checksum = 0x3d7a

Inner Payload
Type: IPv4

Inner Dst IPv4 address: 192.168.18.18
Inner Src IPv4 address: 192.168.9.9

L4 Protocol : 47
L4 info not available

Drop Info:
----------

LUA:
LUB:
LUC:
LUD:
Final Drops:

Troubleshooting QoS

You can check the QoS configuration as shown .

N9K-EX1# show running-config ipqos

!Command: show running-config ipqos
!Running configuration last done at: Thu Apr 4 11:45:37 2024
!Time: Fri Apr 5 11:50:54 2024

version 9.3(8) Bios:version 08.39 
class-map type qos match-all CM-TAC-QoS-GRE
match access-group name TAC-QoS-GRE
policy-map type qos PM-TAC-QoS-GRE
class CM-TAC-QoS-GRE
set dscp 10

vlan configuration 9
service-policy type qos input PM-TAC-QoS-GRE

You can display the QoS policies configured on the specified VLAN, and also the packets that are matching with the ACL associated to the policy-map.

N9K-EX1# show policy-map vlan 9


Global statistics status : enabled

Vlan 9

Service-policy (qos) input: PM-TAC-QoS-GRE 
SNMP Policy Index: 285219173

Class-map (qos): CM-TAC-QoS-GRE (match-all)

Slot 1
5 packets 
Aggregate forwarded :
5 packets 
Match: access-group TAC-QoS-GRE
set dscp 10

You can also clear the QoS statistics with the command shown here.

N9K-EX1# clear qos statistics

Verify ACL programmed in software.

N9K-EX1# show system internal access-list vlan 9 input entries detail

slot 1
=======


Flags: F - Fragment entry E - Port Expansion
D - DSCP Expansion M - ACL Expansion
T - Cross Feature Merge Expansion
N - NS Transit B - BCM Expansion C - COPP


INSTANCE 0x2
---------------

Tcam 1 resource usage:
----------------------
LBL B = 0x1
Bank 2
------
IPv4 Class
Policies: QoS 
Netflow profile: 0
Netflow deny profile: 0
Entries: 
[Index] Entry [Stats]
---------------------
[0x0000:0x0000:0x0700] permit ip 0.0.0.0/0 192.168.18.0/24 [5]

Verify ACL programmed in hardware.

N9K-EX1# show hardware access-list vlan 9 input entries detail

slot 1
=======


Flags: F - Fragment entry E - Port Expansion
D - DSCP Expansion M - ACL Expansion
T - Cross Feature Merge Expansion
N - NS Transit B - BCM Expansion C - COPP


INSTANCE 0x2
---------------

Tcam 1 resource usage:
----------------------
LBL B = 0x1
Bank 2
------
IPv4 Class
Policies: QoS 
Netflow profile: 0
Netflow deny profile: 0
Entries: 
[Index] Entry [Stats]
---------------------
[0x0000:0x0000:0x0700] permit ip 0.0.0.0/0 192.168.18.0/24 [5]

With the command shown here, you can verify the ports that are using the VLAN. In this example, it would be VLAN ID 9, and you can also note the QoS policy that is in use.

N9K-EX1# show system internal ipqos vlan-tbl 9

Vlan range asked: 9 - 9

=================================================

Vlan: 9, pointer: 0x132e3eb4, Node Type:  VLAN

IfIndex array:

    alloc count:    5, valid count:    1, array ptr : 0x13517aac          0: IfI

ndex: 0x1a000400 (Ethernet1/3)     Policy Lists (1): Flags: 01

        Type: INP QOS, Name: PM-TAC-QoS-GRE, Ghost Id: 0x45001c7, Real Id: 0x450

01c8

                       Defnode Id: 0x45001c9




=================================================

N9K-EX1#