Configure Switched Port Analyzer on ACI

Introduction

This document describes how to configure Switched Port Analyzer (SPAN) on Cisco Application Centric Infrastructure (ACI). 

Background Information

In general, there are three types of SPAN. Local SPAN, Remote SPAN (RSPAN) and Encapsulated Remote SPAN (ERSPAN). The differences between these SPANs are mainly the destination of copy packets. Cisco ACI supports Local SPAN and ERSPAN.

Note: This document assumes that readers are already familiar with SPAN in general such as Local SPAN and ERSPAN differences.

SPAN Type in Cisco ACI

Cisco ACI has three types of SPAN;  Fabric SPAN, Tenant SPAN and Access SPAN. The difference between each SPANs is the source of copy packets.

As mentioned previously,

• Fabric SPAN  is to capture packets that come in and go out from interfaces between Leaf and Spine switches.
• Access SPAN is to capture packets that come in and go out from interfaces between Leaf switches and external devices.
• Tenant SPAN  is to capture packets that come in and go out from EndPoint Group (EPG) on ACI Leaf switches.

This SPAN name corresponds to where to be configured on Cisco ACI GUI.

• Fabric SPAN is configured under Fabric > Fabric Policies
• Access SPAN is configured under Fabric > Access Policies
• Tenant SPAN is configured under Tenants > {each tenant}

As for the destination of each SPAN, only Access SPAN is capable of both Local SPAN and ERSPAN. The other two SPAN (Fabric and Tenant) are only capable of ERSPAN.

Limitations and Guidelines

Please review the Limitations & Guidelines from Cisco APIC Troubleshooting Guide. It is mentioned in Troubleshooting Tools and Methodology > Using SPAN.

Configuration

This section introduces brief examples that relate to the configuration for each SPAN Type. There are specific sample cases on how to select the span type in the later section.

SPAN Configuration is also described in Cisco APIC Troubleshooting Guide: Troubleshooting Tools and methodology > Using SPAN.

The UI can appear different than the current versions but the config approach is the same.

Access SPAN (ERSPAN)

Sample Topology

Configuration Example

Where:

Navigate to FABRIC > ACCESS POLICIES > Troubleshoot Policies > SPAN.

• SPAN Source Groups
• SPAN Destination Groups

SPAN Source Group ties Destination and Sources.

How:

1. Create SPAN Source Group (SRC_GRP1).
2. Create SPAN Source (SRC1) under SPAN Source Group (SRC_GRP1).
3. Configure these parameters for SPAN Source (SRC1).
   - Direction - Source EPG (option)
   - Source Paths (could be multiple interfaces)
   
   

   Note: Please refer to the picture for details of each parameter.

4. Create SPAN Destination Group (DST_EPG).
5. Create SPAN Destination (DST).
6. Configure these parameters for SPAN Destination (DST)
   - Destination EPG
   - Destination IP
   - Source IP/Prefix    (This can be any IP. If the prefix is used, node-id of the source node is used for the undefined bits. For example, prefix: 1.0.0.0/8 on node-101 => src IP 1.0.0.101)
     - Other parameters can be left as default
   
   

   Note: Please refer to the picture for details of each parameter.

7. Ensure the SPAN Destination Group is tied to an appropriate SPAN Source Group.
8. Ensure Admin Stateis Enabled.
   
   

   Note: SPAN stops when you select Disabled on this Admin State. There is no need to delete all policies if you re-use them later.

Also please ensure that the destination IP for ERSPAN is learned as an endpoint under the specified destination EPG. In the previously mentioned example, 192.168.254.1 has to be learned under Tenant TK > Application profile SPAN_APP > EPG SPAN. Or the destination IP can be configured as a static EndPoint under this EPG if the destination device is a silent host.

Access SPAN (Local)

Sample Topology

Configuration Example

• Where:

Fabric > ACCESS POLICIES > Troubleshoot Policies > SPAN

- SPAN Source Groups

- SPAN Destination Groups

SPAN Source Group ties Destination  and Sources.



• How :

1. Create SPAN Source Group (SRC_GRP1)
2. Create SPAN Source(SRC1) under SPAN Source Group (SRC_GRP1)
3. Configure these parameters for SPAN Source (SRC1)
   - Direction
   - Source EPG     (option)
   - Source Paths   (could be multiple interfaces)
   ※ please refer to the picture for details of each parameter.
4. Create SPAN Destination Group(DST_Leaf1)
5. Create SPAN Destination(DST)
6. Configure these parameters for SPAN Destination (DST)
   - Destination interface and node.
7. Ensure the SPAN Destination Group is tied to an appropriate SPAN Source Group.

8. Ensure Admin State is Enabled.
   ※ SPAN stops when you select Disabled on this Admin State. There is no need to delete all policies if you re-use them later.

The destination interface does not require any configuration by Interface Policy Groups. It works when you plug a cable into the interface on ACI Leaf.

Limitations:

• For Local SPAN, a destination interface and source interfaces must be configured on the same Leaf.
• The destination interface does not require it to be on an EPG as long as it is UP.
• When the virtual Port-Channel (vPC) interface is specified as a source port, Local SPAN cannot be used 
  However, there is a workaround. On a first-generation leaf, an individual physical port that is a member of vPC or PC can be configured as a SPAN source. With this Local SPAN can be used for traffic on vPC ports.
  This option, however, is not available on a second-generation leaf (CSCvc11053). Instead, support for SPAN on "VPC component PC" was added viaCSCvc44643in 2.1(2e), 2.2(2e) and forward. With this, any generation leaf can configure a port channel, which is a member of vPC, as a SPAN source. This allows any generation leaf to use Local SPAN for traffic on vPC ports.
• Specifying the individual ports of a port channel on second-generation leaves cause only a subset of the packets to be spanned (also due toCSCvc11053).
• PC and vPC cannot be used as the destination port for Local SPAN. From 4.1(1), the PC can be used as a destination port for Local SPAN.

Access SPAN  - With ACL Filters

You can use ACL filters on access span sources. This feature provides the ability to SPAN a particular flow or flow of traffic in/out of a SPAN source.
Users can apply the SPAN Acl(s) to a source when there is a need to SPAN flow specific traffic.
It is not supported in Fabric SPAN and Tenant Span source groups/sources.
Care must be taken when you add filter entries in a filter group since it could add tcam entries for every source that currently uses the filter group.

A Filter Group can be associated to:
-Span Source: the filter group is used to filter traffic on ALL interfaces defined under this Span Source.
-Span Source Group: the filter group (say x) is used to filter traffic on ALL interfaces defined under each of the Span Source(s) of this Span Source Group.

In this configuration snapshot, the filter group is applied to the Span source group.

In the case where a particular Span Source already associates with a Filter Group (say y), that filter group (y) is used instead to filter group on all interfaces under this specific Span Source

- A Filter group that is applied at a source group automatically applies to all sources in that source group.
- A Filter group that is applied at a source is applicable to that source only.
- A filter group is applied at both the source group and a source in that source group, the filter group applied at the source takes precedence.
- A filter group applied at a source is deleted, filter group applied at the parent source group is automatically applied.
- A filter group applied at a source group is deleted, it is deleted from all sources currently that inherit in that source group.

Tenant SPAN (ERSPAN)

Sample Topology

Configuration Example

• Where:

Tenants > {tenant name} > Troubleshoot Policies > SPAN

- SPAN Source Groups

- SPAN Destination Groups

※ SPAN Source Group ties Destination and Sources.

• How:

1. Create SPAN Source Group  (SRC_GRP)
2. Create SPAN Source (SRC_A) under SPAN Source Group (SRC_GRP)
3. Configure these parameters for SPAN Source (SRC_A)
        - Direction
        - Source EPG
   ※ Please refer to the picture for details of each parameter.
4. Create SPAN Destination Group (DST_GRP)
5. Create SPAN Destination (DST_A)
6. Configure these parameters for SPAN Destination(DST_A)
        - Destination EPG
        - Destination IP
        - Source IP/Prefix
        - Other parameters can be left as default
   ※ Please refer to the picture for details of each parameter.
7. Ensure SPAN Destination Group is tied to an appropriate SPAN Source Group.
8. Ensure Admin State is Enabled.
   ※ SPAN stops when you select Disabled on this Admin State. There is no need to delete all policies if you re-use them later.

Fabric SPAN (ERSPAN)

Sample Topology

Configuration Example

• Where:

Fabric > FABRIC POLICIES > Troubleshoot Policies > SPAN

- Fabric

- SPAN Destination Groups

※ SPAN Source Group ties Destination and Sources

• How:

1. Create SPAN Source Group  (SRC_GRP)
2. Create SPAN Source (SRC_A) under SPAN Source Group (SRC_GRP)
3. Configure these parameters for SPAN Source (SRC_A)
        - Direction
        - Private Network     (option)
        - Bridge Domain      (option)
        - Source Paths        (could be multiple interfaces)
   ※ please refer to the picture for details of each parameter.
4. Create SPAN Destination Group (DST_GRP)
5. Create SPAN Destination (DST_A)
6. Configure these parameters for SPAN Destination (DST_A)
        - Destination EPG
        - Destination IP
        - Source IP/Prefix
        - Other parameters can be left as default
   ※ please refer to the picture for details of each parameter.
7. Ensure SPAN Destination Group is tied to an appropriate SPAN Source Group.
8. Ensure Admin State is Enabled.
   ※ SPAN stops when you select Disabled on this Admin State. There is no need to delete all policies if you re-use them later.

Although it is described in a later section "ERSPAN Version (type)", you can tell ERSPAN version II is used for Fabric SPAN and version I is used for Tenant and Access SPAN.

GUI Verification

• Verification of SPAN Configuration Policy

1. Fabric > ACCESS POLICIES > Troubleshoot Policies > SPAN > SPAN Source Groups > Operational tab
2. Fabric > FABRIC POLICIES > Troubleshoot Policies > SPAN > SPAN Source Groups > Operational tab
3. Tenants > {tenant name} > Troubleshoot Policies > SPAN > SPAN Source Groups > Operational tab

Please ensure Operational State is up.

• Verification on SPAN Session on the node itself

1. Double-click on each session from SPAN Configuration Policyor Fabric > INVENTORY > Node > Span Sessions > { SPAN session name }

Please ensure Operational State is up.

SPAN Session naming convention:

     - Fabric SPAN: fabric_xxxx

     - Access SPAN: infra_xxxx
     - Tenant SPAN: tn_xxxx

Select the ACI SPAN Type

In this section, detailed scenarios are described for each ACI SPAN type (Access, Tenant, Fabric) The base topology for each scenario is mentioned in the previous section.

If you understand these scenarios, you can select the appropriate ACI SPAN type for your requirement such as packets on only specific interfaces must be captured or all packets on a specific EPG regardless of interfaces must be captured, and more.

In Cisco ACI, SPAN is configured with the source group  and destination group. The Source group contains multiple source factors such as interfaces or EPG. The destination group contains destination information such as the destination interface for Local SPAN or destination IP for ESPAN.

After packets are captured, please see the section "How to Read SPAN Data" to decode captured packets.

Note: Please focus on VMs highlighted with a green light in each topology. Each scenario is to capture packets from these highlighted VMs.

Access SPAN (ERSPAN)

Case 1. Src "Leaf1 e1/11 e1/34 & Leaf2 e1/11"  |  Dst "192.168.254.1"

• Source Group
  • Leaf1 e1/11
  • Leaf1 e1/34
  • Leaf2 e1/11
• Destination Group
  • 192.168.254.1 on EPG X

Access SPAN can specify multiple interfaces for a single SPAN session. It can capture all packets that come in or go out from specified interfaces regardless of their EPG.

When multiple interfaces are specified as a source group from multiple Leaf switches, the destination group must be ERSPAN, not Local SPAN.

In this example, it copies packets from all VMs on EPG1 and EPG2.

CLI Check Point

• Please make sure the status is "up (active)"
• "destination-ip" is destination IP for ERSPAN
• "origin-ip" is source IP for ERSPAN

Case 2. Src "Leaf1 e1/11 & Leaf2 e1/11"  |  Dst "192.168.254.1"

• Source Group
  • Leaf1 e1/11
  • Leaf2 e1/11
• Destination Group
  • 192.168.254.1 on EPG X

In this example, Leaf1 e1/34 is removed from the SPAN Source Group configured at previous Case1.

The key point in this example is that Access SPAN can specify source interfaces regardless of EPG.

CLI Check Point

• source interface on Leaf1 is changed to "Eth1/11" from "Eth1/11   Eth1/34"

Case 3. Src "Leaf1 e1/11 & Leaf2 e1/11 & EPG1 filter"  |  Dst "192.168.254.1"

• Source Group
  • Leaf1 e1/11
  • Leaf2 e1/11
  • Filter EPG1
• Destination Group
  • 192.168.254.1 on EPG X

This example shows that Access SPAN also can specify a specific EPG on the source ports. This is useful when multiple EPGs flow on a single interface and it is required to capture traffic only for EPG1 on this interface.

Since EPG1 is not deployed on Leaf2, SPAN for Leaf2 fails with faults F1553 and F1561. However, SPAN on Leaf1 still works.

Also, two VLAN filters are automatically added for the SPAN session on Leaf1 because EPG1 uses two VLANs (VLAN-751,752) on Leaf1.

Please be noted that the VLAN ID on CLI (35, 39) is the internal VLAN so-called PI-VLAN(Platform Independent VLAN) which is not the actual ID on the wire. As shown in the picture, show vlan extended command shows the mapping of the actual encap VLAN ID and PI-VLAN.

This SPAN session allows us to capture packets only for EPG1(VLAN-752) on Leaf1 e1/11 even though EPG2 (VLAN-753) flows on the same interface.

CLI Check Point

• Filter VLANs are added as per the EPGs that are used for the filter.
• If there are no corresponding EPGs on Leaf, the SPAN session on that Leaf fails.

Case 4. Src "Leaf1-Leaf2 vPC"  |  Dst "192.168.254.1"

• Source Group
  • Leaf1 - 2e1/11
• Destination Group
  • 192.168.254.1 on EPG X

When the vPC interface is configured as a source, a destination must be remote IP (ERSPAN) not the interface (Local SPAN)

Access SPAN (Local SPAN)

Case 1. Src "Leaf1 e1/11 e1/34"  |  Dst "Leaf1 e1/33"

• Source Group
  • Leaf1 e1/11
  • Leaf1 e1/34
• Destination Group
  • Leaf1 e1/33

Access SPAN can also use Local SPAN (that is a specific interface as a destination)

However, in this case, source interfaces must be on the same Leaf as the destination interface.

Case 2. Src "Leaf1 e1/11 e1/34 & EPG1 filter  |  Dst " Leaf1 e1/33"

• Source Group
  • Leaf1 e1/11
  • Leaf1 e1/34
  • EPG1 Filter
• Destination Group
  • Leaf1 e1/33

Access SPAN with Local SPAN can also use EPG Filter as well as ERSPAN.

Case 3. Src "Leaf1 e1/11 & Leaf2 e/11"  |  Dst "Leaf1 e1/33" (bad case)

• Source Group
  • Leaf1 e1/11
  • Leaf2 e1/11
• Destination Group
  • Leaf1 e1/33

Case 4. Src "Leaf1 e1/11 & EPG3 filter"  |  Dst "Leaf1 e1/33" (bad case)

• Source Group
  • Leaf1 e1/11
  • EPG3 Filter
• Destination Group
  • Leaf1 e1/33

It is similar to case 3 on Access SPAN (ERSPAN) but in this example, the one and only SPAN session on Leaf1 fails because EPG3 does not exist on Leaf1. So SPAN does not work at all.

Case 5 : Src "EPG1 filter"  |  Dst "Leaf1 e1/33" (bad case)

• Source Group
  
  • EPG1 Filter
• Destination Group
  • Leaf1 e1/33

EPG filter on Access SPAN works only when source ports are configured. If EPG is the only source to be specified, Tenant SPAN must be used instead of Access SPAN.

Case 6. Src "Leaf1 - Leaf2 vPC"  |  Dst "Leaf1 e1/33"  (bad case)

• Source Group
  
  • Leaf1-2 vPC
• Destination Group
  • Leaf1 e1/33

A vPC interface cannot be configured as a source with Local SPAN. Please use ERSPAN. Please refer to case4 for Access SPAN (ERSPAN).

Case 7. Src "Leaf1 e1/11  |  Dst "Leaf1 e1/33 & e1/33 belongs to EPG"  (works with fault)

If a destination I/F for SPAN already belongs to EPG, a fault "F1696 : Port has an invalid configuration of both EPG and span destination" is raised under the physical I/F.

But even with this fault, SPAN works without any problems. This fault is just a warning about extra traffic caused by SPAN since it can impact customers' normal EPG traffic on the same I/F.

Tenant SPAN (ERSPAN)

Case 1. Src "EPG1"  |  Dst "192.168.254.1"

• Source Group
  
  • EPG1  (no filter)
• Destination Group
  • 192.168.254.1 on EPG X

Tenant SPAN uses EPG itself as a source while Access SPAN use EPG just for a filter.

The key point of Tenant SPAN is that you do not have to specify each individual port and ACI automatically detects appropriate VLANs on each Leaf switch. So this would be useful when all packets for specific EPG must be monitored and EndPoints for that EPG belong to multiple interfaces across Leaf switches.

Fabric SPAN (ERSPAN)

Case 1. Src "Leaf1 e1/49-50"  |  Dst "192.168.254.1"

• Source Group
  
  • Leaf1 e1/49-50
• Destination Group
  • 192.168.254.1 on EPG X

Fabric SPAN specifies Fabric ports as a source where Fabric ports are interfaces between Leaf and Spine switches.

This SPAN is useful when it is required to copy packets between Leaf and Spine switches. However, packets between Leaf and Spine switches are encapsulated with iVxLAN header. So it is required a bit of a trick to read it. Please refer to "How to Read SPAN Data".

Note: iVxLAN header is an enhanced VxLAN header only for ACI Fabric internal use.

Case 2. Src "Leaf1 e1/49-50 & VRF filter"  |  Dst "192.168.254.1"

• Source Group
  
  • Leaf1 e1/49-50
  • VRF Filter
• Destination Group
  • 192.168.254.1 on EPG X

Fabric SPAN can use filters as well as Access SPAN. But the filter type is different. Fabric SPAN uses Virtual Routing and Forwarding (VRF) or BD as a filter.

In Cisco ACI, as described before, packets that go through Fabric ports are encapsulated with iVxLAN header. This iVxLAN header has VRF or BD information as Virtual Network Identifier (VNID). When packets are forwarded as Layer2 (L2), iVxLAN VNID stands for BD. When packets are forwarded as Layer3 (L3), iVxLAN VNID stands for VRF.

So when it is required to capture routed traffic on Fabric ports, use VRF as a filter.

Case 3. Src "Leaf1 e1/49-50 & BD filter"  |  Dst "192.168.254.1"

• Source Group
  
  • Leaf1 e1/49-50
  • BD Filter
• Destination Group
  • 192.168.254.1 on EPG X

As described in previous case 2, Fabric SPAN can use BD as a filter.

When it is required to capture bridged traffic on Fabric ports, use BD as a filter.

Note: Only a single filter of either BD or VRF can be configured at a time.

What do you need on the SPAN destination device?

Just run a packet capture application such as tcpdump, wireshark on it. It is not required to configure the ERSPAN destination session or anything.

For ERSPAN

Please ensure to run a capture tool on the interface with the destination IP for ERSPAN since SPAN packets are forwarded to the destination IP.

The received packet is encapsulated with a GRE header. Please refer to this section "How to Read ERSPAN Data" on how to decode the ERSPAN GRE header.

For Local SPAN

Please ensure to run a capture tool on the interface that connects to the SPAN Destination interface on ACI Leaf.

Raw packets are received in this interface. it is not required to deal with the ERSPAN header.

How to Read ERSPAN Data

ERSPAN Version (type)

ERSPAN encapsulates copied packets to forward them to the remote destination. GRE is used for this encapsulation. The protocol type for ERSPAN on the GRE header is 0x88be.

In Internet Engineering Task Force (IETF) document, the ERSPAN version is described as type instead of version.

There are three types of ERSPAN. I, II and III. ERSPAN Type is mentioned in this RFC draft. Also, this GRE RFC1701 can be helpful to understand each ERSPAN type as well.

Here is the packet format of each type:

ERSPAN Type I (used by Broadcom Trident 2)

Type I does not use the sequence field on the GRE header. It does not even use the ERSPAN header which must succeed the GRE header if it was ERSPAN type II and III. Broadcom Trident 2 only supports this ERSPAN type I.

ERSPAN Type II or III

If the sequence field is activated by the S bit, this must be ERSPAN type II or III. The version field on the ERSPAN header identifies the ERSPAN type. In ACI, type III is not supported as of 03/20/2016.

If a SPAN source group for Access or Tenant SPAN has sources on both 1st-gen and 2nd-gen nodes, the ERSPAN destination receives both ERSPAN Type I and II packets from each generation of nodes. However, Wireshark can decode only one of the ERSPAN Types at a time. By default, it only decodes ERSPAN Type II. If you enable the decode of ERSPAN Type I, Wireshark does not decode ERSPAN Type II. See the later section on how to decode ERSPAN Type I on Wireshark.

To avoid this type of issue, you can configure ERSPAN Type on a SPAN destination group.

• SPAN Version (Version 1 or Version 2): This refers to the ERSPAN Type I or II
• Enforce SPAN Version (checked or unchecked): This decides if the SPAN session must fail in case the configured ERSPAN Type is not supported on the source node hardware.

By default, SPAN Version is Version 2 and Enforce SPAN Version is unchecked. This means that if the source node is 2nd gen or later which supports ERSPAN Type II, it generates ERSPAN with Type II. If the source node is 1st gen which does not support ERSPAN Type II (except for Fabric SPAN), it falls back to Type I since the Enforce SPAN Version is not checked. As a result, the ERSPAN destination receives a mixed type of ERSPAN.

This table explains each combination for Access and Tenant SPAN.

ERSPAN Data Example

Tenant SPAN/Access SPAN (ERSPAN)

Packets need to be decoded since it is encapsulated by ERSPAN Type I. This can be done with Wireshark. Please refer to the section "How to Decode ERSPAN Type 1".

Details of Captured Packet (ERSPAN Type I)

Fabric SPAN (ERSPAN)

Wireshark automatically decodes ERSPAN Type II. However, it is still encapsulated by iVxLAN header.

By default, Wireshark does not understand the iVxLAN header since it is ACI internal header. Please refer to "How to Decode iVxLAN Header".

Details of the Captured Packet (ERSPAN Type II)

How to Decode ERSPAN Type I

Option 1. Navigate to Edit > Preference > Protocols > ERSPAN and check FORCE to decode the fake ERSPAN frame.

• Wireshark (GUI)

• Tshark (CLI version of Wireshark):

user1@linux# tshark -f 'proto GRE' -nV -i eth0 -o erspan.fake_erspan:true

Note: Please ensure to disable this option when you read ERSPAN type II or III.

Option 2. Navigate to Decode As > Network > ICMP (if it’s ICMP).

How to Decode iVxLAN Header

iVxLAN header uses destination port 48879. So, you can decode iVxLAN header as well as VxLAN if you configure UDP destination port 48879 as VxLAN on Wireshark.

1. Please ensure that you select iVxLAN encapsulated packets first.
2. Navigate to Analyze > Decode As > Transport > UDP destination (48879) > VxLAN.
3. And then Apply.

Note: There are communication packets between APICs on Fabric ports. Those packets are not encapsulated by iVxLAN header.

When you take an erspan capture on a user network that runs Precision Time Protocol (PTP) sometimes it is seen that Wireshark does not interpret the data due to an unknown ethertype within the GRE encap (0x8988). 0x8988 is the ethertype for the time tag that is inserted into dataplane packets when PTP is enabled. Decode the ethertype  0x8988 as "Cisco ttag" to expose the details of the packet.