Troubleshoot and Debug Cisco NFVIS

Log and Show Commands

Support Commands and Show Commands

The following commands translate to corresponding linux commands like virsh, ovs and ip:

Command

Description

System

show system status

To display system defaults and services status.

show system disk-space

To display information about the system disk space.

show system memory

To display information about the system memory.

If DPDK is enabled, check if HugePage is available to use.

show resources cpu-info

To get information on the resource assignment.

VM

support virsh all-info

To display the output of all supported VM and index by number.

support virsh dumpxml <num>

To display all information about one VM index by <num>

support virsh domiflist <num>

To display the list of interfaces on VM index by <num> and MAC address of the VNICs.

support flush cache memory

To clear cache memory and free up some system memory for seamless performance of Cisco NFVIS. Clearing caches using support flush cache command can help resolve issues related to outdated or corrupted cache data. For example, clearing the cache in a web browser can help resolve issues such as slow page load times of Cisco NFVIS portal.

Network

support show ifconfig

To display the configuration details of all network interfaces or a specific interface.

support virsh net-list

To display all the networks in the host

support virsh net-dumpxml <network name>

To display the network information about one network and bridge attachment.

support virsh iface-list

To display a list of interfaces on the host.

Bridge

support ovs vsctl show: : :

To display an overview of the bridge, port and vlan tag.

support ovs appctl fdb-show <bridge-name>

To display information about the ports of a bridge.

support ovs all-info

To display the output of all supported ovs commands

Firewall

support show firewall get-all-rule

Log Files

The tech-support includes all the logs. Download tech-support and record the time of the occurance of error.

Command

Description

show log

To display a list of available log files or content of a specific log file.

show log nfvis_syslog.log

To display syslogs.

show log nfvis_config.log

To display system configuration related logs.

show log esc/escmanager.log

To display VM deployment related logs.

show log switch_confiog.log

To display the built-in switch configuration logs.

SPAN Session or Port Mirroring

About SPAN Sessions

The Switched Port Analyzer (SPAN) or Port Mirroring feature helps you analyze network traffic passing through interfaces or VLANs by using SPAN sessions. The SPAN sessions send a copy (mirror) of the traffic to another interface or VLAN on the switch that has been connected to a network analyzer or monitoring device. SPAN does not affect the switching of network traffic on the source interfaces.


Note

You must dedicate a destination port for SPAN use. Except for traffic that is required for the SPAN session, destination ports do not receive or forward traffic. When the SPAN is configured on the system, there might be some performance hit.

SPAN Session Interfaces

The interface can be:

  • Physical interface

  • LAN SRIOV

  • VM's vNIC (virtio net)

In the case of virtio net or SRIOV VF, you have to specify the VM group name and NIC ID of the VM interface. If the VM vNIC is virtio net type, then the SPAN session is applied on the OVS bridge. If VM vNIC is SRIOV VF, then the mirror is applied to the hardware bridge. The interface name is specified for a physical interface, for example, GE0-0 or eth0.

Configuring SPAN Sessions

The SPAN session configuration has the following four parameters:

  • Session number—Each SPAN session is identified with a unique number.

  • Bridge name—The SPAN session is applied to a bridge. For VLAN mirroring, the bridge must be specified. The bridge name is optional if the source or destination interface is configured for the session.

  • Source configuration—The source of the mirror traffic can be one of the following:

    • Packets entering (Rx), or exiting (Tx), or both. You can specify multiple interfaces of any type.

    • You can also specify all interfaces on the OVS bridge.

    • All packets entering a VLAN. You can also specify a list of VLANs.

  • Destination configuration—The destination for the mirrored traffic can be one of the following:

    • You can send the mirrored traffic to interfaces of Cisco ENCS switch port when you use a physical interface.

    • The mirrored traffic can be sent to a speciifc VLAN. In this case, the original VLAN tag is stripped in the mirrored traffic in favor of the destination VLAN. This loss of original VLAN information might make the mirrored traffic hard to interpret.


Note

Use VM Group Name for source vm-vnic and destination vm-vnic commands. Use the show vm_lifecycle deployments all | include Group command to get Group Name. 


ENCS# show vm_lifecycle deployments all | include Group
VM Group Name : deployment-ASAv
VM Group Name : deployment-ISRv
VM Group Name : deployment-Linux
ENCS#

To configure a SPAN session:


configure terminal
monitor session  2
bridge wan-br
source interface GE0-0
destination vm-vnic Linux2 0
commit

Verifying the SPAN Session Configuration

Use the show system monitor session command to verify the SPAN session configuration. 


nfvis# show system monitor session
system monitor session 2
 bridge                wan-br
 destination_vlan      ""
 destination_interface vnic0
 source_vlans          ""
 source_rx_interfaces  "GE0-0"
 source_tx_interfaces  "GE0-0"
 source_all            false
 statistics            "tx_bytes=142660, tx_packets=1380"

Use the show running-config monitor session command to verify the interface configuration for a SPAN session: 


nfvis# show running-config monitor session
monitor session 2
 destination vm-vnic Linux2 0
 source vm-vnic Linux1 0 both
 source interface GE0-0 both

SPAN Session APIs and Commands

APIs

Commands

  • /api/config/monitor

  • /api/operational/monitor\?deep

  • /api/config/monitor\?deep

  • /api/operational/system/monitor/session\?deep

  • monitor session

  • bridge

  • source

  • destination

  • show system monitor session

  • show monitor session status

  • show running-config monitor session

Configuration Examples for SPAN Session Scenarios

Example: SPAN Session Traffic on a Physical Interface

The following example shows how to configure all traffic coming in or going out on GE0-0 (physical interface) and VM Linux1 (vnic0). And traffic is mirrored to the VM Linux2 (vnic1). With this configuraton, any traffic arriving on vnet1 will be dropped.


Note

An existing SPAN session will be in FAIL state after the system reboot. In this case, you need to recreate (delete and create) the SPAN session after the system bootup.

VM deployment interfaces:

  • SPAN source: GE0-0 (traffic in both directions)

  • SPAN source: Linux1/vnic0, and wan-net (traffic in both directions)

  • • SPAN destination: Linux2/vnic0, and wan-net


nfvis# show running-config monitor session
monitor session 20
 destination vm-vnic Linux2 0
 source vm-vnic Linux1 0 both
 source interface GE0-0 both
!
nfvis# 

nfvis# show system monitor session
system monitor session 20
 bridge                wan-br
 destination_vlan      ""
 destination_interface vnic11
 source_vlans          ""
 source_rx_interfaces  "vnic10, GE0-0"
 source_tx_interfaces  "vnic10, GE0-0"
 source_all            false
 statistics            "tx_bytes=142660, tx_packets=1380"
nfvis# 

nfvis# show monitor session status
NUMBER  STATUS          
------------------------
20      CREATE_SUCCESS

Example: SPAN Session Traffic on a LAN SRIOV

The following example shows how to configure all traffic coming in or going out on an SRIOV interface (VF0). It is also mirrored to VF1.


Note

This scenario is applicable only to the Cisco ENCS.

VM deployment for VF-VF scenario:

CentOS_SRIOV, C3, and C5 are CentOS VMs with SRIOV support.

  • CentOS_SRIOV: vnic0: wan-net/vnic1: LAN-SRIOV-1 (192.168.1.36)

  • C3: vnic0: LAN-SRIOV3 (192.168.1.3)

  • C5: vnic0: LAN-SRIOV5 (192.168.1.5)

SPAN destination and source:

  • SPAN destination: CentOS_SRIOV (vnic0: wan-net/vnic1: LAN-SRIOV-1)

  • SPAN source: C3 (vnic0: LAN-SRIOV-3); traffic in both directions (rx, tx)

  • Ping target: C5 (vnic0: LAN-SRIOV-5)


nfvis# show running-config monitor session 
monitor session 6
 destination vm-vnic CentOS_SRIOV 1
 source vm-vnic C3 0
!
nfvis#

nfvis# show system monitor session
system monitor session 6
 bridge                ""
 destination_vlan      ""
 destination_interface LAN-SRIOV-1
 source_vlans          ""
 source_rx_interfaces  LAN-SRIOV-3
 source_tx_interfaces  LAN-SRIOV-3
 source_all            ""
 statistics            ""
nfvis# 

nfvis# show monitor session status
NUMBER  STATUS          
------------------------
6       CREATE_SUCCESS

Example: SPAN Session Traffic on a VLAN

The following example shows how to configure the SPAN session for all traffic entering in VLAN 10 and 11. It is also mirrored to VLAN 20. 


nfvis# show running-config monitor session
monitor session 11
 bridge lan-br
 destination vlan 20
 source vlan [ 10 11 ]
!

nfvis# show system monitor session   
system monitor session 11
 bridge                lan-br
 destination_vlan      20
 destination_interface ""
 source_vlans          "10, 11"
 source_rx_interfaces  ""
 source_tx_interfaces  ""
 source_all            true
 statistics            "tx_bytes=0, tx_packets=0"

nfvis# show monitor session 11        
NUMBER  STATUS          
------------------------
11      CREATE_SUCCESS

Configuring Packet Capture

The Packet Capture feature helps you capture all packets being transmitted and received over physical and virtual network interface controllers (physical port and vNIC) for analysis. These packets are inspected to diagnose and solve network problems. Packets are stored in the/data/intdatastore/pktcaptures folder on the host server.

Benefits

  • You can customize the configuration to capture specific packets such as Internet Control Message Protocol (ICMP), TCP, UDP, and Address Resolution Protocol (ARP).

  • You can specify a time period over which packets are captured. The default is 60 seconds.

To configure packet capture on a physial port:


configure terminal
tcpdump port eth0

Output: pcap-location /data/intdatastore/pktcaptures/tcpdump_eth0.pcap

To configure packet capture on a vNIC:


configure terminal
tcpdump vnic tenant-name admin deployment-name 1489084431 vm-name ROUTER vnic-id 0 time 30

Output: pcap-location /data/intdatastore/pktcaptures/1489084431_ROUTER_vnic0.pcap

Types of Errors

Error

Scenario

Port/vnic not found

When non-existing interface is given as input.

File/directory not created

When the system is running out of disk space.

The tcpdump command fails

When the system is running out of disk space.

These errors are logged in the nfvis_config.log. By default, warnings and errors are logged,

Example: Debug Built-in Switch Issues

To monitor traffic problems related to built-in switch on an internal interface:

The regular traffic flow between int-LAN and GE1/0 is:

GE0-0-- vnic1---(VM)--vnic2--intLAN--GE1/0

The NFVIS portal has the capability to capture packets. In the network diagram, right click on any vertical line and a window pops up where you can select the duration of the capture. The packet capture starts on the selected interface link. At the end of the capture, a file is downloaded to your local machine. SPAN sessions are supported on both NFVIS host and the built-in switch.

The following is an example of SPAN in built-in switch:

  1. From NFVIS system shell-access, get the password which can be used later.

    
cd /opt/switch-confd/
python decrypt_switch.py

<it will print out a string, it will be the password you need to use later>
8H7)gR348V4Byq4mwjiNt

  2. From Cisco IMC complete the challenge-response authentication:

    
#connect debug-shell
#sldp
login <hit return>
it will print out the challenge string
enter the  respond string
# switch-con ge
user-name:cisco
password: <enter the sring we get from nfvis system shell>

User Name:cisco
Password:*********************.   <this is the password you get from step 1 above>

  3. To configure SPAN specify the source and distribution interface and direction of the packet flow. For example, if you want to mirror XG2 output packet to Ge0, connect an external packet capture tool in GE1/0 and you will see all packets flow from internal XG2. In the following example, the traffic between int_LAN and GE1/0 go through internal interface XE2 and traffic for XE2 interface is monitored: 

    
nfvis(config)#monitor session 1 source interface XG 2 out
nfvis(config)#monitor session 1 destination interface GigabitEthernet 0
remember to unconfig it once you finish debuging.
nfvis(config)#no monitor session 1 destination
nfvis(config)#no monitor session 1 source interface XG 2

Packet Capture APIs and Commands

APIs

Commands

  • /api/operations/packet-capture/tcpdump

  • tcpdump port

  • tcpdump vnic