Troubleshooting

Table 1. Feature History

Feature Name

Release Information

Description

Improved Access to Troubleshooting Tools in Cisco SD-WAN Manager

Cisco vManage Release 20.10.1

The troubleshooting tools are now easily accessible from the various monitoring pages of Cisco SD-WAN Manager, such as Site Topology, Devices, Tunnels, and Applications, thereby providing you with context-based troubleshooting guidance. Earlier, the troubleshooting tools were accessible only from the device dashboard.

Connect to and troubleshoot Cisco Catalyst SD-WAN solution using Cisco RADKit

Cisco IOS XE Catalyst SD-WAN Release 17.15.1a

Cisco Catalyst SD-WAN Manager Release 20.15.1

Use tools and Python modules from Cisco Remote Automation Development Kit (RADKit) to securely connect to remote terminals, WebUIs, or desktops. Using RADKit, a TAC engineer can request the required information during the troubleshooting process, from the various devices and services, in a secure and controlled way.

Troubleshoot Common Cellular Interface Issues

Resolve Problems with Cellular Interfaces

This topic describes the most common issues and error messages that occur with cellular connections from the router to the cellular network, and the steps to resolve them.

Insufficient Radio Signal Strength

Problem Statement

The cellular module in the router cannot detect a radio signal from the service provider network.

Identify the Problem

  • The signal strength displayed in the Cisco SD-WAN Manager Cellular Status screen or with the show cellular status CLI command, or in the Cellular Radio screen or with the show cellular radio command is no signal, poor, or good. It should be excellent. The following table lists the ranges of signal strengths:

Table 2.

Signal

Excellent

Good

Fair

Poor

Received signal strength indicator (RSSI)

>= -65 dBm

-65 to -75 dBm

-75 to -85 dBm

<= -85 dBm

Reference signal receive power (RSRP)

>= -80 dBm

–80 to -90 dBm

–90 to -100 dBm

<= -100 dBm

Reference signal receive quality (RSRQ)

>= -10 dBm

-10 to -15 dB

-15 to -20 dB

< –20 dB

Signal-to-noise ratio (SNR)

>= 20 dB

13 to 20 dB

0 to 13 dB

<= 0 dB

Note

All parameters must be considered together and not in isolation. For example, a strong RSSI does not mean signal quality is good if RSRP is bad.

  • The wireless LED on the router is lit (solid or blinking) and is red, orange or yellow, or it is blinking green. It should be solid green.

Resolve the Problem

  1. Examine the router to verify that both basic antennas are correctly installed.

  2. Contact the service provider to verify that the location has coverage.

  3. Move the router to a new location within the building.

  4. Procure an additional external cabled antenna and connect it to the router.

Modem Status Remains in Low-Power Mode

Problem Statement

End users cannot connect to the cellular network, and the modem status remains in low-power mode.

Identify the Problem

  • End users cannot connect to the cellular network.

  • The error message "Missing or unknown APN" is generated.

  • The signal strength is less than excellent.

Resolve the Problem

  1. Verify that there is sufficient radio signal strength. If there is not, follow the instructions in the Insufficient Radio Signal Strength section.

  2. Verify that the cellular0 interface is operational. When the cellular interface is shut down, the modem status is set to Low Power mode. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices.

    Cisco vManage Release 20.6.1 and earlier: To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Network.

    Then click Real Time, and from the Device Options drop-down list, choose Interface Detail.

    To do this from the CLI, use the show interface command. Check that the Admin Status and Oper Status values are both Up.

  3. Verify that the modem temperature is not above or below the threshold temperatures. To view the modem temperature, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and select the router.

    Cisco vManage Release 20.6.1 and earlier: To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Network.

    Then click Real Time, and from the Device Options drop-down list, choose Cellular Modem.

    From the CLI, use the show cellular modem command.

  4. Check that the access point name (APN) in the profile for the cellular0 interface matches the name expected by your service provider. Some service provides required that you configure the APN, and they include configuration instructions in the SIM card package.

    1. To check which APN name is configured, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and select the router.

      Cisco vManage Release 20.6.1 and earlier: To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Network.

      Then click Real Time, and from the Device Options drop-down list, choose Cellular Profiles.

      From the CLI, use the ; show cellular profiles command. The APN column shows the name of the APN. Each profile specifies an access point name (APN), which is used by the service provider to determine the correct IP address and connect to the correct secure gateway. For some profiles, you must configure the APN.

    2. If the APN is not the one required by the service provider, configure the correct APN. From the Cisco SD-WAN Manager menu, choose Configuration > Templates and use the Cellular Profile feature template.

      To configure this from the CLI, use the cellular cellular0 profile apn command.

  5. If none of the previous steps works, reset the cellular interface.

Error Messages

The following table list the most common error messages that are displayed regarding cellular interfaces:

Table 3.

Error Message

Problem Statement

How Do I Fix the Problem

Authentication failed

End user authentication failed, because the service provider cannot authenticate either the user's SIM card or the Cisco vEdge device SIM card.

Contact the cellular service provider.

Illegal ME

The service provider denied access to an end user, because the end user is blocked from the network.

Contact the cellular service provider.

Illegal MS

The service provider denied access to an end user, because the end user failed the authentication check.

Contact the cellular service provider.

Insufficient resources

The service provider network is experiencing congestion because of insufficient resources and cannot provide the requested service to an end user.

The Cisco vEdge device automatically tries to reconnect. (The duration between retries depends on the service provider.) If the issue does not resolve itself, contact the cellular service provider.

IPV4 data call throttled

The SIM card being used in the Cisco vEdge device requires that you configure static APN.

Verify whether the data plan associated with the SIM card requires a static APN. If so, change the APN to the name specified the SIM card instructions, as described in Modem Status Remains in Low-Power Mode , above.

Missing or unknown APN

End users cannot connect to the cellular network, either because an APN is required and is not included in the cellular profile or because the APN could not be resolved by the service provider.

See the profile's APN, as described in Modem Status Remains in Low-Power Mode , above.

MS has no subscription for this service

The service provided denied access to an end user, because the end user has no subscription.

Contact the cellular service provider.

Network failure

The service provider network is experiencing difficulties.

The Cisco vEdge device automatically tries to reconnect. (The duration between retries depends on the service provider.) If the issue does not resolve itself, contact the cellular service provider.

Network is temporarily out of resources

The service provider network is experiencing congestion because of insufficient resources and cannot provide the requested service to an end user.

The Cisco vEdge device automatically tries to reconnect. (The duration between retries depends on the service provider.) If the issue does not resolve itself, contact the cellular service provider.

Operator has barred the UE

The service provided denied access to an end user, because the operator has barred the end user.

Contact the cellular service provider.

Requested service option not subscribed

The SIM card being used in the Cisco vEdge device requires that you configure a static APN entry.

Verify whether the data plan associated with the SIM card requires a static APN. If so, change the APN to the name specified the SIM card instructions, as described in Modem Status Remains in Low-Power Mode , above.

Service not supported by the PLMN

The Public Land Mobile Network (PLMN) does not support data service.

Contact the cellular service provider.

Troubleshoot WiFi Connections

This topic describes how to check and resolve connection problems between a WiFi client and a WiFi network that is provided by a WiFi router. The procedures described here are applicable to devices that support WiFi only.

Check for WiFi Connection Problems

If a WiFi client is unable to connect to a WiFi network when a router is providing the WiFi network, follow these steps to determine the source of the problem. To perform each step, use a method appropriate for the WiFi client.

  1. Verify that the WiFi client can locate the service identifier (SSID) advertised by the router. If the client cannot find the SSID, see the section, SSID Not Located.

  2. Verify that the WiFi client can connect to the SSID advertised by the router. If the client cannot connect to the SSID, see the section, SSID Connection Fails.

  3. Verify that the WiFi client has been assigned an IP address. If the client cannot obtain an IP address, see the section, Missing IP Address.

  4. Verify that the WiFi client can access the Internet. If the client cannot connect to the Internet, see section, Internet Connection Failure.

  5. If the WiFi client connection is slow or if you notice frequent disconnects, see section, WiFi Speed Is Slow.

Resolve Problems with WiFi Connections

This section describes the most common issues that occur with WiFi connections between a WiFi client and a router, and it describes steps to resolve the issues.

SSID Not Located

Problem Statement

The WiFi client cannot locate the SSID advertised by the router.

Resolve the Problem

  1. Ensure that the basic service set identifier (BSSID) address for the SSID is valid:

    1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

      Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

    2. Choose a device from the device list that appears.

    3. From the left pane, choose WiFi. The right pane displays information about WiFi configuration on the router.

    4. In the right pane, locate the SSID. Check that the BSSID for this SSID does not have a value of 00:00:00:00:00:00.

    5. If the BSSID is 00:00:00:00:00:00, the WLAN (VAP) interface for this SSID may be misconfigured. Ensure that the WLAN interface has been added to a bridge during the configuration process. To view the running configuration of the device, from the Cisco SD-WAN Manager menu, choose Configuration > Devices. For the desired device, click ...and choose Running Configuration.

      To view the running configuration of the device from the CLI, run the show running-config command. To add the WLAN interface to a bridge — from the Cisco SD-WAN Manager, choose Configuration > Templates.

      Click Feature Templates, and choose the Bridge feature template.


      Note

      In Cisco vManage Release 20.7.1 and earlier releases, Feature Templates is titled Feature.

  2. Eliminate static channels. A static channel is one where you explicitly configure the radio channel rather than allowing the router to automatically select the best radio channel. A slow static channel may appear to be an unreachable SSID.

    1. View the current SSID channel setting for the router. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the list of devices that appears. Then click Real Time, and in the Device Options drop-down list, choose WLAN Clients or WLAN Radios.

      From the CLI, run the show wlan clients or show wlan radios command.

    2. If the channel is set to a specific number, change the value to "auto". To do this, use the WiFi Radio feature template in Cisco SD-WAN Manager.

      From the CLI, run the wlan channel auto command.

  3. Ensure that the WiFi client is using the same radio band as the router, either 2.4 GHz (for IEEE 802.11b/g/n) or 5 GHz (for IEEE802.11a/n/ac):

    1. Check which radio band the WiFi client supports.

    2. Check the router’s Select Radio setting. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the device list that appears. Then click Real Time, and in the Device Options drop-down list, choose WLAN Radios.

      From the CLI, run the show wlan radios command.

    3. If the router and WiFi client radio band settings do not match, either change the WiFi client's radio band or change the settings on the router so that they match. To do this, use the Wifi Radio feature template.

      From the CLI, run the wlan command.

SSID Connection Fails

Problem Statement

The WiFi client can locate the SSID advertised by the router but cannot connect to it.

Resolve the Problem

  1. If you configure passwords locally on the router, ensure that the WiFi client's password matches the SSID's password.

  2. If you are using a RADIUS server, ensure that the RADIUS server is reachable and that the WiFi client's username and password match the RADIUS configuration:

    1. To verify that the RADIUS server is reachable from the router, ping the server. To do this in Cisco SD-WAN Manager, ping a device. From the CLI, run the ping command.

    2. Check for matching passwords on the RADIUS server and WiFi client.

  3. Ensure that you do not exceed the maximum number of clients for this SSID:

    1. Verify the number of used clients and the maximum number of clients:

      • From the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the device list that appears. From the left pane, select WiFi. In the right pane, locate the SSID. Check the No. of Clients field. If the used/maximum values are equal, no more clients can connect to this SSID.

      • From the CLI, run the show wlan interfaces detail command.

    2. If needed, increase the maximum clients setting for your SSID. To do this use the WiFi SSID feature template in Cisco SD-WAN Manager.

      From the CLI, run the max-clients command.

  4. Ensure that the WiFi client supports WPA2 management security:

    1. Check your Management Security setting. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the device list that appears. Then click Real Time, and in the Device Options drop-down list, choose WLAN Interfaces.

      From the CLI, run the show wlan interfaces command. If the management security value is set to "required," the WiFi client must support WPA2 security.

    2. If necessary, change the Management Security setting for your SSID to "optional" or "none." To do this in Cisco SD-WAN Manager, use the WiFi SSID feature template.

      From the CLI, run the mgmt-security command.

Missing IP Address

Problem Statement

The WiFi client can connect to the SSID, but cannot obtain an IP address.

Resolve the Problem

Ensure that a DHCP server is reachable and has an available IP address in its address pool:

  1. If the router is acting as a DHCP helper (DHCP relay agent), ping the DHCP server to ensure that it is reachable from the router.. From the CLI, run the ping command.

  2. If you are using a remote DHCP server, check that the remote DHCP server has an available IP address in its address pool.

  3. If the router is acting as the local DHCP server:

    1. View the number of addresses being used. From the Cisco SD-WAN Manager menu, Monitor > Devices and choose a device from the device list that appears. Next, click Real Time, and from the Device Options drop-down list, choose DHCP Servers.

      From the CLI, run the show dhcp server command.

    2. Compute the number of IP addresses in the pool based on the configured DHCP address pool size and the number of addresses excluded from the DHCP address pool. To view these values in Cisco SD-WAN Manager, from the Cisco SD-WAN Manager menu, choose Configuration > Devices. For the desired router, click ... and choose Running Configuration.

      To view them from the CLI, run the show running-config command.

    3. If necessary, increase the range of addresses in the router's DHCP address pool using the DHCP-Server feature template in Cisco SD-WAN Manager.

Internet Connection Failure

Problem Statement

The WiFi client is connected to the SSID and has an IP address, but it cannot connect to the Internet.

Resolve the Problem

Ensure that the WiFi client has received the correct default gateway and DNS settings from the DHCP server:

  1. If the DHCP server is remote, check the settings on the server.

  2. If the router is the DHCP server, ensure that the default gateway and DNS server settings are the same as those on the WiFi client. To view the settings in Cisco SD-WAN Manager, from the Cisco SD-WAN Manager menu, choose Monitor > Devices, and choose a device from the device list that is displayed. Click Real Time, and in the Device Options drop-down list, choose DHCP Interfaces.

    From the CLI, run the show dhcp interface  command.

WiFi Speed Is Slow

Problem Statement

The WiFi client can connect to the Internet, but the connection speed is slow.

Resolve the Problem

Allow the router to choose the best WiFi channel:

  1. View the current SSID channel setting for the router. To do this in Cisco SD-WAN Manager, from the Cisco SD-WAN Manager menu, choose Monitor > Devices, and choose a device from the device list that is displayed. Click Real Time, and in the Device Options drop-down list, choose WLAN Clients.

    From the CLI, run the show wlan clients or show wlan radios command.

  2. If the channel is set to a specific number, change the value to "auto". To do this in Cisco SD-WAN Manager, use the WiFi Radio feature template.

    From the CLI, run the wlan channel auto command.

Troubleshoot a Device

You can troubleshoot the connectivity or traffic health for all the devices in an overlay network.

Check Device Bringup

  1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

  2. Choose a device from the list of devices that is displayed.

  3. Click Troubleshooting in the left pane.

  4. In the Connectivity area, click Device Bringup.

The Device Bringup window opens.

Ping a Device

Table 4. Feature History

Feature Name

Release Information

Description

IPv6 Support in Cisco SD-WAN Manager UI Troubleshooting

Cisco IOS XE Catalyst SD-WAN Release 17.13.1a

Cisco Catalyst SD-WAN Manager Release 20.13.1

Added support for using an IPv6 address when pinging a device. Also added support for using an IPv6 address when running a traceroute, configuring packet capture, and simulating flows.

Before You Begin

Ensure that Device Monitoring and Events features have read and write permissions and Tools has read permission. For more information on different permission settings, see Manage Users.

With the set permissions to the usergroup, ensure that you are able to access the required features.

To verify that a device is reachable on the network, ping the device to send ICMP ECHO_REQUEST packets to it:

  1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

  2. To choose a device, click the device name in the Hostname column.

  3. Click Troubleshooting in the left pane.

  4. In the Connectivity area, click Ping.

  5. In the Destination IP field, enter the IP address of the device to ping.

    For releases before Cisco Catalyst SD-WAN Manager Release 20.13.1, enter an IPv4 address. From Cisco Catalyst SD-WAN Manager Release 20.13.1, enter an IPv4 or IPv6 address.

  6. In the VPN field, choose the VPN to use to reach the device.

  7. In the Source/Interface field, choose the interface to use to send the ping packets.

  8. In the Probes field, choose the protocol type to use to send the ping packets.

  9. In the Source Port field, enter the number of the source port.

  10. In the Destination Port field, enter the number of the destination port.

  11. Click Advanced Options to specify additional parameters:

    1. In the Count field, enter the number of ping requests to send. The range is 1 to 30. The default is 5.

    2. In the Payload Size field, enter the size of the packet to send. The default is 64 bytes, which comprises 56 bytes of data and 8 bytes of ICMP header. The range for data is 56 to 65507 bytes.

    3. Enter the MTU.


      Note
      The MTU option does not apply beginning with Cisco IOS XE Catalyst SD-WAN Release 17.13.1a.

    4. Click the Rapid slider to send five ping requests in rapid succession and to display statistics only for the packets transmitted and received, and the percentage of packets lost.

    5. In the Type of Service field, enter the value to be included in the ping packets.

    6. In the Time to Live field, enter the round-trip time, in milliseconds, for sending this ping packet and receiving a response.

    7. Click the Don't Fragment option to set the Don't Fragment bit in the ping packets.

  12. Click Ping.

From Cisco vManage Release 20.10.1, the Ping option can be accessed using one of these methods:

  • Choose Monitor > Devices, click … adjacent to the device name, and choose Ping.

  • In the Site Topology page, click a device name or tunnel name, and then click Ping in the right navigation pane.

Speed Test

Table 5. Feature History

Feature Name

Release Information

Description

Speed Test

Cisco IOS XE Catalyst SD-WAN Release 17.3.1a

This feature enables you to carry out speed testing between two edge devices.

Speed Test Enhancement

Cisco vManage Release 20.10.1

Cisco IOS XE Catalyst SD-WAN Release 17.10.1a

This feature enables you to carry out speed testing between two edge devices. This feature is enhanced to get accurate speed test and bandwidth results on Cisco IOS XE Catalyst SD-WAN devices and iPerf3 servers.

Information About Speed Test

Iperf3 is a network performance measurement tool used for detecting bandwidth-related network problems.

There are two types of speed testing:

  • Site-to-site speed test: Cisco SD-WAN Manager tests the network speed and available bandwidth between two devices. Cisco SD-WAN Manager designates one device as the source and the other as the destination.

  • Internet speed test: Cisco SD-WAN Manager tests the network speed and available bandwidth between a device and an iperf3 server reachable by the network. Cisco SD-WAN Manager designates the device as the client site and the iperf3 server as the remote site. You can specify the IP address (or domain name) and port number for an iperf3 server.

The speed tests measure upload speed from the source device to the destination device, and measure download speed from the destination device to the source device.

Prerequisites for Speed Test

Speed testing requires the system ID and the device host name of the destination device.

Run Speed Test

Perform the following steps to run a speed test.

Run Site-to-Site Speed Test
Before You Begin

Ensure that Data Stream is enabled under Administration > Settings in Cisco SD-WAN Manager.

  1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

  2. To choose a device, click the device name in the Hostname column.

  3. Click Troubleshooting in the left pane.

  4. In the Connectivity area, click Speed Test.

  5. Specify the following:

    • Source Circuit: From the drop-down list, choose the color of the tunnel interface on the local device.

    • Destination Device: From the drop-down list, choose the remote device by its device name and system IP address.

    • Destination Circuit: From the drop-down list, choose the color of the tunnel interface on the remote device.

  6. Click Start Test.

    The speed test sends a single packet from the source to the destination and receives the acknowledgment from the destination.

The right pane shows the results of the speed test—circuit speed, download speed, and upload speed between the source and destination. The download speed shows the speed from the destination to the source, and the upload speed shows the speed from the source to the destination in Mbps. The configured downstream and upstream bandwidths for the circuit are also displayed.

When a speed test completes, the test results are added to the table in the lower part of the right pane.

From Cisco vManage Release 20.10.1, the Speed Test option is also accessible as follows:

  • On the Monitor > Devices page, click … adjacent to the device name and choose Speed Test.

  • On the Monitor > Applications page, click … adjacent to the application name and choose Speed Test.

  • On the Site Topology page, click a device name, and then click Speed Test in the right navigation pane.

Run Internet Speed Test

  1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Managermenu, choose Monitor > Network.

  2. To choose a device, click the device name in the Hostname column.

  3. Click Troubleshooting in the left pane.

  4. In the Connectivity area, click Speed Test.

  5. Specify the following:

    • Source Circuit: From the drop-down list, choose the color of the tunnel interface on the local device.

    • Destination Device: From the drop-down list, choose Internet.

    • iPer3 Server: (Optional) Enter the hostname or iPer3 server’s IP address in IPv4 format.

    • Server Port Range: (Optional) Enter the server port or a port range. For example, 5201, 5210, or 5201-5205.

  6. Click Start Test.

    The speed test result is displayed.

Troubleshooting Speed Test Issues

The following table provides troubleshooting information for speed testing:

Table 6. Troubleshooting Scenarios

Error Information

Possible Root Cause

Failed to resolve iperf server address

DNS server is not configured at edge device or is unable to resolve the iperf server from the configured DNS server at edge device.

Speed test servers not reachable

The speed test server ping failed. The edge device cannot reach the server IP.

iPerf client: unable to connect stream: Resource temporarily unavailable

Unable to connect to the speed test server. Access may be blocked by access-control list (ACL) permissions.

iPerf client: unable to connect to server

The iPerf3 server is not providing the test service at the user-specified port or default port 5201.

Device Error: Speed test in progress

The selected source or destination device is performing a speed test and cannot start a new one.

Device error: Failed to read server configuration

The data stream configuration is missing.

Workaround: Running a CLI command at the edge device and clearing the Cisco Catalyst SD-WAN control connections can fix the issue.

Speed test session has timed out

The speed test has not successfully completed in 180 seconds. This might be because the edge device has lost the control connection to Cisco SD-WAN Manager during the speed testing.

Run a Traceroute

  1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

  2. To choose a device, click the device name in the Hostname column.

  3. Click Troubleshooting in the left pane.

  4. In the Connectivity area, click Trace Route.

  5. In the Destination IP field, enter the IP address of the corresponding device in the network.

    For releases before Cisco Catalyst SD-WAN Manager Release 20.13.1, enter an IPv4 address. From Cisco Catalyst SD-WAN Manager Release 20.13.1, enter an IPv4 or IPv6 address.

  6. From the VPN drop-down list, choose a VPN to use to reach the device.

  7. From the Source/Interface for VPN drop-down list, choose the interface to use to send the traceroute probe packets.

  8. Click Advanced Options.

  9. In the Size field, enter the size of the traceroute probe packets, in bytes.

  10. Click Start to trigger a traceroute to the requested destination.

The lower part of the right pane displays the following information:

  • Raw output of the path the traceroute probe packets take to reach the destination.

  • Graphical depiction of the path the traceroute probe packets take to reach the destination.

If the traceroute is for the service-side traffic, a Cisco vEdge device generates traceroute responses from any of the interfaces on the service VPN.

From Cisco vManage Release 20.10.1, the Trace Route option can be accessed using one of these methods:

  • Choose Monitor > Devices, click … adjacent to the device name, and choose Trace Route.

  • In the Site Topology page, click a device or tunnel name, and then click Trace Route in the right navigation pane.

Discover Underlay Paths

Minimum release: Cisco vManage Release 20.10.1

On-Demand Troubleshooting

Table 7. Feature History

Feature Name

Release Information

Description

On-Demand Troubleshooting

Cisco IOS XE Catalyst SD-WAN Release 17.6.1a

Cisco SD-WAN Release 20.6.1

Cisco vManage Release 20.6.1

This feature lets you view detailed information about the flow of traffic from a device. You can use this information to assist with troubleshooting.

Enhancement to On-Demand Troubleshooting

Cisco vManage Release 20.11.1

You can view the detailed troubleshooting progress of the flow of traffic from a device.

Information About On-Demand Troubleshooting

On-demand troubleshooting lets you view detailed information about the flow of traffic from a device.

By default, Cisco SD-WAN Manager captures aggregated information about flows. You can obtain detailed information for specific devices and for specific historical time periods by adding an on-demand troubleshooting entry. When you add an entry, Cisco SD-WAN Manager compiles detailed information according to parameters that you configure.

To conserve system resources, Cisco SD-WAN Manager compiles detailed information only when you request it by adding an entry. In addition, Cisco SD-WAN Manager stores the information for a limited time (3 hours by default), then removes it. You can request the same information again, if needed.


Note

On a Cisco SD-WAN Manager cluster setup, only a connected node can remove an on-demand troubleshooting task or mark it as complete.

Restrictions for On-Demand Troubleshooting

Ensure that no Cisco or third-party APIs that instruct on-demand troubleshooting to stop are called when you are using on-demand troubleshooting. These APIs prevent on-demand troubleshooting from compiling information.

Page Elements

The On Demand Troubleshooting window provides options for configuring and adding an on-demand troubleshooting entry. The On Demand Troubleshooting window displays information about existing on-demand troubleshooting entries and provides the following information and options.

Item (Field)

Description

ID

System-assigned identifier of the entry.

Device ID

System IP of the device to which the entry applies.

Data Type

Type of data for which the entry provides detailed information.

Creation Time

Date and time that you added the entry.

Expiration Time

Date and time that the entry expires.

At this expiration time, the entry is removed from the table automatically, and the corresponding detailed information is no longer available.

By default, an entry is removed 3 hours after its creation time.

Data Backfill Start Time

Start date and time of the data backfill period.

Data Backfill End Time

End date and time of the data backfill period.

Status

Status of the entry:

  • IN_PROGRESS: Detailed troubleshooting information is in the process of being compiled.

  • QUEUED: Detailed troubleshooting information is queued for compilation.

  • COMPLETED: Detailed troubleshooting information has been compiled.

Configure On-Demand Troubleshooting

You can configure on-demand troubleshooting for a device from the Tools > On Demand Troubleshooting window in Cisco SD-WAN Manager. This window provides options for adding an on-demand troubleshooting entry, and for managing existing entries.

Cisco vManage Release 20.6.1 and earlier: You can configure on-demand troubleshooting for a device from the Monitor > On Demand Troubleshooting window in Cisco SD-WAN Manager.

You can also start on-demand troubleshooting from various locations in the Monitor > Devices window for a device. See View On-Demand Troubleshooting Information for a Device.

Cisco vManage Release 20.6.1 and earlier: You can start on-demand troubleshooting from various locations in the Monitor > Network window for a device.

On-demand troubleshooting is qualified for troubleshooting entries for up to 10 devices concurrently.

Add an On-Demand Troubleshooting Entry

Adding an entry in the On Demand Troubleshooting window instructs Cisco SD-WAN Manager to compile detailed troubleshooting information for the device that you specify, using the parameters that you configure.

To add an on-demand troubleshooting entry, follow these steps:

  1. From the Cisco SD-WAN Manager menu, choose Tools > On Demand Troubleshooting.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > On Demand Troubleshooting.

  2. From the Select Device drop-down list, choose the Cisco IOS XE Catalyst SD-WAN device or the Cisco vEdge device for which you want to enable on-demand troubleshooting.

  3. From the Select Data Type drop-down list, choose SAIE or ConnectionEvents.

  4. Choose an option for the data backfill period:

    • Last 1 hour: Provides detailed stream information for the period beginning 1 hour before you add the troubleshooting entry and ending at the time that you add the entry.

    • Last 3 hours: Provides detailed stream information for the period beginning 3 hours before you add the troubleshooting entry and ending at the time that you add the entry.

    • Custom Date and Time Range: Use the Start date and time and the End date and time fields to designate the backfill period that you want. Note that the End date and time value cannot be later than the current date and time.

  5. Click Add.

    The troubleshooting entry appears in the table of entries. When the value in the Status field for the entry shows the value Completed, you can view the troubleshooting information from the Monitor > Devices window, as described in View On-Demand Troubleshooting Information for a Device.

Update an On-Demand Troubleshooting Entry

Update an on-demand troubleshooting entry to make changes to its configuration settings. For example, update an entry to adjust its backfill period.

Only entries that are in the QUEUED state can be updated.

To update an on-demand troubleshooting entry, follow these steps:

  1. From the Cisco SD-WAN Manager menu, choose Tools > On Demand Troubleshooting.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > On Demand Troubleshooting.

  2. In the table of entries, click … adjacent to the entry that you want to update and choose Update.

  3. In the Update Troubleshoot Status dialog box that is displayed, configure the settings as needed, and click Add.

Delete an On-Demand Troubleshooting Entry

Deleting an on-demand troubleshooting entry removes the entry from Cisco SD-WAN Manager. After you delete an entry, you can no longer view its detailed information.

Deleting an entry can help free resources in Cisco SD-WAN Manager.

To delete an on-demand troubleshooting entry, follow these steps:

  1. From the Cisco SD-WAN Manager menu, choose Tools > On Demand Troubleshooting.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > On Demand Troubleshooting.

  2. In the table of entries, click … adjacent to the entry that you want to delete and choose Delete on demand queue.

  3. In the Delete On Demand Status window that is displayed, click OK.

View On-Demand Troubleshooting Information for a Device

You can view on-demand troubleshooting information for a device from the Network window for that device.

Before you can view this information, at least one on-demand troubleshooting entry must exist for the device. Add an entry from the On Demand Troubleshooting window as described in Add an On Demand Troubleshooting Entry, or add an entry from the Network window as described in the following procedure.

  1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

  2. In the Hostname column, click the device for which you want to view the information.

  3. Perform either of these actions:

    • To view the troubleshooting information for an SAIE application:

      1. Click SAIE Applications.


        Note

        In Cisco vManage Release 20.7.1 and earlier releases, SAIE Applications is called DPI Applications.

      2. In the Applications Family table, click an application family.

      3. In the Applications table, click an application.

    • To view troubleshooting information for a specific metric, in the left pane, under ON-DEMAND TROUBLESHOOTING click an option. Not all options apply to all device types.

      • FEC Recovery Rate

      • SSL Proxy

      • AppQoe TCP Optimization

      • AppQoE DRE Optimization

      • Connection Events

      • WAN Throughput

      • Flows

      • Top Talkers

    The Flows and Top Talkers metrics are only for TCP Optimized flows.

    If on-demand troubleshooting is configured for the device, detailed troubleshooting information appears. This information includes traffic statistics and metrics such as source IP address, destination IP address, number of packets, number of bytes, and more. Use the options that are available and hover your cursor over elements on the graphs to view the information that you need.


    Note

    Starting from Cisco IOS XE Release 17.9.1a, use the policy ip visibility features enable command to manually enable or disable the feature fields in Flexible Netflow (FNF). Use the show sdwan policy cflowd-upgrade-status command to check which features were enabled before the version upgrade. You have to manually control the features after a version upgrade using the disable or enable commands.

    For more information, see policy ip visibility command page.

    If on-demand troubleshooting information is not configured, the Enable On Demand Troubleshooting option is displayed. Continue to Step 4.

  4. If the Enable On Demand Troubleshooting option is displayed, perform these actions to start this feature for the selected device:

    1. Click Enable On Demand Troubleshooting.

    2. Choose one of the following options:

      • Quick Enable: Starts an on-demand troubleshooting entry with a backfill period of 3 hours. With this option, detailed stream information for the past 3 hours becomes available.

        After you choose this option, click Refresh to view the detailed troubleshooting information. It can take a few minutes for this information to become available. Alternatively, click Go to On Demand Troubleshooting to display the On Demand Troubleshooting window that includes the entry that you just added.

      • Go to On Demand Troubleshooting: Displays the On Demand Troubleshooting window. Add an entry in this window as described in Add an On Demand Troubleshooting Entry. Repeat Steps 1 to Step 3 in this procedure to view the detailed information.

View Progress of On-Demand Troubleshooting

Minimum supported release: Cisco vManage Release 20.11.1

After you enable on-demand troubleshooting, the On-demand Troubleshooting in Progress message appears on the Monitor > Devices page. The message remains until the troubleshooting is complete.

Click a chart option to view the troubleshooting progress in a graphical format. Select a time period to display data or click Custom to display a selection of a custom time period.

You can use the request nms olap-db command to start, stop, or restart the Cisco SD-WAN Manager online analytical processing (OLAP) database or view the status of the database.

For more information about this command, see request nms olap-db.

View Detailed Top Source Data

After on-demand troubleshooting is configured, you can view detailed information about top application usage for a device. To do so, follow these steps:

  1. From the Cisco SD-WAN Manager menu, choose Monitor > Overview > Top Applications.

    Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Dashboard > Main Dashboard > Top Applications.

  2. In the SAIE Application tab, click an application usage bar in the chart.


    Note

    In Cisco vManage Release 20.7.1 and earlier releases, SAIE Application is called DPI Application.

  3. In the chart for the application that you selected, click the device usage bar.

    If on-demand troubleshooting is configured for the device, detailed top source data appears.

    If on-demand troubleshooting information is not configured, the Go to On Demand Troubleshooting option appears. Continue to Step 4.

  4. If the Go to On Demand Troubleshooting option appears, perform these actions:

    1. Click Go to On Demand Troubleshooting to display the On Demand Troubleshooting window.

    2. In the On Demand Troubleshooting window, add an entry, as described in Add an On Demand Troubleshooting Entry.

    3. Repeat Step 1 to Step 3 in this procedure to view the detailed information.

Troubleshoot Cisco Catalyst SD-WAN Solution Using Cisco RADKit

Minimum supported release: Cisco Catalyst SD-WAN Manager Release 20.15.1

Use Cisco RADKit to troubleshoot devices in Cisco Catalyst SD-WAN. RADKit, a Software Development Kit (SDK), is a set of ready-to-use tools and Python modules, which helps you

  • securely connect to remote terminals, WebUI’s or desktops,

  • leverage APIs for remote or local automations, and

  • share support data privately with Cisco Services without any impact on data privacy.

Before You Begin

  • Ensure that you have an internet connection and have configured DNS in the transport VPN (VPN0).

  • Ensure that you are running compatible operating systems. For information about supported operating systems, see Compatibility.

Installation

The RADKit installation includes a client and a service that connects to the Cisco RADKit cloud to interactively connect you to remote terminals, WebUIs, or desktops.

To install the RADKit service, go to Cisco’s Support Services Technical Assistance Center (TAC) and open a support case. After you have installed the RADKit service, you can enroll to the RADKit client. For more information, see Initial Client Setup.

For more information and downloads, see RADKit.