You can capture packets flowing to, through, and from a CSP device of a colocation cluster. You can capture packets at either the PNIC or the VNIC level on the CSP device.

Supported Ports for Packet Capture for Cloud OnRamp Colocation Clusters

Packet capture is supported for the following ports:

Enable Packet Capture on Cisco SD-WAN Manager

Enable the packet capture feature on Cisco SD-WAN Manager before capturing packets at the PNIC or VNIC level on a CSP device of a colocation cluster:

1. From the Cisco SD-WAN Manager menu, choose Administration > Settings.

2. In Data Stream, choose Enabled.

   From Cisco Catalyst SD-WAN Manager Release 20.13.1, click the toggle button to enable data stream.

Capture Packets at PNIC Level

1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

2. Click Colocation Cluster, and choose a cluster.

3. From the list of devices that is displayed, click a CSP device name.

4. In the left pane, click Packet Capture.

5. From the PNIC ID drop-down list, choose a PNIC.

6. (Optional) Click Traffic Filter to filter the packets that you want to capture based on the values in their IP headers.

   Table 5. Packet Capture Filters

   Field	Description
   Source IP	Source IP address of the packet.
   Source Port	Source port number of the packet.
   Protocol	Protocol ID of the packet. The supported protocols are: ICMP, IGMP, TCP, UDP, ESP, AH, ICMP Version 6 (ICMPv6), IGRP, PIM, and VRRP.
   Destination IP	Destination IP address of the packet.
   Destination Port	Destination port number of the packet.

7. Click Start.

   The packet capture begins, and its progress is displayed:

   • Packet Capture in Progress: Packet capture stops after the file size reaches 20 MB, or 5 minutes after you started packet capture, or when you click Stop.

   • Preparing file to download: Cisco SD-WAN Manager creates a file in libpcap format (a .pcap file).

   • File ready, click to download the file: Click the download icon to download the generated file.

Capture Packets at VNIC Level

1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

2. Click Colocation Cluster, and choose a cluster.

3. From the list of devices that is displayed, click a CSP device name.

4. Choose a VNF, and then click Packet Capture in the left pane.

5. Alternatively, choose Monitor > Devices > Colocation Cluster. Next, choose a cluster and click Network Functions, choose a VNF, and then click Packet Capture in the left pane.

6. From the VNIC ID drop-down list, choose a VNIC.

7. (Optional) Click Traffic Filter to filter the packets to capture based on values in their IP headers. For more information on these filters, see the above section.

8. Click Start. The packet capture begins, and displays its progress.

View Tunnel Health

To view the health of a tunnel from both directions:

1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

   Cisco vManage Release 20.6.x and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

2. To choose a device, click the device name under the Hostname column.

3. Click Troubleshooting in the left pane.

4. In the Traffic area, click Tunnel Health.

5. From the Local Circuit drop-down list, choose a source TLOC.

6. From the Remote Device drop-down list, choose a remote device.

7. From the Remote Circuit drop-down list, choose a destination TLOC.

8. Click Go. The lower part of the screen displays:

9. From the Chart Options drop-down list, choose one of these: Loss Percentage, Latency/Jitter, Octets.

10. (Optional) Choose a predefined or a custom time period on the left to view data for the specified time period.

• App-route data (either loss, latency, or jitter) in graphical format for all tunnels between the two devices in each direction.

• App-route graph legend—Identifies selected tunnels from both directions.

From Cisco vManage Release 20.10.1, the Tunnel Health option is also accessible as follows:

• On the Monitor > Tunnels page, click … adjacent to the tunnel name and choose Tunnel Health.

• On the Monitor > Applications page, click … adjacent to the application name and choose Tunnel Health.

• On the Site Topology page, click a tunnel name, and then click Tunnel Health in the right navigation pane.

Check Application-Aware Routing Traffic

To check application-aware routing traffic from the source device to the destination device:

1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

   Cisco vManage Release 20.6.x and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

2. Choose a device from the list of devices that appears.

3. Click Troubleshooting in the left pane.

4. In the right pane, click App Route Visualization under Traffic.

5. From the Remote Device drop-down list, choose a destination device.

6. (Optional) Click Traffic Filter. Choose No Filter or SAIE. No Filter is chosen by default.

   Note	In Cisco vManage Release 20.7.x and earlier releases, the SD-WAN Application Intelligence Engine (SAIE) flow is called the deep packet inspection (DPI) flow.

7. Click Go. The lower part of the screen displays:

8. From the Chart Options drop-down list, choose one of these: Loss Percentage, Latency/Jitter, Octets.

9. (Optional) Choose a predefined or a custom time period on the left to view data for the specified time period.

From Cisco vManage Release 20.10.1, the App Route Visualization option is also accessible from the Monitor > Applications page. Click … adjacent to the application name and choose App Route Visualization.

Send the admin-tech.tar.gz file to Cisco Catalyst SD-WAN Technical Support for analysis and resolution of the issue.

Note	All in-progress admin-tech requests are purged every three hours.

1. From the Cisco SD-WAN Manager menu, choose Tools > Operational Commands.

2. For the desired device, click . . . and choose Generate Admin Tech from the options.

3. In the Generate admin-tech File window, choose the information to include in the admin-tech file if desired:

   1. The Include Logs check box is checked by default. Uncheck this check box if you do not want to include the log files in the admin-tech file.

      Note	The log files are stored in the /var/log/directory on the local device.

   2. Check the Include Cores check box to include any core files.

      Note	Core files are stored in the /var/crash directory on the local device.

   3. Check the Include Tech check box to include any files related to device processes (daemons), memory detail, and operations.

      Note	The log files are stored in the /var/tech directory on the local device.

4. Click Generate.

   Cisco SD-WAN Manager creates the admin-tech file. The file name has the format date-time-admin-tech.tar.gz.

Configure Packet Trace on Cisco IOS XE Catalyst SD-WAN devices

1. Enable packet trace for the traffic and specify the maximum number of packets:

   Device# debug platform packet-trace packet [number of traced packets]

2. Specify the matching criteria for tracing packets. Matching criteria provides the ability to filter by protocol, IP address and subnet mask, interface, and direction:

   Device# debug platform condition [interface interface name] {match ipv4|ipv6|mac src dst}  {both|ingress|egress} [bidirectional]
   

3. Enable MPLS output label trace. A MPLS output label trace is included in debug path to reduce the impact on performance.

   Device# debug platform hardware qfp active feature cef-mpls datapath mpls all
   

4. Enable the specified matching criteria and start packet tracing:

   Device# debug platform condition start

5. Deactivate the condition and stop packet tracing:

   Device# debug platform condition stop

6. Exit the privileged EXEC mode:

   exit

Configure Packet Trace on Cisco vEdge devices

The following example shows how to configure conditions for packet tracing:

Device# debug packet-trace condition source-ip 10.1.1.1
Device# debug packet-trace condition vpn-id 0
Device# debug packet-trace condition interface ge0/1
Device# debug packet-trace condition stop

For more information, see debug packet-trace condition command page.

The SNMP traps are asynchronous notifications that a Cisco device sends to an SNMP management server. Traps notify the management server of events, whether normal or significant, that occur on the device. By default, SNMP traps aren’t sent to an SNMP server. Note that for SNMPv3, the PDU type for notifications is either SNMPv2c inform (InformRequest-PDU) or trap (Trapv2-PDU).

To configure SNMP traps, define the traps and configure the SNMP server that receives the traps.

Note	The trap group UI option isn’t supported from Cisco SD-WAN Release 20.1.1 and later.

To configure groups of traps to be collected on Cisco vEdge devices, use the trap group command:

Note	You don't need to configure groups of traps on Cisco IOS XE Catalyst SD-WAN devices.

vEdge(config-snmp)#  trap group group-name
vEdge(config-group)# trap-type level severity

A single trap group can contain multiple trap types. In the configuration, specify one trap type per line, and each trap type can have one, two, or three severity levels. See the following configuration example for an illustration of the configuration process.

To configure the SNMP server to receive the traps, use the trap target command on Cisco vEdge devices:

Note	You don't need to configure the SNMP server to receive the traps on Cisco IOS XE Catalyst SD-WAN devices.

vedge(config-snmp)#  trap target vpn  vpn-id ipv4-address udp-port
vedge(config-target)# group-name name
vedge(config-target)# community-name community-name
vedge(config-target)# source-interface interface-name

For each SNMP server, specify the identifier of VPN where the server is located, the server's IPv4 address, and the UDP port on the server to connect to. When configuring the trap server's address, you must use an IPv4 address. You can’t use an IPv6 address.

In the group-name command, associate a previously configured trap group with the server. The traps in that group are sent to the SNMP server.

In the community-name command, associate a previously configure SNMP community with the SNMP server.

In the source-interface command, configure the interface to use to send traps to the SNMP server that is receiving the trap information. This interface cannot be a subinterface.

In the following configuration example, all traps are sent to one SNMP server and only critical traps to another SNMP server. Two SNMP trap groups and the two target SNMP servers are configured:

vEdge# config
Entering configuration mode terminal
vEdge(config)# snmp
vEdge(config-snmp)# view community-view 
vEdge(config-view-community-view)# exit
vEdge(config-snmp)# community public
vEdge(config-community-public)# authorization read-only 
vEdge(config-community-public)# view community-view 
vEdge(config-community-public)# exit
vEdge(config-snmp)# trap group all-traps
vEdge(config-group-all-traps)# all level critical major minor
vEdge(config-group-all)# exit
vEdge(config-group-all-traps)# exit
vEdge(config-snmp)# trap group critical-traps
vEdge(config-group-critical-traps)# control level critical
vEdge(config-group-control)# exit
vEdge(config-group-critical-traps)# exit
vEdge(config-snmp)# trap target vpn 0 10.0.0.1 162
vEdge(config-target-0/10.0.0.1/162)# group-name all-traps 
vEdge(config-target-0/10.0.0.1/162)# community-name public
vEdge(config-target-0/10.0.0.1/162)# exit
vEdge(config-snmp)# trap target vpn 0 10.0.0.2 162
vEdge(config-target-0/10.0.0.2/162)# group-name critical-traps 
vEdge(config-target-0/10.0.0.2/162)# community-name public
vEdge(config-target-0/10.0.0.2/162)# exit
vEdge(config-snmp)# show full-configuration 
snmp
 view community-view
 !
 community public
  view          community-view
  authorization read-only
 !
 group groupAuthPriv auth-priv
  view v2
 !
 user u1
  auth          sha
  auth-password $8$UZwdx9eu49iMElcJJINm0f202N8/+RGJvxO+e9h0Uzo=
  priv          aes-cfb-128
  priv-password $8$eB/I+VXrAWDw/yWmEqLMsgTcs0omxcHldkVN2ndU9QI=
  group         groupAuthPriv
 !
 trap target vpn 0 10.0.0.1 162
  group-name     all-traps
  community-name public
 !
 trap target vpn 0 10.0.0.2 162
  group-name     critical-traps
  community-name public
 !
 trap group all-traps
  all
   level critical major minor
  !
 !
 trap group critical-traps
  bfd
   level critical
  !
  control
   level critical
  !
  hardware
   level critical
  !
  omp
   level critical
  !
 !
!
vEdge(config-snmp)#

Information About On-Demand Troubleshooting

On-demand troubleshooting lets you view detailed information about the flow of traffic from a device.

By default, Cisco SD-WAN Manager captures aggregated information about flows. You can obtain detailed information for specific devices and for specific historical time periods by adding an on-demand troubleshooting entry. When you add an entry, Cisco SD-WAN Manager compiles detailed information according to parameters that you configure.

To conserve system resources, Cisco SD-WAN Manager compiles detailed information only when you request it by adding an entry. In addition, Cisco SD-WAN Manager stores the information for a limited time (3 hours by default), then removes it. You can request the same information again, if needed.

Note	On a Cisco SD-WAN Manager cluster setup, only a connected node can remove an on-demand troubleshooting task or mark it as complete.

Restrictions for On-Demand Troubleshooting

Ensure that no Cisco or third-party APIs that instruct on-demand troubleshooting to stop are called when you are using on-demand troubleshooting. These APIs prevent on-demand troubleshooting from compiling information.

Page Elements

The On Demand Troubleshooting window provides options for configuring and adding an on-demand troubleshooting entry. The On Demand Troubleshooting window displays information about existing on-demand troubleshooting entries and provides the following information and options.

Configure On-Demand Troubleshooting

You can configure on-demand troubleshooting for a device from the Tools > On Demand Troubleshooting window in Cisco SD-WAN Manager. This window provides options for adding an on-demand troubleshooting entry, and for managing existing entries.

Cisco vManage Release 20.6.1 and earlier: You can configure on-demand troubleshooting for a device from the Monitor > On Demand Troubleshooting window in Cisco SD-WAN Manager.

You can also start on-demand troubleshooting from various locations in the Monitor > Devices window for a device. See View On-Demand Troubleshooting Information for a Device.

Cisco vManage Release 20.6.1 and earlier: You can start on-demand troubleshooting from various locations in the Monitor > Network window for a device.

On-demand troubleshooting is qualified for troubleshooting entries for up to 10 devices concurrently.

Add an On-Demand Troubleshooting Entry

Adding an entry in the On Demand Troubleshooting window instructs Cisco SD-WAN Manager to compile detailed troubleshooting information for the device that you specify, using the parameters that you configure.

To add an on-demand troubleshooting entry, follow these steps:

1. From the Cisco SD-WAN Manager menu, choose Tools > On Demand Troubleshooting.

   Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > On Demand Troubleshooting.

2. From the Select Device drop-down list, choose the Cisco IOS XE Catalyst SD-WAN device or the Cisco vEdge device for which you want to enable on-demand troubleshooting.

3. From the Select Data Type drop-down list, choose SAIE or ConnectionEvents.

4. Choose an option for the data backfill period:

   • Last 1 hour: Provides detailed stream information for the period beginning 1 hour before you add the troubleshooting entry and ending at the time that you add the entry.

   • Last 3 hours: Provides detailed stream information for the period beginning 3 hours before you add the troubleshooting entry and ending at the time that you add the entry.

   • Custom Date and Time Range: Use the Start date and time and the End date and time fields to designate the backfill period that you want. Note that the End date and time value cannot be later than the current date and time.

5. Click Add.

   The troubleshooting entry appears in the table of entries. When the value in the Status field for the entry shows the value Completed, you can view the troubleshooting information from the Monitor > Devices window, as described in View On-Demand Troubleshooting Information for a Device.

Update an On-Demand Troubleshooting Entry

Update an on-demand troubleshooting entry to make changes to its configuration settings. For example, update an entry to adjust its backfill period.

Only entries that are in the QUEUED state can be updated.

To update an on-demand troubleshooting entry, follow these steps:

1. From the Cisco SD-WAN Manager menu, choose Tools > On Demand Troubleshooting.

   Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > On Demand Troubleshooting.

2. In the table of entries, click … adjacent to the entry that you want to update and choose Update.

3. In the Update Troubleshoot Status dialog box that is displayed, configure the settings as needed, and click Add.

Delete an On-Demand Troubleshooting Entry

Deleting an on-demand troubleshooting entry removes the entry from Cisco SD-WAN Manager. After you delete an entry, you can no longer view its detailed information.

Deleting an entry can help free resources in Cisco SD-WAN Manager.

To delete an on-demand troubleshooting entry, follow these steps:

1. From the Cisco SD-WAN Manager menu, choose Tools > On Demand Troubleshooting.

   Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > On Demand Troubleshooting.

2. In the table of entries, click … adjacent to the entry that you want to delete and choose Delete on demand queue.

3. In the Delete On Demand Status window that is displayed, click OK.

View On-Demand Troubleshooting Information for a Device

You can view on-demand troubleshooting information for a device from the Network window for that device.

Before you can view this information, at least one on-demand troubleshooting entry must exist for the device. Add an entry from the On Demand Troubleshooting window as described in Add an On Demand Troubleshooting Entry, or add an entry from the Network window as described in the following procedure.

1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

   Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

2. In the Hostname column, click the device for which you want to view the information.

3. Perform either of these actions:

   • To view the troubleshooting information for an SAIE application:

     1. Click SAIE Applications.

        Note	In Cisco vManage Release 20.7.1 and earlier releases, SAIE Applications is called DPI Applications.

     2. In the Applications Family table, click an application family.

     3. In the Applications table, click an application.

   • To view troubleshooting information for a specific metric, in the left pane, under ON-DEMAND TROUBLESHOOTING click an option. Not all options apply to all device types.

     • FEC Recovery Rate

     • SSL Proxy

     • AppQoe TCP Optimization

     • AppQoE DRE Optimization

     • Connection Events

     • WAN Throughput

     • Flows

     • Top Talkers

   The Flows and Top Talkers metrics are only for TCP Optimized flows.

   If on-demand troubleshooting is configured for the device, detailed troubleshooting information appears. This information includes traffic statistics and metrics such as source IP address, destination IP address, number of packets, number of bytes, and more. Use the options that are available and hover your cursor over elements on the graphs to view the information that you need.

   Note

   Starting from Cisco IOS XE Release 17.9.1a, use the policy ip visibility features enable command to manually enable or disable the feature fields in Flexible Netflow (FNF). Use the show sdwan policy cflowd-upgrade-status command to check which features were enabled before the version upgrade. You have to manually control the features after a version upgrade using the disable or enable commands. For more information, see policy ip visibility command page.

   If on-demand troubleshooting information is not configured, the Enable On Demand Troubleshooting option is displayed. Continue to Step 4.

4. If the Enable On Demand Troubleshooting option is displayed, perform these actions to start this feature for the selected device:

   1. Click Enable On Demand Troubleshooting.

   2. Choose one of the following options:

      • Quick Enable: Starts an on-demand troubleshooting entry with a backfill period of 3 hours. With this option, detailed stream information for the past 3 hours becomes available.

        After you choose this option, click Refresh to view the detailed troubleshooting information. It can take a few minutes for this information to become available. Alternatively, click Go to On Demand Troubleshooting to display the On Demand Troubleshooting window that includes the entry that you just added.

      • Go to On Demand Troubleshooting: Displays the On Demand Troubleshooting window. Add an entry in this window as described in Add an On Demand Troubleshooting Entry. Repeat Steps 1 to Step 3 in this procedure to view the detailed information.

View Progress of On-Demand Troubleshooting

Minimum supported release: Cisco vManage Release 20.11.1

After you enable on-demand troubleshooting, the On-demand Troubleshooting in Progress message appears on the Monitor > Devices page. The message remains until the troubleshooting is complete.

Click a chart option to view the troubleshooting progress in a graphical format. Select a time period to display data or click Custom to display a selection of a custom time period.

You can use the request nms olap-db command to start, stop, or restart the Cisco SD-WAN Manager online analytical processing (OLAP) database or view the status of the database.

For more information about this command, see request nms olap-db.

View Detailed Top Source Data

After on-demand troubleshooting is configured, you can view detailed information about top application usage for a device. To do so, follow these steps:

1. From the Cisco SD-WAN Manager menu, choose Monitor > Overview > Top Applications.

   Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Dashboard > Main Dashboard > Top Applications.

2. In the SAIE Application tab, click an application usage bar in the chart.

   Note	In Cisco vManage Release 20.7.1 and earlier releases, SAIE Application is called DPI Application.

3. In the chart for the application that you selected, click the device usage bar.

   If on-demand troubleshooting is configured for the device, detailed top source data appears.

   If on-demand troubleshooting information is not configured, the Go to On Demand Troubleshooting option appears. Continue to Step 4.

4. If the Go to On Demand Troubleshooting option appears, perform these actions:

   1. Click Go to On Demand Troubleshooting to display the On Demand Troubleshooting window.

   2. In the On Demand Troubleshooting window, add an entry, as described in Add an On Demand Troubleshooting Entry.

   3. Repeat Step 1 to Step 3 in this procedure to view the detailed information.

When something of interest happens on an individual device in the overlay network, one of the ways the device reports it is by generating a system logging (syslog) message and place it in a syslog file in the /var/log directory on the local device and, if configured, on a remote device.

On Cisco Catalyst SD-WAN devices, you can log event notification system log (syslog) messages to files on the local device or on a remote host, or both. On the local device, syslog files are placed in the /var/log directory.

Configure System Logging

Logging syslog messages with a priority level of "error," to the local device's hard disk, is enabled by default. Log files are placed in the local /var/log directory. By default, log files are 10 MB in size, and up to 10 files are stored. After 10 files have been created, the oldest one is discarded to create a file for newer syslog messages.

To modify the default syslog parameters from Cisco SD-WAN Manager, use the Logging feature template. From the CLI, include the logging disk or logging server commands in the device configuration.

View Syslog Logging Information

1. From the Cisco SD-WAN Manager menu, choose Administration > Settings and, ensure that Data Stream is enabled.

2. From the Cisco SD-WAN Manager menu, choose Monitor > Devices, and choose a device from the list of devices that appears.

   Cisco vManage Release 20.6.x and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network, and choose a device from the list of devices that appears.

3. Click Troubleshooting in the left pane.

4. In the Logs area, click Debug Log.

5. In the Log Files field, choose the name of the log file. The lower part of the screen displays the log information.

To view the contents of a syslog file from the CLI, use the show log command. For example:

Device# show log auth.log tail 10==> /var/log/auth.log <==auth.info: Nov 14 14:33:35 vedge sshd[2570]: Accepted publickey for admin from 10.0.1.1 port 39966 ssh2: RSA SHA256:pkFQ5wE//DmiA0d0JU1rOt91CMTVGkscm9wLSYQrI1sauth.info: Nov 14 14:39:42 vedge sshd[2578]: Received disconnect from 10.0.1.1 port 39966:11: disconnected by userauth.info: Nov 14 14:39:42 vedge sshd[2578]: Disconnected from 10.0.1.1 port 39966auth.info: Nov 16 10:51:45 vedge sshd[6106]: Accepted publickey for admin from 10.0.1.1 port 40012 ssh2: RSA SHA256:pkFQ5wE//DmiA0d0JU1rOt91CMTVGkscm9wLSYQrI1sauth.info: Nov 16 11:21:55 vedge sshd[6108]: Received disconnect from 10.0.1.1 port 40012:11: disconnected by userauth.info: Nov 16 11:21:55 vedge sshd[6108]: Disconnected from 10.0.1.1 port 40012auth.info: Nov 17 12:59:52 vedge sshd[15889]: Accepted publickey for admin from 10.0.1.1 port 40038 ssh2: RSA SHA256:pkFQ5wE//DmiA0d0JU1rOt91CMTVGkscm9wLSYQrI1sauth.info: Nov 17 13:45:13 vedge sshd[15894]: Received disconnect from 10.0.1.1 port 40038:11: disconnected by userauth.info: Nov 17 13:45:13 vedge sshd[15894]: Disconnected from 10.0.1.1 port 40038auth.info: Nov 17 14:47:31 vedge sshd[30883]: Accepted publickey for admin from 10.0.1.1 port 40040 ssh2: RSA SHA256:pkFQ5wE//DmiA0d0JU1rOt91CMTVGkscm9wLSYQrI1s

To view the configured system logging settings for a device, use the show logging command from the CLI. For example:

Device# show logging 
System logging to host  in vpn 0 is disabled
Priority for host logging is set to: emerg

System logging to disk is disabled
Priority for disk logging is set to:  err
File name for disk logging is set to:  /var/log/vsyslog
File size for disk logging is set to:  10 MB
File recycle count for disk logging is set to:  10

Syslog facility is set to: all facilities 

System Log Files

Syslog messages at or above the default or configured priority value are recorded in a number of files in the /var/log directory on the local device. These files include the following:

• auth.log—Login, logout, and superuser access events, and usage of authorization systems.

• kern.log—Kernel messages

• messages—Consolidated log file that contains syslog messages from all sources.

• vconfd—All configuration-related syslog messages

• vdebug—All debug messages for modules whose debugging is turned on and all syslog messages above the configured priority value. Debug logging supports various levels of logging based on the module. Different modules implement the logging levels differently. For example, the system manager (sysmgr) has two logging levels (on and off), while the chassis manager (chmgr) has four different logging levels (off, low, normal, and high). You cannot send debug messages to a remote host. To enable debugging, use the debug operational command.

• vsyslog—All syslog messages from Cisco SD-WAN processes (daemons) above the configured priority value. The default priority value is "informational" (severity level 6), so by default, all "notice", "warning", "error", "critical", "alert", and "emergency" syslog messages (severity levels 5 through 0, respectively) are saved.

The Cisco Catalyst SD-WAN software does not use the following standard LINUX files, which are present in /var/log, for logging: cron.log, debug, lpr.log, mail.log, and syslog.

The writing of messages to syslog files is not rate-limited. This means that if many syslog messages are generated in a short amount of time, the overflow messages are buffered and placed in a queue until they can be written to a syslog file. The overflow messages are not dropped.

For repeating syslog messages—identical messages that occur multiple times in succession—only one copy of the message is placed in the syslog file. The message is annotated to indicate the number of times that the message occurred.

The maximum length of a syslog message is 1024 bytes. Longer messages are truncated.

Syslog messages related to AAA authentication and Netconf CLI access and usage are placed in the auth.log and messages files. Each time Cisco SD-WAN Manager logs in to a Cisco vEdge device to retrieve statistics and status information and to push files to the router, the router generates AAA and Netconf log messages. So, over time, these messages can fill the log files. To prevent these messages from filling the log files, you can disable the logging of AAA and Netconf syslog messages:

Device(config)# system aaa logsViptela(config-logs)# audit-disableViptela(config-logs)# netconf-disable
            

Syslog Message Format

Syslog message generated by the Cisco Catalyst SD-WAN software have the following format:

                facility.source
                date - source - module - level - MessageID: text-of-syslog-message
            

Here is an example syslog message. This is logged with local7 facility and level "notice".

Syslog Message Acronyms

The following acronyms are used in syslog messages and in the explanations of the messages:

To see a list of the various syslog messages generated, see Syslog Messages in the Appendix.

The tables below list the syslog messages generated by Cisco vEdge devices and Cisco IOS XE Catalyst SD-WAN devices. The messages are grouped based on the software module that generates them. The software modules are typically processes (daemons) that run on the device.

All syslog messages are generated on all the devices unless otherwise indicated.

Each syslog message has a corresponding number. The tables list all syslog messages and their number even if the messages are defined in the header files but are not currently used in the operating software. For these messages, the Message Format, Description, and Action fields are empty.

In these tables, the Action field indicates the recommended action you should take in response to the syslog message:

• A—Automatically open a ticket in your organization's support team.

• AE—Automatically open a support ticket and escalate the ticket

• E—Send email to the appropriate team within your organization.

If you see a syslog message that is not listed in one of the tables below, please send the message, along with the device and software version, to Cisco support.

Note	For information about Cisco SD-WAN Manager syslog message format, syslog message levels, and system log files, see Syslog Messages.

CFGMGR: Configuration Manager Process

Priority: Informational

CFLOWD: Cflowd Traffic Flow Monitoring Process

Priority: Informational

Priority: Notice

CHMGR: Chassis Manager

The chassis manager process runs only on physical routers.

Priority: Informational

Priority: Notice

Priority: Warning

Priority: Error

Priority: Critical

Priority: Alert

CVMX: Internal Cavium Driver Process

Priority: Informational

CXP: Cloud onRamp for SaaS Process

Priority: Informational

CONTAINER: Containers

Priority: Informational

DBGD: Debug Process

Priority: Informational

DHCPC: DHCP Client

The DHCP client process runs only on Cisco vEdge devices.

Priority: Informational

Priority: Critical

DHCP: DHCP Server

The DHCP server process runs only on Cisco vEdge devices.

Priority: Informational

FPMD: Forwarding Policy Manager Process

Priority: Informational

Priority: Alert

FTMD: Forwarding Table Management Process

The forwarding table management process runs only on Cisco vEdge devices.

Priority: Informational

Priority: Notice

Priority: Critical

Priority: Warning

Priority: Error

GPS: Global Positioning System

Priority: Informational

IGMP: Internet Group Management Protocol

Priority: Informational

LIBBSS: UNIX BSS Library

Unused Messages

LIBCHMGR: Chassis Manager Library Process

Unused Messages

MSGQ: Message Queue Process

Ununsed Messages

OMP: Overlay Management Protocol

Priority: Informational or Other

Priority: Notice

PIM: Protocol-Independent Multicast Process

Priority: Informational

Priority: Notice

Priority: Error

POLICY: Policy Process

Unused Messages

RESOLV: Resolver Process

Unused Messages

SNMP Listener Process

Unused Messages

SYSMGR: System Manager Process

The system manager process (daemon) spawns, monitors, and terminates all the processes in the system, and it collects and logs vital system information, such as memory and CPU status.

Priority: Informational

Priority: Notice

Priority: Warning

Priority: Error

Priority: Critical

TCPD: TCP Options Process

Priority: Informational

Priority: Debug

TRACKER: Interface Tracker Process

Priority: Informational

VCONFD: Cisco Catalyst SD-WAN Configuration Process

Priority: Informational

VDAEMON: Cisco Catalyst SD-WAN Software Process

Priority: Informational

Priority: Error

VRRP: Virtual Router Redundancy Protocol

The VRRP process runs only on Cisco vEdge devices.

Priority: Informational

WLAN: Wireless LAN Process

The wireless LAN process runs only on Cisco vEdge devices.

Priority: Informational

WWAND: Cellular Process

The wireless WAN process runs only on Cisco vEdge devices.

Priority: Informational

Priority: Error

Resolve Problems with Cellular Interfaces

This topic describes the most common issues and error messages that occur with cellular connections from the router to the cellular network, and the steps to resolve them.

Insufficient Radio Signal Strength

Problem Statement

The cellular module in the router cannot detect a radio signal from the service provider network.

Identify the Problem

• The signal strength displayed in the Cisco SD-WAN Manager Cellular Status screen or with the show cellular status CLI command, or in the Cellular Radio screen or with the show cellular radio command is no signal, poor, or good. It should be excellent. The following table lists the ranges of signal strengths:

Note	All parameters must be considered together and not in isolation. For example, a strong RSSI does not mean signal quality is good if RSRP is bad.

• The wireless LED on the router is lit (solid or blinking) and is red, orange or yellow, or it is blinking green. It should be solid green.

Resolve the Problem

1. Examine the router to verify that both basic antennas are correctly installed.

2. Contact the service provider to verify that the location has coverage.

3. Move the router to a new location within the building.

4. Procure an additional external cabled antenna and connect it to the router.

Modem Status Remains in Low-Power Mode

Problem Statement

End users cannot connect to the cellular network, and the modem status remains in low-power mode.

Identify the Problem

• End users cannot connect to the cellular network.

• The error message "Missing or unknown APN" is generated.

• The signal strength is less than excellent.

Resolve the Problem

1. Verify that there is sufficient radio signal strength. If there is not, follow the instructions in the Insufficient Radio Signal Strength section.

2. Verify that the cellular0 interface is operational. When the cellular interface is shut down, the modem status is set to Low Power mode. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices.

   Cisco vManage Release 20.6.1 and earlier: To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Network.

   Then click Real Time, and from the Device Options drop-down list, choose Interface Detail.

   To do this from the CLI, use the show interface command. Check that the Admin Status and Oper Status values are both Up.

3. Verify that the modem temperature is not above or below the threshold temperatures. To view the modem temperature, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and select the router.

   Cisco vManage Release 20.6.1 and earlier: To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Network.

   Then click Real Time, and from the Device Options drop-down list, choose Cellular Modem.

   From the CLI, use the show cellular modem command.

4. Check that the access point name (APN) in the profile for the cellular0 interface matches the name expected by your service provider. Some service provides required that you configure the APN, and they include configuration instructions in the SIM card package.

   1. To check which APN name is configured, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and select the router.

      Cisco vManage Release 20.6.1 and earlier: To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Network.

      Then click Real Time, and from the Device Options drop-down list, choose Cellular Profiles.

      From the CLI, use the ; show cellular profiles command. The APN column shows the name of the APN. Each profile specifies an access point name (APN), which is used by the service provider to determine the correct IP address and connect to the correct secure gateway. For some profiles, you must configure the APN.

   2. If the APN is not the one required by the service provider, configure the correct APN. From the Cisco SD-WAN Manager menu, choose Configuration > Templates and use the Cellular Profile feature template.

      To configure this from the CLI, use the cellular cellular0 profile apn command.

5. If none of the previous steps works, reset the cellular interface.

Error Messages

The following table list the most common error messages that are displayed regarding cellular interfaces:

This topic describes how to check and resolve connection problems between a WiFi client and a WiFi network that is provided by a WiFi router. The procedures described here are applicable to devices that support WiFi only.

Check for WiFi Connection Problems

If a WiFi client is unable to connect to a WiFi network when a router is providing the WiFi network, follow these steps to determine the source of the problem. To perform each step, use a method appropriate for the WiFi client.

1. Verify that the WiFi client can locate the service identifier (SSID) advertised by the router. If the client cannot find the SSID, see the section, SSID Not Located.

2. Verify that the WiFi client can connect to the SSID advertised by the router. If the client cannot connect to the SSID, see the section, SSID Connection Fails.

3. Verify that the WiFi client has been assigned an IP address. If the client cannot obtain an IP address, see the section, Missing IP Address.

4. Verify that the WiFi client can access the Internet. If the client cannot connect to the Internet, see section, Internet Connection Failure.

5. If the WiFi client connection is slow or if you notice frequent disconnects, see section, WiFi Speed Is Slow.

Resolve Problems with WiFi Connections

This section describes the most common issues that occur with WiFi connections between a WiFi client and a router, and it describes steps to resolve the issues.

SSID Not Located

Problem Statement

The WiFi client cannot locate the SSID advertised by the router.

Resolve the Problem

1. Ensure that the basic service set identifier (BSSID) address for the SSID is valid:

   1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

      Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

   2. Choose a device from the device list that appears.

   3. From the left pane, choose WiFi. The right pane displays information about WiFi configuration on the router.

   4. In the right pane, locate the SSID. Check that the BSSID for this SSID does not have a value of 00:00:00:00:00:00.

   5. If the BSSID is 00:00:00:00:00:00, the WLAN (VAP) interface for this SSID may be misconfigured. Ensure that the WLAN interface has been added to a bridge during the configuration process. To view the running configuration of the device, from the Cisco SD-WAN Manager menu, choose Configuration > Devices. For the desired device, click ...and choose Running Configuration.

      To view the running configuration of the device from the CLI, run the show running-config command. To add the WLAN interface to a bridge — from the Cisco SD-WAN Manager, choose Configuration > Templates.

      Click Feature Templates, and choose the Bridge feature template.

      Note	In Cisco vManage Release 20.7.1 and earlier releases, Feature Templates is titled Feature.

2. Eliminate static channels. A static channel is one where you explicitly configure the radio channel rather than allowing the router to automatically select the best radio channel. A slow static channel may appear to be an unreachable SSID.

   1. View the current SSID channel setting for the router. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the list of devices that appears. Then click Real Time, and in the Device Options drop-down list, choose WLAN Clients or WLAN Radios.

      From the CLI, run the show wlan clients or show wlan radios command.

   2. If the channel is set to a specific number, change the value to "auto". To do this, use the WiFi Radio feature template in Cisco SD-WAN Manager.

      From the CLI, run the wlan channel auto command.

3. Ensure that the WiFi client is using the same radio band as the router, either 2.4 GHz (for IEEE 802.11b/g/n) or 5 GHz (for IEEE802.11a/n/ac):

   1. Check which radio band the WiFi client supports.

   2. Check the router’s Select Radio setting. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the device list that appears. Then click Real Time, and in the Device Options drop-down list, choose WLAN Radios.

      From the CLI, run the show wlan radios command.

   3. If the router and WiFi client radio band settings do not match, either change the WiFi client's radio band or change the settings on the router so that they match. To do this, use the Wifi Radio feature template.

      From the CLI, run the wlan command.

SSID Connection Fails

Problem Statement

The WiFi client can locate the SSID advertised by the router but cannot connect to it.

Resolve the Problem

1. If you configure passwords locally on the router, ensure that the WiFi client's password matches the SSID's password.

2. If you are using a RADIUS server, ensure that the RADIUS server is reachable and that the WiFi client's username and password match the RADIUS configuration:

   1. To verify that the RADIUS server is reachable from the router, ping the server. To do this in Cisco SD-WAN Manager, ping a device. From the CLI, run the ping command.

   2. Check for matching passwords on the RADIUS server and WiFi client.

3. Ensure that you do not exceed the maximum number of clients for this SSID:

   1. Verify the number of used clients and the maximum number of clients:

      • From the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the device list that appears. From the left pane, select WiFi. In the right pane, locate the SSID. Check the No. of Clients field. If the used/maximum values are equal, no more clients can connect to this SSID.

      • From the CLI, run the show wlan interfaces detail command.

   2. If needed, increase the maximum clients setting for your SSID. To do this use the WiFi SSID feature template in Cisco SD-WAN Manager.

      From the CLI, run the max-clients command.

4. Ensure that the WiFi client supports WPA2 management security:

   1. Check your Management Security setting. To do this, from the Cisco SD-WAN Manager menu, choose Monitor > Devices and choose a device from the device list that appears. Then click Real Time, and in the Device Options drop-down list, choose WLAN Interfaces.

      From the CLI, run the show wlan interfaces command. If the management security value is set to "required," the WiFi client must support WPA2 security.

   2. If necessary, change the Management Security setting for your SSID to "optional" or "none." To do this in Cisco SD-WAN Manager, use the WiFi SSID feature template.

      From the CLI, run the mgmt-security command.

Missing IP Address

Problem Statement

The WiFi client can connect to the SSID, but cannot obtain an IP address.

Resolve the Problem

Ensure that a DHCP server is reachable and has an available IP address in its address pool:

1. If the router is acting as a DHCP helper (DHCP relay agent), ping the DHCP server to ensure that it is reachable from the router.. From the CLI, run the ping command.

2. If you are using a remote DHCP server, check that the remote DHCP server has an available IP address in its address pool.

3. If the router is acting as the local DHCP server:

   1. View the number of addresses being used. From the Cisco SD-WAN Manager menu, Monitor > Devices and choose a device from the device list that appears. Next, click Real Time, and from the Device Options drop-down list, choose DHCP Servers.

      From the CLI, run the show dhcp server command.

   2. Compute the number of IP addresses in the pool based on the configured DHCP address pool size and the number of addresses excluded from the DHCP address pool. To view these values in Cisco SD-WAN Manager, from the Cisco SD-WAN Manager menu, choose Configuration > Devices. For the desired router, click ... and choose Running Configuration.

      To view them from the CLI, run the show running-config command.

   3. If necessary, increase the range of addresses in the router's DHCP address pool using the DHCP-Server feature template in Cisco SD-WAN Manager.

Internet Connection Failure

Problem Statement

The WiFi client is connected to the SSID and has an IP address, but it cannot connect to the Internet.

Resolve the Problem

Ensure that the WiFi client has received the correct default gateway and DNS settings from the DHCP server:

1. If the DHCP server is remote, check the settings on the server.

2. If the router is the DHCP server, ensure that the default gateway and DNS server settings are the same as those on the WiFi client. To view the settings in Cisco SD-WAN Manager, from the Cisco SD-WAN Manager menu, choose Monitor > Devices, and choose a device from the device list that is displayed. Click Real Time, and in the Device Options drop-down list, choose DHCP Interfaces.

   From the CLI, run the show dhcp interface  command.

WiFi Speed Is Slow

Problem Statement

The WiFi client can connect to the Internet, but the connection speed is slow.

Resolve the Problem

Allow the router to choose the best WiFi channel:

1. View the current SSID channel setting for the router. To do this in Cisco SD-WAN Manager, from the Cisco SD-WAN Manager menu, choose Monitor > Devices, and choose a device from the device list that is displayed. Click Real Time, and in the Device Options drop-down list, choose WLAN Clients.

   From the CLI, run the show wlan clients or show wlan radios command.

2. If the channel is set to a specific number, change the value to "auto". To do this in Cisco SD-WAN Manager, use the WiFi Radio feature template.

   From the CLI, run the wlan channel auto command.

Set Audit Log Filters

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Audit Log.

   Cisco vManage Release 20.6.1 and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Audit Log.

2. Click the Filter.

3. In the Module field, choose the entity for which you are collecting audit logs. You can choose more than one entity.

4. Click Search to search for logs that match the filter criteria.

Cisco SD-WAN Manager displays a log of activities both in table and graphical format.

Starting from Cisco IOS XE Catalyst SD-WAN Release 17.12.1a and Cisco Catalyst SD-WAN Manager Release 20.12.1, the following enhancements have been introduced in Audit Logs on Cisco SD-WAN Manager.

• Notifications for multiple failed attempts: To enable notifications for unauthorized activity.

  Configure the lockout policy using the system aaa lockout-policy command.

  To verify the configured lockout policy, use the show running-config command. In the below snippet, the policy specifies the lockout interval as 240 seconds, the fail interval for 900 seconds, and 3 fail attempts are permitted before a notification is sent:

  Use the show alarms history command to view the following additional details:

  • aaa-user-locked

  • aaa-user-login-anomaly

• Notifications for higher number of logins: To enable notifications when the number of logins reaches the limit.

  To verify the configured system alarms, use the show running-config command. In the below snippet, the number of logins specified is 3, and the login interval is set at 60 seconds:

Export Audit Log Data in CSV Format

To export data for all audit logs to a file in CSV format, click Export.

Cisco SD-WAN Manager downloads all data from the audit logs table to an Excel file to a CSV format. The file is downloaded to your browser’s default download location and is named Audit_Logs.csv.

View Audit Log Details

To view detailed information about any audit log:

1. Choose the audit log row of from the table

2. For the desired row, click ... and choose Audit Log Details.

The Audit Log Details dialog box opens, displaying details of the audit log.

View Changes to a Configuration Template

You can view changes for previous and current configuration made on a template. To view configuration changes made to a template, do the following:

1. Click the audit log row in the table where the module type is a template.

2. Click ... adjacent to the template module and click Config Diff.

   The Config Difference pane displays a side-by-side view of the differences between the configuration that was originally in the template and the changes made to the configuration. To view the changes inline, click Inline Diff.

To view the updated configuration on the device, click Configuration.

Starting from Cisco IOS XE Catalyst SD-WAN Release 17.6.1a and Cisco SD-WAN Release 20.6.1, for template and policy configuration changes, the Audit Logs option displays the action performed. To view the previous and current configuration for any action, click Audit Log Details. Audit logs are collected when you create, update, or delete device or feature templates, and localized or centralized, and security policies. Audit logs shows the changes in API payloads when templates or policies are attached or not attached.

This topic describes how to monitor the status of cellular interfaces in Cisco Catalyst SD-WAN devices.

Monitor Cellular Interfaces

You can verify signal strength and service availability using either Cisco SD-WAN Manager or the LED on the router. You can view the last-seen error message for cellular interfaces from Cisco SD-WAN Manager.

Verify Signal Strength

1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

   Cisco vManage Release 20.6.x and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

2. From the Device Groups drop-down list, choose a group that the device belongs to.

3. Choose a device by clicking its name in the Hostname column.

4. Click Real Time in the left pane.

5. From the Device Options drop-down list in the right pane, choose Cellular Radio.

   The values for the different cellular signals are displayed. If signal strength is poor, or there is no signal, see Troubleshoot Common Cellular Interface Issues.

CLI equivalent: show cellular status

Verify Radio Signal Strength Using the Router LED

To check signal strength and service availability of a cellular connection from the router, look at the WWAN Signal Strength LED. This LED is typically on the front of the routers, and is labeled with a wireless icon.

The following table explains the LED color and associated status:

View Error Messages for Cellular Interfaces

1. From the Cisco SD-WAN Manager menu, choose Monitor > Devices.

   Cisco vManage Release 20.6.x and earlier: From the Cisco SD-WAN Manager menu, choose Monitor > Network.

2. Choose a device by clicking its name in the Hostname column.

3. Click Real Time in the left pane.

4. From the Device Options drop-down list in the right pane, choose Cellular Status.

   The output displayed includes a column for Last Seen Error

CLI equivalent: show cellular status