Network-Wide Path Insights on SD-Routing Devices
Information About Network-Wide Path Insight on SD-Routing Devices
NWPI is a diagnostic tool natively built into Cisco SD-WAN Manager that helps network administrators find the source of the network issues that users face from time to time while accessing their applications residing on-prem or in the cloud. It provides end-to-end application-tracing serviceability in a Cisco Catalyst SD-WAN network and lets the network administrators initiate application tracing and displays the trace results collected from multiple devices in a consolidated view. NWPI also provides comprehensive analyses of traffic flows in the network with information on applications accessed by users, classification of business-critical flows, monitoring and reporting of network delays, troubleshooting tips, and graphical deep insights into flow analyses.
How Network-Wide Path Insights work on SD-Routing Devices
SD-Routing uses Cisco Catalyst SD-WAN Manager to manage and optimize network paths. NWPI integrates with the SD-WAN Manager to gather detailed path information for UDP and TCP traffic. The network operator starts the process by creating an NWPI trace.
In a DMVPN or FlexVPN scenario with NHRP configured, NWPI trace works in Network-Wide mode. Here, DMVPN CMD carries NWPI metadata. As the data travels through routers, each router uses the NWPI metadata to send flow information back to the Manager. It then integrates all the received data into a unified view. In other scenarios, NWPI trace works in standalone mode. No metadata is added, and only the first router's flow information is sent to the SD-WAN Manager
Working Modes of NWPI on SD-Routing Devices
Mode |
Description |
Traffic Flow Details |
---|---|---|
Standalone Mode |
Shows the first hop router's information only. |
|
Network Wide Mode |
Provides end-to-end path across multiple sites with network-wide correlation. |
Note Network Wide Mode leverages DMVPN CMD for NWPI metadata across sites. DMVPN's network-id is shown as Local and Remote transport unique ID. |
Prerequisites for Network-Wide Path Insights on SD-Routing Devices
Ensure that the Data Stream option is enabled in Cisco SD-WAN Manager. In a Cisco Catalyst multitenant deployment, you must have the provider role to enable this option. To enable this option, perform the following steps.
-
From the Cisco SD-WAN Manager menu, choose
. -
For the Data Stream option, click View.
-
Click Edit and choose Enable.
-
Click Save.
Restrictions for Network-Wide Path Insights on SD-Routing Devices
-
NWPI traces only UDP and TCP traffic
-
Not all packet traces are captured per flow. The system automatically samples the most typical packets.
-
You can monitor up to two traces per device and a maximum of 10 concurrent active traces per Cisco Catalyst SD-WAN Manager tenant.
-
The number of active flows monitored ranges from 50 to 100 per device, with a maximum of 60,000 supported completed flows.
-
Path insight does not display correctly if IPv6 is used as the underlay for the DMVPN tunnel.
-
NWPI does not provide accurate information in VASI deployments.
-
NWPI does not work properly when platform QoS port-channel-aggregate is enabled.
-
Path insight does not show correct information if NAT64 or NAT46 occurs.
-
NWPI cannot display MPLS information if the first hop is an MPLS PE router.
-
NWPI Domain probe traffic cannot go over a tunnel.
-
When starting a trace, you cannot specify the source interface on SD-Routing devices.
-
When NWPI traces traffic over VTI tunnel on some platforms, downstream hop's LocalEdge is Unknown.
Configure Network-Wide Path Insights Tracing
By default, starting an SD-Routing trace on a HUB site fails and generates an alarm. If you still want to start the trace on a HUB site, disable Network-Wide under Monitoring Settings and then start the trace again.
To start an SD-Routing trace, follow these steps.
Procedure
Step 1 | Initiate a New Trace.
|
Step 2 | Configure Basic Trace Settings.
|
Step 3 | (Optional) Configure Advanced Filters Select specific devices, source ports, destination ports, protocols, and DSCP types for the trace. |
Step 4 | (Optional) Configure Monitoring Settings
|
Step 5 | (Optional) Configure Grouping Fields Enable Client Prefix and Server Prefix to additionally group information. Enabling these fields can refine the display of information to meet your preferences. |
Step 6 | (Optional) Configure Synthetic Traffic
|
Step 7 | Start and review the trace
|
View and Manage Trace Information
The path trace instances appear with unique trace IDs in the Trace area. Information about each instance is also displayed, including its state and the actions that you can perform. The following table describes the fields in the All Trace area and the action that the network administrators can perform. Note that you can view the Multi-Traces Insight summary by selecting the checkbox next to Trace Names.
Parameter |
Description |
---|---|
Trace Name |
Trace name and the Insight Summary link, which provides you more information about the traces. |
Trace ID |
System-generated identifier of the trace. |
Start Time |
Date and time at which you started the trace. |
Stop Time |
Date and time at which the trace ended. |
Src.Site |
Source of a trace. |
VPN |
Identifier of the VPN in which the application flow was traced. |
Application/App Group |
The Applications or Application Groups that a trace monitors. |
Domain |
Mentions if the DNS Domain Discovery was enabled or disabled. |
Trace State |
View the status of a trace and error messages, if any, that have been generated. |
Action |
|
Display Insight Summary
An insight summary provides trace-level insight information for application traffic and flows. This information appears in a slide-in pane. In the All Trace section, click Insight Summary in the Trace Name column for the trace that you want.
Overview
The Overview tab displays the following information on Sampled Flow Insight and Application Statistics respectively.
VPN |
This is the identifier of a VPN in which the application flow will be traced. In the VPN field, information for all the VPNs that the trace detects is selected by default. |
Application |
Displays information for all flows that include application traffic. Select the Application check box to enable the same. This option is enabled by default. |
DNS |
Displays information about each domain that the trace discovers. Select the DNS check box to display information for all the flows with DNS resolution. |
Applications Graph |
Displays the number of flows that the trace detects in each application in the monitored traffic. Hover your cursor over the data points in the graph to display the percentage of total flows that the corresponding application flow represents. |
Critical, Warning, and Informational |
To display information for events that have the corresponding severity level enable Critical, Warning or Informational as per your preference. |
Events Graph |
Displays the events that the trace detects in the monitored traffic and the number of application flows that each event affects. Hover your cursor over the data points in the graph to display the percentage of total application flows that the corresponding event affected. |
Events |
Choose the preferred Events from the drop-down list. By default, all the events that the trace detects appear in this field. |
Hotspot issues [Local Drop, Server No Response, WAN Loss(>5%)] |
Provides information about each application flow that was affected, including the traffic path in which the event occurred and the duration of the event. |
VPN |
Select the preferred VPN from the VPN drop down list. |
Transport Interface |
Select the preferred Interface from the Transport Interface drop down list. |
Device |
Select the preferred Device from the Device drop down list. |
Applications (top 10) - Total Flows |
Total number of application flows for the duration of the trace. The selected color is for the egress WAN interface on which flows were initialized. |
Applications (top 10) - Total Bytes |
Overall application bidirectional bandwidth for the duration of the trace. Upstream and downstream flow bandwidths are counted on the egress WAN on which flows were initialized. |
APP Metrics/APP Metrics(Min/Ave/Max) |
The Min, Ave, Max data in charts show the minimum, average, and maximum values for the duration of the trace. Choose APP Metrics or APP Metrics(Min/Ave/Max) according to your preference. Metrics are calculated at interval of one minute. |
Applications (top 10) – Flow Setup Rate |
New incoming flows per second, calculated at 1-minute intervals. |
Applications (top 10) – Active Flow |
Number of active flows, calculated at 1-minute intervals. |
Applications (top 10) – Bandwidth |
Number of KB per second, calculated at 1-minute intervals. |
Applications (top 10) – Flow Live Time |
Overall lifetime of application flows, based on the flows completed within the monitor interval and calculated at 1-minute intervals. |
App Performance Insight
The App Performance Insight tab displays the following information.
Field |
Description |
---|---|
VPN |
Information for all the VPNs that the trace detects are selected by default. |
Group Fields |
|
Server Side Group By |
You can arrange the information that displays into groups, according to the items that you choose in the Group Fields area. |
Application |
The five applications with the most hotspot issues appear in the Application field by default. |
Hop Metrics |
|
Hop |
Select the preferred Hop from the drop-down list. |
Score graph |
Provides an evaluation of application performance. |
Detailed Application Metrics |
|
Loss graph |
Provides information about packet loss. |
Delay graph |
Provides information about delays in the traffic flows. |
Jitter graph |
Provides information about the inconsistencies in latency in traffic flows. |
CND/SND graph |
Provides information about client network delay (CND) and server network delay (SND). |
Application Path & Performance graph |
Provides a snapshot of bandwidth used and loss information at a particular time. You can choose the time by clicking a dot on a timeline in a metrics graph. |
Event Insight
The Event Insight tab displays the following information about application flows that were affected during each minute of the duration of an event. You can use this information to assist with a root-cause analysis.
Field |
Description |
---|---|
VPN |
Information for all the VPNs that the trace detects are selected by default. |
Group Fields |
|
Server Side Group By |
You can arrange the information that displays into groups, according to the items that you choose in the Group Fields area. |
Application |
The five applications with the most hotspot issues appear in the Application field by default. |
Hop Metrics |
|
Upstream and Downstream |
Click Upstream to display information for upstream traffic in the graph and chart. Click Downstream to display information for downstream traffic in the graph and chart. |
Hop |
Select the preferred Hop from the drop-down list. |
Flow Graph |
Provides information about the number of flows at a particular time. |
Event |
Choose the preferred Events from the drop-down list. By default, all the events that the trace detects appear in this field. |
Application Path and Event Flow (Flow Count) |
Provides detailed information about the effect of designated events at a particular time. Hover your cursor over a data point to see more information. |
QoS Insight
The QoS Insight tab provides comprehensive network-wide information about which application traffic entered specific QoS queues on the devices detected during a trace. It includes details on all hops for the traffic. To display information on this tab, enable the QoS Insight option when you start the trace.
The QoS Insight tab includes both graphical and chart-based information as follows.
Field |
Description |
---|---|
Devices |
The graph and chart display information for each device that appears in the Devices field. |
QoS Drop Rate Graph |
Shows information about the packet or byte drop rates for the selected devices over the period of the trace. |
Applications |
Select the preferred Application from the Application drop down list. To provide complete information about bandwidth consumers that cause dropped packets, this tab displays information for all the applications on a device. |
VPNs |
Select the preferred VPN from the VPN drop down list. It also displays information for default VPN and all the service VPNs, regardless of the service VPNs that you choose by using the VPN filter when you start the trace. |
Interfaces |
Select the preferred Interface from the Interface drop down list. All the items that the trace detects are displayed in these fields by default, except items with a packet per second (PPS) rate of less than 0.05 |
Queues |
Select the preferred Queues from the Queue drop down list. All the items that the trace detects are displayed in these fields by default, except items with a packet per second (PPS) rate of less than 0.05. |
Forward/Drop |
Select the preferred Forward/Drop from the V drop down list. All the items that the trace detects are displayed in these fields by default, except items with a packet per second (PPS) rate of less than 0.05. |
QoS - Applications Distribution |
Provides detailed information about the traffic spectrum and QoS processing at a particular time. It illustrates forwarded or dropped traffic in a flow from an application to a VPN, then to a physical interface, and finally to a queue. |
Display packet drop rate information |
Click Packet to show packet drop rate information in the graph and PPS information in the Sankey chart |
Display byte drop rate information |
Click Byte to show byte drop rate information in the graph and Kbps information in the Sankey chart. |
View Trace Insights
To view detailed information about a flow in a specific trace, click View Insight in the Actions column of the trace list. The detailed information will appear in the Insight area, displaying the following:
DNS Domains
The DNS Domains section is available only when DNS domain discovery is enabled and displays information about each domain that the trace discovers. You can expand any row in the list to display detailed information about the application.
Click Discovered Domains to display information for every domain that the trace discovered but that are not yet traced. Click Monitored Domains to display information only for domains that the trace monitored.
The following table describes the information that appears for DNS Domain:
Field |
Description |
---|---|
Check box |
Select the check box for the domains for which you want monitoring to be enabled or disabled and click Start Flow Monitor or Stop Flow Monitor. |
Application Group or App Group |
Name of the application group that the trace discovered in the domain. |
VPN |
Identifier of the VPN in which the application flow was traced. |
DNS Server |
Destination of DNS packets sent from clients. |
DNS Redirect |
DNS resolver to which a device redirects DNS traffic if a resolver is configured by a centralized policy or by Cisco Umbrella. |
Resolved IP |
DNS-resolved IP address for the application. |
DNS Transport |
Transport type used by the domain. |
DNS Egress |
Egress interface and type used by the domain. |
TTL (sec) |
DNS time to live, in seconds. |
Request |
Number of DNS packets sent. |
Monitor State |
Status of flow monitoring for the domain. |
Applications
The Application section displays information about applications that were traced. You can expand any row in this list to display bidirectional path information with hop-by-hop metrics for each application. Column with * mark means it was showing real-time metrics and statistics during the past 1 minute.
The following table describes the information that appears for Applications:
Field |
Description |
---|---|
Last Updated time |
Date and time at which the information was last refreshed. Instances are refreshed every 10 seconds by default. |
App Name |
Name of the application. |
App Group |
Application group to which the application belongs. |
VPN |
Identifier of the VPN in which the application flow was traced. |
Total bytes (K) |
Number of KBs in the upstream and downstream flows of this application. |
Total Packets |
Number of packets in the upstream and downstream flows of this application. |
KBPS* |
Number of KBs per second in the upstream and downstream flows of this application during the past minute. |
PPS* |
Number of packets per second in the upstream and downstream flows of this application during the past minute. |
Total Flows |
Number of flows that were counted for the application. |
Active Flows* |
Flows that had activity during the past 1 minute. |
Flow Setup Rate (s)* |
Average number of new flows per second during the past 1 minute. |
Flow Live Time (ms) |
Max/Min/Avg: Maximum, minimum, and average number of milliseconds of detectable flow activity during the duration of the trace. |
Sampled Flows |
Number of flows that were sampled in the upstream or downstream traffic of this application. |
Sampled Bytes (K) |
Number of KBs that were sampled in the upstream or downstream traffic of this application. |
Active Flows and Completed Flows
The Active Flows section displays information about the flows that are in the Running state. You can expand a flow instance to display bidirectional flow path information with hop-by-hop metrics. The Completed Flows section shows information about the flows that are in the Stopped state. You can expand a flow instance to display bidirectional flow path information with hop-by-hop metrics.
The following table describes the information that appears for Active Flows and Completed Flows:
Field |
Description |
---|---|
+ |
Click the + icon to view the following Expanded Application Information:
|
Start-Update Time |
Destination IP address of the traffic that the trace monitors. |
Flow ID |
System-assigned identifier of the flow. |
Readout |
Click the green tick icon to display detailed information about the flow readout on a slide-in pane. If the flow identifies an application issue, you can use this information to assist with a root-cause analysis. The slide-in pane includes the following information:
|
VPN |
Identifier of the VPN in which the application flow was traced. |
Source IP |
Source IP address of the traffic that the trace monitors. |
Src Port |
Source port of the traffic that the trace monitors. |
Destination IP |
Destination IP address of the traffic that the trace monitors. |
Destination Port or Dest Port |
Destination port of the traffic that the trace monitors. |
Protocol |
Protocol of the traffic that the trace monitors. |
DSCP Upstream/Downstream |
DSCP type that the trace monitors for upstream traffic and downstream traffic. |
Application |
Application that the trace monitors. |
Application Group or App Group |
Application group that the trace monitors. |
Domain |
Domain that the flow belongs to. Click a domain name to display the protocol from which the domain was recognized. This field shows information only for DNS and HTTPS protocol flows. For other flow types, this field displays Unknown. |
ART CND(ms) /SND (ms) |
Application response time, in milliseconds, for client network delay (CND) and server network delay (SND). |
View Advanced Trace Insights
From Cisco SD-WAN Manager navigate to Insight-Advanced Views section to view trace flow information.
and expand theView Domain Trend
The Domain Trend section is available when DNS discovery is enabled and displays metrics and event trends in an application flow. By hovering over the data points, you can see detailed information.
Metrics like client network delay (CND) and server network delay (SND) are measured by the application's TCP traffic. The DNS request frequency indicates how often a SaaS application is accessed. HTTP probe response time and loss rate are measured by router probes sent to the SaaS application server. These help detect a reachable direct internet access (DIA) network path and evaluate the benefits of deploying a DIA traffic policy.
You can choose metric types and specific devices from the Chart Metrics and Devices drop-down lists, respectively. By default, the section shows trend information for all metric types and devices. Additionally, you can limit the displayed trends to a specific time frame, choosing from predefined periods of 1, 10, or 30 minutes, or 1, 2, or 5 hours. For custom time ranges, you can enter a date and time or select Real Time to display live information.
View Flow Trend
The Flow Trend section shows trends for metrics and events in a trace flow. Hover over data points to see detailed information.
Use the Chart Metrics drop-down list to select specific metric types for viewing. The Flow Direction drop-down list allows you to choose the traffic flow direction for data display. By default, trend information is shown for latency, jitter, WAN loss and average queue depth.
To view information about a specific event, use the Navigate to Event drop-down list. You can limit the display to trends within a specified time frame or period, choosing from 1, 10, or 30 minutes, or 1, 2, or 5 hours. For custom time ranges, enter a date and time to see information as it is collected.
View Upstream and Downstream Feature Views
To view the details of the flow, expand a flow record in the flow path and metrics table by clicking on the + icon. The Upstream and Downstream Feature View provides a list of ingress and egress features applied to the flow, along with the execution results of each feature.
Troubleshooting Network-Wide Path Insight
Issue |
Action |
---|---|
Data stream collection might not be operating properly. |
To resolve this issue, chooseDisabled, then click Save. Click Data stream again, click Enabled, choose System for the IP address type, then click Save. , click |
DNS domain discovery might be enabled for the trace, but monitored traffic is not enabled from DNS domains. |
To resolve this issue, chooseEnable DNS Domain Discovery check box in the Trace area, and run the trace again. , deselect the |