Data Stream Troubleshooting Tools FAQ

Introduction

This document describes Q&As for the various Data Stream tools and talks about their troubleshooting features.

Version 18.2, introduced new troubleshooting tools that make use of a new vManage setting called Data Stream. The new troubleshooting tools are Speed Test, Packet Capture, and Debug Logs. These tools are seen on the vManage GUI when you navigate to Monitor > Network > (device) > Troubleshooting page.

Data Stream

Why don't you see the links for these new troubleshooting tools?

These new troubleshooting tools are only displayed when the Data Stream feature is enabled. If you navigate to the Monitor > Network > (device) > Troubleshooting page in the vManage GUI and don't see these tools, you probably see a yellow box at the top of the page which reads, "'Data Stream' is disabled. Navigate to the Settings page in order to enable Data Stream to use Packet Capture, Speed Test, and Debug Logs." The Data Stream must be enabled and configured before these links will appear in the Troubleshooting page. If you do not see that yellow box and still don't see the links for these tools, verify that your vManage is running version 18.2 or later.

How do you enable Data Stream?

In order to enable Data Stream, navigate to the Administration > Settings page in the vManage GUI and locate the line for Data Stream. If it shows "Disabled", then you need to enable it. Or, if it shows "Enabled" and you simply want to reconfigure it, you can follow this same procedure.
Click on the Edit link at the end of the Data Stram line. If the Disabled button is selected, select Enabled. Two new fields and two new buttons are displayed. In the Hostname field, enter the IP address or host name that the vEdge can use to reach the vManage. In the VPN field, enter the VPN associated with that IP address. Click Save.

What do you enter for the Hostname and VPN when you enable Data Stream?

You will need to decide which interface you will use for the vEdge devices in order to send test results back to the vManage. Typically, it is recommended you use vpn 512 management interface if that is accessible from the vEdge devices. If not, then you will need to use a VPN 0 interface. However, if you use a VPN 0 interface, you must ensure that the vEdge device has https as an allowed service on its VPN 0 tunnel interface that connects to that vManage VPN 0 interface. You will want to test that the vEdge device is able to ping the vManage interface you want to use over the VPN you want to use. Resolve any connectivity issues before this Data Stream can be used.

What services are started or ports are opened when you enable Data Stream?

Enabling Data Stream doesn't start any services or open any ports initially. You are simply defining which IP address under which VPN that the vEdge devices will use. When you run one of the troubleshooting tools is when the Data Stream is used. The vManage will open a netconf connection to the vEdge to tell it to execute the troubleshooting command. The vEdge will open an HTTPS connection back to the vManage, using the hostname/IP address and VPN you specified when you enabled Data Stream. These connections are all closed when the troubleshooting tool completes its operation. Or, if something goes wrong and the too fails to complete within 15 minutes, a background timer will close them.

Why do you get a red box stating, "Device Error: Server unreachable," when you try to run a troubleshooting tool?

Check that the vEdge device is able to ping the hostname/ip address over the vpn you configured for the Data Stream in the Administration > Settings page. If you specified a vpn 0 interface, configure the vEdge interface tunnel to allow the https service.

The troubleshooting tool seems to work, but why is no data is displayed? 

The vManage is able to open the netconf to the vEdge, but the vEdge is unable to open the https connection back to the hostname/IP address on the VPN you configured for the Data Stream. Check that the Data Stream configuration contains a valid hostname/IP address and VPN configured and that the vEdge is able to ping it. Verify that nothing is blocking HTTPS from the vEdge to the vManage.

If the vManage associated with the Data Stream settings is in a cluster and were to fail, will Data Stream automatically move to another vManage?

No, the Data Stream settings would need to be manually edited to use a hostname/IP of a vManage that is operational.

Speed Test

What hosts can you test against with Speed Test?

You can either test between two vEdges or between a vEdge and an Internet server.

What Internet services can you use for Speed Test?

If the vManage can access the Internet and the vEdge can access the Internet over the VPN you select, then you can specify an Internet host to use with Speed Test. Speed Test will select the shortest path and use one of these iperf test hosts on the Internet:

• ping.online.net
• iperf.biznetnetworks.com
• speedtest.serverius.net
• bouygues.iperf.fr
• iperf.he.net
• iperf.scottlinux.com
  
  

Why are you not able to use Speed Test with an Internet server?

The Internet server must be accessible from the vEdge over the circuit you have selected. You will want toconfigure the vEdge as a NAT device to provide internet access. You must also create and apply an ACL on the transport interface to allow port 5201, since the vEdge has an implicit ACL that would normally block these connections.

This is an example of the ACL that you would need to create, and how it would be applied to the vpn 0 interface. In this example, ge0/2 under vpn 0 is used for the test, and the Internet iperf3 server is ping.online.net.

vpn 0
 interface ge0/2
  access-list ACL in
!
!
policy
 access-list ACL
  sequence 10
   match
    source-ip   62.210.18.40/32
    source-port 5201
   !
   action accept
   !
  !
  default-action accept
 !
!

Why are you not able to run Speed Test between two vEdges if NAT/DIA is used on server side?

This is because when NAT is configured and no corresponding translation exists, traffic will be dropped by NAT. You should configure ACL and port forwarding to self like shown here:

vpn 0
 interface ge0/2

   ip address 198.51.100.2 255.255.255.0

  nat

   port-forward port-start 5201 port-end 5201 proto tcp

    private-vpn        0

    private-ip-address 198.51.100.2

   !

  !
  access-list ACL_IN in
!
!
policy
 access-list ACL_IN
  sequence 10
   match
    destination-port 5201
   !
   action accept
   !
  !
  default-action accept
 !
!

Why is the Speed Test dial not showing the speed while the test is running?

Two individual tests are run as part of the Speed Test operation: a download test and an upload test. The dial will indicate the result at the end of each individual test, when the vEdge uploads the results to the vManage. So, you will see the needle move twice during the test. Then, and the end, the results are also populated in the table at the bottom.

Why does the Configured bandwidth show Downstream and/or Upstream 0 Mbps?

These reflect the vEdge vpn interface's configuredbandwidth-downstreamandbandwidth-upstreamsettings, and is informational. These settings do not actually limit the bandwidth.

Why does Speed Test not show your full circuit bandwidth?

The maximum bandwidth that Speed Test will measure is about 215 to 250 Mbps. The Speed Test data is transmitted over the same circuit as your data. It will be subject to QoS (DSCP 0), shaping, and policing settings, and will be sharing the circuit with other data that may be in flight.

Why does Speed Test not show more than 215 to 250 Mbps?

This is a limit of the CPU processing. Speed Test is aniperf3test. It is single-threaded and pinned to the control core of the vEdge. This limits the maximum performance that the tool can achieve regardless of the interface or circuit bandwidth. The Speed Test tool should be used to test circuits that are less than 200Mbps between vEdge devices or Internet devices.

Does Speed Test take into consideration tunnel overhead, such as the ipsec header?

No. It is just running an iperf test and taking a measurement of the data transfer.

Can you use different iperf3 options for the Speed Test?

The Speed Test tool from the vManage GUI only allows you to defind the source and destination of the test. No other options can be configured. However, you can use the "tools iperf" CLI from both test machines to run a test with more specific options.

Can the Speed Test results be exported?

Currently, there is no facility for exporting the Speed Test results. However, you can drag over the results to select multiple rows, copy them to your clipboard, and paste them into a file.

Can multiple Speed Tests be run simultaneously?

Only one Data Stream activity can be running on a vEdge at a time. You cannot execute Speed Test on the same vEdge where another Speed Test, Packet Capture, or Debug Log is already running. You can, however, run Speed Test on two different vEdge devices simultaneously, as long as it is not a vEdge that is already involved in a running Speed Test.

Why do you get a red box stating "Server Error: Speed test is active on <ip_addr> as destination device?

You have tried to start Speed Test on a vEdge that is already being used as a destination for a Speed Test run on another vEdge. Wait for the other test to complete.

What is the impact on the vManage and vEdge when you run Speed Test?

The impact on the vManage is minor, and not more than other vManage operations. There is very little processing involved in opening a netconf connection to the vEdge, instructing it to run a test, and receiving the data from the vEdge. To the vEdge, there is more processing power on the core dedicated to control, as this is where the iperf process will execute. Also, on the vEdge, the data transfer performed by iperf will consume bandwidth and packet processing as the data is transmitted over the transport interface.

Packet Capture

What packets are captured?

All packets on the selected interface will be captured, including control and data packets.

Packets were captured but why do they appear to be encrypted?

When you capture on a transport interface, the packets are captured after the ipsec operation, so all traffic will be encrypted. In order to see unencrypted traffic, you will need to capture on a service interface.

What are the limits of the packet capture?

The packet capture can be stopped at any time. The packet capture will automatically stop once the capture file size reaches 5 MB, or after 5 minutes, whichever occurs first.

Can you filter what packets are captured?

You can filter on a source IP, source port, destination IP, destination port, and/or protocol number.

Can you collect a rolling packet capture?

No. Only a single capture file is created that is a maximum size of 5 MB. Once that file size is reached, or if it is not reached within 5 minutes, the packet capture is automatically stopped.

Can you capture multiple interfaces at the same time?

No. You can only specify a single interface on which to capture packets. And, since only one Data Stream operation can be running on the vEdge at a time, you cannot open another browser window to start a capture on another interface at the same time. You can, however, run a packet capture on two different vEdge devices at the same time.

How is the capture file delivered to the client?

When the packet capture stops, it will be transferred to the vManage and you will be presented a download link for downloading the capture to your computer. You will need to have tools on your computer to open the capture file. The downloaded file will be in tcpdump pcap format.

Debug Log

What debug logs can be collected?

These debug logs can be downloaded through the Debug Log troubleshooting tool: vconfd, vsyslog, and vdebug.

What is the vconfd debug log?

The vconfd debug log shows confd log messages, primarily related to netconf and the configuration of the device.

What is the vsyslog debug log?

The vsyslog is the system log, with log entries related to the general regular operation of the device.

What is the vdebug debug log?

The vdebug log is a more verbose system log, with entries related to the internal operations of the device.

Are the logs displayed real-time while they are being updated on the device?

There will be some delay. But, yes, the logs displayed in the web page are updated with new entries as they are written to the log file on the vEdge.

How is the log delivered to the client?

The log is displayed in a frame in your browser. A download link is also available for downloading the file directly to your computer.