Command Details: service sd-avc status

Execute the command on the network service host device.

Output indicates status of SD-AVC installation, configuration, and activation.

• Installation error:

  Service SDAVC is uninstalled, not configured and deactivated

• Activation error:

  Service SDAVC is installed, configured and Activate Failed

Command Details: service sd-avc trace

Execute the command on the network service host device.

Output indicates memory or disk problems.

• Memory problem (shown in bold below):

  service sd-avc trace
  2017/11/27 02:06:42.384 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MACH_PARSE_FAILURE: Virtual Service[SDAVC]::Parsing::XML parsing failure::Unable to parse VM machin
  e definition::Requests 3072 MB of memory which exceeds the maximum of 1024
  2017/11/27 02:06:42.383 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MEMORY_LIMIT_WARN: Virtual service (SDAVC) defines 3072 MB of Memory exceeding the maximum 1024 MB.
  ...
  
  

• Disk problem (shown in bold below):

  2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM mac address binding from FDB
  2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get mac binding from persistent DB file
  2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Could not retrieve HA disk info for VM 'SDAVC'
  2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Unable to locate fdb attributes for vm(SDAVC)
  2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM storage info list from FDB
  2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get storage pool from persistent DB file
  2017/11/27 03:36:52.499 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Virtual Service failure log[SDAVC]::Install::The installation of the virtual service failed
  
  

Command Details: show virtual-service list

Execute the command on the network service host device.

Output indicates activation status (failed in this example):

Virtual Service List:
Name                    Status             Package Name                         
------------------------------------------------------------------------------
SDAVC                   Activate Failed    avc_iosxe_221533.ova     

Command Details: show virtual-service global

Execute the command on the network service host device.

Output indicates virtual service CPU and memory usage:

Example showing a service using 5% of CPU:

show virtual-service global 
Maximum VCPUs per virtual service : 1
Resource virtualization limits:
Name                               Quota     Committed     Available  
---------------------------------------------------------------------
system CPU (%)                       75            5            70  
memory (MB)                        3072           800          2272  
bootflash (MB)                    20000          6764         10672 

Component(s)

Device hosting the SD-AVC network service

Background

Memory or disk allocation issues can prevent successful installation of the SD-AVC network service.

Troubleshooting

1. Use service sd-avc status on the network service host device to check status of installation. If installation is unsuccessful, the output shows "Service SDAVC is uninstalled."

   service sd-avc status
   Service SDAVC is uninstalled, not configured and deactivated
   
   

2. Use service sd-avc trace on the network service host device to indicate whether the installation problem is due to memory or disk.

   • Memory problem:

     service sd-avc trace
     2017/11/27 02:06:42.384 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MACH_PARSE_FAILURE: Virtual Service[SDAVC]::Parsing::XML parsing failure::Unable to parse VM machin
     e definition::Requests 3072 MB of memory which exceeds the maximum of 1024
     2017/11/27 02:06:42.383 [errmsg] [3071]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-MEMORY_LIMIT_WARN: Virtual service (SDAVC) defines 3072 MB of Memory exceeding the maximum 1024 MB.
     ...
     
     

   • Disk problem:

     2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM mac address binding from FDB
     2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get mac binding from persistent DB file
     2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Could not retrieve HA disk info for VM 'SDAVC'
     2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Unable to locate fdb attributes for vm(SDAVC)
     2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get per-VM storage info list from FDB
     2017/11/27 03:36:52.500 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to get storage pool from persistent DB file
     2017/11/27 03:36:52.499 [vman] [3222]: UUID: 0, ra: 0, TID: 0 (ERR): Virtual Service failure log[SDAVC]::Install::The installation of the virtual service failed
     
     

Solutions

Component(s)

Device hosting the SD-AVC network service

Background

The platform hosting the SD-AVC network service should not have other virtual services operating. Sharing CPU resources with other virtual services can prevent successful activation.

Use service sd-avc status on the network service host device to check status of installation. If installation has succeeded, but activation is unsuccessful, the output shows "Activate Failed."

service sd-avc status
Service SDAVC is installed, configured and Activate Failed

Troubleshooting

Use service sd-avc trace on the network service host device to troubleshoot. The following output shows a problem (shown in bold) with activation, due to shared CPU.

service sd-avc trace
2017/11/26 15:46:49.133 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Failed to find domain SDAVC - state query
2017/11/26 15:46:49.133 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Domain not found: No domain with matching name 'SDAVC'
2017/11/26 15:46:49.133 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Error from libvirt: code=42
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (note): VM (SDAVC) State Transition: next_state: LIFECYCLE_ACTIVATE_FAILED
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Virtual Service failure log[SDAVC]::Activate::Internal error::Machine definition customization failed
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Machine definition customization failed
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Customization of common XML parameters failed
2017/11/26 15:46:48.131 [vman] [2224]: UUID: 0, ra: 0, TID: 0 (ERR): Customize CPU tunes: Cannot commit CPU tunes
2017/11/26 15:46:48.131 [errmsg] [2224]: UUID: 0, ra: 0, TID: 0 (noise):(2): %VMAN-2-CPUSHARES_LIMIT: Virtual Service[SDAVC]::CPU shares limit::The virtual service definition exceeds the maximum number of CPU shares::Defined: 75, available: 70

Use show virtual-service global to provide details. In this example, another process is using 5% of the CPU resources (shown in bold).

show virtual-service global 
Maximum VCPUs per virtual service : 1
Resource virtualization limits:
Name                               Quota     Committed     Available  
--------------------------------------------------------------
system CPU (%)                  75            5            70  
memory (MB)                   3072           800          2272  
bootflash (MB)               20000          6764         10672 

Solutions

Deactivate Interface Using CPU Resources

1. Check the running configuration using show run on the network service host device. If an active interface is using CPU resources, deactivate the interface.

   Example

   GigabitEthernet1 is using CPU resources.

   show run | section csr_mgmt
   virtual-service csr_mgmt
   ip shared host-interface GigabitEthernet1
   activate
   
   

2. Deactivate the interface.

   Example

   conf t 
   virtual-service csr_mgmt
   no activate
   no ip shared host-interface GigabitEthernet1
   
   

3. Repeat the installation of the SD-AVC network service.

Component(s)

Device hosting the SD-AVC network service

Background

If the host interface that is used as a gateway interface for the SD-AVC network service is attached to a VRF, the SD-AVC network service installation may be successful, but a configuration step may fail.

Troubleshooting

1. Check VRF status of the SD-AVC network service gateway interface.

   Example showing a VRF configured on the gateway interface GigabitEthernet1:

   interface GigabitEthernet1
   ip vrf forwarding Mgt
   ip address 10.56.196.177 255.255.252.0
   
   service sd-avc configure gateway interface gigabitEthernet 1 service-ip 10.56.196.180
   % Error: VRF 'Mgt' is configured on gateway. This type of configuration is not supported.
   
   

Solutions

Remove the VRF assignment from the management interface. Example:

interface GigabitEthernet1
no ip vrf forwarding

Component(s)

Devices in the network that are using SD-AVC

Background

The NBAR2 component must be active on any interface that processes network traffic, in order to report on traffic handled by the interface. For details, see Configuration Prerequisites: Network Devices Using SD-AVC.

If NBAR2 is not active on an interface processing network traffic:

• The device will not report on any traffic on that interface.

• On the Dashboard > Application Visibility page, the Timeline graph of bandwidth will show no activity.

  

• The device will not receive application rules packs from the SD-AVC network service.

Troubleshooting

Verify that NBAR2 is active on interfaces that process network traffic.

Solutions

If necessary, activate NBAR2 on the interface(s).

Component(s)

Devices in the network that are using SD-AVC

Background

The SD-AVC agent must be enabled on any device participating with SD-AVC. This requires entering sd-service-controller submode on the device.

It is possible to connect to the device through multiple sessions. An error may occur in the following conditions, with an error message indicating the problem:

• One active session is in sd-service-controller submode.

• You attempt to open sd-service-controller submode in a new session.

Example:

Device(config)#avc sd-service
Device(config-sd-service)# segment sdavc
Device(config-sd-service)# controller
%% NBAR Error: There is an active session already in sd-service-controller submode

Solutions

Close any interfering active sessions.

1. On the device, use show users to display active sessions.

2. In the command output, note the line number of a session to close. Use clear line line-number to close a session.

Example:

Device#show users
  Line     User    Host(s)   Idle    Location
*  1       vty 0   prod      idle    00:00:00
                                     dhcp-10-11-12-13-14-15.cisco.com
   3       vty 2   prod      idle    1d04h 198.51.100.10

Device#clear line 3
[confirm]
[OK]

Device#show users   
    Line   User    Host(s)   Idle    Location
*  1       vty 0   prod      idle    00:00:00
                                     dhcp-10-11-12-13-14-15.cisco.com

Component(s)

SD-AVC network service

Devices in the network that use SD-AVC

Background

The SD-AVC Network Service uses UDP over port 50000 to communicate with the devices that it manages.

Troubleshooting

1. If a Connection warning appears in the SD-AVC Dashboard, for a specific device in the network, check connectivity on UDP port 50000. Warnings appear here:

   SD-AVC Dashboard > Application Visibility page > SD-AVC Monitoring pane

2. If no problem is found, contact Cisco TAC.

Solutions

Ensure that UDP connectivity is possible on port 50000 between the affected device and the SD-AVC network service.

Component(s)

SD-AVC network service

Devices in the network that use SD-AVC

Background

The SD-AVC network service communicates with SD-AVC agents in the network using:

• TCP over port 21 (FTP) for devices using Cisco IOS XE 16.11.x Gibraltar or earlier

• TCP over port 8080 (HTTP) for devices using Cisco IOS XE 16.12.1 Gibraltar or later

(See System Requirements: Network Devices Using SD-AVC.)

Troubleshooting

1. If an FTP warning appears in the SD-AVC Dashboard, for a specific device in the network, check connectivity on TCP port 21 (FTP) or port 8080 (HTTP). Warnings appear here:

   SD-AVC Dashboard > Application Visibility page > SD-AVC Monitoring pane

2. If no problem is found, contact Cisco TAC.

Solutions

Ensure that TCP communication is possible over port 21 (FTP) and port 8080 (HTTP) between the affected device and the SD-AVC network service.

Component(s)

SD-AVC network service

Devices in the network that use SD-AVC

Background

The SD-AVC network service uses FTP/HTTP to communicate with the devices that it manages.

A device with partial connectivity, but problems specific to FTP/HTTP may show a warning in the SD-AVC Dashboard.

For FTP/HTTP issues caused by connecting a device to an internal FTP/HTTP server for non-SD-AVC FTP/HTTP traffic, see Scenario: Internal FTP/HTTP Server.

Troubleshooting

1. If an FTP warning appears in the SD-AVC Dashboard while the Connection status is green, for a specific device in the network, check the FTP/HTTP connection status. Warnings appear here:

   SD-AVC Dashboard > Application Visibility page > SD-AVC Monitoring pane

2. On the device with the connectivity issue, use show avc sd-service info summary to check the FTP/HTTP connection status. "Status: DISCONNECTED" in the output below shows an FTP/HTTP connectivity problem.

   show avc sd-service info summary
    
   Status: DISCONNECTED
    
   Device ID: csi-mcp-asr1k-4ru-32
   Device segment name: cisco
   Device address: 10.56.192.31
    
   Active controller:
      Type  : Primary
      IP    : 64.103.125.30
      Status: Disconnected
      Last connection: Never
   
   

Solutions

Ensure that FTP/HTTP communication is possible between the affected device and the SD-AVC network service.

1. Verify that nothing is preventing FTP/HTTP network connectivity between the SD-AVC network service and the network device. This includes checking ACL, firewalls, and so on.

2. To determine whether communication with the SD-AVC network service uses FTP or HTTP, execute the following command on the device. The example shows HTTP.

   show avc sd-service info detailed | inc Transport for file copy:

   Transport for file copy: http

3. On the device with the FTP/HTTP warning, ensure that FTP/HTTP connectivity is possible from the routable interface to the SD-AVC network service. To enable FTP/HTTP connections from a specific interface, use:

   ip ftp source-interface interface-name

   Example:

   ip ftp source-interface GigabitEthernet1
   ip http  client source-interface g1

Component(s)

SD-AVC network service

Cisco NBAR2 Protocol Packs

Background

Use the SD-AVC network service to deploy Protocol Packs to one or more devices. See Deploying Protocol Packs to Devices. When deploying Protocol Packs to one or more devices, if the deployment fails, the results page may show an error.

Troubleshooting

1. Operating an excessive number of services on a router simultaneously can cause insufficient Quantum Forwarding Processor (QFP) memory to be available to load a new Protocol Pack. In this case, SD-AVC may display a “Low Memory” error message when you try to update a Protocol Pack.

   You can use the show platform hardware qfp active infrastructure exmem statistics command on a router to check the status of QFP memory resources. If less than 50 MB are available, you can reload the router to free memory, and attempt to load the Protocol Pack again. If it fails a second time, you can use the show platform hardware qfp active infrastructure exmem statistics user command to display the individual processes using QFP memory.

2. Load the Protocol Pack manually on the device indicated by the error to verfiy that the Protocol Pack is valid and can be loaded onto the device. This rules out any problems with the Protocol Pack file.

   (config)#ip nbar protocol-pack bootflash:pack_file_name.pack
   
   

3. If no problem is found, contact Cisco TAC.