Configuring Accelerated Networking on Microsoft Azure

Table 1. Feature History

Release

Description

Cisco IOS XE Gibraltar 16.10.1

Feature introduced. Support for accelerated networking with Mellanox 4.

Cisco IOS XE Gibraltar 16.12.1

Support for accelerated networking with Mellanox 5 introduced.

Cisco IOS XE Amsterdam 17.3.1

Support for Azure-PMD introduced.

Overview of Accelerated Networking

What is Accelerated Networking

Accelerated networking enables single root I/O virtualization (SR-IOV) on VMs such as a Cisco CSR 1000V VM. The accelerated networking path bypasses the virtual switch, increases the speed of network traffic, improves the networking performance, and reduces the network latency and jitter.

Usually, all the networking traffic in and out of the VM traverses the host and the virtual switch. However, with accelerated networking, the network traffic arrives at the virtual machine's network interface (NIC), and is then forwarded to the VM. Thus, all the network policies that the virtual switch applies are now offloaded and applied in the hardware.

For more information about the accelerated networking functionality that is available in Microsoft Azure, see Create a Linux VM With Accelerated Networking Using Azure CLI.

Accelerated networking is available in CSR 1000V public cloud deployments and in government cloud deployments for a Cisco CSR 1000V version Cisco IOS XE Gibraltar 16.10.1 or later.

If you are upgrading to Cisco IOS XE Gibraltar 16.10.x and 16.11.x, use a Microsoft Azure Accelerated Networking (AN) .bin image from www.cisco.com. For example, csr1000v-universalk9azn.16.10.x.SSA.bin.

Support for Azure-PMD

The Azure-PMD (Poll Mode Driver) functionality on Azure offers a faster, user-space packet processing framework for performance-intensive applications. This framework bypasses the virtual machine's kernel network stack. In a typical packet processing that uses the kernel network stack, the process is interrupt-driven. When the network interface receives the incoming packets, there is an interruption to the kernel to process the packet and a context switch from the kernel space to the user space. Azure-PMD eliminates the context switching and the interrupt-driven method in favor of a user-space implementation that uses poll mode drivers for fast packet processing.

Starting the Cisco IOS XE 17.3 release, you can enable the Azure-PMD functionality for CSR 1000V running on Microsoft Azure. This functionality increases the performance of the CSR 1000V when compared to the previous versions that use accelerated networking.

Supported VM Instance Types

The following VM instance types support the Accelerated Networking functionality:

IOS XE Version Supported VM Instance Types
16.12.x

DS2_v2 / D2_v2

DS3_v2 / D3_v2

DS4_v2 / D4_v2
17.1.x

DS2_v2 / D2_v2

DS3_v2 / D3_v2

DS4_v2 / D4_v2
17.2.x

DS2_v2 / D2_v2

DS3_v2 / D3_v2

DS4_v2 / D4_v2

17.3.x

DS2_v2 / D2_v2

DS3_v2 / D3_v2

DS4_v2 / D4_v2

F16s_v2

F32s_v2

Support for Mellanox Hardware

Microsoft Azure cloud has two types of hardware that support the accelerated networking functionality. The following table specifies the Mellanox versions supported for the accelerated networking functionality.

Table 2. Compatibility Matrix of IOS Versions and Accelerated Networking

IOS XE Version

Support for Accelerated Networking

Support for MLX4

Support for MLX5

Support for Azure-PMD

16.10.x

Yes

Yes

No

No

16.11.x

Yes

Yes

No

No

16.12.x

Yes

Yes

Yes

No

17.1

Yes

Yes

Yes

No

17.2

Yes

Yes

Yes

No

17.3

Yes

Yes

Yes

Yes

Note

MLX4 (Mellanox 4) is also referred to as ~ connectx3 = cx3, and MLX5 (Mellanox 5) is also referred as connectx4 = cx4.

You can't specify which NIC Azure uses MLX4 or MLX5 for your VM deployment. Cisco recommends that you upgrade to CSR 1000V 16.12 version or later to use the accelerated networking functionality.

Enable Accelerated Networking

To enable accelerated networking on a Cisco CSR 1000V, the instance must be running a Microsoft Azure AN variant image (applicable for Cisco IOS XE Gibraltar 16.10.x or 16.11.x images only), or a 16.12.x image, or later. Read on to know how to check for the variant image before you enable accelerated networking.

Check For Variant Image

Before you enable Microsoft Azure AN for a Cisco CSR 1000V VM in Microsoft Azure, ensure that you are running a Microsoft Azure AN variant image by executing the following Cisco IOS EXEC command: router# show platform software system hypervisor. This command displays specific details of your instance from the hypervisor.

The following example shows the output for a CSR1000V 16.10.x or 16.11.x image:

router# show platform software system hypervisor

Hypervisor: AZURE
Product Name : Virtual Machine in Azure
Manufacturer: Microsoft Corporation
Serial Number: 0000-0009-8597-0349-7291-4826-11
UUID: 17B4D488-BC82-F345-A829-F6279F54047D
image_variant : azn ===>> verifies you are running a Microsoft Azure AN variant image
Cloud Metadata
-------------------
Region: eastus
Zone: 1
Instance ID: cc60aff2-b7dc-4d81-9854-1c2be6eeacc2
Instance Type: Standard_DS2_v2
Version: 16.11.120210903
Image ID:
Publisher: cisco
Offer: cisco-csr1000v
SKU: 16_11_01a-byol
 
Interface Info
-------------------
Interface Number : 0
    IPv4 Public IP:
    IPv4 Private IP: 10.1.0.4
    IPv4 Subnet Mask: 24
    IPv4 Network: 10.1.0.0
    IPv4 Gateway: 10.1.0.1
    MAC Address: 000D3A180834

Note

The version that is displayed in this command output is the initial deployment version of your Cisco CSR1000V instance. To view the current or upgraded version of your instance, run the show version command.


Note

If you are upgrading from an earlier release to a 16.10.x or a 16.11.x image, select the AZN variant of the 16.x.x.bin image to achieve AN performance. The AZN variant contains the string "azn" in the file name. For example, csr1000v-universalk9azn.2018-12-03_23.12_abcd4.SSA.bin.

Enable Accelerated Networking

To enable accelerated networking, create or modify a vNIC using the az network nic command and the --accelerated-networking option. See the Microsoft Azure documentation for the az network nic command and also refer to the following examples.


Note

Depending on how you created the CSR 1000V instance, accelerated networking might initially be disabled on the CSR NICs. If accelerated networking is disabled on the NIC, and you want to enable accelerated networking on an interface, use one of the commands as shown in the following examples.

Example 1

This example shows how to create a vNIC "mynic1" and enable accelerated networking using the az network nic create command with the --accelerated-networking true option. 

az network nic create -n mynic1 -g "RG1" --accelerated-networking true -l "east us" --vnet-name "vnetname" --subnet "subnet1"

Example 2

This example shows how to create a vNIC "mynic2" and enable accelerated networking using the az network nic create command with the --accelerated-networking true option option. 

az network nic create -n "mynic2" -g "RG1" --accelerated-networking true -l "east us" --vnet-name "vnetname" --subnet "subnet1"

Example 3

This example shows how to modify a vNIC "mynic3" to enable accelerated networking using the az network nic update command with the --accelerated-networking true option. 

az network nic update -n mynic3 -g rg1 --accelerated-networking true

Caution

Due to a Microsoft Azure limitation, enabling accelerated networking on all the interfaces of a Cisco CSR 1000V router might cause a significant performance drop if packets greater than 1500 bytes are sent across the Azure infrastructure. The performance degradation occurs because Azure starts fragmenting the packets at 1438 bytes and drops out the sequence packets. This is a known issue.

Verifying Accelerated Networking

After Enabling accelerated networking on the NICs, use the following IOS commands to verify whether accelerated networking is enabled on the NIC. The Azure infrastructure uses Mellanox NICs to achieve SR-IOV or accelerated networking.

You can use the following commands to verify CSR NICs by using the Mellanox kernel drivers as the NIC’s I/O drivers to process the packets. In addition, the Mellanox NICs in the HyperV server of the Azure infrastructure presents a bonded interface to the CSR 1000V guest VM. This VM is used for accelerated networking, and the VM is in a bonded state whenever accelerated networking is enabled.

Run the following command to verify the accelerated networking status:

device#show platform software vnic-if database 
vNIC Database
eth00_1539659125237802400
    Device Name : eth0
    Driver Name : mlx4_en  ==>> this verifies that AN is enabled on NIC and CSR is using mellanox kernel modules/drivers as NIC’s I/O driver.
    MAC Address : 000d.3a1e.11f9
    PCI DBDF    : 86ab:00:02.0
    UIO device  : no
    Management  : no
    Status      : bonded   ==>> this verifies that AN is enabled on the NIC and mellanox kernel modules have recognized SR-IOV and kernel is presenting bonded interface mode to CSR.
  eth01_1539659128292894000
    Device Name : eth1
    Driver Name : mlx4_en
    MAC Address : 000d.3a1e.1c73
    PCI DBDF    : 9354:00:02.0
    UIO device  : no
    Management  : no
    Status      : bonded
  eth_17__1539659131397365100
    Device Name : Gi1
    Driver Name : hv_netvsc
    MAC Address : 000d.3a1e.1c73
    PCI DBDF    : UNKNOWN
    UIO device  : no
    Management  : no
    Status      : supported
  eth_18__1539659134427961100
    Device Name : Gi2
    Driver Name : hv_netvsc
    MAC Address : 000d.3a1e.11f9
    PCI DBDF    : UNKNOWN
    UIO device  : no
    Management  : no
    Status      : supported

Note

The previous example is a configuration output from a release that supported the MLX4 driver. However, from the 16.12.1 release, this output might display MLX4 or MLX5, depending on the MLX driver in your Azure Infrastructure.

Verifying Accelerated Networking for CSR 1000V 16.10.x Through 17.2.x Images

In the following example, you can see that the interface eth1 is in use, is processing packets, and reflects the packet counters in the vf. 

pdev1010anenabled-csr#show controllers
GigabitEthernet1 - Gi1 is mapped to eth_17_ on VXE
  DPIF Rx Drop 0 Packets 46339
  Driver Rx Stops 0 DPIF Rx Congestion Drop 0
Detailed interface statistics:
  tx_scattered 0
  tx_no_memory 0
  tx_no_space 0
  tx_too_big 0
  tx_busy 0
  tx_send_full 0
  rx_comp_busy 0
  vf_rx_packets 29750   ==>>> This verifies Accelerated networking is working properly. this numbers should be moving if traffic is passing through CSR.
  vf_rx_bytes 32439581
  vf_tx_packets 48761
  vf_tx_bytes 7109172
  vf_tx_dropped 0
  tx_queue_0_packets 53
  tx_queue_0_bytes 7630
  rx_queue_0_packets 8554
  rx_queue_0_bytes 3577166
  tx_queue_1_packets 11
  tx_queue_1_bytes 1628
  rx_queue_1_packets 8056
  rx_queue_1_bytes 3121870
Bonded interface (eth1) statistics:
  rx_packets 29750
  tx_packets 48748
  rx_bytes 32856081
  tx_bytes 7108700
  rx_dropped 0
  tx_dropped 0
  tso_packets 0
  xmit_more 0
  queue_stopped 0
  wake_queue 0
  tx_timeout 0
  rx_alloc_pages 6144
  rx_csum_good 29748
  rx_csum_none 0
  rx_csum_complete 0
  tx_chksum_offload 0
  rx_xdp_drop 0
  rx_xdp_tx 0
  rx_xdp_tx_full 0
  tx0_packets 16
  tx0_bytes 1216
  tx1_packets 48732
  tx1_bytes 7107484
  rx0_packets 19051
  rx0_bytes 22867089
  rx0_dropped 0
  rx0_xdp_drop 0
  rx0_xdp_tx 0
  rx0_xdp_tx_full 0
  rx1_packets 10699
  rx1_bytes 9988992
  rx1_dropped 0
  rx1_xdp_drop 0
  rx1_xdp_tx 0
  rx1_xdp_tx_full 0

Verifying Accelerated Networking for CSR 1000V 17.3.x or Later (With Azure-PMD)

After enabling accelerated networking on the NICs, use the following IOS commands to verify whether accelerated networking with Azure-PMD is enabled on NIC. The Azure infrastructure uses Mellanox NICs to achieve SR-IOV or accelerated networking.

Use the following commands to verify the CSR 1000V NICs by using the Mellanox Azure-PMD drivers as the NIC’s I/O drivers to process the packets. In addition, the Mellanox NICs in the HyperV server of the Azure infrastructure presents a bonded interface to the CSR 1000V guest VM. This VM is used for accelerated networking, and the VM is in a bonded state while accelerated networking is enabled. Note that the bonded interfaces share the same MAC address. The aggregate counters appear on Gi interfaces, while the non-accelerated packets counters appear on the net_tap interfaces. The accelerated packets counters appear on the net_mlx interfaces.

In the following example, the interface Gi2 indicates that a majority of the packets are flowing over the net_mlx interface.

csrANpmd#sh plat hard qfp act dat inf pmd controllers | inc NIC|good_packets
NIC extended stats for port 0  (Gi1) net_failsafe 000d.3a8f.1bf1 xstats count 13
  rx_good_packets: 411
  tx_good_packets: 326
NIC extended stats for port 1  (Bonded) net_mlx5 000d.3a8f.1bf1 xstats count 35
  rx_good_packets: 389
  tx_good_packets: 326
NIC extended stats for port 2  (Bonded) net_tap 000d.3a8f.1bf1 xstats count 13
  rx_good_packets: 22
  tx_good_packets: 0
NIC extended stats for port 3  (Gi2) net_failsafe 000d.3a8f.1040 xstats count 13
  rx_good_packets: 10638289
  tx_good_packets: 3634525
NIC extended stats for port 4  (Bonded) net_mlx5 000d.3a8f.1040 xstats count 35
  rx_good_packets: 10639534.  ==>>> This verifies Accelerated Networking is working properly for RX
  tx_good_packets: 3636099     ==>>> This verifies Accelerated Networking is working properly for TX
NIC extended stats for port 5  (Bonded) net_tap 000d.3a8f.1040 xstats count 13
  rx_good_packets: 291
  tx_good_packets: 0
NIC extended stats for port 6  (Gi3) net_failsafe 000d.3a8f.1a90 xstats count 13
  rx_good_packets: 3637187
  tx_good_packets: 10522981
NIC extended stats for port 7  (Bonded) net_mlx5 000d.3a8f.1a90 xstats count 35
  rx_good_packets: 3638631
  tx_good_packets: 10524554
NIC extended stats for port 8  (Bonded) net_tap 000d.3a8f.1a90 xstats count 13
  rx_good_packets: 28
  tx_good_packets: 0

Disable Accelerated Networking

To disable accelerated networking for the Cisco CSR 1000v, you can create or modify a vNIC using the az network nic command and the --accelerated-networking option.

For more information about the command, see the Microsoft Azure documentation for the az network nic command.

The following examples specify how to modify a vNIC.

Example

This example shows how to modify a vNIC "mynic1" to disable Accelerated Networking using the az network nic update command with the --accelerated-networking false option. 

az network nic update -n "mynic1" -g rg1 --accelerated-networking false