Configuration
Since the Cisco Nexus SmartNIC (formerly ExaNIC) appears to Linux as a normal network card, most configuration can be performed through the normal Linux mechanisms. For example, in Redhat-based distributions, configuration is generally performed using files in the /etc/sysconfig/network-scripts directory. Alternatively, the interface can be configured temporarily by issuing a command such as ifconfig eth7 192.168.1.100.
If the interface is being used through the low-level userspace API (libexanic) only, it is sufficient to bring the interface up without assigning an IP address. This can be done through either ifconfig (e.g. ifconfig eth7 up) or exanic-config (e.g. exanic-config exanic0:0 up).
The exanic-config Utility
The exanic-config utility can be used to inspect diagnostic information about the card and SFP modules. For example:
$ exanic-config
Device exanic0:
Hardware type: ExaNIC X4
Board ID: 0x02
Temperature: 53.2 C VCCint: 1.00 V VCCaux: 1.81 V
Fan speed: 6987 RPM
Function: network interface
Firmware date: Thu Nov 28 20:54:56 2013
Bridging: off
Port 0:
Interface: eth7
Port speed: 10000 Mbps
Port status: enabled, SFP present, signal detected, link active
Mirroring: off
Promiscuous mode: off
Bypass-only mode: off
MAC address: 64:3f:5f:01:13:18
RX packets: 6 ignored: 0 error: 0
TX packets: 6
...
$ exanic-config exanic0:0 sfp status
Device exanic0 port 0 SFP status:
Vendor: FINISAR CORP. PN: FTLX8571D3BCL rev: A
SN: ALF0QE7 date: 111006
Wavelength: 850 nm
Nominal bit rate: 10300 Mbps
Rx power: -2.5 dBm (0.56 mW)
Tx power: -1.9 dBm (0.65 mW)
Temperature: 33.8 C
The exanic-config utility will either accept the Linux interface name (e.g. eth7) or the device name of the SmartNIC device (e.g. exanic0 for the first SmartNIC in the system by PCI ID, and exanic0:0 for the first port of that card). Globbing syntax is also accepted in order to match several cards/ports. For example, exanic*:[0-3] will match ports 0-3 inclusive on any SmartNICs installed. See the output of exanic-config --help for more details.
Interface Packet Counters
Packet Counters in exanic-config
The exanic-config utility will display the packet counter values for traffic being transmitted or received at a particular port.
Port 0:
Interface: eth7
Port speed: 10000 Mbps
Port status: enabled, SFP present, signal detected, link active
Mirroring: off
Promiscuous mode: off
Bypass-only mode: off
MAC address: 64:3f:5f:01:13:18
RX packets: 1514936895 ignored: 20578657 error: 145683 dropped: 0
TX packets: 615149369
RX Packets: refers to the total number of packets received by the NIC hardware.
ignored: refers to packets that did not match the local MAC address. It is incremented when the NIC receives a packet that is unicast but destined to a different destination address. This is not as a result of load or any problem with the NIC/host, but that the device on the other end of the wire is sending packets not destined for this device.
error: refers to packets which failed CRC checking. A common cause of this type of behaviour is poor signal from the other end, e.g. a bad cable or optical splitter. To investigate run exanic-config exanicX:Y sfp status at both ends of the link.
dropped: refers to packets lost as a result of insufficient PCIe bandwidth to transfer all packets into host memory. If this occurs, check lspci -vvv and in particular the LnkSta: line for the SmartNIC to verify that the card is operating at Speed 8GT/s (PCIe Gen 3.0) and Width x8. Dropped packets can also occur due to system topology - for example in a multi-socket system where the card is being accessed from a remote socket and QPI bottlenecks occur - or if the incoming network bandwidth exceeds that available on the PCI Express link (e.g. 8x10G or 2x40G can exceed the bandwidth available on a 8x8Gbit/s link during line rate bursts).
Packet Counters in ifconfig
dropped in ifconfig counters can be a result of:
- insufficient CPU cycles to service the NIC
- if jumbo frames are coming in (see note 1 below)
- if IP6 frames are coming in when expecting IP4 only.
Note
Although the kernel driver and exasock don't support jumbo frames, the raw libexanic API can actually receive jumbo frames with no problems. Transmitting jumbo frames using the raw libexanic API is also possible, however you will need to allocate a larger TX buffer than the default size. For sending 4k frames you will need a TX buffer size of 16k.
Setting Port Speed
The default speed for SmartNIC ports is 10 Gigabit. To set a port to a different speed, use the speed command to either ethtool or exanic-config:
$ ethtool -s eth7 speed 1000
or:
$ exanic-config exanic0:0 speed 1000
Supported values are 10000, 1000 and 100.
Note1 Gigabit Ethernet support requires firmware dated 20140206 or later.
Note100 Mbit Ethernet support requires firmware version 20150327 or later, and one of the following SFP modules: Finisar FCLF-8520-3/FCLF-8521-3/FCMJ-8520-3/FCMJ-8521-3 or Avago ABCU57xxxxxZ.
NoteSome 100 Mbit SFP modules will not configure the speed unless the port has first been enabled (e.g. exanic-config exanic0:0 up). Ensure the module is inserted into the port at the time the port is configured for 100M.
Promiscuous Mode
Normally the SmartNIC only passes on received unicast traffic that matches the MAC address of the port to the CPU. If Promiscuous mode is enabled on a port then the network interface controller will instead pass all unicast traffic it receives on that port to the CPU.
$ exanic-config exanic0:0 promisc on
Bypass-only Mode
If the interface is being used through the userspace API only and kernel CPU load on high traffic connections is a concern, the bypass-only setting in exanic-config (or bypass_only in ethtool) can be used to detach the kernel driver from the interface. This means that the kernel driver will not receive any packets. However, tools such as tcpdump will not function correctly while this is enabled. Additionally, this setting is not currently persistent across reboots so, if required, should be added to post-up scripts for the interface.
Warningbypass-only mode should not be enabled for exasock to function correctly.
$ exanic-config exanic0:0 bypass-only on
Local-loopback Mode
When an interface is configured to local-loopback mode the SmartNIC will transmit frames through the FPGA to the wire and will also loop the frame back to the host.
WarningWhen configured to local-loopback mode traffic is disabled from coming in from the wire. That is you will only receive the traffic you transmit on that interface.
$ exanic-config exanic0:0 local-loopback on
exanic0:0: local-loopback mode enabled
This then allows the transmitted traffic to be confirmed by observing the received traffic with tools such as exanic-capture and tcpdump. Once validated remember to turn the local-loopback function back off.
$ exanic-config exanic0:0 local-loopback off
exanic0:0: local-loopback mode disabled
Pulse-per-second
SmartNICs with a SMA connector (Cisco Nexus SmartNIC K35-S (formerly X10) and onwards) can be configured to drive a PPS out or accept a PPS input - see SmartNIC clock synchronization for full details.
Packet Capture
Using tcpdump
tcpdump can be used to capture SmartNIC traffic that the kernel sees. This would normally be all received traffic (unless steered away using flow steering) and any traffic transmitted via the kernel. Note that traffic transmitted via utilities that use libexanic or exasock bypass the kernel, hence will not be shown in tcpdump. The kernel may also not keep up with received traffic in high load scenarios.
Using exanic-capture
A utility called exanic-capture is provided that can be used to capture traffic received by an SmartNIC.
This utility receives Ethernet frames from defined interface and dumps to terminal or defined file.
$ exanic-capture
Usage: ./exanic-capture -i interface [-w savefile] [-s snaplen] [-p] [-H] [-N] [filter...]
-i: specify Linux interface (e.g. eth0) or ExaNIC port name (e.g. exanic0:0)
-w: dump frames to given file in pcap format (- for stdout)
-s: maximum data length to capture
-p: do not attempt to put interface in promiscuous mode
-H: use hardware timestamps (refer to documentation on how to sync clock)
-N: write nanosecond-resolution pcap format
Filter examples:
tcp port 80 (to/from tcp port 80)
host 192.168.0.1 tcp port 80 (to/from 192.168.0.1:80)
dst 192.168.0.1 dport 53 (to 192.168.0.1:53, either tcp or udp)
src 192.168.0.5 sport 80 or dst 192.168.0.1 (combine clauses with 'or')
exact-capture
There is a utility called exact-capture which is designed for high rate capture across multiple SmartNIC interfaces. This utility is open source and available on Exablaze's github account.
This is the recommended utility for capturing traffic in high load environments. Please refer to this page for documentation and further details on exact-capture.
Multicast Traffic
Confirming Registration
Please note this guide assumes the use of IGMP v3, to check the configured version of IGMP on the host operating system (OS) run:
cat /proc/net/igmp
WarningIt is a common source of error that the host OS and the multicast switch operate with different IGMP versions. While the IGMP versions are backwards compatible, please ensure that the implementations of the versions are suitable.
exanic-config should be used to confirm that the SmartNIC card is present and that the interface to be used is appropriately configured. E.g. enabled and link active. The rest of the guide assumes a switch with IP 10.230.91.1
$ exanic-config
This should port output similar to.
Port 1:
Interface: enp1s0d1
Port speed: 10000 Mbps
Port status: enabled, SFP present, signal detected, link active
...
IP address: 10.230.91.62 Mask: 255.0.0.0
Confirm the SmartNIC is able to register for group membership successfully at the host. To assist, Cisco provides with the driver source an example multicast registration utility. Browse to the src/examples/exanic/ folder and build the mcast_register utility for registering to a multicast group.
$ make
Then without exasock, run the built utility to register for the multicast group on the SmartNIC host. Providing the interface to utilise and the multicast address to register to. e.g.
$ ./mcast_register 226.20.20.21:enp1s0d1
Next confirm the SmartNIC interface and the multicast address show in the multicast group memberships on the SmartNIC host OS. e.g.
$ netstat -g
This should return a table similar to.
IPv6/IPv4 Group Memberships
Interface RefCnt Group
--------------- ------ ---------------------
lo 1 all-systems.mcast.net
enp1s0 1 all-systems.mcast.net
enp1s0d1 1 226.20.20.21
Then confirm the SmartNIC interface and the switch are up and have link and routes. If possible ping between the IP addresses on the subnet, note this may not be possible with Fusion objects.
Confirm that multicast registration requests are being sent from the SmartNIC interface e.g.
$ tcpdump -i enp1s0 igmp
NoteIGMP Version 3 membership reports are destined to the address 224.0.0.22. All IGMPv3-capable multicast routers must listen to this address.
09:14:57.915283 IP 10.230.91.62 > 224.0.0.22: igmp v3 report, 1 group
record(s)
09:14:58.908281 IP 10.230.91.62 > 224.0.0.22: igmp v3 report, 1 group
record(s)
NoteIGMP Version 1 and 2 use a query-response model. Queries are sent to 224.0.0.1.
10:09:25.890209 IP 10.230.91.1 > 224.0.0.1: igmp query v2
10:11:24.291555 IP 10.230.91.1 > 224.0.0.1: igmp query v2
Next please confirm the SmartNIC shows in the multicast group memberships on the switch. For Cisco Nexus 3550-F Fusion switch devices this is the show igmp groups command. For further details on the Fusion IGMP groups see : IGMP groups
Then confirm that once registered multicast traffic queries etc are observed at the SmartNIC interface e.g. $ tcpdump -i enp1s0
10:33:57.351212 IP 10.230.91.1 > 226.20.20.21: igmp query v3
10:33:58.352121 IP 10.230.91.1 > 226.20.20.21: igmp query v3
Feedback