Hardware Based Timestamping

Every packet that traverses the card (both received and transmitted) is timestamped to nanosecond level resolution (0.25ns for Cisco Nexus NIC HPT (formerly ExaNIC HPT), 6.2 nanoseconds for all others). This timestamping is done in hardware. To obtain a useful, timestamp, the hardware clock must be synchronized. Synchronizing is possible between the hardware internal timer and the host's clock, a separate pulse-per-second (PPS), or networked PTP grand master.

Synchronizing Cisco Nexus SmartNIC Cards in the Same Host

To synchronize Cisco Nexus SmartNIC (formerly ExaNIC) cards to a host or primary card, the exanic-clock-sync daemon can be used to keep the SmartNIC card's clock synchronized. Each argument to exanic-clock-sync specifies a <device>:<synchronization-source> pair, with the synchronization source being one of host (host clock) or exanicN (another SmartNIC device). For example:

$ exanic-clock-sync exanic0:host

or

$ exanic-clock-sync exanic0:host exanic1:exanic0

If desired this can be placed in system startup scripts. A sample init.d script is provided in configs/exanic-clock-sync.

A utility exanic-clock-check is provided to check how tightly synchronized the host and NIC clocks are:

$ exanic-clock-check
Compares the ExaNIC clock to the host or another ExaNIC
Usage: ./exanic-clock-check <exanic> [<exanic>]

should provide output similar too

$ exanic-clock-check  exanic0
Device exanic0: 1628741401 ticks (1498447571360653132 ns since epoch)
Host clock: 1498447571360653 us since epoch
Difference: 132 ns

Synchronizing Cisco Nexus SmartNIC Cards in Different Hosts with Pulse Per Second Signals (PPS)

Utilizing the PPS Input

To synchronize SmartNIC cards to this PPS signal the exanic-clock-sync daemon can be used. Again each argument to exanic-clock-sync specifies a <device>:<synchronization-source> pair, with PPS as the synchronization source in this example:

$ exanic-clock-sync exanic0:pps

will result in output similar to:

exanic0: Starting clock discipline using PPS
exanic0: Nominal frequency: 161132800 Hz
exanic0: Using differential PPS input
exanic0: PPS signal detected
exanic0: Clock offset at PPS pulse: -9305478.518 us  drift: 0.000 ppm
exanic0: Clock offset at PPS pulse: -17.253 us  drift: 0.000 ppm
...
exanic0: Clock offset at PPS pulse: -0.019 us  drift: -16.794 ppm

By default, the SMA input on SmartNIC cards accepts 3.3V TTL and has a standard 50 ohm termination enabled. To turn termination off, specify pps-no-term instead of pps. This allows for daisy chaining the PPS inputs of several SmartNIC cards together, where the last card in the chain terminates the PPS signal. The intermediate cards should be configured with pps-no-term and the final card with pps.

Intermediate card:

$ exanic-clock-sync exanic0:pps-no-term

Final card:

$ exanic-clock-sync exanic0:pps

Configuring the Pulse Per Second (PPS) Output

SmartNICs with a SMA connector (Cisco Nexus SmartNIC K35-S (formerly X10) and onwards) can be configured to drive a PPS out. The simplest starting point is to install a single SMA coaxial lead from the first SmartNIC device to a second SmartNIC device. With the cable present run:

$ exanic-config exanic0 pps-out on

running exanic-config will then show the new state of the PPS output.

Device exanic0:
  Hardware type: ExaNIC X10
...
  PPS out: enabled
note.gif

Noteblank.gifThe PPS output is via a female SMA connector, capable of driving 3.3V TTL into a 50 ohm load.

warn.gif

Warningblank.gifSmartNIC support for PPS out requires firmware from about Aug 2016 or later - please refer to the firmware changelog for the SmartNIC for details.

Synchronizing Cisco Nexus SmartNIC Cards in Different Hosts with the Precision Time Protocol (PTP)

The Precision Time Protocol (PTP) is a protocol that allows for very accurate time synchronization between a master and a number of client (slave) systems. The SmartNIC supports the Linux SO_TIMESTAMPING API, so standard PTP daemons such as ptp4l (LinuxPTP) or ptpd can be used to synchronize the card's clock.

note.gif

Noteblank.gifThe underlying time standard used by PTP is the International Atomic Time (TAI) standard, which at the time of writing is 37 seconds ahead of the Universal Coordinated Time (UTC) standard. LinuxPTP will set and keep the SmartNIC hardware clock in TAI time, not UTC time. Applications that make use of SmartNIC timestamps will need to be aware of this. Recent kernels are aware of the UTC-TAI offset, and this can be queried using adjtimex().

note.gif

Noteblank.gifThe host operating system requirements for PTP HW and PHC support are:

  • linux kernel >= 3.17, or
  • centos 7 with stock 3.10 kernel, or
  • centos 6 with stock 2.6.32 kernel and nohz=off
warn.gif

Warningblank.gifIt is not recommended or required to run the PTP HW synchronization and PPS exanic-clock-sync at the same time.

note.gif

Noteblank.gifLinuxPTP (PTP4l) works out of the box, however does not support unicast PTP traffic. We have had good results with ptp4l and there is a good deployment guide at the link below:

LinuxPTP Red Hat Deployment Guide

note.gif

Noteblank.gifAs of December 2016, the current release of ptpd (2.3.1) does not contain the required hardware timestamping code, so you need to either use the hardware timestamping branch described at http://nwtime.org/new-features-for-ptpd/ or to additionally run exanic-clock-sync to synchronize the time from the host.

As an example, a PTP slave ptp4l can then be run with a command such as the following:

$ ptp4l -i eth4 -m

Configuration File Method

In all the above examples, exanic-clock-sync's parameters and configuration options are set through command line options. These can alternatively be stored in a configuration file for ease of maintenance.

To specify the config file /etc/exanic_clksync.conf, run the following command:

$ exanic-clock-sync --config /etc/exanic_clksync.conf

The configuration file is broken into sections, each section containing key-value pairs for a synchronization target. The following subsections detail the format of the configuration file.

Key-Value Format

source

This option takes a string as input and is equivalent to specifying the synchronization source on the command line.

pps_term or pps_termination

This option takes a boolean input. When the key is present, the termination resistor is connected unless the value is "falsey", that is, one of off, no, false, 0 or an empty string. This is equivalent to specifying the source as pps_no_term or some other PPS option on the command line. The value for this key is case-insensitive.

pps_edge

This option takes a string as input and the value is one of the following: rising, positive, falling or negative. rising or positive causes the synchronization target to accept rising edge PPS signal, and falling or negative, falling edge. All of the possible values above are case insensitive.

offset

This option takes an integer input and is equivalent to appending a + or - followed by an integer offset after the source on the command line, e.g.

$ exanic-clock-sync exanic0:pps+5

This option specifies the offset, in nanoseconds, to add or subtract from the synchronization source. For example, to compensate for a delay of 5 nanoseconds between the synchronization source to your device, configure PPS with -5.

interval

This option applies to PPS only and specifies the averaging interval for calculating clock jitter and adjustment of the synchronization target. The default is 4 seconds.

Sample Configuration File

Here is a sample configuration file that demonstrates how to set up exanic0 for rising edge PPS time synchronization, with the termination resistor enabled and the averaging interval for jitter calculation set to 5 seconds:

[exanic0]
source=pps
pps_edge=rising
pps_term=yes
interval=5

Troubleshooting PTP

Below are some suggestions for troubleshooting PTP configuration for a PTP client. For troubleshooting suggestions for an Cisco Nexus NIC GM (formerly ExaNIC GM) please look at NIC GM troubleshooting.

1. To confirm the timestamping capabilities are present on a slave SmartNIC run the following on your interface: (requires ethtool 3.4 or above)

$ ethtool -T eth1
Time stamping parameters for eth1:
Capabilities:
    hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
    software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
    hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
    software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
    software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
    hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)

2. Make sure the SmartNIC port has link up and is configured to the correct speed e.g.

Port 0:
   Interface: eth1
   Port speed: 10000 Mbps
   Port status: enabled, SFP present, link active

to change the port state or speed use the exanic-config commands.

$ sudo exanic-config exanic0:0 up
$ sudo exanic-config exanic0:0 speed 10000

2. Make sure the SmartNIC has been assigned an appropriate IP address and that the master and slave are using the same transport protocol, i.e. multicast or unicast - and be aware of issues arising from hybrid master operation.

3. Generally debugging is simpler with multicast - at which point you can look to see that the SmartNIC port is receiving traffic.

$ sudo exanic-config exanic0:0

Which should show the received packet counters incrementing

RX packets: 708  ignored: 0  error: 0  dropped: 0

4. Then make sure kernel interface is receiving traffic.

$ sudo ifconfig eth1

should also show received packets incrementing

RX packets:587 errors:0 dropped:0 overruns:0 frame:0

and

$ sudo tcpdump -i eth1 port 319 or port 320

should show PTP traffic being observed. If the traffic is not observed at the kernel confirm that the SmartNIC port has not been configured to bypass-only mode.

5. If traffic is observed at the kernel but no output is observed at the PTP client, confirm that firewalls have been switched off or configured to allow the PTP traffic to flow.

6. Alternatively, if traffic is observed at the kernel interface but no output is observed leaving the PTP client, confirm that an appropriate route exists in the kernel routing table by inspecting ip route. You can determine the route the multicast address will take by using ip r g <multicast address> if it's not the SmartNIC PTP interface then a route needs to be added.

7. If traffic is being observed but the grandmaster settings are not correctly set make sure that the PTP domain that the slave is operating in matches the grandmaster. Make sure that the PTP is configured for hardware timestamps. Make sure that the PTP grandmaster has obtained a suitable clock synch and is serving a valid time.