The Cisco Nexus NIC GM (formerly ExaNIC GM) is a combined Precision Time Protocol (PTP) Grandmaster and ultra low latency network card. The device is packaged in a PCIe network card form factor with twin SFP+ ports. Both SFP+ ports 0 and 1 operate as low latency network interfaces. Port 0 additionally functions as a hardware accelerated PTP Grandmaster.

NIC GM Device Installation

Installation of the NIC GM is almost identical to other Cisco Nexus SmartNIC (formerly ExaNIC) devices. This page should be used in conjunction with the SmartNIC Installation section. Once the device is installed as above, connect the GPS SMA to an externally mounted GPS antenna, one preferably with an unobstructed view of the sky. Proper cable, grounding techniques, and lightning arrests should be used. Do not mount the antenna near obstructions such as a wall or obstacles blocking part of the sky.

A second SMA marked PPS can be connected for 1PPS/10MHz connections to allow synchronization. The behavior of this connection is configurable and is covered here.

PTP traffic will be served out of the first port of the NIC GM card. (Refer to the below picture to identify the first port.)

459795.png

NIC GM Components

Software Installation

Software installation of the NIC GM is identical to that of other SmartNIC devices. Please refer to the SmartNIC Installation guide for details. Note that the first driver software to support the NIC GM is 1.8.0.

After Installation

Assuming installation completed successfully, verify that the exanic-config utility works and reports the hardware type and function of the NIC GM PTP grandmaster:

$ sudo exanic-config

Device exanic0:
 Hardware type: ExaNIC GM
 Board ID: 0x00
 Temperature: 64.2 C VCCint: 0.94 V VCCaux: 1.85 V
 Function: PTP grandmaster
 Firmware date: 20170728 (Fri Jul 28 00:36:12 2017)

The exanic-config utility can be used to inspect diagnostic information about the card and SFP modules. Its general use is covered in more detail in the SmartNIC Configuration Guide.

The exanic-config PTP Status

The exanic-config utility can be used to inspect extended diagnostic information about the PTP configuration and status. For example:

$ sudo exanic-config exanic0 ptp status

Device exanic0:
  PTP grandmaster: enabled
  Synchronize to GPS time: enabled
  Hardware time: 2017-08-01 00:32:10.145673160 TAI
                 2017-08-01 00:31:33.145673160 UTC
  TAI-UTC offset: 37s
  MAC address: 64:3f:5f:01:2D:68
  PTP configuration:
    Ethernet multicast: off
    IPv4 multicast: on
    IPv4 unicast negotiation: off
    IPv4 address: 172.17.11.228
    PTP domain: 0
    PTP priority 1: 128
    PTP priority 2: 128
    Multicast announce interval: 1 (2s)
    Multicast sync interval: 0 (1s)
    Announce receipt timeout: 1 (2s)
    One-step or two-step clock: one-step
  GPS configuration:
    Antenna cable delay: -23ns
  GPS receiver status:
    Fix type: 3D fix
    Number of tracked satellites: 13
  Clock status:
    Sync state: synced
    Estimated clock accuracy: 8ns
  PTP grandmaster status:
    PTP port state: master
    Number of unicast clients: 0
    PTP packets sent: 2349
    PTP packets received: 73

The exanic-config utility will either accept the Linux interface name (e.g. eth7) or the device name of the SmartNIC device (e.g. exanic0 for the first SmartNIC in the system by PCI ID, and exanic0:0 for the first port of that card).

The Hardware time is reported to the user in Co-ordinated Universal Time (UTC) and International Atomic Time (TAI), and all PTP network traffic is in TAI. The current offset between TAI and UTC is also displayed.

note.gif

Noteblank.gifThe underlying time standard used by PTP is the International Atomic Time (TAI) standard, which at the time of writing is 37 seconds ahead of the Universal Coordinated Time (UTC) standard. PTP traffic will be in TAI time, not UTC time. Applications that make use of SmartNIC timestamps will need to be aware of this. Recent kernels are aware of the UTC-TAI offset, and this can be queried using adjtimex().

PTP Configuration

The exanic-config utility also provides methods for configuring the NIC GM as a PTP grandmaster. To access a list of the PTP configuration commands execute:

$ sudo exanic-config exanic0 ptp --help
exanic-config version 1.8.1
Detailed PTP grandmaster configuration and status:
   exanic-config <device> ptp status
   exanic-config <device> ptp { enable | disable }
   exanic-config <device> ptp ip-address <addr>
   exanic-config <device> ptp show-profile
   exanic-config <device> ptp profile { default | telecom | none }
   exanic-config <device> ptp { one-step | two-step }
   exanic-config <device> ptp eth-multicast { on | off }
   exanic-config <device> ptp ip-multicast { on | off }
   exanic-config <device> ptp ip-unicast { on | off }
   exanic-config <device> ptp gps-sync { on | off }
   exanic-config <device> ptp domain <value>
   exanic-config <device> ptp priority1 <value>
   exanic-config <device> ptp priority2 <value>
   exanic-config <device> ptp announce-interval <value>
   exanic-config <device> ptp sync-interval <value>
   exanic-config <device> ptp announce-receipt-timeout <value>
   exanic-config <device> ptp antenna-cable-delay <value>
   exanic-config <device> ptp save

Note after configuring the desired PTP configuration for it to remain persistent over a power cycle it is required to save the setup to EEPROM using the PTP save call. For example:

$ sudo exanic-config exanic0 ptp save

Enabling or Disabling the PTP Stack

Normally the NIC GM operates as a PTP grandmaster serving out the PTP protocol, however it can be configured to suppress the PTP behaviour to allow operation as a normal NIC or to allow a different grandmaster to control the network. To disable the PTP stack on the NIC GM execute:

$ sudo exanic-config exanic0 ptp disable

Supported values are enable or disable.

MAC and IP Addresses

The NIC GM provides 2 interfaces for the host, each of which has a MAC assigned at the factory, and can have an IP address set by the host. The CPU within the NIC GM that handles the PTP stack also has a MAC assigned at the factory, and needs to be assigned an IP address by the user for use on IP networks.

459796.png

NIC GM Interfaces

To configure the IP address for the PTP stack:

$ sudo exanic-config exanic0 ptp ip-address 192.168.1.12

The IP address for the host interface is set using standard methods:

$ sudo ifconfig eth1 192.168.1.10

Don’t forget to save to EEPROM with exanic-config exanic0 ptp save

PTP Profiles

To simplify the configuration of a PTP network standard profiles were defined. These profiles include default values and acceptable ranges. To select and configure the NIC GM to one of the standard PTP profiles use the command:

$ sudo exanic-config exanic0 ptp profile default

The acceptable values are default, telecom and none. none allows the operator to enter any values without restriction. To show the current selected profile and ranges execute the command:

$ sudo exanic-config exanic0 ptp show-profile

Setting PTP One-step or Two-step Event Messages

The NIC GM can be configured to use one-step or two-step event messages. That is, a sync event message that gets timestamped can either get an accurate timestamp placed in the message "on the fly" (one-step), or placed in a second follow up message (two-step).

$ sudo exanic-config exanic0 ptp one-step

$ sudo exanic-config exanic0 ptp two-step

Setting the PTP Communication Method

The PTP protocol can be configured to use several communication methods and management techniques. The multicast methods (IP and Ethernet) will configure the NIC GM to broadcast out PTP messages, whereas IP unicast will require the PTP clients to initiate connections.

$ sudo exanic-config exanic0 ptp ip-multicast on

$ sudo exanic-config exanic0 ptp eth-multicast off

$ sudo exanic-config exanic0 ptp ip-unicast off

It is suggested that ip-multicast be used to validate the network configuration before deciding on alternate communication methods.

Enabling or Disabling GPS Synchronization

By default, the NIC GM utilities an on-board GNSS receiver (GPS) to tightly synchronize the NIC GM oscillator to GPS time from anywhere on the globe. An operator can however disable the NIC GM synchronization, this allows the NIC GM to be synchronized from alternate sources such as the Host PC.

To disable the GPS to oscillator synchronization execute:

$ sudo exanic-config exanic0 ptp gps-sync off

The acceptable values are on or off.

For best operation the NIC GM should be attached to an antenna installed with an unobstructed view of the sky. This will result in a high quality positional and time sync lock, and can be confirmed by verifying “Fix Type” is reported as “3D fix” when running exanic-config exanic0 ptp status.

Configuring PTP Best Master Priority Values

A number of configuration fields are used by the NIC GM in the execution best master clock algorithm (BMCA). These are used in conjunction with quality and performance metrics of the NIC GM to determine which device should operate as a grandmaster on a particular network. The exanic-config application can be used to set the priority1 and priority2 fields to modify the priority of the NIC GM in best master clock identification decisions. For example:

$ sudo exanic-config exanic0 ptp priority1 127
$ sudo exanic-config exanic0 ptp priority2 1

The acceptable values are between 0 and 255, where a lower value is chosen over a higher value in the BMCA.

Configuring PTP Grandmaster Broadcast Intervals

A number of configuration fields are used by the NIC GM to control timing of PTP protocol messages. These are used to manage announce messages as part of the best master clock identification and sync messages as part of time synchronization. The exanic-config program can be used to set these values to adjust the timing and traffic on the network. For example:

$ sudo exanic-config exanic0 ptp announce-interval 3
$ sudo exanic-config exanic0 ptp sync-interval -1

The supplied interval values are used to calculate the interval in seconds, using the formula 2x seconds, where X is the supplied value. For example an announce-interval of 3 equals 23 or 8 seconds and a sync-interval of -1 equals 2-1 or half a second.

The acceptable values are between -127 and 127, however PTP profiles set a minimum and maximum and these are respected.

Configuring GPS Antenna Delay

The GPS antenna delay can be configured to factor in the latency incurred by the length of the installed cabled between the GPS antenna and the NIC GM. Note - The provided value should be in nanoseconds and changing the value will reset the clock sync algorithm to apply the new value correctly.

$ sudo exanic-config exanic0 ptp antenna-cable-delay 20

Configuring PPS Output

The NIC GM includes a PPS connector that can be used as either an input or output. When operating in output mode a 10MHz or 1PPS signal can be generated, which can be used to synchronize the time of other devices that can accept the PPS input.

As this is available for other SmartNIC devices it is configurable through the exanic-config level commands, not the ptp commands.

Usage:

$ sudo exanic-config <device> pps-out { on | off }

$ sudo exanic-config <device> 10m-out { on | off }

For example:

$ sudo exanic-config exanic0 pps-out on
note.gif

Noteblank.gifThe PPS utilizes 3.3V TTL, 50 ohm termination and by default a rising edge trigger. The NIC GM exposes a female SMA connector.

Configuring PPS Input

There is no specific configuration value for setting the NIC GM for PPS input, instead the input is setup and utilized by the exanic-clock-sync utility that manages the SmartNIC clock synchronization.

The exanic-clock-sync program takes arguments in the form of one or more <device>:<synchronization-source> pairs. The synchronization source being one of either the host (host clock), pps (pulse-per-second input) or exanicX (another SmartNIC device). For example:

$ sudo exanic-clock-sync exanic0:pps exanic1:exanic0

If desired this can be placed in system startup scripts. A sample init.d script is provided in configs/exanic-clock-sync.

NIC GM LED States

LED State NIC GM State
Solid Green All OK (GPS Fix)
Green Flashing Adjust (GPS Fix) or Warmup (GPS Fix)
Orange (R+G) Flashing Holdover (No GPS fix)
Red Flashing Re-sync in progress (GPS could be any state)
Solid Orange (R+G) GPS sync disabled (GPS could be any state
LED Off GM Disabled

Packet Counters

There is several packet counters used on the NIC GM. The first are those packets sent by the onboard CPU handling the PTP stack. These are shown in the ptp status command:

PTP grandmaster status:
 Number of unicast clients: 0
 PTP packets sent: 248
 PTP packets received: 73

The host driver will also report packets sent/received by the card related to host activity. For example, exanic-config reports the following:

$ exanic-config exanic0:0

Device exanic0:
 Hardware type: SmartNIC GM
 Board ID: 0x00
 Temperature: 58.6 C VCCint: 0.94 V VCCaux: 1.83 V
 Function: PTP grandmaster
 Firmware date: 20160805 (Fri Aug 5 07:08:15 2016)
 Port 0:
  Interface: enp1s0
  Port speed: 10000 Mbps
  Port status: enabled, no SFP, no signal, no link
  Bypass-only mode: off
  MAC address: 64:3f:5f:01:2D:66
  RX packets: 1254 ignored: 0 error: 0 dropped: 0
  TX packets: 914

This says that the host system has transmitted 914 packets and has received 1254 packets , and the CPU handling the PTP stack has transmitted 248 and received 73. Note that some packets such as broadcast packets would be received by both the host and PTP CPU.

Troubleshooting

Note all interface numbers, IP addresses and device names below are for example only. These may differ on your system.

1. Make sure the NIC GM port is configured to the correct speed e.g.

$ sudo exanic-config exanic0:0 speed 10000

2. Make sure the NIC GM has been assigned an IP address at the device level e.g.

$ sudo exanic-config exanic0 ptp ip-address 192.168.1.12

3. Make sure the NIC GM port is configured to the correct auto negotiation e.g.

$ sudo ethtool -s eth1 autoneg on

4. Is GPS connected? At boot time, the hardware clock on the NIC GM is at epoch time zero, or 1st Jan 1970. The NIC GM will not serve time onto the network unless GPS has been detected and a lock established, at which point the correct time is known. From then on, GPS can be removed and the NIC GM will fallback to its onboard oscillator.

In the case below, even though PTP is enabled, as the system has never had the time set via GPS (as can be seen with the year as 1970), it will be blocked from serving time.

$ exanic-config exanic0 ptp status

Device exanic0:
 PTP grandmaster: enabled
 Synchronize to GPS time: enabled
 Hardware time: 1970-01-01 00:01:00.769655540 TAI
 1970-01-01 00:00:24.769655540 UTC
 TAI-UTC offset: 36s
 MAC address: 64:3f:5f:01:2d:68

5. Confirm that the PTP port has been through the best master clock algorithm (BMCA) and is operating as the master. The NIC GM will initially report as listening while the BMCA is operating and passive if a better GM is detected on the network.

PTP port state: master

6. Look to see that exanic-config ptp status is reporting an increase in the number of ‘PTP packets sent’ e.g.

$ sudo exanic-config exanic0 ptp status

Device exanic0:
 PTP grandmaster: enabled
 Synchronize to GPS time: enabled
 Hardware time: 2016-08-07 23:51:38.225087568 TAI
 2016-08-07 23:51:02.225087568 UTC
…

 PTP grandmaster status:
 PTP packets sent: 39267
note.gif

Noteblank.gifThe standard linux applications such as tcpdump can not be used to monitor the transmitted traffic from the PTP soft core.

note.gif

Noteblank.gifFrom firmware 20170728 the NIC GM ports can be configured with local-loopback. This allows the SmartNIC to internally loop the port 0 outgoing traffic back to port 0 Rx. e.g.

$ exanic-config exanic0:0 local-loopback on

This then allows the transmitted traffic to be confirmed by observing the received traffic with tools such as exanic-capture and tcpdump. Once validated remember to turn the local-loopback function back off.

$ exanic-config exanic0:0 local-loopback off

7. Look to see that the PTP slave system is receiving PTP traffic e.g.

$ sudo tcpdump -i eth1 port 319