The following are the guidelines and limitations for Hardware Telemetry.

• The ASIC on the switch forwards the flows only to the default VRF. There is no support for having collectors in an area other than the default VRF.

The following is an example of installing the telemetry RPM.

switch# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# feature bash-shell
switch(config)# end
switch# run bash sudo su
bash-4.3# service docker start              >> Starting docker service
Free bootflash: 86936 MB, total bootflash: 116590 MB
Carving docker bootflash storage: 2000 MB
2000+0 records in
2000+0 records out
2000000000 bytes (2.0 GB) copied, 16.5267 s, 121 MB/s
mke2fs 1.42.9 (28-Dec-2013)
fs_types for mke2fs.conf resolution: 'ext4'
Discarding device blocks: done                           
Discard succeeded and will return 0s - skipping inode table wipe
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
122160 inodes, 488281 blocks
4882 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=503316480
15 block groups
32768 blocks per group, 32768 fragments per group
8144 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912
 
Allocating group tables: done                           
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
 
tune2fs 1.42.9 (28-Dec-2013)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          7c841fbf-1972-4082-9ea0-37b77844be3b
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              122160
Block count:              488281
Reserved block count:     4882
Free blocks:              471692
Free inodes:              122149
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      119
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8144
Inode blocks per group:   509
Flex block group size:    16
Filesystem created:       Tue Dec 10 18:46:46 2019
Last mount time:          n/a
Last write time:          Tue Dec 10 18:46:47 2019
Mount count:              0
Maximum mount count:      -1
Last checked:             Tue Dec 10 18:46:46 2019
Check interval:           0 (<none>)
Lifetime writes:          33 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      bbea6663-7b8a-47e6-8a10-ac87ca23dfcc
Journal backup:           inode blocks
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Starting dockerd with args '--debug=true':      .
 
bash-4.3# cd /bootflash/
bash-4.3# ls sw_telemetry-2.0-2.x86_64.rpm  
sw_telemetry-2.0-2.x86_64.rpm
bash-4.3# rpm -Uvh sw_telemetry-2.0-2.x86_64.rpm    >> Installing container RPM  
Preparing...                ########################################### [100%]
   1:sw_telemetry           ########################################### [100%]
9da9fb5b1b21: Loading layer [==================================================>] 197.4 MB/197.4 MB
855a0ab0dad9: Loading layer [==================================================>] 13.54 MB/13.54 MB
0fbbc3806bd9: Loading layer [==================================================>] 539.1 kB/539.1 kB
17518e2561b2: Loading layer [==================================================>] 81.41 kB/81.41 kB
a00c7083c0c8: Loading layer [==================================================>] 469.5 kB/469.5 kB
b5b4081bee95: Loading layer [==================================================>] 4.608 kB/4.608 kB
83af9595fe45: Loading layer [==================================================>] 1.536 kB/1.536 kB
96b5c176e3d9: Loading layer [==================================================>] 25.73 MB/25.73 MB
Loaded image: sw_telemetry:latest
bash-4.3# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
sw_telemetry        latest              b8f86559a869        36 hours ago        228 MB
bash-4.3# pwd
/bootflash
bash-4.3# cat env.list       >> create env.list file with local loopback ip which is reachable to external collector
transport_source_ip=14.13.12.11
bash-4.3# exit
exit
switch# show running-config interface loopback 10
 
!Command: show running-config interface loopback10
!Running configuration last done at: Tue Dec 10 18:55:59 2019
!Time: Tue Dec 10 18:57:25 2019
 
version 9.3(3) Bios:version 05.39
 
interface loopback10
  ip address 14.13.12.11/32
  ip router ospf 10 area 0.0.0.0
 
switch# ping 5.1.1.1 source-interface loopback 10   >> 5.1.1.1 is my external collector
PING 5.1.1.1 (5.1.1.1): 56 data bytes
64 bytes from 5.1.1.1: icmp_seq=0 ttl=62 time=0.766 ms
64 bytes from 5.1.1.1: icmp_seq=1 ttl=62 time=0.64 ms
64 bytes from 5.1.1.1: icmp_seq=2 ttl=62 time=0.557 ms
64 bytes from 5.1.1.1: icmp_seq=3 ttl=62 time=0.553 ms
64 bytes from 5.1.1.1: icmp_seq=4 ttl=62 time=0.532 ms
 
--- 5.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.532/0.609/0.766 ms
switch# run bash sudo su
bash-4.3# cd /bootflash/
bash-4.3# chkconfig --add docker           >> make docker persistent on realod
bash-4.3# docker run -v /var/run/netns:/var/run/netns:ro,rslave --network host --cap-add SYS_ADMIN -it --env-file /bootflash/env.list -v /bootflash:/bootflash -p 50051:50051 -p 50052:50052 -d -v /tmp/nginx_local/:/tmp/nginx_local/ -v /var/run/:/var/run/ -v /etc/localtime:/etc/localtime:ro -v /etc:/etc -v /nxos/tmp:/nxos/tmp --restart unless-stopped --ipc="host" sw_telemetry_grpc        >> starting container
57a65cbceb599f3750c76de1d88749054d8a396c23c1c34e2bd22743a14dbc3e
bash-4.3# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
57a65cbceb59        sw_telemetry        "ip netns exec def..."   2 seconds ago       Up 1 second                             determined_meninsky
bash-4.3#
bash-4.3# exit

The docker container can be configured using gRPC and the filters that are supported for SSX in the following table. The gRPC message number is the enum number of the filters in the proto file.

SSX Egress Queue Drop

  "egressQueueDrop": [
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 0,
    "ts": "1562008724381260"
   },
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 1,
    "ts": "1562008724381260"
   },
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 2,
    "ts": "1562008724381260"
   },
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 3,
    "ts": "1562008724381260"
   },
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 4,
    "ts": "1562008724381260"
   },
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 5,
    "ts": "1562008724381260"
   },
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 6,
    "ts": "1562008724381260"
   },
   {
    "QDropBytes": "0",
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 7,
    "ts": "1562008724381260"
   },

SSX Ingress Queue Occupancy

 "ingressQueueOccupancy": [
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 0,
    "ts": "1562640434373522"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 1,
    "ts": "1562640434373522"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 2,
    "ts": "1562640434373522"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 3,
    "ts": "1562640434373522"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 4,
    "ts": "1562640434373522"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 5,
    "ts": "1562640434373522"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 6,
    "ts": "1562640434373522"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 7,
    "ts": "1562640434373522"
   },

SSX Egress Queue Occupancy

 "egressQueueOccupancy": [
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 0,
    "ts": "1562640434380646"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 1,
    "ts": "1562640434380646"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 2,
    "ts": "1562640434380646"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 3,
    "ts": "1562640434380646"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 4,
    "ts": "1562640434380646"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 5,
    "ts": "1562640434380646"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 6,
    "ts": "1562640434380646"
   },
   {
    "QOcc": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 7,
    "ts": "1562640434380646"
   },

SSX Ingress Pause Drops

"data": {
  "ingressQueuePauseDrop": [
   {
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 4,
    "ts": "1562970319568955"
   },
   {
    "QDropPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 3,
    "ts": "1562970319568955"
   },

SSX Egress Mem Occupancy (per slice)

 "egressMemOccupancy": [
   {
    "EgBuf": "0",
    "slice": "0",
    "ts": "1573329731594964"
  },
   {
    "EgBuf": "8066",
    "slice": "1",
    "ts": "1573329731594964"
   },
   {
    "EgBuf": "0",
    "slice": "2",
    "ts": "1573329731594964"
   },
   {
    "EgBuf": "0",
    "slice": "3",
    "ts": "1573329731594964"
   }
  ],

SSX ECN

"ecnStats": [
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 0,
    "ts": "1562018638423871"
   },
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 1,
    "ts": "1562018638423871"
   },
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 2,
    "ts": "1562018638423871"
   },
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 3,
    "ts": "1562018638423871"
   },
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 4,
    "ts": "1562018638423871"
   },
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 5,
    "ts": "1562018638423871"
   },
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 6,
    "ts": "1562018638423871"
   },
   {
    "ECNBytes": "0",
    "ECNPkts": "0",
    "intf": "Ethernet1/1",
    "queueIndex": 7,
    "ts": "1562018638423871"
   },

The following is a sample of port counter output.

 "counters": [
   {
    "interfaceName": "Ethernet1/1",
    "portIn": "21900548",
    "portInDiscard": "0",
    "portInError": "0",
    "portInIpv4": "5468000",
    "portInIpv4Rate": "0",
    "portInIpv6": "5468094",
    "portInIpv6Rate": "0",
    "portOut": "22919963",
    "portOutDiscard": "0",
    "portOutError": "0",
    "portOutIpv4": "5468000",
    "portOutIpv4Rate": "0",
    "portOutIpv6": "5468000",
    "portOutIpv6Rate": "0",
    "portPacketIn": "109722",
    "portPacketInIpv4": "27340",
    "portPacketInIpv4Rate": "0",
    "portPacketInIpv6": "27341",
    "portPacketInIpv6Rate": "0",
    "portPacketOut": "112917",
    "portPacketOutIpv4": "27340",
    "portPacketOutIpv4Rate": "0",
    "portPacketOutIpv6": "27340",
    "portPacketOutIpv6Rate": "0",
    "portQosGroup0MulticastTxBytes": "0",
    "portQosGroup0MulticastTxPkts": "0",
    "portQosGroup0UnicastTxBytes": "440800",
    "portQosGroup0UnicastTxPkts": "2204",
    "portQosGroup1MulticastTxBytes": "0",
    "portQosGroup1MulticastTxPkts": "0",
    "portQosGroup1UnicastTxBytes": "21231200",
    "portQosGroup1UnicastTxPkts": "106156",
    "portQosGroup2MulticastTxBytes": "0",
    "portQosGroup2MulticastTxPkts": "0",
    "portQosGroup2UnicastTxBytes": "0",
    "portQosGroup2UnicastTxPkts": "0",
    "portQosGroup3MulticastTxBytes": "0",
    "portQosGroup3MulticastTxPkts": "0",
    "portQosGroup3UnicastTxBytes": "0",
    "portQosGroup3UnicastTxPkts": "0",
    "portQosGroup4MulticastTxBytes": "0",
    "portQosGroup4MulticastTxPkts": "0",
    "portQosGroup4UnicastTxBytes": "0",
    "portQosGroup4UnicastTxPkts": "0",
    "portQosGroup5MulticastTxBytes": "0",
    "portQosGroup5MulticastTxPkts": "0",
    "portQosGroup5UnicastTxBytes": "0",
    "portQosGroup5UnicastTxPkts": "0",
    "portQosGroup6MulticastTxBytes": "0",
    "portQosGroup6MulticastTxPkts": "0",
    "portQosGroup6UnicastTxBytes": "0",
    "portQosGroup6UnicastTxPkts": "0",
    "portQosGroup7MulticastTxBytes": "0",
    "portQosGroup7MulticastTxPkts": "0",
    "portQosGroup7UnicastTxBytes": "0",
    "portRx0Pfc": "0",
    "portRx1Pfc": "0",
    "portRx2Pfc": "0",
    "portRx3Pfc": "0",
    "portRx4Pfc": "0",
    "portRx5Pfc": "0",
    "portRx6Pfc": "0",
    "portRx7Pfc": "0",
    "portTx0Pfc": "0",
    "portTx1Pfc": "0",
    "portTx2Pfc": "0",
    "portTx3Pfc": "0",
    "portTx4Pfc": "0",
    "portTx5Pfc": "0",
    "portTx6Pfc": "0",
    "portTx7Pfc": "0",
    "timestamp": "1561055719695692554"
   },

The following are the Guidelines and Limitations for Buffer Drop Capture (BDC) and Buffer Latency Capture (BLC).

• These features are supported on the following platforms:

  • Cisco Nexus 3408-S

  • Cisco Nexus 3432D-S

• Flow of Interest (FoI): You must apply IPv4 and IPv6 ACLs. If you add only an IPv4 ACL, an explicit deny for IPv6 is configured.

• FoI: Use the same ACL for BDC, BLC, and IPT.

• BDC does not work on Inno Block 4 and Inno Block 5.

• BDC does not work on no-drop queue.

• ODM merge does not work for BDC/BLC FoI. By default, statistic-per-entry is present for these ACLs.

• The hw-telemetry table does not have enough match bits available for ether type (16 bits) to differentiate between IPv4 and IPv6.

• If the original data traffic and SPAN copy of that traffic is going out on the same IB and if the original data traffic is dropped on a queue where BDC is enabled, you will not see any BDC packets for this traffic.

• If the BLC port of interest is also configured as SPAN TX source (egress mirror enable), only SPAN destination will receive a copy and the BLC copy is not created. Hence, BLC does not work on this port.

• We support 5000 PPS for BDC/BLC together.

• You need to carve the TCAM before using FoI. An error is not displayed when applying FoI without TCAM carving.

Innovium Path Telemetry (IPT) has the following configuration guidelines and limitations:

• On the sink node, IPT packets are accounted for on the sink interface queue besides being accounted for on the collector interface queue.

• Egress remarking of Innovium Path Telemetry (IPT) packets is not supported. Egress remarking of original packets should not be used with IPT since IPT data cannot be guaranteed in this case.

• SPAN and ERSPAN cannot monitor IPT packets.

• Innovium Path Telemetry is supported on Cisco Nexus 3408-S and 3432D-S switches.

• For Innovium Path Telemetry to work correctly, TCAM carving for ingress and egress is required.

• Only unicast, non-fragmented TCP and UDP packets are supported.

• Flow of Interest (FoI) is supported.

• The FoI ACL is shared with buffer latency and buffer drop features.

• Deny ACEs within the ACL are not supported.

• A maximum of one monitor can be applied.

• IPT/buffer drop and buffer latency share same ACL. If buffer drop and/or buffer latency have FoI with action telemetry_path, even though FoI is not applied to IPT, filter action happens for IPT as well.

To display the Innovium Path Telemetry configuration information enter the following command:

-----------------------------------
  IPT Enabled ***** System Details
-----------------------------------
Probe Marker   : 1234

-------------------------------------
Source Monitor : src_mon(1) details
-------------------------------------
In use (applied to system): YES 
V4 acl: ipt 
V6 acl: iptv6 
Sampling Rate: 1 
Record Attached: src_rec 

-------------------------------------
Source Record : src_rec(1) details
-------------------------------------
Total interfaces under record:  1
Ethernet1/23

-------------------------------------

                               IPV4 PACL [ifacl] size =  512 
                          IPV6 PACL [ipv6-ifacl] size =    0 
                            MAC PACL [mac-ifacl] size =    0 
                                IPV4 VACL [vacl] size =    0 
                           IPV6 VACL [ipv6-vacl] size =    0 
                             MAC VACL [mac-vacl] size =    0 
                                IPV4 RACL [racl] size =  256 
                           IPV6 RACL [ipv6-racl] size =    0 
                       Egress IPV4 RACL [e-racl] size =    0 
                  Egress IPV6 RACL [e-ipv6-racl] size =    0 
                                     SPAN [span] size =    0 
                     Ingress L2 QOS [ing-l2-qos] size =    0 
           Ingress L3/VLAN QOS [ing-l3-vlan-qos] size =  128 
                           Ingress SUP [ing-sup] size =  256 
                      Egress L2 QOS [egr-l2-qos] size =    0 
            Egress L3/VLAN QOS [egr-l3-vlan-qos] size =  128 
                Ingress PACL v4 & v6 [ifacl-all] size =    0 
                 Ingress RACL v4 & v6 [racl-all] size =    0 
               Ingress QoS L2/L3 [ing-l2-l3-qos] size =    0 
                     HW Telemetry [hw-telemetry] size =  128 
                 Egress PACL v4 v6 [e-ifacl-all] size =    0 
            Egress HW Telemetry [e-hw-telemetry] size =  128

show hardware access-list tcam region 
                               IPV4 PACL [ifacl] size =  512 
                          IPV6 PACL [ipv6-ifacl] size =    0 
                            MAC PACL [mac-ifacl] size =    0 
                                IPV4 VACL [vacl] size =    0 
                           IPV6 VACL [ipv6-vacl] size =    0 
                             MAC VACL [mac-vacl] size =    0 
                                IPV4 RACL [racl] size =  256 
                           IPV6 RACL [ipv6-racl] size =    0 
                       Egress IPV4 RACL [e-racl] size =    0 
                  Egress IPV6 RACL [e-ipv6-racl] size =    0 
                                     SPAN [span] size =    0 
                     Ingress L2 QOS [ing-l2-qos] size =    0 
           Ingress L3/VLAN QOS [ing-l3-vlan-qos] size =  128 
                           Ingress SUP [ing-sup] size =  256 
                      Egress L2 QOS [egr-l2-qos] size =    0                 
            Egress L3/VLAN QOS [egr-l3-vlan-qos] size =  128 
                Ingress PACL v4 & v6 [ifacl-all] size =    0 
                 Ingress RACL v4 & v6 [racl-all] size =    0 
               Ingress QoS L2/L3 [ing-l2-l3-qos] size =    0 
                     HW Telemetry [hw-telemetry] size =  128 
                 Egress PACL v4 v6 [e-ifacl-all] size =    0 

show ip access-lists

IP access list ipt
        10 permit ip 1.1.1.1/32 2.2.2.2/32 telemetry_path 
        20 permit ip 3.3.3.3/32 4.4.4.4/32 telemetry_queue 

show ipv6 access-lists

IPv6 access list iptv6
        10 permit ipv6 1:1::1:1/128 2:2::2:2/128 telemetry_path 
        20 permit ipv6 3:3::3:3/128 4:4::4:4/128 telemetry_queue