gRPC Agent

About gRPC Agent

gRPC is a modern opensource high performance Remote Procedure Call (RPC) framework. Cisco NX-OS provides a gRPC agent to support gRPC related services including: gNMI and gNOI.

Revision History

Release Description
9.3(3)

Add support for

  • grpc port

  • grpc certificate
10.1(1)

Add support for client cert-based authentication

  • grep client root certificate
10.3(3) Support NGINX to act as GRPC proxy

Guidelines and Limitations for gRPC Agent

Following are the guidelines and limitations for gRPC agent:

  • When you enable gRPC on both the management VRF and default VRF and later disable on the default VRF, the gNMI operations on the management VRF stop working.

    As a workaround, disable gRPC completely by entering the no feature grpc command and reprovision it by entering the feature grpc command or with any existing gRPC configuration commands like, grpc certificate or grpc port . You must also resubscribe to any existing notifications on the management VRF.

  • If the gRPC certificate is explicitly configured, after a reload with the saved startup configuration to a prior Cisco NX-OS 9.3(x) image, the gRPC feature does not accept connections.

    To confirm this issue, enter the show grpc gnmi service statistics command. The following status error message is displayed:

    Status: Not running - Initializing...Port not available or certificate invalid.

    Unconfigure and configure the proper certificate command to restore the service.

  • If you have configured a custom gRPC certificate, upon entering the reload ascii command the configuration is lost. It reverts to the default day-1 certificate. After entering the reload ascii command, the switch reloads. Once the switch is up again, you must reconfigure the gRPC custom certificate.


    Note

    This applies when entering the grpc certificate command.

  • The reachability in non-default VRF for gRPC is supported only over L3VNI’s/EVPN and IP. However, reachability over MPLS in non-default VRF and VXLAN Flood and Learn is not supported.

  • For Cisco NX-OS release prior to 9.3(x), information about supported platforms, see Platform Support for Programmability Features in the guide for that release. Starting with Cisco NX-OS release 9.3(x), for information about supported platforms, see the Nexus Switch Platform Matrix.

  • The gRPC process uses the HIGH_PRIO control group, which limits the CPU usage to 75% of CPU and memory to 4 GB.

  • The gRPC agent supports management VRF and one user specified VRF for a total of two gRPC servers on each switch. Supporting a gRPC in the user specified VRF (for example: the default VRF) adds flexibility to offload processing gRPC calls from the management VRF, where significant traffic load is not desirable.

  • If two gRPC servers are configured, be aware of the following:

    • VRF boundaries are strictly enforced, so each gRPC server process requests independent of the other. Requests do not cross between VRFs.

    • The two servers are not HA or fault tolerant. One gRPC server does not back up the other, and there is no switchover or switchback between them.

    • Any limits for the gRPC server are per VRF.

  • Beginning with Cisco NX-OS Release 10.4(3)F, gRPC is supported on 92348GC-X.

Configuring the gRPC Agent

Configuring gRPC

Configure the gNMI feature through the grpc commands.

To import certificates used by the grpc certificate command onto the switch, see the Installing Identity Certificates section of the Cisco Nexus 9000 Series NX-OS Security Configuration Guide.


Note

When modifying the installed identity certificates or grpc port and grpc certificate values, the gRPC server might restart to apply the changes. When the gRPC server restarts, any active subscription is dropped, and you must resubscribe.

Before you begin

Prepare and sign the required certificate files for the server authentication.

You can re-use the existing trustpoint files as this is not specific to gRPC.

SUMMARY STEPS

  1. configure terminal
  2. (Optional) crypto ca trustpoint <server-trustpoint>
  3. crypto ca import <server-trustpoint> pkcs12 bootflash: :<server-ca-file> <pkcs-password>
  4. feature grpc
  5. (Optional) grpc port port-id
  6. grpc certificate certificate-id
  7. (Optional) use-vrf default

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:
switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

(Optional) crypto ca trustpoint <server-trustpoint>

Example:
switch# crypto ca trustpoint tls_server_trustpoint
 (Optional)

Creates a trustpoint for server authentication.

Step 2-3 is optional if there already exist usable server trustpoint.

Step 3

crypto ca import <server-trustpoint> pkcs12 bootflash: :<server-ca-file> <pkcs-password>

Example:
switch# crypto ca import tls_server_trustpoint pkcs12 bootflash:server.pfx test

Imports the server pkcs12 file to the trustpoint.

Step 4

feature grpc

Example:
switch# feature grpc
switch(config)#

Enables the gRPC agent, which supports the gNMI interface for dial-in.

Step 5

(Optional) grpc port port-id

Example:
switch(config)# grpc port 50051
 (Optional)

Configures the port number. The range of port-id is from 1024 to 65535. 50051 is the default.

Step 6

grpc certificate certificate-id

Example:
switch(config)# grpc certificate cert-1

Specify the certificate trustpoint ID. For more information, see the Installing Identity Certificates section of the Cisco Nexus 9000 Series NX-OS Security Configuration Guide for importing the certificate.

Step 7

(Optional) use-vrf default

Example:
switch(config)# grpc use-vrf default
 (Optional)

Enables the gRPC agent to accept incoming (dial-in) RPC requests from the default VRF. This step enables the default VRF to process incoming RPC requests. By default, the management VRF processes incoming RPC requests when the gRPC feature is enabled.

Generating Key/Certificate

The following is an example for generating a self-signed key/certificate in the switch bash shell. This is only for experimental usage. For more information on generating identity certificates, see the Installing Identity Certificates section of the Cisco Nexus 9000 Series NX-OS Security Configuration Guide.


Note

This task is an example of how a certificate can be generated on a switch. You can also generate a certificate in any Linux environment. In a production environment, you should consider using a CA signed certificate.

SUMMARY STEPS

  1. Generate the self-signed key and pem files.
  2. After generating the key and pem files, you must bundle the key and pem files for use in the trustpoint CA Association.
  3. Set up the trustpoint CA Association by inputting in the pkcs12 bundle into the trustpoint.
  4. Verify the setup.

DETAILED STEPS

  Command or Action Purpose

Step 1

Generate the self-signed key and pem files.

 
switch# run bash sudo su
bash-4.3# openssl req -x509 -newkey rsa:2048 -keyout self_sign2048.key -out self_sign2048.pem -days 365 -nodes

Step 2

After generating the key and pem files, you must bundle the key and pem files for use in the trustpoint CA Association.

 
After generating the key and pem files, you must bundle the key and pem files for use in the trustpoint CA Association.
switch# run bash sudo su bash-4.3# cd /bootflash/
bash-4.3# openssl pkcs12 -export -out self_sign2048.pfx -inkey self_sign2048.key -in self_sign2048.pem -certfile self_sign2048.pem -password pass:Ciscolab123!
bash-4.3# exit

Step 3

Set up the trustpoint CA Association by inputting in the pkcs12 bundle into the trustpoint.

 
switch(config)# crypto ca trustpoint mytrustpoint switch(config-trustpoint)# crypto ca import mytrustpoint pkcs12 self_sign2048.pfx Ciscolab123!

Step 4

Verify the setup.

 
switch(config)# show crypto ca certificates Trustpoint: mytrustpoint certificate:
subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
issuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R serial=0413
notBefore=Nov 5 16:48:58 2015 GMT
notAfter=Nov 5 16:48:58 2035 GMT
SHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3E purposes: sslserver sslclient
CA certificate 0: subject= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R
issuer= /C=US/O=Cisco Systems, Inc./OU=CSG/L=San Jose/ST=CA/street=3700 Cisco Way/postalCode=95134/CN=ems.cisco.com/serialNumber=FGE18420K0R serial=0413
notBefore=Nov 5 16:48:58 2015 GMT notAfter=Nov 5 16:48:58 2035 GMT
SHA1 Fingerprint=2E:99:2C:CE:2F:C3:B4:EC:C7:E2:52:3A:19:A2:10:D0:54:CA:79:3E purposes: sslserver sslclient

Configuring gRPC Client Certificate Authentication

gRPC also allows to authenticate the client based on the cert files (public key). This provides password-less authentication, which is considered more secure than password-based authentication.

Before you begin

Prepare and sign the required certificate files for the server authentication.

You can re-use the existing trustpoint files as this is not specific to gRPC.

SUMMARY STEPS

  1. configure terminal
  2. (Optional) crypto ca trustpoint <server-trustpoint>
  3. rsakeypair <client-key>
  4. (Optional) crypto ca authenticate <client-root-trustpoint>
  5. grpc client root certificate <client-root-trustpoint>

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:
switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

(Optional) crypto ca trustpoint <server-trustpoint>

Example:
switch# crypto ca trustpoint tls_server_trustpoint
 (Optional)

Creates a trustpoint for server authentication.

Step 2-3 is optional if there already exist usable server trustpoint.

Step 3

rsakeypair <client-key>

Example:
switch# rsakeypar client-key

Generates a rsa key pair for the client trustpoint.

Step 4

(Optional) crypto ca authenticate <client-root-trustpoint>

Example:
switch# crypto ca authenticate client_trustpoint
 (Optional)

Imports the client certificate. This step requires manual copy paste. Please follow the instruction.

Step 5

grpc client root certificate <client-root-trustpoint>

Example:
switch(config)# grpc client root certificate client_trustpoint

Enters the trustpoint to host the client CA root certificate.
Example

Config example

This section provides an example config sequence for illustration.

  1. Prepare the Client Root CA Certificates.

  2. Import the certificate

    When you have generated a new certificate to the client root successfully, following are the sample commands to configure them in the switch, and their output.
    switch(config)# crypto ca trustpoint my_client_trustpoint 
switch(config-trustpoint)# crypto ca authenticate my_client_trustpoint
input (cut & paste) CA certificate (chain) in PEM format; end the input with a line containing only END OF INPUT :
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIJAJLisBKCGjQOMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJDQTERMA8GA1UEBwwIU2FuIEpvc2UxDjAMBgNVBAoM
BUNpc2NvMB4XDTIwMTAxNDIwNTYyN1oXDTQwMTAwOTIwNTYyN1owPTELMAkGA1UE
BhMCVVMxCzAJBgNVBAgMAkNBMREwDwYDVQQHDAhTYW4gSm9zZTEOMAwGA1UECgwF
Q2lzY28wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDEX7qZ2EdogZU4 EW0NSpB3EjY0nSlFLOw/iLKSXfIiQJD0Qhaw16fDnnYZj6vzWEa0ls8canqHCXQl gUyxFOdGDXa6neQFTqLowSA6UCSQA+eenN2PIpMOjfdFpaPiHu3mmcTI1xP39Ti3 /y548NNORSepApBNkZ1rJSB6Cu9AIFMZgrZXFqDKBGSUOf/CPnvIDZeLcun+zpUu
CxJLA76Et4buPMysuRqMGHIX8CYw8MtjmuCuCTHXNN31ghhgpFxfrW/69pykjU3R
YOrwlSUkvYQhtefHuTHBmqym7MFoBEchwrlC5YTduDzmOvtkhsmpogRe3BiIBx45
AnZdtdi1AgMBAAGjUzBRMB0GA1UdDgQWBBSh3IqRrm+mtB5GNsoLXFb3bAVg5TAf
BgNVHSMEGDAWgBSh3IqRrm+mtB5GNsoLXFb3bAVg5TAPBgNVHRMBAf8EBTADAQH/
MA0GCSqGSIb3DQEBCwUAA4IBAQAZ4Fpc6lRKzBGJQ/7oK1FNcTX/YXkneXDk7Zrj 8W0RS0Khxgke97d2Cwl5P5reXO27kvXsnsz/VZn7JYGUvGSlxTlcCb6x6wNBr4Qr t9qDBu+LykwqNOFe4VCAv6e4cMXNbH2wHBVS/NSoWnM2FGZ10VppjEGFm6OM+N6z 8n4/rWslfWFbn7T7xHH+Nl0Ffc+8q8h37opyCnb0ILj+a4rnyus8xXJPQb05DfJe ahPNfdEsXKDOWkrSDtmKwtWDqdtjSQC4xioKHoshnNgWBJbovPlMQ64UrajBycwV z9snWBm6p9SdTsV92YwF1tRGUqpcI9olsBgH7FUVU1hmHDWE
-----END CERTIFICATE----END OF INPUT
Fingerprint(s): SHA1 Fingerprint=0A:61:F8:40:A0:1A:C7:AF:F2:F7:D9:C7:12:AE:29:15:52:9D:D2:AE
Do you accept this certificate? [yes/no]:yes switch(config)#
NOTE: Use the CA Certificate from the .pem file content.
switch# show crypto ca certificates Trustpoint: my_client_trustpoint CA certificate 0:
subject=C = US, ST = CA, L = San Jose, O = Cisco
issuer=C = US, ST = CA, L = San Jose, O = Cisco
serial=B7E30B8F4168FB87 notBefore=Oct 1 17:29:47 2020 GMT notAfter=Sep 26 17:29:47 2040 GMT
SHA1 Fingerprint=E4:91:4E:D4:41:D2:7D:C0:5A:E8:F7:2D:32:81:B3:37:94:68:89:10 purposes: sslserver sslclient

  3. Associating Trustpoints to gRPC

    When you have configured a new certificate to the client root successfully, the following is the output example for associating trustpoints to gRPC server on the switch:
    switch(config)# feature grpc
switch(config)# grpc client root certificate my_client_trustpoint switch(config)# show run grpc
!Command: show running-config grpc
!Running configuration last done at: Wed Dec 16 20:18:35 2020
!Time: Wed Dec 16 20:18:40 2020
version 10.1(1) Bios:version N/A feature grpc
grpc gnmi max-concurrent-calls 14 grpc use-vrf default grpc certificate my_trustpoint
grpc client root certificate my_client_trustpoint grpc port 50003

  4. Validating the Certificate Details

    When you have successfully associated the trustpoints to gRPC on the switch, the following is the output example for validating the certificate details:
    switch# show grpc gnmi service statistics
============= gRPC Endpoint =============
Vrf : management
Server address : [::]:50003
Cert notBefore : Mar 13 19:05:24 2020 GMT
Cert notAfter : Nov 20 19:05:24 2033 GMT
Client Root Cert notBefore : Oct 1 17:29:47 2020 GMT
Client Root Cert notAfter : Sep 26 17:29:47 2040 GMT
...

  5. Verifying the Connection using Client Certificate Authentication for any gNMI Clients.

    The client certificate requests with a private key (pkey) and ca chain (cchain). The password is now optional. Make sure the client needs to supply full chain from to root CA to its client cert.

  6. For removing trustpoint reference from gRPC (no command) use the following command:
    switch(config)# no grpc client root certificate my_client_trustpoint

    The command will remove the trustpoint reference only from gRPC agent, but the trustpoints CA certificates will NOT be removed. Connections that use client certificate authentication to gRPC server on switch will not establish, but basic authentication with username and password will go through.

Configuring NGINX proxy for GRPC

Like Netconf and Restconf, gRPC agent runs on a dedicate server/port. The gRPC client would need to directly connect to the gRPC agent/server.

Starting from release 10.3(3)F, NX-OS NGINX can also act as GRPC proxy by relaying the gRPC traffic and can be useful for certain use cases.

  • GRPC port blocked: The GRPC agent listens on port 50051. If this port is blocked by the firewall, GRPC clients can access the gRPC services indirectly through the NGINX HTTPS port 443.

  • Increased VRF support: Currently GRPC services are accessible only via management or one user specified VRFs. NGINX proxy can forward the gRPC requests from any VRF.

This new support does not affect existing behavior. The GRPC client can still connect to the GRPC agent directly. It can also instead connect to the NGINX server, which then proxies the GRPC requests to the GRPC agent. Note that such redirection deems to incur additional request-response latency.

All server and client authentication will be handled by NGINX. Just enable GRPC and configure NGINX server certificate and/or client certificates.

Before you begin

Enable the grpc feature.

Prepare the NX-API certificates. See Using NX-API CLI for detail.

SUMMARY STEPS

  1. configure terminal
  2. feature nxapi
  3. nxapi certificate httpscrt certfile cert-file
  4. nxapi certificate httpscrt keyfile key-file password <password>
  5. nxapi certificate enable
  6. (Optional) crypto ca trustpoint <trustpoint>
  7. (Optional) crypto ca authenticate <trustpoint>
  8. (Optional) nxapi client certificate authentication

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:
switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

feature nxapi

Example:
switch# feature nxapi
switch(config)#

Enables the feature nxapi.

Step 3

nxapi certificate httpscrt certfile cert-file

Example:
switch# nxapi certificate httpscrt certfile bootflash:nxapi.crt

Configures the cert file.

Step 4

nxapi certificate httpscrt keyfile key-file password <password>

Example:
switch# nxapi certificate httpskey keyfile bootflash:nxapi.key password cisco123

Configures the key file.

Step 5

nxapi certificate enable

Example:
switch# nxapi certificate enable

Enables the certificate authentication.

Step 6

(Optional) crypto ca trustpoint <trustpoint>

Example:
switch# crypto ca trustpoint grpcClientCA
 (Optional)

Creates a trustpoint for server authentication.

Step 7

(Optional) crypto ca authenticate <trustpoint>

Example:
switch# crypto ca authenticate grpcClientCA
 (Optional)

Imports the client root certificate to the trustpoint.

Step 8

(Optional) nxapi client certificate authentication

Example:
switch# nxapi client certificate authentication
 (Optional)

Enables the client certificate authentication.

Troubleshooting

Check Feature Status

  • In Cisco NX-OS device, enter the show feature grpc command to check the agent config.

  • To view the status of the gRPC agent, use the show feature command. 

switch-1# show feature | grep grpc
restconf	1	enabled
switch-1#

Check Connectivity

From a client system, ping the management port of the switch to verify that the switch is reachable.

Gathering gRPC Agent Logs

The /volatile directory houses the grpc agent log.
bash-4.3# cd /volatile/ bash-4.3# ls /volatile -al
…
-rw-rw-rw- 1 root root 103412 Jun 21 16:14 grpc-internal-log
…

Gathering TM-Trace Logs

tmtrace.bin -f gnmi-logs gnmi-events gnmi-errors following are available 2. 
Usage:
bash-4.3# tmtrace.bin -d gnmi-events | tail -30 Gives the last 30
…
[06/21/19 15:58:38.969 PDT f8f 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,
sub_id_str: 2329, dc_start_time: 0, length: 124, sync_response:1
[06/21/19 15:58:43.210 PDT f90 3133] [3621780288][tm_ec_yang_data_processor.c:93] TM_EC: [Y] Data received for 2799743488: 49
{
  "cdp-items": {
    "inst-items": {
      "if-items": {
        "If-list": [
          {
            "id": "mgmt0",
            "ifstats-items": {
              "v2Sent": "74",
              "validV2Rcvd": "79"
            }
          }
        ]
      }
    }
  }
} [06/21/19 15:58:43.210 PDT f91 3133] [3981658944][tm_transport_internal.c:43] dn: Cisco-NX-OS-device:System/cdp-items, sub_id: 0,
sub_id_str: 2329, dc_start_time: 0, length: 141, sync_response:1
[06/21/19 15:59:01.341 PDT f92 3133] [3981658944][tm_transport_internal.c:43] dn:
Cisco-NX-OS-device:System/intf-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157935518, length: 3063619, sync_response:0 [06/21/19 15:59:03.933 PDT f93 3133] [3981658944][tm_transport_internal.c:43] dn:
Cisco-NX-OS-device:System/cdp-items, sub_id:
4091, sub_id_str: , dc_start_time: 1561157940881, length: 6756, sync_response:0 [06/21/19 15:59:03.940 PDT f94 3133] [3981658944][tm_transport_internal.c:43] dn:
Cisco-NX-OS-device:System/lldp-items, sub_id:

Gathering MTX-Internal Logs

  1. Modify the following file with below /opt/mtx/conf/mtxlogger.cfg

    Please refer to the “Diagnose and Serviceability” section to toggle the preferred filter.

  2. Disable then enable feature grpc .

  3. The /volatile directory houses the mtx-internal.log, the log rolls over time so make sure to download the logs before rolled over. 

    bash-4.3# cd /volatile/ bash-4.3# ls /volatile -al
…
-rw-r--r-- 1 root root 24 Jun 21 14:44 mtx-internal-19-06-21-14-46-21.log
-rw-r--r-- 1 root root 24 Jun 21 14:46 mtx-internal-19-06-21-14-46-46.log
-rw-r--r-- 1 root root 175 Jun 21 15:11 mtx-internal-19-06-21-15-11-57.log
-rw-r--r-- 1 root root 175 Jun 21 15:12 mtx-internal-19-06-21-15-12-28.log
-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-17.log
-rw-r--r-- 1 root root 175 Jun 21 15:13 mtx-internal-19-06-21-15-13-42.log
-rw-r--r-- 1 root root 24 Jun 21 15:13 mtx-internal-19-06-21-15-14-22.log
-rw-r--r-- 1 root root 24 Jun 21 15:14 mtx-internal-19-06-21-15-19-05.log
-rw-r--r-- 1 root root 24 Jun 21 15:19 mtx-internal-19-06-21-15-47-09.log
-rw-r--r-- 1 root root 24 Jun 21 15:47 mtx-internal.log