Troubleshooting Cisco NIR Application

This chapter contains the following sections:

Troubleshooting Cisco NIR Common GUI Issues

The following are troubleshooting tips for GUI issues on Cisco NIR app in Cisco DCNM .

  • The Cisco NIR app has the ability to display historical data. The specific time duration can be selected from the available calendar to see data within that particular time range.

  • The majority of issues will be due to receiving data from the APIs other than what was expected. Opening the Developer Tools Network tab and repeating the last action will show the API data received. If the issue is with the APIs, then troubleshooting will need to continue on the backend.

  • If the API requests and responses are accurate then check the Developer Tools Console tab for any errors.

  • After initial installation the application needs time to start. During this time, the GUI may exhibit incomplete or unstable behavior. It is recommended to wait several minutes before starting to use the application.

  • Take screenshots just before and just after reproducing an issue. The screenshots along with a full network capture saved as HAR with contents can be used to issue reports. If an issue report has a HAR recording attached then there is a significantly higher chance that the root cause can be identified and resolved quickly.

  • If the Cisco NIR GUI page loads to a skeleton template with a spinner then this means almost none of the APIs are responding.

  • If the Cisco NIR GUI page is taking a while to load fabrics then this means the fabrics.json API is not responding or not returning any fabrics.

  • If the fabric anomaly score does not agree with reported anomalies, or the node counts are incorrect, then check the fabricsSummary.json response for the fabric anomalyScore value, and check the nodes.json response for the types and counts of nodes reported.

  • If the expected fabrics are not shown in the fabric selection dropdown, first verify that they are not included in the fabrics.json response entries, then rerun setup and edit the data collection setup configuration to view the state of the configured fabrics. Make sure the appropriate fabrics are enabled and that no errors are reported. This data comes from the get_nir_fabrics request.

  • For Flow Analytics issues make sure the following requirements are met:

    • The capability.json request is made when the GUI loads and returns true. If it returns false, it means the fabric does not support this feature.

    • Navigate to Application Settings tab and make sure Flow Collection has been enabled, the Management In-Band EPG has been selected, and verify the flow collection filters have been correctly configured.

    • To verify the MOs are using visore, navigate to uni > fabric > flowcol to check the configuration and check the classes telemetrySelector, telemetrySubnetFltGrp, and telemetrySubnetFilter.

    • Navigate to Collection Status tab and check if the nodes are returning flow telemetry.

Total Audit Logs, Events, and Faults

Faults

If faults occur within the application, they can be viewed from the Warning icon at the top-right of Application GUI screen next to the Settings icon.

Table 1. Total Audit Logs, Events, and Faults
Property Description

Creation Time

The day and time of when the audit log, event, or fault instance occurred.

Severity

The current severity level of the event. The levels are:

  • Critical—A service-affecting condition that requires immediate corrective action. For example, this severity could indicate that the managed object is out of service and its capability must be restored.

  • Major—Serious problems exist with one or more components. These issues should be researched and fixed immediately.

  • Minor—Problems exist with one or more components that might adversely affect system performance. These issues should be researched and fixed as soon as possible before they become a critical problem.

  • Warning—Potential problems exist with one or more components that might adversely affect system performance if they are allowed to continue. These issues should be researched and fixed as soon as possible before they become a critical problem.

  • Info—A basic notification or informational message, possibly independently insignificant.

  • Cleared—A notification that the condition that caused the fault has been resolved, and the fault has been cleared.

Code

The code that helps to categorize and identify different types of fault instance objects.

Last Transition

The day and time on which the severity last changed. If the severity has not changed, this field displays the original creation date.

Description

Additional descriptive information on the audit log, event or fault.

Basic Debugging Commands

apic-ifc1# acidiag scheduler status
Scheduler status:
 [True]    APIC-01
 [True]    APIC-02
 [True]    APIC-03

apic-ifc1# acidiag scheduler members

 ID             Name    Status         Address      OOBAddress    Type       Serial  NodeFqdn
-----------------------------------------------------------------------------------------------------------------------------
  1*     apic-ifc1    active        10.0.0.1   172.1.2.3    Apic  FCH1748V24D  apic-ifc1.node.ifav22.apic.local
  2      apic-ifc2    active        10.0.0.2    172.4.5.6    Apic  FCH1809V18S  apic-ifc2.node.ifav22.apic.local
  3      apic-ifc3    active        10.0.0.3    172.7.8.9    Apic  FCH1809V191  apic-ifc3.node.ifav22.apic.local

apic-ifc1# 
apic-ifc1# acidiag scheduler appstatus

Job                              Type         Status      
------------------------------------------------------- 
Cisco_NIR                       
 `-Cisco_NIR-ClusterService      service      running     
 `-Cisco_NIR-SystemService       system       running     
bird_kafka                      
 `-bird_kafka-kafka              system       running     
bird_kafkax                     
 `-bird_kafkax-kafka             system       running     
bird_zk                         
 `-bird_zk-zk                    service      running     
elastic                         
 `-elastic-systemjob             system       running     
elasticx                        
 `-elasticx-systemjob            system       running     

apic-ifc1# acidiag scheduler appstatus bird_kafka

Container            Group                                    Node             Status       Modified             Image
------------------------------------------------------------------------------------------------------------------------------------------------------ 
kafka                bird_kafka-kafka.kafka                   apic-ifc3      running      0d 19h 37m 16s       apic-system/kafka:0.1.0
kafka                bird_kafka-kafka.kafka                   apic-ifc1      running      0d 19h 37m 16s       apic-system/kafka:0.1.0
kafka                bird_kafka-kafka.kafka                   apic-ifc2      running      0d 19h 37m 16s       apic-system/kafka:0.1.0

apic-ifc1# acidiag scheduler appstatus elastic

Container            Group                                    Node             Status       Modified             Image
------------------------------------------------------------------------------------------------------------------------------------------------------ 
es                   elastic-systemjob.db                     apic-ifc1      running      0d 19h 41m  8s       apic-system/elastic:v1
es                   elastic-systemjob.db                     apic-ifc3      running      1d 13h  2m 52s       apic-system/elastic:v1
es                   elastic-systemjob.db                     apic-ifc2      running      1d 13h 13m 15s       apic-system/elastic:v1

apic-ifc1# acidiag scheduler appstatus Cisco_NIR

Container            Group                                    Node             Status       Modified             Image
------------------------------------------------------------------------------------------------------------------------------------------------------ 
app-brain            Cisco_NIR-ClusterService.brain           apic-ifc2      running      0d 18h 58m 53s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/brain:v1-0-1-827
app-scheduler        Cisco_NIR-ClusterService.scheduler       apic-ifc1      running      0d 18h 58m 54s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/scheduler:v1-0-1-827
app-correlator       Cisco_NIR-ClusterService.correlator      apic-ifc3      running      0d 18h 58m 53s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/correlator:v1-0-1-827
app-predictor        Cisco_NIR-ClusterService.predictor       apic-ifc3      running      0d 18h 58m 53s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/predictor:v1-0-1-827
app-apicagent        Cisco_NIR-ClusterService.apicagent       apic-ifc2      running      0d 18h 58m 54s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apicagent:v1-0-1-827
app-logstash         Cisco_NIR-SystemService.logstash         apic-ifc1      running      0d 18h 59m  4s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/logstash:v1-0-1-827
app-eventcollector   Cisco_NIR-SystemService.eventcollector   apic-ifc3      running      0d 18h 59m  5s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/eventcollector:v1-0-1-827
app-eventcollector   Cisco_NIR-SystemService.eventcollector   apic-ifc1      running      0d 18h 59m  4s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/eventcollector:v1-0-1-827
app-logstash         Cisco_NIR-SystemService.logstash         apic-ifc2      running      0d 18h 59m  5s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/logstash:v1-0-1-827
app-apiserver        Cisco_NIR-SystemService.apiserver        apic-ifc2      running      0d 18h 59m  4s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apiserver:v1-0-1-827
app-apiserver        Cisco_NIR-SystemService.apiserver        apic-ifc1      running      0d 18h 59m  5s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apiserver:v1-0-1-827
app-logstash         Cisco_NIR-SystemService.logstash         apic-ifc3      running      0d 18h 59m  4s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/logstash:v1-0-1-827
app-apiserver        Cisco_NIR-SystemService.apiserver        apic-ifc3      running      0d 18h 59m  4s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/apiserver:v1-0-1-827
app-eventcollector   Cisco_NIR-SystemService.eventcollector   apic-ifc2      running      0d 18h 59m  4s       local-docker-repo/cisco-nir/aci-docker-reg-cisco-com/telemetry/eventcollector:v1-0-1-827

apic-ifc1# 
apic-ifc1# acidiag scheduler elastic members
ip       heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.0.0.3           26          99  20    4.88    4.40     3.49 mdi       -      apic-ifc3
10.0.0.1           26          91  19    3.04    3.75     3.56 mdi       -      apic-ifc1
10.0.0.2           26          88  19    0.97    1.77     2.05 mdi       *      apic-ifc2

apic-ifc1# acidiag scheduler elastic health
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 120,
  "active_shards" : 360,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

apic-ifc1# acidiag scheduler elastic indices
health status index                                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   cisco_nir-fabricnodesdb                 B8X8lktsSnWzCckzms8JfQ   1   2         16            0    182.7kb           61kb
green  open   cisco_nir-aggflowdb-2019.01.31.18.00.00 RnIB3S7fTBikO07xPhquFw   9   2          0            0      6.1kb            2kb
green  open   cisco_nir-sysmetrics-2019.01.31         HBP_iJgsRQyGTyOa-Horvg   7   2    1807463            0    747.1mb        249.1mb
green  open   cisco_nir-statsdb-000003                Sgh1bZ7CQ_et4j__AQ56Ww   5   2    9517896            0      2.9gb        998.8mb
green  open   cisco_nir-eventsdb                      tJTC02wpSmy_9Fa8p33WDg   5   2      22940            0     25.4mb          8.4mb
green  open   searchguard                             9nSh8NeqSYKYF7w4W0eHkQ   1   2          5            2     65.1kb         21.7kb
green  open   cisco_nir-statsdb-000002                Zv9P247tSfyK_6o37NGkjg   5   2    9494058            0      2.9gb        999.5mb
green  open   cisco_nir-fault_historydb               mUY-NT2lQqmP54flD44xzg   5   2       2405            0      4.3mb          1.4mb
green  open   cisco_nir-collectorstatsdb              6RrCkrhxT6OWz-M8eIfjrw   5   2          0            0      3.4kb          1.1kb
green  open   cisco_nir-sysmetrics-2019.01.30         wz3Jif_8SMOhc4Or8MEXNg   7   2      41870            0     19.2mb          6.4mb
green  open   cisco_nir-fabric_issuesdb               tj-Y0cP4SF2OdfMkumqcqQ   2   2          0            0      1.3kb           466b
green  open   cisco_nir-anomalytsdb                   rzGukbWCTk276i2FQpRCJQ   3   2          1            0     24.9kb          8.3kb
green  open   cisco_nir-aggflowdb-2019.01.31.12.00.00 hVUmPx5JQJi9gtiEB4no_A   9   2          0            0      6.1kb            2kb
green  open   cisco_nir-resourcecollectdb             kDTBYxq0RtSp0tzXkFgVWw   3   2     168380            0     38.1mb         12.7mb
green  open   cisco_nir-resourcescoresdb              ApM3S1QEQ3m9co-UeX-tvQ   3   2      38120            0     29.4mb          9.8mb
green  open   cisco_nir-aggflowdb-2019.01.31.16.00.00 fdaRZvNVS2eVqEF1uRFKcg   9   2          0            0      6.1kb            2kb
green  open   cisco_nir-eprecordsdb                   JIzHooPPQwShJeFCa11GyA   5   2          0            0      3.4kb          1.1kb
green  open   cisco_nir-statsdb-000004                pqhaqo3OTv6E6y1zBwfwYg   5   2    1539566            0    507.6mb        170.8mb
green  open   cisco_nir-aggflowdb-2019.01.31.14.00.00 G6yngLSoQzyn1odMCDLIaQ   9   2          0            0      6.1kb            2kb
green  open   cisco_nir-licensedb                     87XBQmQHRfap024AAXnEXg   1   2          1            0     10.2kb          3.4kb
green  open   cisco_nir-aggflowdb-2019.01.31.20.00.00 ZQdMl2yxSaaNCdGXW-4YOg   9   2          0            0      6.1kb            2kb
green  open   cisco_nir-aggflowdb-2019.01.31.10.00.00 btO1x9A0Teakv2AdK_6n-A   9   2          0            0      6.1kb            2kb
green  open   cisco_nir-anomalydb                     QrHtrk2LSZ-LNsS37E0btQ   3   2          1            0     23.5kb          7.8kb

apic-ifc1# acidiag scheduler elastic shards
index                                   shard prirep state      docs   store ip       node
cisco_nir-sysmetrics-2019.01.30         4     r      STARTED    5914 924.5kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30         4     p      STARTED    5914 928.9kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30         4     r      STARTED    5914 899.8kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30         1     r      STARTED    6033 920.7kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30         1     p      STARTED    6033 954.1kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30         1     r      STARTED    6033 982.7kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30         2     r      STARTED    6070 944.1kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30         2     r      STARTED    6070 914.2kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30         2     p      STARTED    6070 951.1kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30         6     p      STARTED    5923 961.2kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30         6     r      STARTED    5923 944.4kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30         6     r      STARTED    5923 958.8kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30         3     p      STARTED    5962 954.4kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30         3     r      STARTED    5962 911.1kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30         3     r      STARTED    5962 926.3kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30         5     r      STARTED    6003 937.9kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30         5     r      STARTED    6003 931.6kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30         5     p      STARTED    6003   912kb 10.0.0.1 ifav22-ifc1
cisco_nir-sysmetrics-2019.01.30         0     p      STARTED    5965 947.9kb 10.0.0.3 ifav22-ifc3
cisco_nir-sysmetrics-2019.01.30         0     r      STARTED    5965 909.2kb 10.0.0.2 ifav22-ifc2
cisco_nir-sysmetrics-2019.01.30         0     r      STARTED    5965 966.8kb 10.0.0.1 ifav22-ifc1

<-- SNIP LIST OF ALL OTHER RESOURCES -->
apic-ifc1#