Consistency Checker

Limitations for Consistency Checker

The Consistency Checker has the following limitations:

Consistency Checkers are CPU intensive. It is not recommended to run the checkers at very short intervals.
Legacy Consistency Checkers do not have support for snapshot. So, the previous runs cannot be displayed.
There is no command to stop/abort the already running Consistency Checkers.
Forwarding Engine hardware entry validations are partially implemented. Only programming failures can be detected and reported.
Layer2 MAC Consistency Checker can validate the MAC address in hardware with software copy.
Consistency checker is designed to reduce false positives in all cases. However, there could be rare cases of reporting a false positive in the following scenarios:
- Large table state changes (i.e clear, relearn etc).
- Under very high CPU usage due to any other feature while a consistency checker running. The consistency checker may report inconsistency in processes where CPU usage is high.
Forwarding engine hardware (FED) check is not entirely supported in Layer3 Multicast Consistency Checker. You can only detect and report on programming failures.
Forwarding Manager-RP software entry is not supported in Layer3 Multicast Consistency Checker.

Information about Consistency Checker

Overview of Consistency Checker

The Consistency Checker collects information on various table states within the software and the hardware. It compares the software state with the hardware state. If there is any inconsistency, it flags the issue immediately. This helps to reduce increased troubleshooting time at a later period. The consistency checker supplements basic troubleshooting and helps to identify scenarios where inconsistent states between software and hardware tables are causing issues in the network, thereby reducing the mean time to resolve the issue.

There are two types of consistency checker implementation available:

Legacy Consistency Checker - supports validating the entry from control plane to the forwarding engine (or hardware copy).
End-to-End Consistency Checker - supports validating the software entry from control plane to all processes involved in distributing and handling the entry, as well as the forwarding engine's hardware copy.

End-to-End Consistency Checker

End-to-End (E2E) Consistency Checker supports full scan and single entry and should be started manually or run via gold diagnostic. The consistency checker can be started for a single entry using the command which helps to isolate the issue at which forwarding process entry is not consistent and helps speed up the debugging.

Every time the consistency checker is started, a runID is provided. Using the runID, its status, summary, details can be viewed. The last 5 snapshots are available any time for you to check the previous run's result.

E2E consistency checker performs the following functions:

Validates the IOS entry to software tables/processes (Forwarding manger-RP, Forwarding manager-FP and FED) for all modules.
Reports various inconsistencies (entry inconsistent, entry missing, stale entry) and sends a syslog to alert the administrator.
Helps to speed up the fault isolation.
Records any inconsistent entry with relevant data.
Consistency checker supports the recursive single entry check which can validate the dependent objects along with the actual entry. (i.e, A Layer 3 Multicast with N outgoing interfaces can be validated for multicast entries along with OIFs programming, OIF's Adjacency validation, etc)
Constant memory usages irrespective of total entries in a table.

Note

The consistency checker is bound to CPU utilization and can not exceed the configured value while validating the tables across processes.

Features Supported in Consistency Checker

The following features are supported in consistency checker:

Legacy Consistency Checker
- Layer2 MAC Consistency Checker: This consistency checker validates the IOS entry to FED software entry. It also validates the MAC address into hardware tables.
- Layer3 FMANFP Entry Consistency Checker: This consistency checker validates the Layer 2, Layer 3, and multicast objects status in the Forwarding Manager-FP process. This includes stale objects and long pending objects.
E2E Consistency Checker
- Layer2 Multicast Consistency Checker: This consistency checker validates the IOS Layer 2 multicast IGMP/MLD VLAN, the group entry to Forwarding Manager-FP software entry, FED software entry, and FED hardware programming errors.
- Layer3 Multicast Consistency Checker: This consistency checker validates the IOS Layer 3 multicast IGMP/MLD VLAN, the group entry to Forwarding Manager-FP software entry and FED software entry.

Running the Consistency Checker

The table shown below lists the commands to run the various consistency checkers:


Command	Purpose
show consistency-checker l2	Runs the consistency-checker on the Layer 2 forwarding tables.
show consistency-checker l3	Runs the consistency-checker on the Layer 3 forwarding tables.
show consistency-checker mcast l2m	Runs the consistency-checker on the Layer 2 multicast forwarding tables.
show consistency-checker mcast l3m	Runs the consistency-checker on the Layer 3 multicast forwarding tables.
show consistency-checker objects	Runs the End-to-End consistency-checker on objects.
show consistency-checker run-id `run-id`	Runs the End-to-End consistency-checker by run ID.
show consistency-checker switch	Runs the consistency-checker on the specified switch.

Output Examples for Consistency Checker

The following is a sample output for the show consistency-checker mcast l2m command where the consistency checker runs a full scan:

Device# show consistency-checker mcast l2m start all
L2 multicast Full scan started. Run_id: 2
Use 'show consistency-checker run-id 2 status' for completion status.

Device#
*Feb 17 06:19:14.889: %FED_CCK_ERRMSG-4-INCONSISTENCY_FOUND: F0/0: fed: Consistency Checker(CCK) detected inconsistency for l2m_vlan. Check 'show consistency run-id 2 detail'.
*Feb 17 06:19:14.890: %FED_CCK_ERRMSG-4-INCONSISTENCY_FOUND: F0/0: fed: Consistency Checker(CCK) detected inconsistency for l2m_group. Check 'show consistency run-id 2 detail'.
Device#
*Feb 17 06:19:19.432: %IOSXE_FMANRP_CCK-6-FMANRP_COMPLETED: Consistency Check for Run-Id 2 is completed. Check 'show consistency-checker run-id 2'.
Device#
Device# show consistency-checker run-id 2 status
Process: IOSD
  Object-Type      Status           Time(sec)     Exceptions
  l2m_vlan         Completed        13            No
  l2m_group        Completed        13            No

Process: FMAN-FP
  Object-Type      Status           Time(sec)     State
  l2m_vlan         Completed        9             Consistent
  l2m_group        Completed        9             Consistent

Process: FED
  Object-Type      Status           Time(sec)     State
  l2m_vlan         Completed        9             Inconsistent
  l2m_group        Completed        9             Inconsistent

Device#
Device# show consistency-checker run-id 2
Process: IOSD
  Object-Type    Start-time              Entries       Exceptions
  l2m_vlan       2021/02/17 06:19:05          22                0
  l2m_group      2021/02/17 06:19:05          24                0

Process: FMAN-FP
  *Statistics(A/I/M/S/Oth): Actual/Inherited/Missing/Stale/Others

  Object-Type    Start-time              State             A/  I/  M/  S/Oth
  l2m_vlan       2021/02/17 06:19:05     Consistent        0/  0/  0/  0/  0
  l2m_group      2021/02/17 06:19:05     Consistent        0/  0/  0/  0/  0

Process: FED
  *Statistics(A/I/M/S/HW/Oth): Actual/Inherited/Missing/Stale/Hardware/Others

  Object-Type    Start-time              State              A/  I/  M/  S/ HW/Oth
  l2m_vlan       2021/02/17 06:19:05     Inconsistent       1/  0/  0/168/  0/  0
  l2m_group      2021/02/17 06:19:05     Inconsistent       4/  0/  2/  0/  0/  0


Device#
Device# show consistency-checker run-id 2 detail
Process: IOSD

Process: FMAN-FP

Process: FED
  Object-Type:l2m_vlan   Start-time:2021/02/17 06:19:05
    Status:Completed   State:Inconsistent
    Key/data                                Reason
    (Ipv4, vlan: 768)                       Stale
      snoop:off stp_tcn:off flood:off pimsn:off
    (Ipv4, vlan: 769)                       Stale
      snoop:off stp_tcn:off flood:off pimsn:off
    (Ipv6, vlan: 900)                       Inconsistent
      snoop:on stp_tcn:on flood:on pimsn:off
    (Ipv6, vlan: 767)                       Stale
      snoop:off stp_tcn:off flood:off pimsn:off

  Object-Type:l2m_group   Start-time:2021/02/17 06:19:05
    Status:Completed   State:Inconsistent
    Key/data                                Reason
    (Ipv4, vlan:100 (*,227.0.0.0))          Inconsistent
      Group ports: total entries: 0
    (Ipv4, vlan:100 (*,227.1.0.0))          Missing

Device#

The following is a sample output for the show consistency-checker mcast l2m command where the consistency checker runs a recursive single-entry scan:

Device# show consistency-checker mcast l2m start vlan 900 229.1.1.1 recursive
Single entry scan started with Run_id: 2

*Feb 17 06:54:09.880: %IOSXE_FMANRP_CCK-6-FMANRP_COMPLETED: Consistency Check for Run-Id 2 is completed. 
Check 'show consistency-checker run-id 2'.
Device#
Device# show consistency-checker run-id 2
Process: IOSD
  Object-Type    Start-time              Entries       Exceptions
  l2m_vlan       2021/02/17 06:54:01           1                0
  l2m_group      2021/02/17 06:54:01           1                0

Process: FMAN-FP
  *Statistics(A/I/M/S/O): Actual/Inherited/Missing/Stale/Others

  Object-Type    Start-time              State            A / I / M / S / O
  l2m_vlan       1970/01/01 00:10:03     Consistent        0/  0/  0/  0/  0
  l2m_group      1970/01/01 00:10:03     Consistent        0/  0/  0/  0/  0


Process: FED
  *Statistics(A/I/M/S/HW/O): Actual/Inherited/Missing/Stale/Hardware/Others

  Object-Type    Start-time              State             A / I / M / S / HW/ O
  l2m_vlan       2021/02/17 06:54:01     Inconsistent       1/  0/  0/  0/  0/  0
  l2m_group      2021/02/17 06:54:01     Inconsistent       0/  1/  0/  0/  0/  0

Device#
Device# show consistency-checker run-id 2 detail
Process: IOSD
  Object-Type:l2m_vlan   Start-time:2021/02/17 06:54:01
    Key/data                                Reason
    (Ipv4, vlan:900)                        Success
      snoop:on stp_tcn:off flood:off pimsn:off

  Object-Type:l2m_group   Start-time:2021/02/17 06:54:01
    Key/data                                Reason
    (Ipv4, vlan:900, (*,229.1.1.1))         Success
      Twe1/0/5


Process: FMAN-FP

Process: FED
  Object-Type:l2m_group   Start-time:2021/02/17 06:54:01
    Status:Completed   State:Inconsistent
    Key/data                                Reason
    (Ipv4, vlan:900 (*,229.1.1.1))          Inherited
      Group ports: total entries: 1
        TwentyFiveGigE1/0/5

      ---------------Recursion-level-1-----------------
      Object-Type:l2m_vlan   Start-time:2021/02/17 06:54:01
        Status:Completed   State:Inconsistent
        Key/data                                Reason
        (Ipv4, vlan: 900)                       Inconsistent
          snoop:on stp_tcn:off flood:on pimsn:off

Device#

The following is a sample output for the show consistency-checker objects command where the consistency checker runs a scan on objects:

Device# show consistency-checker objects l2m_group
Process: IOSD
  Run-id    Start-time              Exception
  1         2021/02/17 05:20:42     0
  2         2021/02/17 06:19:05     0

Process: FMAN-FP
  *Statistics(A/I/M/S/Oth): Actual/Inherited/Missing/Stale/Others

  Run-id    Start-time              State           A/  I/  M/  S/Oth
  1         2021/02/17 05:20:42     Consistent      0/  0/  0/  0/  0
  2         2021/02/17 06:19:05     Consistent      0/  0/  0/  0/  0

Process: FED
  *Statistics(A/I/M/S/HW/Oth): Actual/Inherited/Missing/Stale/Hardware/Others

  Run-id    Start-time              State           A/  I/  M/  S/ HW/Oth
  1         2021/02/17 05:20:42     Consistent      0/  0/  0/  0/  0/  0
  2         2021/02/17 06:19:05     Inconsistent    4/  0/  2/  0/  0/  0

Device#
Stark#sh consistency-checker run 2 detail
Process: IOSD
  Object-Type:l2m_vlan   Start-time:2021/02/17 06:54:01
    Key/data                                Reason
    (Ipv4, vlan:900)                        Success
      snoop:on stp_tcn:off flood:off pimsn:off

  Object-Type:l2m_group   Start-time:2021/02/17 06:54:01
    Key/data                                Reason
    (Ipv4, vlan:900, (*,229.1.1.1))         Success
      Twe1/0/5


Process: FMAN-FP

Process: FED
  Object-Type:l2m_group   Start-time:2021/02/17 06:54:01
    Status:Completed   State:Inconsistent
    Key/data                                Reason
    (Ipv4, vlan:900 (*,229.1.1.1))          Inherited
      Group ports: total entries: 1
        TwentyFiveGigE1/0/5

      ---------------Recursion-level-1-----------------
      Object-Type:l2m_vlan   Start-time:2021/02/17 06:54:01
        Status:Completed   State:Inconsistent
        Key/data                                Reason
        (Ipv4, vlan: 900)                       Inconsistent
          snoop:on stp_tcn:off flood:on pimsn:off

Device# show consistency-checker objects l2m_group 2 detail
Process: IOSD

Process: FMAN-FP

Process: FED
  Object-Type:l2m_group   Start-time:2021/02/17 06:19:05
    Status:Completed   State:Inconsistent
    Key/data                                Reason
    (Ipv4, vlan:100 (*,227.0.0.0))          Inconsistent
      Group ports: total entries: 0
    (Ipv4, vlan:100 (*,227.1.0.0))          Missing
    (Ipv4, vlan:100 (*,227.0.0.1))          Inconsistent
      Group ports: total entries: 0
    (Ipv4, vlan:100 (*,227.1.0.1))          Missing
    (Ipv4, vlan:100 (*,227.0.0.2))          Inconsistent
      Group ports: total entries: 0
    (Ipv4, vlan:100 (*,227.0.0.3))          Inconsistent
      Group ports: total entries: 0


Device#

The following is a sample output for the show consistency-checker mcast l3m command where the consistency checker runs a full scan:

Device#sh consistency-checker mcast l3m start all
L3 multicast Full scan started. Run_id: 1
Use 'show consistency-checker run-id 1 status' for completion status.

Device#       
*Apr  2 17:30:01.831: %IOSXE_FMANRP_CCK-6-FMANRP_COMPLETED: Consistency Check for Run-Id 1 is completed. Check 'show consistency-checker run-id 1'. 
Device#sh consistency-checker run-id 1 
Process: IOSD
Flags:    F - Full Table Scan, S - Single Entry Run
          RE -  Recursive Check, GD -  Garbage Detector
          Hw -  Hardware Check, HS - Hardware Shadow Copy
  Object-Type    Start-time              Entries  Exceptions  Flags
  l3m_entry      2021/04/02 17:29:35           8         0    F GD Hw HS 

Process: FMAN-FP
  *Statistics(A/I/M/S/Oth): Actual/Inherited/Missing/Stale/Others

  Object-Type    Start-time              State             A/  I/  M/  S/Oth
  l3m_entry      2021/04/02 17:29:35     Consistent        0/  0/  0/  0/  0

Process: FED
  *Statistics(A/I/M/S/HW/Oth): Actual/Inherited/Missing/Stale/Hardware/Others

  Object-Type    Start-time              State              A/  I/  M/  S/ HW/Oth
  l3m_entry      2021/04/02 17:29:35     Consistent         0/  0/  0/  0/  0/  0

Device#sh consistency-checker mcast l3m start 225.1.1.1 recursive 
Single entry scan started with Run_id: 2
Use 'show consistency-checker run-id 2 status' for completion status.

Device#sh consistency-checker run-id 2 status 
Process: IOSD
  Object-Type      Status           Time(sec)     Exceptions
  l2m_vlan         Completed        11            No               
  l2m_group        Completed        11            No               
  l3m_entry        Completed        11            No               

Process: FMAN-FP
  Object-Type      Status           Time(sec)     State
  l2m_vlan         Completed        12            Consistent       
  l2m_group        Completed        12            Consistent       
  l3m_entry        Completed        12            Consistent       

Process: FED
  Object-Type      Status           Time(sec)     State
  l2m_vlan         Completed        12            Consistent       
  l2m_group        Completed        12            Consistent       
  l3m_entry        Completed        12            Consistent       
 
Device#sh consistency-checker run-id 2 detail 
Process: IOSD
  Object-Type:l2m_vlan   Start-time:2021/04/02 17:34:12
    Key/data                                Reason
    (Ipv4, vlan:100)                        Success
      snoop:on stp_tcn:off flood:off pimsn:off

  Object-Type:l2m_group   Start-time:2021/04/02 17:34:12
    Key/data                                Reason
    (Ipv4, vlan:100, (*,225.1.1.1))         Success
      Fo1/0/3
      
  Object-Type:l3m_entry   Start-time:2021/04/02 17:34:12
    Key/data                                Reason
    (Ipv4, (*,225.1.1.1))                   Success
      Entry flags: C 
      Total entries: 1
      Obj_id: F80004A1 Obj_flags: F 
      
Process: FMAN-FP
Process: FED

The following is a sample output for the show consistency-checker mcast l3m command where the consistency checker runs a recursive single-entry scan:

Device#sh consistency-checker mcast l3m start 225.1.1.1 15.1.1.1 recursive 
Single entry scan started with Run_id: 4
Use 'show consistency-checker run-id 4 status' for completion status.
Device#sh consistency-checker run-id 4 status 
Process: IOSD
  Object-Type      Status           Time(sec)     Exceptions
  l2m_vlan         Completed        10            No               
  l2m_group        Completed        10            No               
  l3m_entry        Completed        10            No               

Process: FMAN-FP
  Object-Type      Status           Time(sec)     State
  l2m_vlan         Completed        11            Consistent       
  l2m_group        Completed        11            Consistent       
  l3m_entry        Completed        11            Consistent       

Process: FED
  Object-Type      Status           Time(sec)     State
  l2m_vlan         Completed        11            Consistent       
  l2m_group        Completed        11            Consistent       
  l3m_entry        Completed        11            Consistent       
Device#sh consistency-checker run-id 4 detail 
Process: IOSD
  Object-Type:l2m_vlan   Start-time:2021/04/02 17:37:36
    Key/data                                Reason
    (Ipv4, vlan:100)                        Success
      snoop:on stp_tcn:off flood:off pimsn:off

  Object-Type:l2m_group   Start-time:2021/04/02 17:37:36
    Key/data                                Reason
    (Ipv4, vlan:100, (*,225.1.1.1))         Success
      Fo1/0/3
      
  Object-Type:l3m_entry   Start-time:2021/04/02 17:37:36
    Key/data                                Reason
    (Ipv4, vrf:, (15.1.1.1,225.1.1.1))      Success
      Entry flags: 
      Total entries: 2
      Obj_id: F80004A1 Obj_flags: F 
      Obj_id: F80003C1 Obj_flags: A 
      
Process: FMAN-FP
Process: FED

The following is a sample output for the show diagnostic content command where end to end consistency is checked through gold diagnostics:

Device#show diagnostic content switch all

switch 2  module 1: 

  Diagnostics test suite attributes:
    M/C/* - Minimal bootup level test / Complete bootup level test / NA
      B/* - Basic ondemand test / NA
    P/V/* - Per port test / Per device test / NA
    D/N/* - Disruptive test / Non-disruptive test / NA
      S/* - Only applicable to standby unit / NA
      X/* - Not a health monitoring test / NA
      F/* - Fixed monitoring interval test / NA
      E/* - Always enabled monitoring test / NA
      A/I - Monitoring is active / Monitoring is inactive

                                                          Test Interval   Thre-
  ID   Test Name                          Attributes      day hh:mm:ss.ms shold
  ==== ================================== ============    =============== =====
    1) TestGoldPktLoopback -------------> *BPN*X**I       not configured  n/a
    2) TestOBFL ------------------------> *B*N*X**I       not configured  n/a
    3) TestFantray ---------------------> *B*N****A       000 00:01:40.00 1
    4) TestPhyLoopback -----------------> *BPD*X**I       not configured  n/a
    5) TestThermal ---------------------> *B*N****A       000 00:01:30.00 1
    6) TestScratchRegister -------------> *B*N****A       000 00:01:30.00 5
    7) TestPortTxMonitoring ------------> *BPN****A       000 00:02:30.00 1
    8) TestConsistencyCheckL2 ----------> *B*N****A       000 00:01:30.00 1
    9) TestConsistencyCheckL3 ----------> *B*N****A       000 00:01:30.00 1
   10) TestConsistencyCheckMcast -------> *B*N****A       000 00:01:30.00 1
   11) TestConsistencyCheckL2m ---------> *B*N****A       000 00:01:30.00 1
   12) TestConsistencyCheckL3m ---------> *B*N****A       000 00:01:30.00 1  
This gives the status of consistency check for multicast

Feature History for Consistency Checker

This table provides release and related information for the features explained in this module.

These features are available in all the releases subsequent to the one they were introduced in, unless noted otherwise.


Release	Feature	Feature Information
Cisco IOS XE Amsterdam 17.3.1	Consistency Checker	The Consistency Checker collects information on various table states within the software and the hardware and flags any inconsistency it finds immediately. It supplements basic troubleshooting and helps to identify scenarios where inconsistent states between software and hardware tables are causing issues in the network, thereby reducing the mean time to resolve the issue.
Cisco IOS XE Bengaluru 17.6.1	Consistency Checker	This feature was enhanced and the multicast consistency checkers were introduced. The following keywords were added to the show consistency-checker command: mcast , objects , and run-id .
Cisco IOS XE Cupertino 17.7.1	Consistency Checker	Support for this feature was introduced on the Cisco Catalyst 9600 Series Supervisor 2 Module.

Use the Cisco Feature Navigator to find information about platform and software image support. To access Cisco Feature Navigator, go to https://cfnng.cisco.com/

http://www.cisco.com/go/cfn.

System Management Configuration Guide, Cisco IOS XE Cupertino 17.9.x (Catalyst 9600 Switches)

Bias-Free Language

Book Title

System Management Configuration Guide, Cisco IOS XE Cupertino 17.9.x (Catalyst 9600 Switches)

Chapter Title