Onboard Failure Logging

Onboard Failure Logging (OBFL) captures and stores hardware failure and environmental information into nonvolatile memory. OBFL permits improved accuracy in hardware troubleshooting and root cause isolation analysis. Stored OBFL data can be retrieved in the event of a router crash or failure.

Finding Feature Information

Finding Feature Information

Your software release may not support all the features that are documented in this module. For the latest feature information and caveats, see the release notes for your platform and software release. The Feature Information Table at the end of this document provides information about the documented features and lists the releases in which each feature is supported.

Hardware Compatibility Matrix for the Cisco cBR Series Routers


Note


The hardware components that are introduced in a given Cisco IOS-XE Release are supported in all subsequent releases unless otherwise specified.
Table 1. Hardware Compatibility Matrix for the Cisco cBR Series Routers

Cisco CMTS Platform

Processor Engine

Interface Cards

Cisco cBR-8 Converged Broadband Router

Cisco IOS-XE Release 16.5.1 and Later Releases

Cisco cBR-8 Supervisor:

  • PID—CBR-SUP-250G

  • PID—CBR-CCAP-SUP-160G

  • PID—CBR-CCAP-SUP-60G

  • PID—CBR-SUP-8X10G-PIC

Cisco IOS-XE Release 16.5.1 and Later Releases

Cisco cBR-8 CCAP Line Cards:

  • PID—CBR-LC-8D30-16U30

  • PID—CBR-LC-8D31-16U30

  • PID—CBR-RF-PIC

  • PID—CBR-RF-PROT-PIC

  • PID—CBR-CCAP-LC-40G

  • PID—CBR-CCAP-LC-40G-R

Cisco cBR-8 Downstream PHY Modules:

  • PID—CBR-D30-DS-MOD

  • PID—CBR-D31-DS-MOD

Cisco cBR-8 Upstream PHY Modules:

  • PID—CBR-D30-US-MOD

  • PID—CBR-D31-US-MOD

Understanding OBFL

OBFL provides a mechanism to store hardware, software, and environment related critical data in a non-volatile memory, such as flash EPROM or EEPROM on routers. The logging information is used by the TAC team to troubleshoot and fix hardware issues.

OBFL collects data like temperatures and voltages. It stores the data in a dedicated area of the flash memory of the router. This data is retrieved by TAC personnel to troubleshoot routers. It can also be analyzed by back-end software to detect failure patterns, and possibly to recommend specific quality improvements.

Retrieval of the OBFL message

If the hardware is defective and the system cannot boot up, any data in flash is inaccessible. In that case, use any one of the following methods to recover OBFL data:
  • Read the flash through JTAG: this requires provisions in hardware design and back-end hardware and software support tools.

  • Repair the system; boot it; use the OBFL CLI commands.

Configuring OBFL

Use the hw-module {all|slot|module} {slotnumber/subslotnumber|modulenumber} logging onboard {disable | enable} command to enable or disable OBFL on a specified hardware module.


Note


OBFL is enabled by default.
Router# hw-module slot R0 logging onboard enable

Displaying OBFL Logging Information

Use the show logging onboard {slot|module|bay} {slotnumber/subslotnumber|modulenumber} {dram | message | serdes | status | temperature | uptime | voltage | firmware] command to view the OBFL log information.


Note


OBFL is enabled by default on the Cisco cBR series router.

For the card PICs, use the show logging onboard bay slotnumber/subslotnumber {dram | message | serdes | status | temperature | uptime | voltage| firmware] command to view its OBFL information.

Clearing OBFL Logging

Use the clear logging onboard {slot|module|bay} {slotnumber/subslotnumber|modulenumber} {dram | message | serdes | temperature | voltage|firmware} command to clear OBFL logging.

The following example shows how to clear firmware version:

Router# clear logging onboard slot R0 firmware
Router# clear logging onboard bay 4/4 firmware

Following example shows how to clear DRAM ECC error log:

Router# clear logging onboard slot R0 dram

Following example shows how to clear OBFL error message:

Router# clear logging onboard slot R0 message

Following example shows how to clear onboard serdes log:

Router# clear logging onboard slot R0 serdes

Following example shows how to clear onboard temperature log:

Router# clear logging onboard slot R0 temperature

Following example shows how to clear onboard voltage log:

Router# clear logging onboard slot R0 voltage

Configuration and Verification Examples

Example—Verifying OBFL Configuration Status


Router#show logging onboard slot R1 status 
Status: Enabled

Router#show logging onboard slot 5 status  
Status: Disabled

Example—Displaying OBFL Logs


The following onboard failure logging example shows firmware version for SUP160: 
Router# show logging onboard slot R0 firmware

slot   timestamp            firmware                      version                                                                                               
--------------------------------------------------------------------------------------------------------------------------------------------------------------
  0    01/16/18 09:36:38    CPLD                          16052011                                                                                              
  0    01/16/18 09:36:38    ViperSO CPLD                  14091201                                                                                              
  0    01/16/18 09:36:38    ViperSIO CPLD                 14092901                                                                                              
  0    01/16/18 09:36:39    Rommon                        16.6(1r)S                                                                                             
  0    01/16/18 09:36:39    SUP-DC CPLD                   ffffffff                                                                                              
  0    01/16/18 09:36:39    SUP PSOC 0                    v4.1.0_i2c1                                                                                           
  0    01/16/18 09:36:39    SUP PSOC 1                    v4.0.8_i2c1                                                                                           
  0    01/16/18 09:36:39    SUP PSOC 2                    v4.1.1_IVB                                                                                            
  0    01/16/18 09:36:39    SUP PSOC 3                    v4.0.6_i2c1                                                                                           
  0    01/16/18 09:36:39    SUP-DC PSOC 0                 N/A                                                                                                   
  0    01/16/18 09:36:39    SUP-DC PSOC 1                 N/A                                                                                                   
  0    01/16/18 09:36:39    SUP-PIC PSOC 0                V2.0.6                                                                                                
  0    01/16/18 09:36:39    SUP-PIC PSOC 1                V2.0.6                                                                                                
  0    01/16/18 09:36:39    Blackbird                     00000112                                                                                              
  0    01/16/18 09:38:12    Raptor ESI                    0001003b   



The following onboard failure logging example shows firmware version for linecards:
Router# show logging onboard slot 3 firmware  

slot   timestamp            firmware                      version                                                                                               
--------------------------------------------------------------------------------------------------------------------------------------------------------------
  3    01/16/18 09:41:43    CPLD                          00000025                                                                                              
  3    01/16/18 09:41:43    Rommon                        2011.03.18                                                                                            
  3    01/16/18 09:41:43    Basestar                      00110022                                                                                              
  3    01/16/18 09:41:43    Raider                        02020018                                                                                              
  3    01/16/18 09:41:43    Caprica                       00000023                                                                                              
  3    01/16/18 09:41:43    HA-PLL                        N/A                                                                                                   
  3    01/16/18 09:41:43    PSOC 0                        v4.6                                                                                                  
  3    01/16/18 09:41:44    PSOC 1                        v4.6                                                                                                  
  3    01/16/18 09:42:04    dsphy0_fpga                   2.f                                                                                                   
  3    01/16/18 09:42:04    dsphy0_micro                  1.e                                                                                                   
  3    01/16/18 09:42:04    dsphy0_psoc                   3.9                                                                                                   
  3    01/16/18 09:42:04    dsphy0_cpld                   0.6                                                                                                   
  3    01/16/18 09:42:04    dsphy1_fpga                   2.f                                                                                                   
  3    01/16/18 09:42:04    dsphy1_micro                  1.e                                                                                                   
  3    01/16/18 09:42:04    dsphy1_psoc                   3.9                                                                                                   
  3    01/16/18 09:42:04    dsphy1_cpld                   0.6     



The following onboard failure logging example shows firmware version for RF-PICs:
Router# show logging onboard bay 4/3 firmware 

slot   timestamp            firmware                      version                                                                                               
------------------------------------------------------------------------------------
  3    01/16/18 09:39:21    RF-PIC Firmware               0000073e     



The following onboard failure logging example shows firmware version for SUP160-PIC:
Router# show logging onboard bay 4/4 firmware 

slot   timestamp            firmware                      version                                                                                               
------------------------------------------------------------------------------------
  4    01/16/18 09:40:20    SUP-PIC CPLD                  14071504                                                                                              
  4    01/16/18 09:40:20    DTI Client FPGA               00000005                                                                                              
  4    01/16/18 09:40:20    DTI Firmware                  00000a03                                                                                              
  4    01/16/18 09:40:20    Raptor MAC                    00010031                                                                                              
  4    01/16/18 09:40:20    Cortina PHY                   201402061607                                                                                          



The following onboard failure logging example shows firmware version for D-PIC:
Router# show logging onboard bay 4/8 firmware
slot   timestamp            firmware           version                    
---------------------------------------------------------------------------------------------
8    01/16/18 09:40:13    DPIC Firmware        00010001 (UBOOT:2015.7 FPGA:00fd0000 00010011) 



The following onboard failure logging example shows the firmware versions that recently booted up: 
Router# show logging onboard slot R0 firmware reverse 

slot   timestamp            firmware                      version                                                                                               
------------------------------------------------------------------------------------
  0    01/16/18 09:38:12    Raptor ESI                    0001003b                                                                                              
  0    01/16/18 09:36:39    Blackbird                     00000112                                                                                              
  0    01/16/18 09:36:39    SUP-PIC PSOC 1                V2.0.6                                                                                                
  0    01/16/18 09:36:39    SUP-PIC PSOC 0                V2.0.6                                                                                                
  0    01/16/18 09:36:39    SUP-DC PSOC 1                 N/A                                                                                                   
  0    01/16/18 09:36:39    SUP-DC PSOC 0                 N/A                                                                                                   
  0    01/16/18 09:36:39    SUP PSOC 3                    v4.0.6_i2c1                                                                                           
  0    01/16/18 09:36:39    SUP PSOC 2                    v4.1.1_IVB                                                                                            
  0    01/16/18 09:36:39    SUP PSOC 1                    v4.0.8_i2c1                                                                                           
  0    01/16/18 09:36:39    SUP PSOC 0                    v4.1.0_i2c1                                                                                           
  0    01/16/18 09:36:39    SUP-DC CPLD                   ffffffff                                                                                              
  0    01/16/18 09:36:39    Rommon                        16.6(1r)S                                                                                             
  0    01/16/18 09:36:38    ViperSIO CPLD                 14092901                                                                                              
  0    01/16/18 09:36:38    ViperSO CPLD                  14091201                                                                                              
  0    01/16/18 09:36:38    CPLD                          16052011



The following onboard failure logging example shows the firmware versions that are logged in the backup log file. The backup log file is created when an existing log file reaches its maximum size of 1MB.
sj-104-cbr-13#show logging onboard bay 4/4 firmware backup

slot   timestamp            firmware                      version                                                                                               
------------------------------------------------------------------------------------
  4    01/16/18 09:40:20    SUP-PIC CPLD                  14071504                                                                                              
  4    01/16/18 09:40:20    DTI Client FPGA               00000005                                                                                              
  4    01/16/18 09:40:20    DTI Firmware                  00000a03                                                                                              
  4    01/16/18 09:40:20    Raptor MAC                    00010031                                                                                              
  4    01/16/18 09:40:20    Cortina PHY                   201402061607  
…
  4    01/17/18 08:38:22    SUP-PIC CPLD                  14071504                                                                                              
  4    01/17/18 08:38:22    DTI Client FPGA               00000005                                                                                              
  4    01/17/18 08:38:22    DTI Firmware                  00000a03                                                                                              
  4    01/17/18 08:38:22    Raptor MAC                    00010031                                                                                              
  4    01/17/18 08:38:22    Cortina PHY                   201402061607


The following onboard failure logging example shows the firmware versions that were recently logged in the backup log file:
Router# show logging onboard bay 4/4 firmware backup reverse

slot   timestamp            firmware                      version                                                                                               
  4    01/17/18 08:38:22    Cortina PHY                   201402061607     
  4    01/17/18 08:38:22    Raptor MAC                    00010031                                                                                              
  4    01/17/18 08:38:22    DTI Firmware                  00000a03                                                                                              
  4    01/17/18 08:38:22    DTI Client FPGA               00000005                                                                                              
  4    01/17/18 08:38:22    SUP-PIC CPLD                  14071504                                                                                              
  …
  4    01/16/18 09:40:20    Cortina PHY                   201402061607  
  4    01/16/18 09:40:20    Raptor MAC                    00010031                                                                                              
  4    01/16/18 09:40:20    DTI Firmware                  00000a03                                                                                              
  4    01/16/18 09:40:20    DTI Client FPGA               00000005                                                                                              
  4    01/16/18 09:40:20    SUP-PIC CPLD                  14071504



Router#show logging onboard slot R1 message 
timestamp            module    sev  message                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
01/01/12 12:00:23    SUP_PSOC  3    SUP MB PSOC alert interrupt                                                                                                       
01/01/12 12:00:23    SUP_PSOC  3    SUP MB PSOC alert interrupt                                                                                                       
01/01/12 12:00:23    SUP_PSOC  3    SUP MB PSOC alert interrupt                                                                                                       
01/01/12 12:00:23    SUP_PSOC  3    SUP MB PSOC alert interrupt                                                                                                       
01/01/12 12:01:15    SUP_PSOC  3    SUP MB PSOC alert interrupt      


Router#show logging onboard slot R1 voltage 
Name              Id    Data (mV)  Poll      Last Update               
---------------------------------------------------------------------
PSOC-MB2_20: VO   40         1791  1         01/01/12 17:03:03         
PSOC-MB2_21: VO   41         3290  1         01/01/12 17:03:03         
PSOC-MB2_22: VO   42         3293  1         01/01/12 17:03:03         
PSOC-MB2_23: VO   43         3299  1         01/01/12 17:03:03         
PSOC-MB2_24: VO   44         4958  1         01/01/12 17:03:03         
PSOC-MB2_25: VO   45         4508  1         01/01/12 17:03:03         
PSOC-MB3_0: VOU   46         4999  1         01/01/12 17:03:03         
PSOC-MB3_1: VOU   47         4982  1         01/01/12 17:03:03         
PSOC-MB3_2: VOU   48         1499  1         01/01/12 17:03:03         
PSOC-MB3_3: VOU   49         1193  1         01/01/12 17:03:03         
PSOC-MB3_4: VOU   50          708  1         01/01/12 17:03:03         
PSOC-MB3_5: VOU   51          757  1         01/01/12 17:03:03         
PSOC-MB3_6: VOU   52          585  1         01/01/12 17:03:03         
PSOC-MB3_7: VOU   53         1501  1         01/01/12 17:03:03         


Router#show logging onboard slot R1 temperature 
Name              Id     Data (C)  Poll      Last Update               
---------------------------------------------------------------------
Temp: BB_DIE      159          25  1         01/02/12 23:04:19         
Temp: VP_DIE      160          21  1         01/02/12 23:04:19         
Temp: RT-E_DIE    161          29  1         01/02/12 23:04:19         
Temp: INLET_1     162          20  1         01/02/12 23:04:19         
Temp: INLET_2     163          18  1         01/02/12 23:04:19         
Temp: OUTLET_1    164          22  1         01/02/12 23:04:19         
Temp: 3882_1      165          44  1         01/02/12 23:04:19         
Temp: 3882_1A     166          38  1         01/02/12 23:04:19         
Temp: 3882_1B     167          36  1         01/02/12 23:04:19         
Temp: 3882_2      168          38  1         01/02/12 23:04:19         
Temp: 3882_2A     169          37  1         01/02/12 23:04:19         
Temp: 3882_2B     170          35  1         01/02/12 23:04:19         
Temp: 3882_3      171          38  1         01/02/12 23:04:19         


Router#show logging onboard slot R1 uptime latest 
 Slot              Reset reason  Power On                  
---------------------------------------------------------
    1      reset local software  01/02/12 23:02:46         



Router#show logging onboard slot R1 uptime 
 Slot              Reset reason  Power On                  
---------------------------------------------------------
    0      reset local software  01/06/12 01:52:26         
    4      reset local software  01/06/12 01:52:42         
    0      reset local software  01/06/12 01:52:45         
    0      reset local software  01/06/12 02:20:27         
    4      reset local software  01/06/12 02:20:43         
    0      reset local software  01/06/12 02:20:46         
    0      reset local software  01/06/12 05:12:02         
    4      reset local software  01/06/12 05:12:19         
    0      reset local software  01/06/12 05:12:22         
    0      reset local software  01/06/12 05:17:31         
    4      reset local software  01/06/12 05:17:48         
    0      reset local software  01/06/12 05:17:51         
    0            reset power on  01/01/12 08:56:44         
    4            reset power on  01/01/12 08:57:00       

  

Router# show logging onboard slot R1 uptime detail
-----------------------------------------------
UPTIME SUMMARY INFORMATION
-----------------------------------------------
First customer power on: 01/15/18 17:33:12
Number of resets: 6
Number of slot changes: 0
Last reset reason: power reset from RP
Current slot: 2
Current power on: 01/17/18 16:14:59
	
-----------------------------------------------
UPTIME CONTINUOUS INFORMATION
-----------------------------------------------
 Slot              Reset reason  Power On               Up: Years   Days  Hours   Mins  
--------------------------------------------------------------------------------------
    2       power reset from RP  01/15/18 17:33:12              0      0      0      0  
    2       power reset from RP  01/16/18 11:44:28              0      0     18      0  
    2       power reset from RP  01/16/18 12:13:19              0      0      0     15  
    2       power reset from RP  01/16/18 17:12:43              0      0      4      0  
    2       power reset from RP  01/17/18 14:34:36              0      0     21      0  
    2       power reset from RP  01/17/18 16:14:59              0      0      1      0  



Router#show logging onboard bay 4/3 message 
timestamp            module    sev  message                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
01/02/12 08:14:22    RFSW-PIC  6    CAT1836E07Q:7.13:Initialize:3/1                                                                                                   
01/02/12 08:20:42    RFSW-PIC  6    CAT1836E07Q:7.13:Initialize:3/1                                                                                                   
01/02/12 09:13:23    RFSW-PIC  6    CAT1836E07Q:7.13:Initialize:3/1                                                                                                   
01/02/12 09:42:33    RFSW-PIC  6    CAT1836E07Q:7.13:Initialize:3/1                                                                                                   
01/02/12 11:56:09    RFSW-PIC  6    CAT1836E07Q:7.13:Initialize:3/1                                                                                                   
01/02/12 12:27:23    RFSW-PIC  6    CAT1836E07Q:7.13:Initialize:3/1


Router#show logging onboard bay 5/3 message 
timestamp            module    sev  message                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
01/22/15 01:06:05    RFSW-PIC  6    JAB092709EL:7.35:Init--stby:3/1                                                                                                   
01/22/15 01:19:01    RFSW-PIC  6    JAB092709EL:7.35:Init--stby:3/1                                                                                                   
01/22/15 01:31:47    RFSW-PIC  6    JAB092709EL:7.35:Init--stby:3/1                                                                                                   
01/22/15 01:44:38    RFSW-PIC  6    JAB092709EL:7.35:Init--stby:3/1                                                                                                   
01/22/15 01:59:04    RFSW-PIC  6    JAB092709EL:7.35:Init--stby:3/1                                                                                                   
01/22/15 02:12:07    RFSW-PIC  6    JAB092709EL:7.35:Init--stby:3/1                                                                                             


Router#show logging onboard bay 4/4 message 
timestamp            module    sev  message                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
01/01/12 10:01:44    SUP-PIC   0    TEST1122334:0.130:PLL-LOS:1[04]                                                                                                   
01/01/12 10:01:45    SUP-PIC   0    TEST1122334:0.130:PLL-LOS:2[04]                                                                                                   
01/01/12 10:01:46    SUP-PIC   0    TEST1122334:0.130:PLL-LOS:3[04]                                                                                                   
01/01/12 10:01:49    SUP-PIC   0    TEST1122334:0.130:PLL-LOS:4[04]                                                                                                   
01/01/12 10:01:50    SUP-PIC   0    TEST1122334:0.130:PLL-LOS:5[04]                                                                                                   
01/01/12 10:01:51    SUP-PIC   0    TEST1122334:0.130:PLL-LOS:6[04]  


Router#show logging onboard bay 5/5 message 
timestamp            module    sev  message                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
01/03/12 13:52:55    SUP-PIC   0    TEST8877665:0.130:PLL-LOS:1[04]                                                                                                   
01/03/12 13:52:56    SUP-PIC   0    TEST8877665:0.130:PLL-LOS:2[04]                                                                                                   
01/03/12 13:52:57    SUP-PIC   0    TEST8877665:0.130:PLL-LOS:3[04]                                                                                                   
01/03/12 13:53:00    SUP-PIC   0    TEST8877665:0.130:PLL-LOS:4[04]                                                                                                   
01/03/12 13:53:01    SUP-PIC   0    TEST8877665:0.130:PLL-LOS:5[04]                                                                                             

Feature Information for Onboard Failure Logging

Use Cisco Feature Navigator to find information about the platform support and software image support. Cisco Feature Navigator enables you to determine which software images support a specific software release, feature set, or platform. To access Cisco Feature Navigator, go to the www.cisco.com/go/cfn link. An account on the Cisco.com page is not required.


Note


The following table lists the software release in which a given feature is introduced. Unless noted otherwise, subsequent releases of that software release train also support that feature.


Table 2. Feature Information for Onboard Failure Logging

Feature Name

Releases

Feature Information

Onboard Failure Logging

Cisco IOS XE Everest 16.6.1

This feature was integrated into Cisco IOS XE Everest 16.6.1 on theCisco cBR Series Converged Broadband Routers.