Cisco UCS Faults and Error Messages Reference

In the Cisco UCS, a fault is a mutable object that is managed by the Cisco UCS Manager. Each fault represents a failure in the Cisco UCS instance or an alarm threshold that has been raised. During the lifecycle of a fault, it can change from one state or severity to another.

Each fault includes information about the operational state of the affected object at the time the fault was raised. If the fault is transitional and the failure is resolved, then the object transitions to a functional state.

A fault remains in the Cisco UCS Manager until the fault is cleared and deleted according to the settings in the fault collection policy.

You can view all faults in the Cisco UCS instance from either the Cisco UCS Manager CLI or the Cisco UCS Manager GUI. You can also configure the fault collection policy to determine how a Cisco UCS instance collects and retains faults.

Note All Cisco UCS faults can be trapped by SNMP.

Fault Severities

A fault raised in a Cisco UCS instance can transition through more than one severity during its lifecycle. Table 1-1 describes the possible fault severities in alphabetical order.

Table 1-1 Fault Severities in Cisco UCS
Severity	Description
Cleared	A notification that the condition that caused the fault has been resolved, and the fault has been cleared.
Condition	An informational message about a condition, possibly independently insignificant.
Critical	A service-affecting condition that requires immediate corrective action. For example, this severity could indicate that the managed object is out of service and its capability must be restored.
Info	A basic notification or informational message, possibly independently insignificant.
Major	A service-affecting condition that requires urgent corrective action. For example, this severity could indicate a severe degradation in the capability of the managed object and that its full capability must be restored.
Minor	A non-service-affecting fault condition that requires corrective action to prevent a more serious fault from occurring. For example, this severity could indicate that the detected alarm condition is not currently degrading the capacity of the managed object.
Warning	A potential or impending service-affecting fault that currently has no significant effects in the system. Action should be taken to further diagnose, if necessary, and correct the problem to prevent it from becoming a more serious service-affecting fault.

Fault Types

A fault raised in a Cisco UCS instance can be one of the types described in Table 1-2.

Table 1-2 Types of Faults in Cisco UCS
Type	Description
fsm	An FSM task has failed to complete successfully, or the Cisco UCS Manager is retrying one of the stages of the FSM.
equipment	The Cisco UCS Manager has detected that a physical component is inoperable or has another functional issue.
server	The Cisco UCS Manager is unable to complete a server task, such as associating a service profile with a server.
configuration	The Cisco UCS Manager is unable to successfully configure a component.
environment	The Cisco UCS Manager has detected a power problem, thermal problem, voltage problem, or a loss of CMOS settings.
management	The Cisco UCS Manager has detected a serious management issue, such as one of the following: •Critical services could not be started. •The primary switch could not be identified. •Components in the instance include incompatible firmware versions.
connectivity	The Cisco UCS Manager has detected a connectivity problem, such as an unreachable adapter.
network	The Cisco UCS Manager has detected a network issue, such as a link down.
operational	Cisco UCS Manager has detected an operational problem, such as a log capacity issue or a failed server discovery.

Properties of Faults

The Cisco UCS Manager provides detailed information about each fault raised in a Cisco UCS instance. Table 1-3 describes the fault properties that can be viewed in the Cisco UCS Manager CLI or the Cisco UCS Manager GUI.

Table 1-3 Fault Properties
Property Name	Description
Severity	The current severity level of the fault. This can be any of the severities described in Table 1-1.
Last Transition	The day and time on which the severity for the fault last changed. If the severity has not changed since the fault was raised, this property displays the original creation date.
Affected Object	The component that is affected by the condition that raised the fault.
Description	The description of the fault.
ID	The unique identifier assigned to the fault.
Status	Additional information about the fault state. This can be any of the states described in Table 1-4.
Type	The type of fault that has been raised. This can be any of the types described in Table 1-2.
Cause	The unique identifier associated with the condition that caused the fault.
Created at	The day and time when the fault occurred.
Code	The unique identifier assigned to the fault.
Number of Occurrences	The number of times the event that raised the fault occurred.
Original Severity	The severity assigned to the fault on the first time that it occurred.
Previous Severity	If the severity has changed, this is the previous severity.
Highest Severity	The highest severity encountered for this issue.

Lifecycle of Faults

The faults in Cisco UCS are stateful, and a fault raised in a Cisco UCS instance transitions through more than one state during its lifecycle. In addition, only one instance of a given fault can exist on each object. If the same fault occurs a second time, the Cisco UCS increases the number of occurrences by one.

A fault has the following lifecycle:

1. A condition occurs in the system and the Cisco UCS raises a fault in the active state.

2. If the fault is alleviated within a short period of time know as the flap interval, the fault severity remains at its original active value but the fault enters the soaking state. The soaking state indicates that the condition that raised the fault has cleared, but the system is waiting to see whether the fault condition reoccurs.

3. If the condition reoccurs during the flap interval, the fault enters the flapping state. Flapping occurs when a fault is raised and cleared several times in rapid succession. If the condition does not reoccur during the flap interval, the fault is cleared.

4. Once cleared, the fault enters the retention interval. This interval ensures that the fault reaches the attention of an administrator even if the condition that caused the fault has been alleviated, and that the fault is not deleted prematurely. The retention interval retains the cleared fault for the length of time specified in the fault collection policy.

5. If the condition reoccurs during the retention interval, the fault returns to the active state. If the condition does not reoccur, the fault is deleted.

When a fault is active, the additional lifecycle state information listed in Table 1-4 may be provided in the Status field of the fault notification.

Table 1-4 Fault Lifecycle States
State	Description
Soaking	A fault was raised and then cleared within a short time known as the flap interval. Since this may be a flapping condition, the fault severity remains at its original active value, but this state indicates that the condition that raised the fault has cleared. If the fault does not reoccur, the fault moves into the cleared state. Otherwise, the fault moves into the flapping state.
Flapping	A fault was raised, cleared, and then raised again within a short time known as the flap interval.

Fault Collection Policy

The fault collection policy controls the lifecycle of a fault in the Cisco UCS instance, including the length of time that each fault remains in the flapping and retention intervals.

Tip For information on how to configure the fault collection policy, see the Cisco UCS configuration guides, accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

Faults in Cisco UCS Manager

This section includes the following topics:

•Faults in Cisco UCS Manager GUI

•Faults in Cisco UCS Manager CLI

Faults in Cisco UCS Manager GUI

If you want to view the faults for a single object in the system, navigate to that object in the Cisco UCS Manager GUI and then click the Faults tab in the Work pane. If you want to view the faults for all objects in the system, navigate to the Faults node on the Admin tab under the Faults, Events and Audit Log.

In addition, a summary of all faults can be viewed in a Cisco UCS instance. Go to the Fault Summary area in the upper left of the Cisco UCS Manager GUI. This area provides a summary of all faults that have occurred in the Cisco UCS instance.

Each fault severity is represented by a different icon. The number below each icon indicates how many faults of that severity have occurred in the system. When you click an icon, the Cisco UCS Manager GUI opens the Faults tab in the Work pane and displays the details of all faults with that severity.

Faults in Cisco UCS Manager CLI

If you want to view the faults for all objects in the system, at the top-level scope, enter the show fault command. If you want to view faults for a specific object, scope to that object and then enter the show fault command.

If you want to view all of the available details about a fault, enter the show fault detail command.

Overview of the Finite State Machine

This section includes the following topics:

•About the Finite State Machine in Cisco UCS

•FSM Stage Names

•FSM in Cisco UCS Manager

About the Finite State Machine in Cisco UCS

A finite state machine (FSM) is a workflow model, similar to a flow chart, that is composed of the following:

•Finite number of stages (states)

•Transitions between those stages

•Operations

The current stage in the FSM is determined by past stages and the operations performed to transition between the stages. A transition from one stage to another stage is dependent on the success or failure of an operation.

Cisco UCS Manager uses FSM tasks that run in the Data Management Engine (DME) to manage end points in the UCS object model, including the following:

•Physical components (chassis, I/O module, servers)

•Logical components (LAN cloud, policies)

•Workflows (server discovery, service profile management, downloads, upgrades, backups)

The DME manages the FSM stages and transition, and instructs the Application Gateway (AG) to perform operations on the managed end points. Therefore, each stage can be considered to be an interaction between the DME, the AG, and the managed end point. The AGs do the real work of interacting with managed end points, such as the CIMC, the adapter, or the I/O module

When all of the FSM stages have run successfully, the Cisco UCS considers that the FSM operation is successful.

If the FSM encounters an error or a timeout at a stage, the FSM retries that stage at scheduled intervals. When the retry count has been reached for that stage, the FSM stops and the Cisco UCS Manager declares that the change have failed. If an FSM task fails, the Cisco UCS Manager raises the appropriate faults and alarms.

The mMultiple FSM tasks can be associated to an end point. However, only one FSM task at a time can run. Additional FSM tasks for the same end point are placed in a queue and are scheduled to be run when the previous FSM task is either successfully completed or the task fails.

You can view the FSM details for a particular end point to determine if a task succeeded or failed. You can also use the FSM to troubleshoot any failures.

FSM Stage Names

The FSM stage names are constructed using the following notation

FsmObjectWorkflowOperationWhere-is-it-executed

where:

•Object is the object that the FSM is running, such as the Blade or Chassis.

•Workflow is the overall task being performed by the FSM, such as Discover or Association.

•Operation is the task being performed at a particular stage, such as Pnuos-Config.

•Where-is-it-executed is generally "", or "A" or "B" or "Local" or "Peer". If this is not specified, it is executed on the managingInst node.

Each FSM stage name has a prefix that identifies the FSM and a suffix that identifies a stage within the FSM. The prefix notation is FsmObjectWorkflow and the suffix notation is OperationWhere-is-it-executed. For example, if the FSM name is FsmComputeBladeDiscoverBmcInventory:

•The prefix is FsmComputeBladeDiscover

•The suffix is BmcInventory

FSM in Cisco UCS Manager

This section includes the following topics:

•FSM in the Cisco UCS Manager GUI

•FSM in the Cisco UCS Manager CLI

FSM in the Cisco UCS Manager GUI

The Cisco UCS Manager GUI displays the FSM information for an end point on the FSM tab for that end point. You can use the FSM tab to monitor the progress and status of the current FSM task and view a list of the pending FSM tasks.

The information about a current FSM task in the Cisco UCS Manager GUI is dynamic and changes as the task progresses. You can view the following information about the current FSM task:

•FSM task being executed

•Current state of that task

•Time and status of the previously completed task

•Any remote invocation error codes returned while processing the task

•Progress of the current task

To view the FSM task for an end point that supports FSM, navigate to the end point in the Navigation pane and click on the FSM tab in the Work pane.

FSM in the Cisco UCS Manager CLI

The Cisco UCS Manager CLI can display the FSM information for an end point when you are in the command mode for that end point.

Enter the show fsm status command in the appropriate mode to view the current FSM task for an end point. The information displayed about a current FSM task in the CLI is static. You must re-enter the command to see the progress updates. The following example displays the information about the current FSM task for the server in chassis 1, slot 6:

UCS-A# scope server 1/6

UCS-A /chassis/server # show fsm status

Slot: 6

Server: sys/chassis-1/blade-6

    FSM 1:

        Remote Result: Not Applicable

        Remote Error Code: None

        Remote Error Description:

        Status: Discover Blade Boot Wait

        Previous Status: Discover Blade Boot Wait

        Timestamp: 2006-01-26T23:31:36

        Try: 0

        Flags: 0

        Progress (%): 33

        Current Task: Waiting for system reset on server 1/6 
(FSM-STAGE:sam:dme:ComputeBladeDiscover:BladeBootWait)

Enter the show fsm task command in the appropriate mode to view all of the pending tasks in the FSM queue. The following example displays the FSM task queue for the server in chassis 1, slot 6:

UCS-A# scope server 1/6

UCS-A /chassis/server # show fsm task

FSM Task:

    Item             ID       Completion   FSM Flags

    ---------------- -------- ------------ ---------

    Powercycle        1154858 Scheduled

    BiosRecovery      1154860 Scheduled

Cisco Error Message Decoder

The Error Message Decoder parses published System Message Guides and provides easy error message search capabilities for Cisco customers.

To use the Error Message Decoder:

Step 1 Click the following link: http://www.cisco.com/cgi-bin/Support/Errordecoder/index.cgi

Step 2 Log in.

Step 3 Paste the error message into the field indicated.

Step 4 Click Submit.

Bias-Free Language

Book Title

Cisco UCS Faults and Error Messages Reference

Chapter Title

Introduction

Results

Chapter: Introduction

Introduction to UCS Faults

Overview of Faults

About Faults in the Cisco UCS

Fault Severities

Fault Types

Properties of Faults

Lifecycle of Faults

Fault Collection Policy

Faults in Cisco UCS Manager

Faults in Cisco UCS Manager GUI

Faults in Cisco UCS Manager CLI

Overview of the Finite State Machine

About the Finite State Machine in Cisco UCS

FSM Stage Names

FSM in Cisco UCS Manager

FSM in the Cisco UCS Manager GUI

FSM in the Cisco UCS Manager CLI

Cisco Error Message Decoder

Was this Document Helpful?

Contact Cisco