Introduction
This document describes how to troubleshoot a Cisco Fabric Services (CFS) lock on a Nexus 5000 Series Switch.
Background Information
CFS provides a common infrastructure for automatic configuration synchronization in the fabric. It provides the transport function as well as a rich set of common services to the applications. CFS can discover CFS-capable switches in the fabric as well as their application capabilities. Some of the applications that can be synchronized using CFS on a Nexus 5000 switch include:
- arp
- callhome
- device-alias
- dhcp_snoop
- dpvm
- eth_port_sec
- fc-port-security
- fcdomain
- fctimer
- fscm
- fwm
- icmpv6
- igmp
- mcectest
- msp
- ntp
- rscn
- session-mgr
- stp
- syslogd
- tapp
- vem_mgr
- vim
- vms
- vpc
When you configure an application that uses the CFS infrastructure, that feature starts a CFS session and locks the fabric. When a fabric is locked, the Nexus software does not allow any configuration changes from a switch, other than the switch that holds the lock. The Nexus software also issues an error message that states "Operation failed. Fabric is already locked".
If you start a CFS session that requires a fabric lock but forget to end the session, an administrator can clear the session. If you lock a fabric at any time, your user name is remembered across restarts and switchovers. If another user (on the same machine) tries to perform configuration tasks, the attempts of that user are rejected and a "session currently owned by a different user" error message appears.
Problem
A user is unable to make any configuration related change for the corresponding application, for which a CFS lock is stuck or is unable to perform an In-Service Software Upgrade (ISSU) if the CFS is locked for session-mgr.
This list shows some common error messages caused by a CFS lock:
- Operation failed. Fabric is already locked
- Session currently owned by a different user
- Service "cfs" returned error: Operation failed. Fabric is already locked (0x40B30029)
Solution
There are two methods you can use in order to clear a CFS lock:
- Enter the clear <application> session command.
- Identify the application SAP-ID and unlock the fabric for the application with the hidden command cfs internal unlock <sap-id>. Sap-ID is the uniquely assigned numerical ID for each process.
This procedure includes both methods:
- Validate if CFS is locked and identify the affected application.
This example output shows that CFS is currently locked for Virtual Port Channel (VPC):
cisco-N5k# show cfs lock
Application: vpc
Scope : Physical-eth
--------------------------------------------------------------
Switch WWN IP Address User Name User Type
--------------------------------------------------------------
20:00:00:2a:6a:6d:03:c0 0.0.0.0 CLI/SNMP v3
Total number of entries = 1
Cisco-N5k# show cfs lock name vpc
Scope : Physical-eth
--------------------------------------------------------------
Switch WWN IP Address User Name User Type
--------------------------------------------------------------
20:00:00:2a:6a:6d:03:c0 0.0.0.0 CLI/SNMP v3
Total number of entries = 1
cisco-N5k#
cisco-N5k# show system internal csm info trace
Thu Feb 19 13:20:40.856718 csm_get_locked_ssn_ctxt[515]: Lock not yet taken.
Thu Feb 19 11:21:11.106929 Unlocking DB, Lock Owner Details:Client:2 ID:-1
Thu Feb 19 11:21:11.104247 DB Lock Successful by Client:2 ID:-1
Mon Feb 16 20:45:16.320494 csm_get_locked_ssn_ctxt[515]: Lock not yet taken.
Mon Feb 16 20:45:14.223875 csm_get_locked_ssn_ctxt[515]: Lock not yet taken.
Mon Feb 16 20:44:59.40095 csm_get_locked_ssn_ctxt[515]: Lock not yet taken.
You can also enter the show cfs application command in order to see the applications that currently use CFS:
cisco-N5k# show cfs application
----------------------------------------------
Application Enabled Scope
----------------------------------------------
arp Yes Physical-eth
fwm Yes Physical-eth
ntp No Physical-fc-ip
stp Yes Physical-eth
vpc Yes Physical-eth
fscm Yes Physical-fc
igmp Yes Physical-eth
role No Physical-fc-ip
rscn No Logical
icmpv6 Yes Physical-eth
radius No Physical-fc-ip
fctimer No Physical-fc
syslogd No Physical-fc-ip
fcdomain No Logical
session-mgr Yes Physical-ip
device-alias Yes Physical-fc
Total number of entries = 16
- Clear the CFS lock. Choose one of the two methods provided in this step:
Method 1: Enter the clear <application> session command in order to clear the lock.
A CFS lock for the NTP application is cleared in this example:
cisco-N5k#clear ntp session
Note: This command is not applicable for all applications. For example, applications that fall under the "Physical-eth" scope such as Address Resolution Protocol (ARP), Forwarding Manager (FWM), Spanning Tree Protocol (STP), VPC, Internet Group Management Protocol (IGMP), and Internet Control Message Protocol (ICMP6). You must use the hidden command in Method 2 in order to unlock the session.
Method 2: Identify the application sap-id and unlock the fabric with hidden command cfs internal unlock <sap-id>.
cisco-N5k# show system internal sysmgr service all
Name UUID PID SAP state Start count Tag Plugin ID
---------------- ------- ---- ----- ----- ---------- ----- ---------
aaa 0x000000B5 3221 111 s0009 1 N/A 0
cert_enroll 0x0000012B 3220 169 s0009 1 N/A 0
Flexlink 0x00000434 [NA] [NA] s0075 None N/A 0
psshelper_gsvc 0x0000021A 3159 398 s0009 1 N/A 0
radius 0x000000B7 3380 113 s0009 1 N/A 0
securityd 0x0000002A 3219 55 s0009 1 N/A 0
tacacs 0x000000B6 [NA] [NA] s0075 None N/A 0
eigrp 0x41000130 [NA] [NA] s0075 None N/A 0
isis_fabricpath0x41000243 3876 436 s0009 1 N/A 0
vpc 0x00000251 3900 450 s0009 1 N/A 0 < <
vsan 0x00000029 3817 15 s0009 1 N/A 2
vshd 0x00000028 3149 37 s0009 1 N/A 0
vtp 0x00000281 3902 478 s0009 1 N/A 0
Identify the sap-id from the output and unlock the fabric as this example shows:
cisco-N5k# cfs internal unlock 450
Application Unlocked
cisco-N5k#
Note: The cfs internal unlock command is a hidden Nexus OS command used to unlock the CFS and is safe to run in production.
- Issue these show commands in order to validate the solution:
cisco-N5k# show cfs lock name vpc
cisco-N5k#
cisco-N5k# show cfs internal session-history name vpc
--------------------------------------------------------
Time Stamp Source WWN Event
User Name Session ID
---------------------------------------------------------
Tue May 26 23:35:51 2015 20:00:00:05:73:d0:c0:00 LOCK_OBTAINED
admin 147513262
Tue May 26 23:53:52 2015 20:00:00:05:73:d0:c0:00 LOCK_CLEAR
admin 147513262
---------------------------------------------------------
Known Issues
These are some of the CFS related known software defects:
- Cisco bug ID CSCtj40756 - ISSU failure -"cfs" returned error:Fabric is already locked (0x40B30029)
- Cisco bug ID CSCue03528 - Session Database / Config Sync / CFS locked on one side without a commit
Feedback