UltraM — 排除StarOS EMCtrl错误“找不到VDU”

简介

本文档介绍在思科超服务平台(UltraM)中的虚拟化数据包核心(VPC)卡因“找不到VDU”错误（如show logs中所示）而停滞启动时的故障排除步骤。

背景信息

Sample:

2017-Sep-26+08:05:05.839 [emctrl 218804 error] [2/0/16829 <emctrl:0> emctrl_vnf.c:828] [software internal system syslog] Failed to find VDU, of card number <1> 

如果进一步检查日志，您会看到非常具体的错误，指出卡类型与分机移动(EM)信息不匹配：

2017-Sep-26+08:03:32.126 [emctrl 218802 info] [2/0/16829 <emctrl:0> emctrl_util.c:381] [software internal system critical-info syslog] siti msg for standby CF, card type doesn't match EM, reboot it
2017-Sep-26+08:03:32.126 [emctrl 218802 info] [2/0/16829 <emctrl:0> emctrl_util.c:376] [software internal system critical-info syslog] siti card 1 card type drvctrl 40010100, siti 0
2017-Sep-26+08:03:32.126 [emctrl 218802 info] [2/0/16829 <emctrl:0> emctrl_util.c:329] [software internal system critical-info syslog] siti sync msg received for card 1 with cardtype 40010100, uuid 9F1F2B1E-35FC-4AF9-807A-E856336702D6
2017-Sep-26+08:03:32.105 [system 1004 info] [2/0/9741 <evlogd:0> evlgd_syslogd.c:279] [software internal system syslog] CPU[2/0]:  sitiserv[9533]: SITI_PRESENT: invoking notify card present cmd notify_card_present 1 0 0x40010100 9F1F2B1E-35FC-4AF9-807A-E856336702D6

检查命令

从错误中可以看出，存在受影响卡的通用唯一标识符(UUID) — 此示例UUID为9F1F2B1E-35FC-4AF9-807A-E856336702D6。

理想情况下，此UUID应与show emctrl vdu detail output命令的输出匹配。

show emctrl vdu detail是隐藏命令。

[local]UltraM-QVPC-DI# show emctrl vdu detail 
Showing emctrl vdu
card[01]: name[CFC_01 ] uuid[1FE70E43-0F33-4E17-8BFA-439169CD52BA]
card[02]: name[CFC_02 ] uuid[3AFC540B-546E-4F35-A645-A23E62C32C59]
card[03]: name[SFC_03 ] uuid[93359FA0-09C2-4F7C-93F6-17BE0A2AF49F]
card[04]: name[SFC_04 ] uuid[E02C8AAA-7E8A-4881-8018-6EC59963C8F6]
card[05]: name[SFC_05 ] uuid[6F297BF6-4AFC-43AB-A36D-FCD0FAE39DA3]

如果此输出为空，则可能是EMCtrl进程已损坏。 

此ID应与EM上显示的相同，如突出显示：

admin@scm# show vdus vdu card-type session-function 
vdus vdu session-function
 card-type session-function
 vnfci BOOT_generic_di-chassis_SF1_1
 constituent-element-group di-chassis
 is-infra true
 initialized false
 vim-id 93359fa0-09c2-4f7c-93f6-17be0a2af49f
 vnfci BOOT_generic_di-chassis_SF2_1
 constituent-element-group di-chassis
 is-infra true
 initialized false
 vim-id e02c8aaa-7e8a-4881-8018-6ec59963c8f6
 vnfci BOOT_generic_di-chassis_SF3_1
 constituent-element-group di-chassis
 is-infra true
 initialized false
 vim-id 54e9a5d6-f4dd-4636-95d3-b29443ebfa14

 使用以下命令可以找到有关StarOS端上此实例的详细信息：

[local]UltraM-QVPC-DI# show vdu detail type session-function instance BOOT_generic_di-chassis_SF1_1
vdu-id: session-function, vdu-instance: BOOT_generic_di-chassis_SF1_1, state: from:Invalid to:Alive
card_number: 3, card_type: 0x42030100, uuid:93359fa0-09c2-4f7c-93f6-17be0a2af49f
 networks:
 cp-id: di_intf1, state: Alive, type: unknown
 vl: vl-di-internal1 vnfc: sf-vnfc-di-chassis
 mac: fa:16:3e:87:ac:e4, ip: 192.168.1.12
 cp-id: di_intf2, state: Alive, type: unknown
 vl: vl-di-internal2 vnfc: sf-vnfc-di-chassis
 mac: fa:16:3e:92:ea:26, ip: 192.168.2.11
 cp-id: orch, state: Alive, type: unknown
 vl: vl-orchestration vnfc: sf-vnfc-di-chassis
 mac: fa:16:3e:1e:f5:b5, ip: 172.16.180.21
 cp-id: svc_intf1, state: Alive, type: unknown
 vl: vl-service-network1 vnfc: sf-vnfc-di-chassis
 mac: fa:16:3e:bf:c8:6f, ip: 10.10.10.2
 cp-id: svc_intf2, state: Alive, type: unknown
 vl: vl-service-network2 vnfc: sf-vnfc-di-chassis
 mac: fa:16:3e:15:a9:22, ip: 20.20.20.7
 cp-id: svc_intf3, state: Alive, type: unknown
 vl: vl-service-network1 vnfc: sf-vnfc-di-chassis
 mac: fa:16:3e:1f:fa:0c, ip: 10.10.10.6
 cp-id: svc_intf4, state: Alive, type: unknown
 vl: vl-service-network2 vnfc: sf-vnfc-di-chassis
 mac: fa:16:3e:2f:6b:00, ip: 20.20.20.10 

不一致情形1：在EMCtrl和EM VDU实例上看到的不同ID

如果您注意卡5的ID，可以看到它是6F297BF6-4AFC-43AB-A36D-FCD0FAE39DA3。

[local]UltraM-QVPC-DI# show emctrl vdu detail 
Showing emctrl vdu
card[01]: name[CFC_01 ] uuid[1FE70E43-0F33-4E17-8BFA-439169CD52BA]
card[02]: name[CFC_02 ] uuid[3AFC540B-546E-4F35-A645-A23E62C32C59]
card[03]: name[SFC_03 ] uuid[93359FA0-09C2-4F7C-93F6-17BE0A2AF49F]
card[04]: name[SFC_04 ] uuid[E02C8AAA-7E8A-4881-8018-6EC59963C8F6]
card[05]: name[SFC_05 ] uuid[6F297BF6-4AFC-43AB-A36D-FCD0FAE39DA3]

然而，如果您在EM上检查了相同的ID，则找不到它：

admin@scm# show vdus | include vim
 vim-id 1fe70e43-0f33-4e17-8bfa-439169cd52ba  ---> CF 1
 vim-id 3afc540b-546e-4f35-a645-a23e62c32c59  ---> CF 2
 vim-id 93359fa0-09c2-4f7c-93f6-17be0a2af49f  ---> SF 3
 vim-id e02c8aaa-7e8a-4881-8018-6ec59963c8f6  ---> SF 4
 vim-id 54e9a5d6-f4dd-4636-95d3-b29443ebfa14  ---> ?

因此您可以看到，对于插槽5中的卡，似乎存在不一致。

当您在StarOS上签入特定ID的更多详细信息时，您现在会看到，使用show vdu detail命令时，ID实际上与EM端显示的ID相同：

[local]UltraM-QVPC-DI# show vdu detail type session-function instance BOOT_generic_di-chassis_SF3_1
vdu-id: session-function, vdu-instance: BOOT_generic_di-chassis_SF3_1, state: from:Invalid to:Alive
card_number: 5, card_type: 0x42030100, uuid:54e9a5d6-f4dd-4636-95d3-b29443ebfa14

这样，您就可以确认EMCtrl过程没有正确的信息。 

如果检查日志，则会看到以下警告：

2017-Sep-26+08:36:31.317 UltraM-QVPC-DI [emctrl 218802 info] [2/0/20871 <emctrl:0> emctrl_util.c:579] [software internal system critical-info syslog] drvctrl uuid mismatch /6F297BF6-4AFC-43AB-A36D-FCD0FAE39DA3 with em uuid 54e9a5d6-f4dd-4636-95d3-b29443ebfa14, use drvctrl uuid

1.如果终止EMCtrl任务，则不会有所帮助。

2.此外，如果重新启动卡，则无济于事。

不一致场景2：显示EMCtrl VDU详细信息为空

这可能是因为EMCtrl表损坏了，根据您目前掌握的知识，它是错误的结果。

show emctrl vdu list的输出将完全为空：

Showing emctrl vdu
card[01]: name[                                                  ] uuid[          ]
card[02]: name[                                                  ] uuid[          ]

要从VNFM代理端检查卡的实际状态，请执行以下操作：

#show vdu detail type control-function instance  BOOT_generic_di-chasis_CF1_1

vdu-id: control-function, vdu-instance: BOOT_generic_di-chasis_CF1_1, state: from:Invalid to:Alive

已知漏洞:CSCvf32599

解决方法：重新启动EMCtrl任务：

task kill facility emctrl all

不一致场景3：卡表中缺少CF，EM中不存在

有时，您会看到卡表中缺少SF或CF。

从输出中您可以看到，StarOS只看到一个CF卡：

[local]AUPGW101# show card tabl
Wednesday September 27 09:26:46 UTC 2017
Slot Card Type Oper State SPOF Attach
----------- -------------------------------------- ------------- ---- ------
1: CFC Control Function Virtual Card Active Yes 
3: FC 4-Port Service Function Virtual Card Active No 
4: FC 4-Port Service Function Virtual Card Active No 
5: FC 4-Port Service Function Virtual Card Active No 
6: FC 4-Port Service Function Virtual Card Active No 
7: FC 4-Port Service Function Virtual Card Active No 
8: FC 4-Port Service Function Virtual Card Active No 
9: FC 4-Port Service Function Virtual Card Active No 
10: FC 4-Port Service Function Virtual Card Standby -

但是，如果您检查卡2的调试控制台，您会看到它尝试联机：

[local]AUPGW101# debug consol card 1 cpu 0
Wednesday September 27 09:26:58 UTC 2017
[local]AUPGW101# 2017-Sep-27+09:23:18.370 card 1-cpu0: collect persistdump for card <2> success
2017-Sep-27+09:24:22.112 card 1-cpu0: Hatsystem rcvd card 2/0 fail req from card (1) emctrl/0 - 32:150:3
2017-Sep-27+09:24:22.115 card 1-cpu0: The Control Function Virtual Card with serial number in slot 2 has failed and will be brought down and brought back online. (Device=CARD, Reason=EMCTRL_CARDTYPE_MISMATCH, Status=0)

正如您在show log中看到的，EMCtrl认为CF在EM中不存在：

2017-Sep-27+09:27:13.964 [emctrl 218802 info] [1/0/7805 <emctrl:0> emctrl_util.c:357] [software internal system critical-info syslog] siti msg for standby CF, but doesn't exist in EM, reboot it
2017-Sep-27+09:27:13.964 [emctrl 218802 info] [1/0/7805 <emctrl:0> emctrl_util.c:329] [software internal system critical-info syslog] siti sync msg received for card 2 with cardtype 40010100, uuid C6217904-8F65-4C48-B607-4F13EAE6745D
2017-Sep-27+09:27:13.939 [system 1004 info] [1/0/7684 <evlogd:0> evlgd_syslogd.c:279] [software internal system syslog] CPU[1/0]: sitiserv[3063]: SITI_PRESENT: invoking notify card present cmd notify_card_present 2 0 0x40010100 C6217904-8F65-4C48-B607-4F13EAE6745D

您确实可以确认：

[local]AUPGW101# show emctrl vdu list
Wednesday September 27 09:30:21 UTC 2017
Showing emctrl vdu
card[01]: name[CFC_01 ] uuid[42913D9A-91A9-4E5E-8473-AEADD73BEC08]
card[03]: name[SFC_03 ] uuid[CB2C4429-0965-4394-8200-ABB4071BB067]
card[04]: name[SFC_04 ] uuid[17997C02-DF9F-40BC-8A41-D2B9D448D47C]
card[05]: name[SFC_05 ] uuid[159F91EE-B6A4-4DE6-A8C9-F900CD087093]
card[06]: name[SFC_06 ] uuid[7EE371A9-4E64-477F-AA09-42B6ED70B92B]
card[07]: name[SFC_07 ] uuid[DF2D38F2-01FD-4E95-97EC-4B1EB75683FD]
card[08]: name[SFC_08 ] uuid[E7D7F817-09C6-4EBA-9537-A66A686713A1]
card[09]: name[SFC_09 ] uuid[B24BE6CC-EB7B-483D-A859-284EF638647C]
card[10]: name[SFC_10 ] uuid[2AAD074F-C65C-4708-AAA9-A76588BD434D]

解决方法：重新启动EMCtrl任务。