简介
本文档介绍如何排除和解决与同步策略操作相关的思科策略套件(CPS)错误。
先决条件
要求
Cisco 建议您了解以下主题:
使用的组件
本文档中的信息基于思科策略套件。
本文档中的信息都是基于特定实验室环境中的设备编写的。本文档中使用的所有设备最初均采用原始(默认)配置。如果您的网络处于活动状态,请确保您了解所有命令的潜在影响。
问题:处理同步策略操作时出错
在/var/log/broadhop/consolidated-qns.log日志文件中观察到的错误:
ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Data store is not available: No master DB available: sessionmgrxx-SITE1:27720
ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Duplicate Value for Unique Data Constraint: credential networkId: 61431792343
示例显示2个不同站点和2个不同时间戳中的错误:
由于误差的影响,直径响应时间会略有增加,在grafana上也可以观察到尖峰。
SITE-1整合队列ns日志:
SITE1-qns11 2016-10-13 09:44:59,343 [pool-2-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Data store is not available: No master DB available: sessionmgrxx-SITE1:27720
at com.broadhop.spr.dao.impl.mongo.MongoPersistence.searchSubscribers(MongoPersistence.java:433) ~[com.broadhop.spr.dao.mongo_2.3.6.r078013.jar:na]
SITE1-qns04 2016-10-13 10:42:03,629 [pool-2-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
SITE1-qns06 2016-10-13 10:47:20,555 [pool-3-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
SITE-1 Mongo日志:
2016-10-13T08:22:02.330+1100 [conn15604045] command spr.$cmd command: count { count: "subscriber", query: { services_key.code_key: "HWBB" }, fields: {} } planSummary: COLLSCAN keyUpdates:0 numYields:2 locks(micros) r:1297276 reslen:48 700ms
2016-10-13T08:47:02.909+1100 [conn15605855] command spr.$cmd command: count { count: "subscriber", query: { services_key.code_key: "HWBB" }, fields: {} } planSummary: COLLSCAN keyUpdates:0 numYields:1 locks(micros) r:1095391 reslen:48 585ms
2016-10-13T09:37:02.563+1100 [conn15609536] command spr.$cmd command: count { count: "subscriber", query: { services_key.code_key: "HWBB" }, fields: {} } planSummary: COLLSCAN keyUpdates:0 numYields:0 locks(micros) r:519701 reslen:48 519ms
2016-10-13T09:52:02.472+1100 [conn15610667] command spr.$cmd command: count { count: "subscriber", query: { services_key.code_key: "HWBB" }, fields: {} } planSummary: COLLSCAN keyUpdates:0 numYields:1 locks(micros) r:1072306 reslen:48 564ms
2016-10-13T10:27:02.649+1100 [conn15613701] command spr.$cmd command: count { count: "subscriber", query: { services_key.code_key: "HWBB" }, fields: {} } planSummary: COLLSCAN keyUpdates:0 numYields:1 locks(micros) r:657876 reslen:48 504ms
SITE-2 consolidated-qns日志:
SITE2-qns04 2016-11-18 05:28:33,776 [pool-1361732-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Duplicate Value for Unique Data Constraint: credential networkId: 61431792343
SITE2-qns09 2016-11-17 20:47:59,539 [pool-1361592-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Duplicate Value for Unique Data Constraint: credential networkId: 61431792343
SITE2-qns06 2016-11-17 01:45:15,476 [pool-63652-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Data store is not available: No master DB available: sessionmgr11-SITE1:27720
SITE2-qns12 2016-11-17 01:45:15,514 [pool-63484-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Data store is not available: No master DB available: sessionmgr11-SITE1:27720
SITE2-qns10 2016-11-12 07:40:53,659 [pool-63538-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Duplicate Value for Unique Data Constraint: credential networkId: 61431792343
“处理同步策略操作时出错”可能由以下两个原因引起:
1.主/主MongoDB无法访问或关闭。
2.由于网络问题,策略服务器(QNS)实例无法访问会话管理器虚拟机(VM)。
日志表明这与网络波动有关,因为在大多数QNS VM上都可以观察同步策略操作的会话管理器VM访问异常。这些例外是无法访问的相关主SPR数据库(sessionmgr11-SITE1:27720)。在相同持续时间内,MongoDB日志中未发现任何错误。此外,在同一时间段内未观察到主数据库到辅助数据库的故障切换。
解决方案
SITE1和SITE2上发生“同步策略操作”错误,原因有二:
1.
SITE1-qns11 2016-10-13 09:44:59,343 [pool-2-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Data store is not available: No master DB available: sessionmgrxx-SITE1:27720
"数据存储不可用"。 这显然类似于通过快速接续解决的轻微网络波动:
2.
SITE2-qns04 2016-11-18 05:28:33,776 [pool-1361732-thread-1] ERROR c.b.policy.impl.RulesPolicyService - An error occured while processing a Sync Policy Action.
com.broadhop.spr.exception.SubscriberException: Duplicate Value for Unique Data Constraint: credential networkId: 61431792343
“唯一数据约束的重复值”对系统或用户没有影响,因为当从PCEF收到重复请求时,会观察到此异常。 最初的请求已得到满足。
这些是当consolidated-qns.log中观察到“Data store is not available”时思科建议执行的步骤。
步骤1.检查QNS VM和sessionmgr11-SITE1VM之间的网络路径或任何路由器。
步骤2.检查来自QNS VM和sessionmgr11-SITE1VM的/var/log/messages。
步骤3.检查QNS和sessionmgr11-SITE1 VM所连接的路由器
为了确认在sessionmgrxx-xxx:277xx上尝试连接数据库的任何实例时出现“datastore不可用”异常,您可以在发现网络问题的sessionmgrxx VM上运行此命令的整个持续时间(可能为24小时)。 输出将存储在/var/tmp/output.txt。
while true; do netstat -apn | grep 27720; sleep 1 ; done | tee /var/tmp/output.txt
要停止netstat命令的运行,只需使用CTRL C将其停止:
ctrl c
注意:如果观察到在sessionmgrxx-xxx上连接和重置TCP连接时出现问题。然后,我们需要通过删除数据并从辅助节点重新创建数据库来修复数据库。 (要发布的修复/恢复数据库的链接 — )