Introduction
This document describes problem and provides solution regards to SMF system sync fail.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
Session Management Function (SMF) fails to start the service, when issue occurs there is raised alert on Common Execution Environment (CEE).
Problem
The SMF-RCDN continues to cycle through Ops Center System Upgrade and then fail.
On CEE you see this alert:
[smf-rcdn/cee-rcdn] cee# show alerts active summary | inc ops
ops-system-sync-runni 687ca7b9266c minor 09-07T17:59:36 smf-rcdn-mas ops center system upgrade for smf-rcdn is in progress
ops-latest-sync-faile 31531915bf54 major 09-07T10:52:26 smf-rcdn-mas ops center latest system sync for smf-rcdn failed
On SMF you see this error:
[smf-rcdn/smf-rcdn] smf#
Message from confd-api-manager at 2022-09-07 17:49:32...
Helm update is STARTING. Trigger for update is STARTUP.
[smf-rcdn/smf-rcdn] smf#
Message from confd-api-manager at 2022-09-07 17:49:51...
Helm update is ERROR. Trigger for update is STARTUP. Message is:
InterruptedException: one or multiple helm chart installations failed
javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
at com.broadhop.confd.config.proxy.dao.HelmRepositoryDAO.sendConfiguration(HelmRepositoryDAO.java:272)
at com.broadhop.confd.config.proxy.service.ConfigurationSynchManager.run(ConfigurationSynchManager.java:233)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException: one or multiple helm chart installations failed
at com.broadhop.confd.config.proxy.dao.HelmRepositoryDAO.sendConfiguration(HelmRepositoryDAO.java:266)
... 8 more
Analysis
To troubleshoot this, you have to look inside logs from SMF ops center pod.
In this scenario, smf-rcdn did not start the smf related pods.
cloud-user@smf-rcdn-master-1:~$ kubectl get pods -n smf-rcdn
NAME READY STATUS RESTARTS AGE
documentation-69768456cb-klq8d 1/1 Running 0 102d
ops-center-smf-rcdn-ops-center-85899d6b90-9kx6h 5/5 Running 1 40m
smart-agent-smf-rcdn-ops-center-6b9cd64f85-8f8cz 1/1 Running 0 22h
cloud-user@smf-rcdn-master-1:~$
Note down the name of the ops center pod and collect logs for the container confd-api-bridge.
cloud-user@smf-rcdn-master-1:~$ kubectl logs ops-center-smf-rcdn-ops-center-85899d6b90-9kx6h -n smf-rcdn -c confd-api-bridge
Preparing upgrade logic for helm ...
Inside the logs is the reason why system failed to start. In this example, the issue occurred due to sgw-service configuration. The profile does not have configured interfaces.
WARN [2022-09-13 19:44:55,860] com.broadhop.confd.config.proxy.dao.helm.ReleaseInstallCallable: [436] Install or upgrade failure for chart: sgw-service,
release-name: smf-rcdn-sgw-service, command: [/usr/local/bin/helm, upgrade, smf-rcdn-sgw-service, /tmp/chart1014799367411807494.tgz,
--install, -f, /tmp/override1205042274924409625.yaml, -f, /tmp/values4318819924777544020.yaml, --namespace, smf-rcdn, --dry-run]
WARN [2022-09-13 19:44:55,860] com.broadhop.confd.config.proxy.dao.helm.ReleaseInstallCallable: Command result:
Release "smf-rcdn-sgw-service" does not exist. Installing it now.
Error: template: sgw-service/templates/sgw-service.yaml:14:30: executing "sgw-service/templates/sgw-service.yaml" at
<$endpoint.service.nodeCount>: nil pointer evaluating interface {}.nodeCount
INFO [2022-09-13 19:44:55,860] com.broadhop.confd.config.proxy.dao.helm.ReleaseInstallCallable: Command result:
Release "smf-rcdn-udp-proxy" does not exist. Installing it now.
NAME: smf-rcdn-udp-proxy
LAST DEPLOYED: Tue Sep 13 19:44:55 2022
NAMESPACE: smf-rcdn
STATUS: pending-install
REVISION: 1
TEST SUITE: None
HOOKS:
MANIFEST:
On SMF, check show running-configuration. This configuration contains the profile for sgw-service but mandatory parameters are not defined.
profile smf smfprof
mode offline
locality LOC1
allowed-nssai [ slice1 ]
instances 1 fqdn xxx
instances 2 fqdn xxx
plmn-list mcc 123 mnc 456
exit
service name nsmf-pdu
type pdu-session
schema http
version 1.0.2
http-endpoint base-url http://smf-service
icmpv6-profile icmpprf1
compliance-profile June19
priority 20
access-profile idft
subscriber-policy polsub
exit
exit
profile sgw cn-sgw
exit
Solution
The solution is to remove the configuration mistake.
Related Information
Feedback