Troubleshoot SMF PODs that Does not Come Up after the Day 1 Configuration

Available Languages

Download Options

  • PDF     (78.7 KB)
    View with Adobe Reader on a variety of devices
  • ePub     (86.3 KB)
    View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
  • Mobi (Kindle)     (74.1 KB)
    View on Kindle device or Kindle app on multiple devices
Updated:July 15, 2021
Document ID:215787

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

    Introduction

    This document describes the issue of SMF NF PODs that does not come up after the Day 1 configuration is loaded in SMF ops-center.

    Prerequisite

    Requirements

    Cisco recommends that you have knowledge of these topics:

    • Subscriber Microservices Infrastructure (SMI)
    • Docker
    • Kubernetes
    • 5G

    Components Used

    The information in this document is based on these software and hardware versions:

    • SMI
    • Ops-Center
    • SMF

    The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

    Problem 

    In the customer setup, they have two SMF NF that runs with the same version. Both these SMF NFs were upgraded to the latest version last night. Before the upgrade, both the NF were having PODs in running state. The issue is seen only with one SMF i.e. SMF-IMS. The other POD SMF-DATA is upgraded and has all PODs in running state.

    • SMF version before upgrade: smf.2020.01.0-12
    • SMF version after upgrade: smf.2020.01.0-18

    Abbreviations

    SMF Session Management Function
    NF Network Function
    CEE Common Execution Environment 
    POD It is the smallest possible unit in the Kubernetes environment i.e. at least one container.
    IMS IP Multimedia Subsystem
    SMI Subscriber Microservices Infrastructure 

    Observations

    • Cluster Sync shows Deployment Successful.
    • Kubernetes Master shows the PODS in running state with Day zero configuration.
    • When the Day-1 config is loaded, the new PODS does not come up.
    • Inside SMF ops-center, you would see the helm chards in the deleted state.
    • Change system mode runs to shut down and vice versa did not help.
    • Add a new day-1 configuration also did not help.

    Symptoms

    • SMF-IMS NF shows the PODs with Day-0 configuration.
    • Ops-center is allowing us to log in.
    • CEE ops-center is up and running.
    • SMF-DATA ops-center is up and running with day-1 config - This one is the another NF with working PODs.
    ~ubuntu@crucs501-cnat-cnat-core-master1:~$ kubectl get pods -n smf-ims

    NAME                                              READY   STATUS    RESTARTS   AGE

    api-smf-ims-ops-center-69f4d8f47b-hsqnx           1/1     Running   0          162m

    base-entitlement-smf-998c8b84f-79r8v              1/1     Running   0          162m

    documentation-65484db875-n4ljq                    1/1     Running   0          162m

    ops-center-smf-ims-ops-center-6fb57bf79c-9dj29    5/5     Running   2          162m

    smart-agent-smf-ims-ops-center-5dd679cf8b-hq4hs   1/1     Running   0          162m

    swift-smf-ims-ops-center-745565bbf8-w5d7g         1/1     Running   0          162m
    • Helm Chart status
    crucs501-cnat/ims] smf# show helm

    CHART              INSTANCE  STATUS   VERSION                                  REVISION  RELEASE                    NAMESPACE 

    -------------------------------------------------------------------------------------------------------------------------------

    infra-charts       -         DELETED  0.0.2-master-0031-200306111921-107580e   1         smf-ims-infra-charts       smf-ims   

    smf-dashboard      -         DELETED  0.0.2-master-0018-200113112417-b028370   1         smf-ims-smf-dashboard      smf-ims   

    smf-configuration  -         DELETED  0.0.6-master-1067-200303174113-9ee9665   1         smf-ims-smf-configuration  smf-ims   

    li-ep              -         DELETED  0.0.1-master-0405-200306144054-3c56b02   1         smf-ims-li-ep              smf-ims   

    smf-nodemgr        -         DELETED  0.0.2-master-3741-200304171906-5013914   1         smf-ims-smf-nodemgr        smf-ims   

    smf-udp-proxy      -         DELETED  0.0.2-master-1420-200305182644-ebb4bc9   1         smf-ims-smf-udp-proxy      smf-ims   

    gtpc-ep            -         DELETED  0.0.3-master-0926-200305203830-3306ff4   1         smf-ims-gtpc-ep            smf-ims   

    smf-protocol       -         DELETED  0.0.2-master-4652-200304144735-d1e3798   1         smf-ims-smf-protocol       smf-ims   

    smf-dns-proxy      -         DELETED  0.1.0-master-0541-200304144718-b028370   1         smf-ims-smf-dns-proxy      smf-ims   

    smf-service        -         DELETED  0.0.5-master-18345-200305110040-5e8938b  1         smf-ims-smf-service        smf-ims   

    smf-rest-ep        -         DELETED  0.3.3-master-6072-200304171221-7b0ff1a   1         smf-ims-smf-rest-ep        smf-ims   

    etcd-cluster       -         DELETED  0.5.2-master-0046-200305044107-60d06f1   1         smf-ims-etcd-cluster       smf-ims   

    ngn-datastore      -         DELETED  1.0.1-master-0619-200305030353-d255520   1         smf-ims-ngn-datastore      smf-ims

    Troubleshoot

    1. Perform the cluster sync multiple times via SMI-Deployer without success

    2. Day-1 configuration is verified.

    3. Remove the Day-1 config and add back.

    4. Delete the ops-center from Kubernetes master.

    5. Entire config removal is performed.

    6. Delete the Config Maps (CM).

    7. Delete the helm charts from the master.

    8. Delete the namespace.

    9. Remove the supporting files from Deployer.

    10. As the same new SMF build works fine on other deployments in the customer environment, it is ruled out that there is any issue with the image.

    11. SMF-DATA on the same setup had come up without any issue.

    Solution

    1. Delete the cluster config of SMF-IMS ops-center from SMI deployer.

    2. Sync the cluster.

    3. Add back the config.

    4. Sync the cluster.

    There is one more workaround to solve this problem:

    Delete the older version of the SMF package from the directory that SMI Deployer refers to while cluster sync.

    Here is the configuration portion that was removed and added back from SMI Deployer ops-center running-config:

    ops-centers smf ims

      repository               https://charts.10.192.1.xxx.nip.io/smf.2020.01.0-18

      sync-default-repository true

      netconf-ip              10.241.69.xx

      netconf-port            2024

      ssh-ip                  10.241.69.xx

      ssh-port                22

      ingress-hostname        10.241.69.xx.nip.io

      initial-boot-parameters use-volume-claims true

      initial-boot-parameters first-boot-password <xxxyyyzzz>

      initial-boot-parameters auto-deploy false

      initial-boot-parameters single-node false

     exit

    As per the deployments call flow, it is SMI Deployer who takes care of the extraction of the images for PODs from the package that is stored in it.

    Normally, the downloaded Software package of SMF are stored local directory, from which SMI deployer extracts and shift them under this directory: /data/software/packages/</strong >

    If the list of packages available under this directory is checked, you can see all the older packages as well available in it along with the new package list.

    ubuntu@xxxxx501-cnat-smi-cm-core-cm1:/data/software/packages$ ls -lrt

    total 24

    drwxrwxr-x 3 root root 4096 Mar 23 13:15 sample

    drwxrwxr-x 3 root root 4096 Mar 24 05:48 smf.2020.01.0-12   >>> Older version of SMF

    drwxrwxr-x 3 root root 4096 Mar 24 05:48 cee.2020.01.0-1

    drwxrwxr-x 3 root root 4096 Apr 13 19:48 smf.2020.01.0-18   >>> Newer version of SMF

    drwxr-xr-x 3 root root 4096 May  4 10:10 smf.2020.02.0.i66  >>> Older version os SMF

    drwxr-xr-x 3 root root 4096 May  8 12:02 cee.2020.02.0

    In this output, you can see there are three different SMF packages available. Even though the correct SMF version (i.e. smf.2020.01.0-18) is defined in SMI-Deployer running configuration, still, somehow the SMI-Deployer is not able to get the correct image files for that package.

    After the workaround mentioned in the Solution section is performed, the issue was resolved.

    Note: Similar issue is observed with CEE PODs as well, for which similar workaround is applied that is mentioned in the Solution section.

    Revision History

    Revision Publish Date Comments
    1.0
    10-Jul-2020
    Initial Release
    TAC Authored

    Contributed by Cisco Engineers

    • Kumarsinh Vihol
      Cisco TAC Engineer

    Was this Document Helpful?

    FeedbackFeedback

    Contact Cisco

    This Document Applies to These Products