The CDL endpoint pod is the front-end for receiving requests from NF application towards CDL. The CDL endpoint exposes gRPC over HTTP2 interface towards the NF client's for processing database service requests. Each pod starts with the following attributes:

• systemID: This parameter specifies the site ID (For instance: Site).

• clusterID: This parameter specifies the unique datastore ID within a site (For instance: X, Y).

The CDL endpoint receives the Create, Read, Update, Delete (CRUD) requests from the NF applications. It communicates with the CDL Index and Slot pods over GRPC to perform these operations.

The CDL endpoint learns about the other pods within a site automatically using the etcd. Also, the Cisco Data Store K8s pod is stateless in nature and does not maintain any session stickiness. The CDL endpoint receives the request from NF application and communicates with Slot and Index pods internally through GRPC to handle the request accordingly. It returns the response after the processing is complete.

When the CDL endpoint receives a Create, Delete, or Update session, the receiving container replicates it to the Slot and Index pods and sends a response only when 'n' number of writes are acknowledged (ACKed) by the Index and Slot microservices.

Note	A maximum of four virtual CPUs (vCPUs) is required for deploying the CDL endpoint pod in production environment.

The CDL Slot pod stores the actual session data. The CDL endpoint connects to all the Slot pods within the cluster to replicate the session data to all the pods. These microservices are K8s pod deployed for exposing internal gRPC interface towards the Cisco Data Store. Each pod starts with the following attributes:

• systemID: This parameter specifies the site ID (For instance: Site-1).

• clusterID: This parameter specifies the unique datastore ID within a site (For instance: Session).

• mapID: This parameter specifies the replica set ID within the cluster. (For instance: map-1, map-2,..map-n).

• instanceID:This parameter specifies the instance ID within the replica set. (For instance: map-1.instance-1,map-1.instance-2)

Each Slot pod holds a finite number of sessions and pre-allocated memory for the storing session data. Also, each replica within the replica set (mapID) has a defined anti affinity rule, which prevents the same blade form hosting multiple member or instances of the same replica set (for high availability in case of a blade or node failure). Each Slot pod maintains a timer and last updated ts. The Slot pod generates the notification callback to the client NF for taking action when the timer expires of if a conflict is detected.

Note	A maximum of two vCPUs is required for deploying the Slot pod in production environment.

In the event of pod failover and recovery, the Slot pod recovers from:

• Local Replica Member: The Slot directly reads from the gRPC stream in bulk directly to recover data.

• Remote Replica Member: When there is no local replica available for synchronization, the Slot reads the data from the remote site instances for the same map.

  The following figures depict the Slot recovery process from local and remote peers:

  

  

Data Slicing

Data slicing logically separates CDL as slices and stores the session data based on the slice name received from the Network Functions (NF).

With data slicing, one or more NFs can store different types of session data in dedicated slices of CDL.

A default slice name called session is used if the slice names are not configured.

The configuration is as follows:

cdl datastore <datastore name> slice-names [ <sliceName 1> <sliceName 2> ... <sliceName n> ]

The sample configuration is as follows:

cdl datastore session slice-names [ session1 session2 ]

Note

If the slice names are configured at the NF's ops-center or CDL's ops-center, every request from the NF must have a valid slice name. If the slice name is different from what is configured or empty, then the request is rejected with an error code. If the slice names are not configured, then the NF requests are routed to the default session. The slice names cannot be updated in a running system post deployment.

Deleting CDL Slot Data

In certain scenarios, the CDL records are found on Slot but not in the index pods. The notifications from Slot towards the application for such records do not receive the values correctly. The record in the slot is not deleted, if the index data is not deleted.

Ensure the following before deleting the CDL Slot Data:

• If the number of notifications to an application crosses a threshold value (default value of 3), a record is suspected to be stale.

• This triggers a validation check to find the corresponding record in any of the index pods (local or on any geo remote sites).

• If there is a mismatch in map ID from index, or if the map ID is not found in all index pods, then a clean-up is invoked to delete the record on local as well as remote sites.

The following parameters are introduced to delete stale records:

disable-auto-deletion:When set to true, the stale CDL records are not deleted. Auto deletion of stale records is enabled by default.

notification-retry-count:Specifies the minimum number of timer expiry notification retries sent to application without receiving an update from application. If there are no updates received even after notification-retry-count times, cdl proceeds to check if slot record is stale. The default number is 3.

The sample CDL configurations are as follows:

To disable the stale slot record auto deletion feature:

cdl datastore session
features slot-stale-session-detection disable-auto-deletion true
exit

You can change the notification-retry-count to a new value, for example 5. This indicates that the timer expiry notification tries 5 times, after which it proceeds for checking whether the data is stale.

cdl datastore session
features slot-stale-session-detection notification-retry-count 5
exit

Troubleshooting

To enable troubleshooting logs for Stale CDL Slot Data on endpoint and slot pods, use the following configuration:

cdl logging logger ep.staleRecord.session
level info
exit

cdl logging logger slot.staleRecord.session
level info
exit

The Indexing containers contain indexing data. There are two key pieces of information stored in the Index pod:

• Primary key to Slot Map ID.

• Unique secondary key to primary key mapping.

The index entries are created (an example use case) at the time of session connect and removed at the time of disconnect. Using the in-memory KV store, Indexing stores the keys in memory and provides a multi-primary write. The indexing write log (set and delete operations) are also written to Kafka pods for remote site replication. One of the Index pods writes the logs in Kafka, which are then received at the remote site. The corresponding operations are also performed at the remote site.

Note	A maximum of two vCPUs is required for deploying the Index pod in production environment.

In the event of pod failover and recovery, the Index pod recovers from:

• Local Replica Member - The Index directly reads from the gRPC stream in bulk directly to recover data.

• Remote Replica Member - If there is no local replica available for synchronization, the Index reads the data from the remote site instances for the same map.

  The following figures depict the Index recovery process from local and remote peers:

  

  

The CDL uses the etcd (an open-source key-value store) as the DB service discovery. When the CDL pods (Endpoint, Slot or Index) is started, killed or shutdown, it results in updating the event to etcd by publishing the state.

The CDL endpoint is interested in knowing when a Slot or Index pod comes up or goes down. Therefore, it subscribes to the notifications for Slot and Index pod events in etcd.

Similarly, the CDL Slot or Index pod is interested in knowing when its peer replica pod (with the same map-id) comes up or goes down. Therefore, it subscribes to the notifications for corresponding Slot or Index (with same map-id) pod events in etcd.

Thus, the notifications are sent to each of the pods subscribed to these events. In addition, when a key event is added or removed, it refreshes the local map. The etcd cache is only applicable for local site events.

The Kafka pod replicates data between the local replicas and across sites for Indexing. For replication across sites, Kafak uses MirrorMaker. The Kafka pods are deployed on the session vms with the replica count set to a minimum of two for high availability.

Kafka uses the Zookeeper pod for managing Kafka clusters and coordinating with Kafka brokers.

The Mirror Maker pod geo-replicates the indexing data to the remote CDL sites. It takes data from the remote sites and publishes it to the local Kafka site for the appropriate Indexing instances to pick up.

CDL endpoint monitors the remote site connection (replication connection and internal operational connection) using the CDL Ping RPC every 30 seconds. If ping fails three times for any of the connections, then recreate that connection and close the old connection.

Remote site monitoring is configurable and it is enabled by default. Use the cdl datastore session features remote-site-connection-monitoring enable CLI command to enable or disable remote site connections.

To enable or disable remote site connection, use the following CLI command:

config 
   cdl datastore session features remote-site-connection-monitoring enable [ true | false ] 
   exit 

To view the logs for remote site connection on an endpoint, use the following configuration:

cdl logging logger ep.remoteConnection.session 
level trace 
exit 

This section describes the following call flows:

• Finding Record by Primary Key

• Finding Record by Unique Key

• Creating a Record

• Updating a Record

• Deleting a Record

• Notification to NF on Timer Expiry

• Geo Replication - Create (GEO)

• Geo Replication - Update (GEO)

• Geo Replication - Delete (GEO)

In certain scenarios, the unique keys in index pods of the CDL are stale (presumed to be deleted). The NFs may try to use these unique keys for another record, as the CDL does not show the stale record details to the NFs.

The CDL allows the operator to enable a parameter to detect the stale records and perform the necessary action on such stale records.

To detect the stale records and perform actions:

1. Identify the stale records. When a new record is created, the unique key is overwritten. To identify such records, enable the index-overwrite-detection parameter, and ensure that the unique-keys-prefix matches the prefix pattern.

2. Perform the necessary action (notify, delete or log) on the identified stale records.

   The CDL detects any unique key that is overwritten and performs one of the following actions:

   • Deletes the stale record or the entire record. The delete action is triggered only if the PurgeOnEval flag is set to false for the stale record.

   • Notifies the NF about the stale record. The notify action sends the STALE_INDEX_NOTIFICATION to the NF.

   • Logs the overwritten unique key. For the log action, the stale record is logged with the WARN log level.

   Note	If there are two different unique keys pointing to the same primary key; one with notify action and the other with delete action, then the notify action is performed.

Use the following configuration:

features index-overwrite-detection max-tps variable
features index-overwrite-detection queue-size variable
features index-overwrite-detection unique-keys-prefix uk
  action [delete-record, notify-record, log-record]
 

where,

• max-tps: Controls the rate of notification for stale records. The default is 200.

• queue-size: Controls the processing queue size of the stale records. The default is 1000.

  Note	The queue-size parameter is for both delete and notify actions. The max-tps parameter is only for the notify action.

• unique-keys-prefix: Specifies the unique-key prefix pattern along with the action that needs to be performed.

Example:

cdl datastore session
features index-overwrite-detection max-tps 250
 features index-overwrite-detection queue-size 2000
 features index-overwrite-detection unique-keys-prefix uk
 action notify-record
 exit    
  exit      
  

Troubleshooting

To troubleshoot the stale index records, set the index.overwrite.session logger to INFO level. The logs from the endpoint pods as well as the index pods help in troubleshooting.

CDL configuration:

cdl logging logger index.overwrite.session
 level info
exit
 

The following metrics are introduced:

• overwritten_index_records_deleted–maintains the total number of records deleted due to stale records identified at index.

• overwritten_index_records_skipped–maintains the total number of records detected as stale, but dropped when the queue is full while processing the records for notify or delete.

You can deploy CDL through the Network Functions (NFs) – Session Management Function (SMF) and Policy Control Function (PCF) – Ops Center.

For deploying CDL through SMF Ops Center, see Cisco Common Data Layer in SMF chapter in Ultra Cloud Core 5G Session Management Function - Configuration and Administration Guide

For deploying CDL through PCF Ops Center, see Cisco Common Data Layer in PCF chapter in Ultra Cloud Core 5G Policy Control Function -Configuration and Administration Guide

This section describes how to deploy CDL for Geo Replication (GR).

This section describes the alarms in the CDL deployment.

The following alarms are introduced to detect failure:

CNDP supports zone-based upgrades where all the nodes in an upgrade zone are upgraded in parallel. This avoids the time that is taken to upgrade large clusters. CDL utilizes the K8s Topology Spread Constraints feature to spread the pods across upgrade zones.

To support this in-service for CDL, the pods must be aware of upgrade zones and ensure that all replicas for a slot, index, or Kafka do not get scheduled in the same upgrade zone. For example, if both slot replicas of the same map are in the same upgrade zone, both replicas go down at the same time and cause session loss in a non-GR scenario. The CDL endpoint stops serving requests until one replica is back.

Ensure that at least three upgrade zones are present. Any node failure can result in Pending state for pods even if another zone has space to schedule the pod.

There is no explicit configuration that is required in CDL or NF Ops Center to enable this feature. The cluster deployer itself passes the configuration.

You can define the maximum capacity per shard in CDL using the following parameters:

• slotRecordCapacity (Default value: 1M/shard)

• indexRecordCapacity (Default value: 10M/shard)

• slotRecordsSizeCapacityInBytes (Default value: 5GB/shard)

Note	You cannot change the default capacity per shard.

Based on the following calculations, the CDL determines the maximum capacity:

Soft Limit

1. cdl_slot_record_capacity  =  No of shards * slotRecordCapacity

2. cdl_index_record_capacity = No of shards * indexRecordCapacity

3. cdl_slot_size_capacity    = No of shards * slotRecordsSizeCapacityInBytes

Hard Limit

1. cdl_slot_record_capacity = No of shards * slotRecordCapacity * 105%

2. cdl_index_record_capacity = No of shards * indexRecordCapacity * 105%

3. cdl_slot_size_capacity = No of shards * slotRecordsSizeCapacityInBytes * 105%

The CDL is overloaded whenever the system reaches its maximum capacity. Also, the CDL triggers the following alerts to protect the system from overloading:

• cdlOverloaded - major - This alert is triggered when the CDL system reaches 80% of its capacity.

• cdlOverloaded – critical - This alert is triggered when the CDL system reaches 90% of its capacity.

In addition to the alerts, the CDL endpoint rejects all the create requests with the following error:

error code 507: Datastore reached its full capacity

However, the CDL processes all the update requests till the hard limit capacity is reached. When the hard limit capacity is exceeded, the CDL rejects all the update requests.

The the existing CDL overload protection configuration is replaced with a new configuration.

Note

Earlier, the overload protection was enabled by default with hard coded limits of 1 million records per slot, 10 million records per index and 5 gb record size per slot. However, in this release, the feature is disabled by default and requires configuration to enable this feature and to configure the appropriate overload limits (optional) for each parameter.

cdl datastore session overload-protection disable true 

cdl datastore session features overload-protection enable <true/false> 

If the overload protection is enabled then the alerts are also enabled. The prometheus-rules-cdl pod is spawned. If overload-protection is disabled then the alerts are disabled. The prometheus-rules-cdl pod is removed.

Configuring Overload Protection Parameter

The following parameters are configured to set limits for overload-protection:

• cdl datastore session features overload-protection index-max-record-count <value> 

• cdl datastore session features overload-protection slot-max-record-count <value> 

• cdl datastore session features overload-protection slot-max-size <value> 

• cdl datastore session features overload-protection hard-limit-percentage <value> 

The table below lists the configuration details:

cdl datastore session features overload-protection enable <true/false> 

cdl datastore session features overload-protection index-max-record-count <value> 

cdl datastore session features overload-protection slot-max-record-count <value> 

cdl datastore session features overload-protection slot-max-size <value> 

cdl datastore session features overload-protection hard-limit-percentage <value> 

cdl datastore session features overload-protection major-alert-threshold <value> 

cdl datastore session features overload-protection critical-alert-threshold <value> 

To configure the alert percentage, run the following command:

cdl datastore session features overload-protection critical-alert-threshold <percentage> 

cdl datastore session features overload-protection major-alert-threshold <percentage> 

You can verify the alerts using the following command in CEE Ops Center CLI:

show alerts active { detail | summary }  

show alerts active detail sumary "CDL is overloaded." 

In the following example, the cdlOverloaded - major alert is triggered when the system capacity is greater than or equal to 80%

alerts active detail cdlOverloaded-major 5446095ab264
 severity    major
 type        "Processing Error Alarm"
 startsAt    2020-10-15T15:09:00.425Z
 source      System
 summary     "CDL is overloaded."

In the following example, the cdlOverloaded - critical alert is triggered when the system capacity is greater than or equal to 90%

alerts active detail cdlOverloaded-critical 5446095ab264
 severity    critical
 type        "Processing Error Alarm"
 startsAt    2020-10-15T15:09:16.425Z
 source      System
 summary     "CDL is overloaded."

CDL supports rack conversion from half height to full height without draining traffic using the rebalancing method.

This rebalancing method doesn’t affect the existing and new calls during migration. This feature addresses index map rebalancing for CDL.

Use the following procedure to migrate CDL half-rack configuration to full-rack configuration.

1. Set rack-1 as Active and rack-2 as Standby.

   

   Prerequisite: Both sites must have full-rack resources though CDL is running using half-rack configuration.

   Action: Make rack-1 as Active and rack-2 as Standby. No configuration change in CDL.

   Post-validation: Traffic lands only on rack-1.

2. Set the system mode shutdown in rack-2.

   

   Prerequisite: Rack-1 must be on active mode and rack-2 must be on standby mode.

   Action: Do system mode shutdown in rack-2.

   Post-validation: The mirror-maker on site-1 is not in running state.

3. Configure rack-2 as Standby. Configure full-rack CDL along with new configuration to enable the scale-up mode with prev-map-count configured.

   The following is a sample configuration:

   configure 
      cdl datastore session mode scale-up 
      cdl datastore session index prev-map-count 2 

   Half-rack configuration without scaleup mode:

   cdl datastore session  
   label-config session 
      index map 2 
      slot map 4 
      exit 

   Full-rack configuration with scaleup mode:

   cdl datastore session  
   label-config session 
   mode scale-up 
      index map 4 
      index prev-map-count 2 
      slot map 8 
      exit 

4. Enable system mode running of rack-2.

   

   Prerequisite: Rack-1 must be active and rack-2 must be on standby with full-rack CDL configuration.

   Action: Execute the system mode running in rack-2.

   Post-validation: The mirror-maker in both sites must be in running state. The old indexes or slots in site-2 must be able to do initial sync with remote peers.

5. Switch the traffic to rack-2.

   

   Prerequisite: Rack-1 must be active and have full-rack resources with CDL running on half-rack configuration. Rack-2 must be on standby and have full-rack resources with CDL running on full -ack configuration.

   Action: Make rack-1 as standby and rack-2 as active. No configuration change is required in CDL.

   Post-validation: The traffic lands on rack-2 only.

6. Enable system mode shutdown of rack-1.

   

   Prerequisite: Rack-2 must be on active and rack-1 must be on standby.

   Action:Do system mode shutdown in rack-1.

   Post-validation: The mirror-maker on site-2 is not in running state.

7. Configure rack-1 as Standby. Configure full-rack CDL with the new configuration to enable scale-up mode with premap-count configured.

   The following is a sample configuration:

   configure 
      cdl datastore session mode scale-up 
      cdl datastore session index prev-map-count 2 

   The full configuration updates the slot maps count and index maps count. The label configuration is added for newly added slots and indexes.

8. Enable system mode running of rack-1.

   

   Prerequisite: Rack-2 must be active and rack-1 must be on standby with full-rack CDL configuration.

   Action: Do system mode running in rack-1.

   Post-validation: The mirror-maker in both sites must be in a running state. All indexes and slots in site-1 must be able to do initial sync with remote peers. Check in CEE alerts for CDL replication errors if any.

9. Trigger index rebalances using the CLI on rack-1 (Standby).

   

   Prerequisite: The CDL must be in a healthy state at both sites. Rack-1 must be on standby and rack-2 must be on active. Both sites must have scale-up mode configured with index prev-map-count.

   Index rebalancing through the CLI on rack-1 triggers copying the index from the old calculated map (as per the old map count) to the newly calculated map (as per the new map count). This is done after deleting index from the old calculated map once the copying is complete.

   Action: Trigger the rebalance-index run using the following command in standby site(rack1):

   cdl rebalance-index run tps tps_value 
   

   Post-validation: To track the rebalancing status, use the following steps:

   • Monitor the rebalance status using the following CLI command:

     
     cdl rebalance-index status 
     

   • Validate the rebalance run using the following CLI command:

     cdl rebalance-index validate 
     

10. Switch the instances to Active-Active.

    

    Prerequisite: The rebalancing must be done successfully.

    Action: Enable rack-1 and rack-2 as active.

    Post-validation: The traffic must land on both sites.

11. Remove the CDL scale-up mode configuration from both sites.

    Prerequisite: Both rack-1 and rack-2 must be active.

    Action: Configure the following command on both sites:

    no cdl datastore session mode 
    

    Post-validation: CDL must be running in a healthy state. The mirror-maker restarts after scale-up mode is disabled.

To view the rebalancing logs, configure the following logger in CLI:

cdl logging logger datastore.idx-rebalance.session 
level warn 
exit 

The following metric is introduced in the index pod to know the total number of index keys that are rebalanced.

In this release, the CDL supports enhancements for the FindAll and FindAllNotify queries.

New query filters are introduced to support the following operations:

• A maximum of 20 IN filters in the FindAllNotify query. The CDL retreives all sessions from a slot and then matches each session with the passed filters. If the session keys match with any one of the passed filters, then notify the same to the NF with the matched IN filter condition.

• A NOT-MATCH operation with more than one condition.

• The IN and NOT operations in the CDL CLI.

• IPC streaming for the notifications.

  In the previous releases, the CDL used a unary RPC for sending notifications to the application. With this release, the CDL supports the bidirectional streaming RPC for optimizing IPCs between it and the application.

This section describes the sequence of operations for CDL to support the different filter conditions in FindAll and FindAllNotify queries.

Support IN Filters

1. The system checks for all the AND and IN conditions inside the query parameter.

   • If both the AND and IN query filters are in the query, then it's processed in the following manner:

     • The CDL first checks all the AND filters. If any of the AND filter fails, then it skips the other AND filters and IN filters because the key did not match the filter.

     • After all the AND filters are matched, then check the IN filters. If any of the IN filters is matched, then the CDL skips the other IN filters because the key already matched the filter.

   • If only the IN filters are in the query and any of the IN filter is matched, then the CDL skips processing the other IN conditions.

2. If there are more than 20 IN filters in the query request, then respond back with a 400 Bad Request message.

3. The CDL sends the matched IN filter condition along with the notification only when it is enabled in the FindAllNotify query request.

Note	If more IN filters are passed in the request, the query performance is impacted.

Support NOT-MATCH Operation with More than One Condition

The CDL supports the additional conditions for this query filtering:

• not-match

• not-starts-with

• not-ends-with

• not-contains

Note	If any key (pk/uk/nuk) in a session satisfies the match, contains, starts-with, or ends-with condition, then that session will not be for selected for the filter corresponding to the not-match, not-contains, not-starts-with, or not-ends-with condition respectively.

Support the IN and NOT Operations in the CDL CLI

• Support the following additional conditions for filtering the sessions in CLI:

  • contains

  • not-contains

  • not-ends-with

  • not-starts-with

  • not-match

• Support the in-filter filter in addition to the AND filter filter for the IN operations in CLI

  The following code snippet is a sample CLI command:

  cdl show sessions summary filter { condition ends-with key 6 } in-filter { in-condition starts-with key key-1 } in-filter { in-condition starts-with key key-2 }

  Note

  The filter and in-filter filters are a list of conditions. The query supports a maximum of 20 in-filters filters. The order of the filters, filter and in-filter does not matter as the filters mentioned with filter are grouped together. Likewise, the filters mentioned with in-filter are also grouped together.

• The following CDL operations are supported for filtering with filter and in-filter:

  • cdl show sessions count summary

  • cdl show sessions count detailed

  • cdl show sessions detailed

  • cdl show sessions summary

  • cdl clear sessions

Support Streaming Notifications

• The CDL uses bi-directional streaming RPC to send the notifications over a stream with a transaction ID to the application. The response is sent in a different stream from the application to the datastore with the transaction ID for correlation.

• By default, the CDL uses unary RPC and the streaming functionality is disabled. Streaming is enabled only when the feature flag use-stream is configured in the CDL CLI. If the streaming is enabled, then all the CDL notifications are sent using the streaming.

  To enable the streaming for notification functionality, use the following CLI command:

  cdl datastore session slot notification use-stream true

• If the streaming is enabled, then by default there are four stream connections from the CDL to a notification endpoint. The connection count can be configured by using the CDL CLI. The stream connections are used in the round robin manner.

• Support metrics to check the number of active stream connections from the slot or endpoint to the notification endpoint. Use these metrics to generate alerts when there is no active stream connection.

In this release, the CDL enables you to configure its network policy to override the global network policy configuration. If the configuration is enabled, then the network policy for CDL endpoint, index, slot, Kafka, and Zookeeper are applied.

You can also configure the network policy for etcd to override the global network policy configuration. If the configuration is enabled, then the network policy for etcd is applied.

To configure the CDL network policy, use the following CLI commands:

config 
   cdl network-policy enable <true/false> 

To configure the etcd network policy, use the following CLI commands:

config 
   etcd network-policy enable <true/false> 

Issue Description

The session data is not replicated properly to the remote site.

Identifying the Issue

The data added at one site is inaccessible at the other site.

Possible Causes

The incorrect Geo-replication configuration may cause this issue.

Resolution

To resolve this issue:

• Verify the local system-id and remote site configuration.

• Verify whether CDL endpoints and Kafka between each site are reachable.

• Verify the map, replica of index and slot at each site. They must be identical across all the sites.

This section describes how to resolve the CDL operation failure issue.

Issue Description

The NF connects to the CDL, but the session operations such as find, create, update, and delete fail.

Identifying the Issue

You can view the NF log files to identify the call failure.

Possible Causes

You might encounter this issue when the CDL Index and Slot pods are not ready.

Resolution

To resolve this issue:

• Verify whether all the pods are in ready and running state.

• The Index pods move to the ready state only if the synchronization is complete with peer replica (local or remote if available).

• The Slot pods move to the ready state only if the synchronization is complete with peer replica (local or remote if available).

• At least one slot and index pod must be available for the Endpoint to move to ready state. Even when it isn’t ready, the client accepts the GRPC connection.

This section describes how to bring up the CDL pods when they are down.

Issue Description

The CDL pods are not in the "running" state because of incorrect CDL configuration.

Identifying the Issue

Verify the "describe pods" output (Containers, Member, State, Reason, or Events) to identify whether the pods are down with the following command:

kubectl describe pods -n  <namespace> <failed pod name> 

Possible Causes

The possible causes are:

• Pods are in "pending" state.

• Pods are in "CrashLoopBackOff" failure state.

• Pods are in "ImagePullBack" failure state.

Resolution

• When the pods are in "pending" state:

  • Verify whether the k8s nodes with the label value cdl/node-type are present. Also, ensure that the number of replicas are less than or equal to the number of k8s nodes with the label value cdl/node-type.

    kubectl get nodes -l smi.cisco.com/node-type=<value of cdl/node-type, default value is 'session' in multi node setup)
    

• When the pods are in "CrashLoopBackOff" failure state:

  • Verify the status of the ETCD pods.

    kubectl describe pods -n <namespace> <etcd pod name>

    If the ETCD pods are not running, resolve the ETCD issues to bring up the pods.

• When the pods are in "ImagePullBack" failure state:

  • Verify whether the helm repository and image registry are accessible.

  • Verify whether the required proxy and DNS servers are configured.

This section describes how to move the Mirror Maker pods that remain in the "Not Ready" state.

Issue Description

The Mirror Maker pods are not in running state because of connectivity issues on the remote site.

Identifying the Issue

Verify the Describe pod's output and pod logs with the following command:

kubectl describe pods -n <namespace> <failed pod name>
kubectl logs -n <namespace> <failed pod name> [-c <container name>]

Possible Causes

Connectivity issues with remote site Kafka brokers may cause this issue.

Resolution

To resolve this issue:

• Verify whether the external IPs configured for Kafka are accurate.

• Verify whether the remote sites of Kafka are reachable through the external IPs.

This section describes how to resolve the early or delayed notification sent for purging record from CDL.

Issue Description

The notification sent for purging record from CDL is either early or delayed.

Identifying the Issue

While purging record from CDL, the notification comes either early or delayed.

Possible Causes

The nodes in the Kubernetes cluster are not time synchronized.

Resolution

To resolve this issue:

• Verify whether all the nodes in the k8s cluster are time synchronized.

• Verify the synchronization status on all the k8s nodes.

  chronyc tracking
  chronyc sources -v
  chronyc sourcestats -v

Issue Description

The number of sessions pile up because of some network issues or an unexpected number of sessions on CDL, which does not match with the peer session count.

Identifying the Issue

The session count from the CLI or Grafana dashboard displays a higher session count than expected or the sessions are rising constantly.

Possible Causes

The Create request received at CDL are more than the Delete request. Consequently, the session count keeps on growing.

Resolution

To resolve this issue, the respective NF must take appropriate action for triggering clear for the subscribers.

This section describes how to resolve unschedulable pods issue that may arise by using single node label for scheduling, node affinity, and pod placement in CDL pods.

Issue Description

The nodeType parameter controls scheduling, node affinity, and pod placement in CDL. Currently, this parameter is configured with a single node label value smi.cisco.com/node-type. When CDL is configured to its full node capacity, the pods are marked as unschedulable (sometimes) because of the single node label usage.

Identifying the Issue

The CDL pods are marked as unschedulable.

Possible Causes

The usage of single node label for scheduling, node affinity, and pod placement.

Resolution

You can control different CDL pod types using separate node labels. To create separate node labels:

1. (Mandatory) Create smi.cisco.com/node-type label on all the nodes used for CDL through SMI Cluster Manager Ops Center CLI.

2. Create labels for CDL pods as specified in the following table:

   Pods	Label	Value
   CDL Endpoint pods	smi.cisco.com/cdl-ep	true
   CDL Slot pods	smi.cisco.com/cdl-slot-<MAP Id 1..n>	true
   CDL Index pods	smi.cisco.com/cdl-index-<MAP Id 1..n>	true

   Note	This configuration is required only for user-defined scheduling.

Example

The following example shows how to create separate node labels for CDL which uses four different nodes with two endpoint replicas, four Slot maps (with two replicas each) and two Index maps (with two replicas each).

clusters user-k8s
  nodes session1
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-3 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit
clusters user-k8s
  nodes session2
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-1 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-3 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit
clusters user-k8s
  nodes session3
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-4 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit
clusters user-k8s
  nodes session3
   k8s node-labels smi.cisco.com/cdl-ep true
   exit
   k8s node-labels smi.cisco.com/cdl-index-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-2 true
   exit
   k8s node-labels smi.cisco.com/cdl-slot-4 true
   exit
   k8s node-labels smi.cisco.com/node-type session
   exit
  exit
exit

cdl label-config session
  endpoint key smi.cisco.com/cdl-ep
  endpoint value true
  slot map 1
    key smi.cisco.com/cdl-slot-1
    value true
  slot map 2
    key smi.cisco.com/cdl-slot-2
    value true
  slot map 3
    key smi.cisco.com/cdl-slot-3
    value true
  slot map 4
    key smi.cisco.com/cdl-slot-4
    value true
  index map 1
    key smi.cisco.com/cdl-index-1
    value true
  index map 2
    key smi.cisco.com/cdl-index-2
    value true
cdl datastore session
  label-config session
  endpoint replica 2
  index replica 2
  index map 2
  slot replica 2
  slot map 4
exit

You can monitor various activities in CDL through the Grafana dashboard called CDL Dashboard, which is bundled by default. The CDL Dashboard displays the TPS towards CDL for various pods like cdl-endpoint, Slot, Index and the response time taken for each operation.

A sample Grafana CDL Summary dashboard displaying the total records by Type, SliceName, SystemID and so on is shown below:

A sample Grafana CDL dashboard displaying the CDL TPS and response time graphs is shown below:

The Grafana CDL dashboard is enhanced to show the Geo Replication status. A sample Grafana dashboard displaying the Geo Replication status and other details is shown below.

The GR connection, Index replication and slot replication panels, and their descriptions are listed in the table below:

You can verify the status of the CDL and inspect the data inserted by the client using the cdl show status command in the CDL Ops Center. For more information on cdl show status command, see the UCC CDL Command Reference Guide.

The following example displays the status of all the CDL components and client's data.

[unknown] cdl# cdl show status    
message params: {cmd:status mode:cli dbName:session sessionIn:{mapId:0 limit:10000 filters:[]}}
site {
    system-id 1
    cluster-id 1
    db-endpoint {
        state STARTED
    }
    slot {
        map-id 1
        instance-id 2
        records 100
        capacity 1000000
        state STARTED
    }
    slot {
        map-id 1
        instance-id 1
        records 100
        capacity 1000000
        state STARTED
    }
    index {
        map-id 1
        instance-id 2
        records 500
        capacity 2500000
        state ONLINE
    }
    index {
        map-id 1
        instance-id 1
        records 500
        capacity 2500000
        state ONLINE
    }
}
[unknown] cdl#