The SMI comes equipped with different notification streams to verify the event status
(cluster or node synchronization) or synchronization state of the current cluster or
nodes in it.
The show notification stream provides notification details for alert notifications, license status and system status.
show notification stream ?
Possible completions:
alert-notification Notifications for alerts
license-status Notifications for smart agent licensing will be generated here
system-status Notifications for system status
The NSO must subscribe to the alert-notification notification stream to receive alerts.
You can access the notification stream using the following command show
command in the SMI Cluster Manager CLI:
show notification stream {node-state | node-status | sync-state | sync-status}
Sync-State Notification
You can access the synchronization state of a given cluster using the
sync-state
notification. After every cluster
synchronization, a sync-state notification is registered as sent.
The following example displays the deployment of a stacked cluster:
SMI Cluster Deployer# show notification stream sync-state
notification
eventTime 2020-10-07T18:58:27.368+00:00
sync-state-notification
cluster <stacked_cluster_name>
state DEPLOYED
!
!
notification
eventTime 2020-10-07T19:07:56.114+00:00
sync-state-notification
cluster <stacked_cluster_name>
state DEPLOYED
!
!
notification
eventTime 2020-10-07T19:46:54.531+00:00
sync-state-notification
cluster <stacked_cluster_name>
state DEPLOYED
!
!
notification
eventTime 2020-10-08T15:59:56.697+00:00
sync-state-notification
cluster <stacked_cluster_name>
state DEPLOYED
!
!
The node-state-notification could be used to assess the state of the node in a cluster deloyment sync process. Below example shows the state of control plane and worker nodes are joined in <stacked_cluster_name> cluster. Before reaching JOINED state if the join process is still running they would show up as JOINING.
[installer-controlplane] SMI Cluster Deployer# show notification stream node-state
notification
eventTime 2020-10-07T19:46:54.531+00:00
node-state-notification
cluster <stacked_cluster_name>
node cmts-worker3
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.532+00:00
node-state-notification
cluster <stacked_cluster_name>
node controlplane2
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.534+00:00
node-state-notification
cluster <stacked_cluster_name>
node cmts-worker2
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.535+00:00
node-state-notification
cluster <stacked_cluster_name>
node controlplane3
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.535+00:00
node-state-notification
cluster <stacked_cluster_name>
node controlplane1
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.54+00:00
node-state-notification
cluster <stacked_cluster_name>
node cmts-worker1
state JOINED
!
!
Node-State Notification
You can view the node state in a cluster deployment synchronization process using the
node-state
notification.
The following example displays the state of control plane and worker nodes that have joined the stacked cluster. The node
state is displayed as JOINING , if the process is running. Otherwise, the node state is displayed as JOINED .
SMI Cluster Deployer# show notification stream node-state
notification
eventTime 2020-10-07T19:46:54.531+00:00
node-state-notification
cluster <stacked_cluster_name>
node cmts-worker3
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.532+00:00
node-state-notification
cluster <stacked_cluster_name>
node controlplane2
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.534+00:00
node-state-notification
cluster <stacked_cluster_name>
node cmts-worker2
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.535+00:00
node-state-notification
cluster <stacked_cluster_name>
node controlplane3
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.535+00:00
node-state-notification
cluster <stacked_cluster_name>
node controlplane1
state JOINED
!
!
notification
eventTime 2020-10-07T19:46:54.54+00:00
node-state-notification
cluster <stacked_cluster_name>
node cmts-worker1
state JOINED
!
!
Sync-Status Notification
You view the progression of the cluster synchronization process using the
sync-status notification.
The following example displays the different states (SYNC-START, ERROR and DONE) of
the cluster synchronization process:
SMI Cluster Deployer# show notification stream sync-status
notification
eventTime 2020-10-08T16:05:53.326+00:00
sync-status-notification
cluster <stacked_cluster_name>
status ERROR
!
!
notification
eventTime 2020-10-08T16:12:48.042+00:00
sync-status-notification
cluster <stacked_cluster_name>
status SYNC-START
!
!
notification
eventTime 2020-10-08T16:26:23.444+00:00
sync-status-notification
cluster <stacked_cluster_name>
status ERROR
!
!
notification
eventTime 2020-10-08T16:26:23.446+00:00
sync-status-notification
cluster <stacked_cluster_name>
status ERROR
!
!
notification
eventTime 2020-10-08T16:31:42.551+00:00
sync-status-notification
cluster <stacked_cluster_name>
status SYNC-START
!
!
notification
eventTime 2020-10-08T16:43:33.498+00:00
sync-status-notification
cluster <stacked_cluster_name>
status DONE
!
!
notification
eventTime 2020-10-08T20:04:21.386+00:00
sync-status-notification
cluster kali-stacked
status SYNC-START
!
!
notification
eventTime 2020-10-08T20:04:21.386+00:00
sync-status-notification
cluster kali-stacked
status SYNC-START
!
!
notification
eventTime 2020-10-08T20:27:32.155+00:00
sync-status-notification
cluster kali-stacked
status DONE
!
!
Node-Status Notification
You can view the status of the node synchronization using the
node-status notification. The following example
displays the status of the nodes in a stacked cluster:
SMI Cluster Deployer# show notification stream node-status
notification
eventTime 2020-09-25T19:59:56.728+00:00
node-status-notification
cluster kali-vm3
node controlplane
status DONE
!
!
notification
eventTime 2020-09-29T16:36:05.962+00:00
node-status-notification
cluster kali
node etcd1
status DONE
!
!
notification
eventTime 2020-09-29T20:54:18.415+00:00
node-status-notification
cluster kali-stacked
node controlplane1
status DONE
!
!
notification
eventTime 2020-09-30T01:28:57.355+00:00
node-status-notification
cluster kali-stacked
node controlplane1
status DONE
!
!
notification
eventTime 2020-09-30T01:46:53.034+00:00
node-status-notification
cluster kali-stacked
node controlplane1
status DONE
!
!
notification
eventTime 2020-09-30T04:17:25.701+00:00
node-status-notification
cluster kali-stacked
node controlplane1
status ERROR
!
!
notification
eventTime 2020-10-06T20:02:30.108+00:00
node-status-notification
cluster kali-stacked
node cmts-worker1
status DONE
!
!
VM Status Alerts
The NSO needs to know the state of the NF (Network Function) that has been deployed and notifications for the following states:
-
DEPLOYED
-
ALIVE
-
UNDEPLOYED
-
ERROR
-
RECOVERING
-
RECOVERY_FAILED
Note
|
The SMI supports VM status notification for UPF only.
|
The following parameters are introduced in the notification streams for the VM status notifications:
Table 1.
Parameter
|
Description
|
vm-state-notification stream
|
The vm-state-notification stream has the following details:
|
alert-notification stream
|
The alert-notification stream has the following labels:
|
Depending on the life cycle stage of the VM, notifications are generated from either the Cluster Manager or CEE Ops-Center
and sent to the NSO. When VM gets deleted or redeployed, the UNDEPLOYED notification is sent to the Cluster Manager notification
stream. All other notifications are generated by the Alert Manager and then sent to the CEE Ops-Center notification stream.
Cluster Manager Notification
The Cluster Manager sends vm-state-notification for the UNDEPLOYED VM state.
CEE Ops-Center Notification
The CEE Ops-Center alert-notification sends the following alerts for different VM states:
-
vm-deployed: minor - DEPLOYED
-
vm-alive: minor – ALIVE (alert lasts for a short time and disappears automatically)
-
vm-error: major - ERROR
-
vm-recovering: warning - RECOVERING
-
vm-recovery_failed: critical - RECOVERY_FAILED
All required fields are included in alert labels for notification from alert-notification. All VM alerts are viewable on the
Grafana dashboard.
VM Action Notifications
Delete Action: When delete VM action is triggered, CM sends notifications that the VM is deleted. The VM states are UNDEPLOYED and ERROR
for vm delete action.
clusters abc-cluster-15 nodes kvm-1 vms upf1 actions delete
Redeploy Action: When VM is in RECOVERY_FAILED state, NSO sends a request to redeploy the VM. A redeploy action does both delete action and
sync action.
clusters abc-cluster-15 nodes kvm-1 vms upf1 actions redeploy
Redeploy Action Notification: The redeploy action sends a notification to the CM. The redeploy vm action has the following 4 states: UNDEPLOYED, ERROR,
REDEPLOYED, REDEPLOY_ERROR.
show notification stream vm-state
notification
eventTime 2021-02-23T21:27:28.692+00:00
vm-state-notification
cluster_name cndp-testbed
node_name kvm-1
vm_name upf2
state UNDEPLOYED
message
!
!
notification
eventTime 2021-02-23T21:29:18.699+00:00
vm-state-notification
cluster_name cndp-testbed
node_name kvm-1
vm_name upf2
state REDEPLOYED
message
!
!
Configuring the Alert Notification in CEE
The user must configure alert notifications when they deploy the UPF VMs. Log in to the CEE cli to add the following configuration:
config
bulk-stats prune-interval-days 3
prometheus kvm-metrics defaults private-key "-----BEGIN OPENSSH PRIVATE KEY-----LGXtil23N4YV=\n-----END OPENSSH PRIVATE KEY-----\n"
prometheus kvm-metrics defaults user cloud-user
prometheus kvm-metrics monitor-server 10.194.62.41
hostname abc-bm-15-controlplane
exit
Note
|
The user must replace the IP, hostname, private key and user details.
|
Sample Notification from the Alert Notification Stream
notification
eventTime 2021-01-08T03:28:54.501+00:00
smi-alert-notification
starts-at 2021-01-08T03:28:24.493874101Z
ends-at 0001-01-01T00:00:00Z
alert-status firing
smi-alert-notification alert-label
name alertname
value vm-recovery-failed
!
smi-alert-notification alert-label
name cluster
value test-cee-kvm_cee-voice
!
smi-alert-notification alert-label
name hostname
value test-bm-15-controlplane
!
smi-alert-notification alert-label
name instance
value metrics-proxy-test-bm-15-controlplane:9100
!
smi-alert-notification alert-label
name job
value metrics-proxy
!
smi-alert-notification alert-label
name message
value 10.1.1.3
!
smi-alert-notification alert-label
name monitor
value prometheus
!
smi-alert-notification alert-label
name node_name
value controlplane
!
smi-alert-notification alert-label
name replica
value test-cee-kvm_cee-voice
!
smi-alert-notification alert-label
name severity
value critical
!
smi-alert-notification alert-label
name state
value RECOVERY_FAILED
!
smi-alert-notification alert-label
name vm_name
value upf2
!
smi-alert-notification alert-annotation
name summary
value upf2 failed to recover.
!
smi-alert-notification alert-annotation
name type
value Equipment Alarm
!
!
!