Alarm States

Cisco SD-WAN Manager alarms are assigned a state based on their severity:

• Critical (red)—Serious events that impair or shut down the operation of an overlay network function.

• Major (yellow)—Serious events that affect, but do not shut down, the operational of a network function.

• Medium (blue)—Events that might impair the performance of a network function.

• Minor (green)—Events that might diminish the performance of a network function.

Note	From Cisco vManage Release 20.11.1, the Medium alarms appear in green and the Minor alarms appear in blue.

The alarms listed as Active generally have a severity of either critical or major.

To view alarm details such as alarm name, sevcerity, and alarm description:

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Alarms.

2. Click Export to export data for all alarms to a file in CSV format.

   Cisco SD-WAN Manager downloads data from the alarms table to the default download location of your browser. The data is downloaded as a CSV file with the name alarms-mm-dd-yyyy.csv.

3. Open the downloaded file to view alarm details.

Alarm Fields

Alarm messages can contain the following fields that provide more information about the alarm:

Use the Alarms screen to display detailed information about alarms generated by control components and routers in the overlay network.

When the notification events that Cisco SD-WAN Manager receives indicate that the alarm condition has passed, most alarms clear themselves automatically. Cisco SD-WAN Manager then lists the alarm as Cleared, and the alarm state generally changes to medium or minor.

Set Alarm Filters

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Alarms.

2. Click Filter.

3. In the Severity field, choose an alarm severity level from the drop-down list. You can specify more than one severity level.

4. In the Active field, choose active, cleared, or both types of alarm from the drop-down list. Active alarms are alarms that are currently on the device but have not been acknowledged.

5. In the Alarm Name field, choose an alarm name from the drop-down list. You can specify more than one alarm name.

6. Click Search to look for alarms that match the filter criteria.

Cisco SD-WAN Manager displays the alarms in both table and graphical formats.

From Cisco Catalyst SD-WAN Manager Release 20.12.1, the heatmap view displays alarms.

Set Advanced Alarm Filters

From Cisco vManage Release 20.11.1, you can set advanced filters to search for alarms that are generated by sites or devices. To set advanced filters:

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Alarms.

2. Click Advanced Filter.

3. In the Object Type drop-down menu, choose either Site or Device for which you want to view alarms.

4. In the Object List drop-down menu, choose either Site ID or Device IP for which you want to view alarms.

   You can choose more than one site or device.

5. In the Severity drop-down menu, choose one or more alarm severity levels from the drop-down list.

6. In the Type drop-down menu, choose one or more alarm names from the drop-down list.

7. Click Apply Filters to view alarms that match the filter criteria.

   The Custom Filter Condition allows you to filter alarms based on the OR condition, for example, 1 OR 2 OR 3.

You can add up to five filters. To delete a filter, click the Bin icon.

Cisco SD-WAN Manager displays the alarms in both table and graphical formats.

From Cisco Catalyst SD-WAN Manager Release 20.12.1, the heatmap view displays alarms.

Enable Email Notifications

Configure SMTP and email recipient parameters to enable email notifications for alarms. Configure the SMTP and email recipient parameters on this screen:

1. From the Cisco SD-WAN Manager menu, choose Administration > Settings.

2. In Alarm Notifications, choose Enabled.

   From Cisco Catalyst SD-WAN Manager Release 20.13.1, click the toggle button to enable cloud services.

3. Check the Email Settings check box.

4. Choose the security level for sending the email notifications. The security level can be None, SSL, or TLS.

5. In the SMTP Server field, enter the name or the IP address of the SMTP server to receive the email notifications.

6. In the SMTP Port field, enter the SMTP port number. For no security, the default port is 25; for SSL it is 465; and for TLS it is 587.

7. In the From address field, enter the full email address to include as the sender in email notifications.

8. In the Reply to address field, enter the full email address to include in the Reply-To field of the email. This address can be a noreply address, such as noreply@cisco.com.

9. Check the Use SMTP Authentication check box to enable SMTP authentication to the SMTP server.

   Enter the username and password to use for SMTP authentication. The default user email suffix is appended to the username. The password that you type is hidden.

10. Click Save.

Note	The email is sent from Cisco SD-WAN Manager Public-IP of VPN0 (Transport Interface) as a source interface.

Send Alarm Notifications

Before you begin: Ensure that Email Notifications are enabled under Administration > Settings, check whether Alarm Notifications is enabled and, Email Settings check box is checked.

From Cisco Catalyst SD-WAN Manager Release 20.13.1, click the toggle button to enable cloud services.

From Cisco Catalyst SD-WAN Manager Release 20.15.1, configure Slack or Webex webhooks to receive alarm notifications.

To send email notifications when alarms occur:

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Alarms.

   From the Cisco SD-WAN Manager menu, choose Monitor > Alarms.

2. Click Alarm Notifications. A list of configured notifications is displayed in the table.

3. Click Add Alarm Notifications.

4. In the Notification Name field, enter a name for the email notification. The name can be up to 128 characters and can contain only alphanumeric characters.

5. Expand the Alarm Type filter and do the following to configure the parameters:

   • From the Object Type drop-down list, choose a site or device you want to view the alarms for.

   • From the Object List drop-down list, choose a site ID or a device based on the type of object you have selected.

   • From the Severity drop-down list, choose the alarm severity.

   • From the Types drop-down list, choose an alarm type.

6. Expand the Delivery Method filter and click the following options to configure the alarm delivery method.

   1. Check the Email check box to trigger an email an alarm notification event occurs.

      1. In the Email field, enter one or more email addresses.

      2. (Optional) Click Add New Email List and enter an email list, if desired.

      3. In the Email Threshold field, set the maximum number of emails to be sent per minute. The number can be a value from 1 through 30. The default is 5.

   2. Check the WebHook check box to trigger an HTTP callback to a webhook channel when an alarm notification event occurs.

      1. From the Choose a Channel for Webhook drop-down list, choose a webhook channel to receive alarm notifications in

         • Cisco Webex

         • Slack

         • Custom

      2. In the WebHook URL field, enter the URL of the webhook server.

         To create a webhook URL for Slack, go to api.slack.com see the section "Sending messages using incoming webhooks".

         To create a webhook URL for Webex, go to WebEx App Hub and see the section Incoming Webhooks.

      3. In the WebHook Threshold field, enter the threshold value.

         The value you enter indicates the number of notifications that you receive for that webhook URL per minute. For example, if the WebHook Threshold value is 2, you receive two notifications for that webhook URL per minute. Notifications that are generated beyond the threshold are not delivered.

      4. (Optional) If you have chosen Custom from the Choose a Channel for Webhook option, configure these additional parameters:

         Field	Description
         Username	Enter a username for authentication.
         Password	Enter a password for authentication.

7. Click Add.

View and Edit Email Notification

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Alarms.

   From the Cisco SD-WAN Manager menu, choose Monitor > Alarms.

2. Click Alarm Notifications. A list of configured notifications is displayed in the table.

3. For the desired notification, click the View icon to the right of the row.

4. When you are done viewing the notification, click OK.

Edit an Email Notification

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Alarms.

   From the Cisco SD-WAN Manager menu, choose Monitor > Alarms.

2. Click Alarm Notifications. A list of configured notifications is displayed in the table.

3. For the desired email notification, click the Edit icon.

4. When you are done editing the notification, click Update.

Delete an Email Notification

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Alarms.

   From the Cisco SD-WAN Manager menu, choose Monitor > Alarms.

2. Click Alarm Notifications. A list of configured notifications is displayed in the table.

3. For the desired email notification, click the Trash Bin icon.

4. In the confirmation dialog box, click OK.

View Events by Using the CLI

To view information about a device on which an event was generated, for Cisco vEdge devices, you can use the show notification stream viptela command. Here is an example of the command output. The first line of the output shows the time when the message was generated (the SNMP eventTime). The time is shown in UTC format, not in the device's local time. The second line of the notification contains a description of the event, and the third line indicates the severity level.

vEdge# show notification stream viptela
notification 
 eventTime 2015-04-17T14:39:41.687272+00:00
 bfd-state-change 
  severity-level major
  host-name vEdge
  system-ip 1.1.4.2
  src-ip 192.168.1.4
  dst-ip 108.200.52.250
  proto ipsec
  src-port 12346
  dst-port 12406
  local-system-ip 1.1.4.2
  local-color default
  remote-system-ip 1.1.9.1
  remote-color default
  new-state down
 !
!
notification 
 eventTime 2015-04-17T15:12:20.435831+00:00
 tunnel-ipsec-rekey 
  severity-level minor
  host-name vEdge
  system-ip 1.1.4.2
  color default
 !
!
notification 
 eventTime 2015-04-17T16:56:50.314986+00:00
 system-login-change 
  severity-level minor
  host-name vEdge
  system-ip 1.1.4.2
  user-name admin
  user-id 9890
 !

To view information about a device on which an event was generated, for Cisco IOS XE Catalyst SD-WAN devices, you can use the show sdwan notification stream command. Here is an example of the command output. The first line of the output shows the time when the message was generated (the SNMP eventTime). The time is shown in UTC format, not in the device's local time. The second line of the notification contains a description of the event, and the third line indicates the severity level.

Device# show sdwan notification stream
notification                                
 eventTime 2020-03-03T02:50:04.211317+00:00
sla-change
  severity-level major
  host-name SanJose
  system-ip 4.4.4.103
  src-ip 10.124.19.15
  dst-ip 10.74.28.13
  proto ipsec
  src-port 12426
  dst-port 12346
  local-system-ip 4.4.4.103
  local-color default
  remote-system-ip 4.4.4.106
  remote-color biz-internet
  mean-loss 17
  mean-latency 13
  mean-jitter 19
  sla-classes None
  old-sla-classes Voice-And-Video
!
!

Starting from Cisco IOS XE Catalyst SD-WAN Release 17.6.3, the alarms alarm bfd-state-change syslog command is used to view the BFD state change syslog message for any BFD state change event in the device. For complete details, see alarms alarm bfd-state-change syslog command.

Device(config-system)# alarms alarm bfd-state-change syslog 
Device(config-alarm-bfd-state-change)# commit

Here is an example for BFD state change syslog message:

Jul 10 07:09:07.583: %Cisco-SDWAN-vm5-FTMD-5-NTCE-1000009: BFD-session 10.1.15.15:12346 -> 10.1.16.16:12366, 
local-tloc-index: 32775 -> remote-tloc-index: 32777, TLOC- local sys-ip: 172.16.255.15, local color: lte -> remote 
sys-ip: 172.16.255.16, remote color: lte, encap: IPSEC, new state->UP delete:false, reason:REMOTE_FSM

Running configuration after enabling BFD state change:

 Device# show sdwan running-config 
system
 gps-location latitude 35.0
 gps-location longitude -120.0
 system-ip             170.16.1.1
 simulated-devices     27 2
 simulated-color       red blue
 simulated-wan-ip      192.168.1.1
 domain-id             1
 site-id               10000
 admin-tech-on-failure
 organization-name     "vIPtela Inc Regression"
 vbond 10.0.12.26
 alarms alarm bfd-state-change
  syslog
 !
!

Set Event Filters

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Events.

2. Click the Filter icon from the search box.

3. Choose the time of the event from the Event Time drop-down list.

4. Choose the name of the host, from the Hostname drop-down list.

5. Choose the system IP of the devices from the System IP drop-down list to view generated events.

6. Choose the event name, from the generated events, from the Name drop-down list. You can choose more than one event name.

7. Choose the event severity level from the Severity drop-down list.

   The events generated by Cisco Catalyst SD-WAN devices are classified as:

   1. Critical—indicates that action needs to be taken immediately.

   2. Major—indicates that the problem needs immediate attention from you but, is not critical enough to bring down the network.

   3. Minor—is informational only.

8. Choose one or more components that caused the event from the Component drop-down list.

9. Choose the relevant event details from the Details drop-down list.

View the filtered events in Cisco SD-WAN Manager both as tabular and graphical formats.

From Cisco Catalyst SD-WAN Manager Release 20.12.1, view the events in a heatmap format.

Set Advanced Event Filters

1. From the Cisco SD-WAN Manager menu, choose Monitor > Logs > Events.

2. Click the Advanced Filter option.

3. In the Object Type drop-down menu, choose either Site or Device for which you want to view events.

4. In the Object List drop-down menu, choose either Site ID or Device IP for which you want to view events.

   You can choose more than one site or device.

5. In the Severity drop-down menu, choose one or more event severity levels from the drop-down list.

6. In the Type drop-down menu, choose one or more event names from the drop-down list.

7. Click Apply Filters to view events that match the filter criteria.

8. The Custom Filter Condition enables you in filtering events based on the OR condition, for example, 1 OR 2 OR 3.

9. Click the + icon and add up to five filters.

10. Click the Bin icon to delete a filter.

View the filtered events in Cisco SD-WAN Manager both as tabular and graphical formats.

From Cisco Catalyst SD-WAN Manager Release 20.12.1, view the events in a heatmap format.

To monitor and control the event trace function for a specified SD-WAN subsystem, use the monitor event-trace command in privileged EXEC mode. Event trace provides the functionality to capture the SD-WAN traces between the SD-WAN daemons and SD-WAN subsystems. For more information on the commands, see monitor event-trace sdwan and show monitor event-trace sdwan.

Verify a Lockout Policy for Cisco SD-WAN Manager

To verify the lockout policy configuration, use the show running-config system aaa lockout-policy command.

Verify a Login-Rate Alarm Threshold for Cisco SD-WAN Manager

To verify the login rate alarm configuration, use the show running-config system alarms command.

vmanage# show running-config system alarms
system 
 alarms
  login-rate
    interval 60
      num-logins 3
      !
    !
  !

Monitor Notifications of Failed Login Attempts to Cisco SD-WAN Manager

To view the history of failed login attempts, use the show alarms history command.

In the following example, there were two failed login attempts, after which Cisco SD-WAN Manager prevented additional login attempts.

vmanage# show alarms history | inc aaa-user
07/10     16:07:18       aaa-login-anomaly    major    user-name:test    remote:host:192.0.2.1
07/10     16:07:10       aaa-login-anomaly    major    user-name:test    remote:host:192.0.2.1
07/10     16:07:00       aaa-user-locked      major    user-name:test    remote:host:192.0.2.1

Monitor System Login Rate Alarms

To view alarms configured by the login-rate command, showing when the number of logins to Cisco SD-WAN Manager exceeds a configured threshold, use the show alarms history command, and view alarms of type system-login-rate .

vmanage# show alarms history

DATE    TIME        TYPE                  SEVERITY    DETAILS
----------------------------------------------------------------
07/10   16:08:05   system-login-rate      minor    num-logins:3 time-interval:60 login-message:3 logins were done in 0 hours 1 minutes 8 seconds
07/10   16:08:05   system-login-change    minor    user-name:admin    user-id:145