Configure and Verify Syslog on UCS Intersight Managed Mode

Introduction

This document describes the process to setup and verify the Syslog protocol on Intersight Managed Mode UCS Domains.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Unified Computing System (UCS) Servers
• Intersight Managed Mode (IMM)
• Networking basic concepts
• Syslog protocol

Components Used

The information in this document is based on these software versions:

• Intersight software as a service (SaaS)
• Cisco UCS 6536 Fabric Interconnect, firmware 4.3(5.240032)
• Rack Server C220 M5, firmware 4.3(2.240090)
• Alma Linux 9

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Syslog policies are applicable for Fabric Interconnects and Servers. They allow for configuration of local and remote logging.

Configure

1. Navigate to Policies > Create new policy.
2. Choose Syslog, then click Start.
   Policy selection
   
3. Choose the Organization and choose a name, then click Next.
   Configure organization and name
   
4. Choose the desired minimum severity to report for Local Logging. Severity levels can be referenced on RFC 5424.
   Choose the minumum severity to report for Local Logging
5. Choose the desired minimum severity to report for Remote Logging, and the required settings. These are the remote server(s) IP address or hostname, the port number, and the port protocol (TCP or UDP).
   
   

   Note: This example uses the default setting UDP port 514. While the port number can be changed, this only applies to Servers. Fabric Interconnects use the default port 514 by design.

   
   Configure Remote Logging parameters
   
   
6. Click Create.
7. Assign the Policy to the desired devices.

Fabric Interconnects

1. Navigate to the Domain Profile, click Edit, then click Next until step 4 UCS Domain Configuration.
2. Under Management > Syslog, choose the desired Syslog Policy.
   Choose the syslog policy on a Fabric Interconnect Domain Profile
3. Click Next, then Deploy. The deployment of this policy is not disruptive.

Servers

1. Navigate to the Server Profile, click Edit, then go Next until step 4 Management Configuration.
   
2. Choose the Syslog Policy.
   Choose the syslog policy on a Server Service Profile
3. Continue until the last step and Deploy.

Verify

At this point, Syslog messages must be logged on the Syslog remote server(s). For this example, the Syslog server was deployed on a Linux server with the rsyslog library.

Note: Verification of the Syslog messages logging can differ depending based on the remote Syslog server in use.

Confirm that the Fabric Interconnects Syslog messages were logged on the remote server:

[root@alma jormarqu]# tail /var/log/remote/msg/192.0.2.3/_.log
Jan 16 15:09:19 192.0.2.3 : 2025 Jan 16 20:11:57 UTC: %VSHD-5-VSHD_Syslog_CONFIG_I: Configured from vty by root on vsh.bin.32025
Jan 16 15:09:23 192.0.2.3 : 2025 Jan 16 20:12:01 UTC: %VSHD-5-VSHD_Syslog_CONFIG_I: Configured from vty by root on vsh.bin.32237

Confirm that the Servers Syslog messages were logged on the remote server:

[root@alma jormarqu]# tail /var/log/remote/msg/192.0.2.5/AUDIT.log  
Jan 16 20:16:10 192.0.2.5 AUDIT[2257]: KVM Port port change triggered with value "2068" by User:(null) from Interface:
Jan 16 20:16:18 192.0.2.5 AUDIT[2257]: Communication Services(ipmi over lan:enabled,ipmi privilege level limit:admin) has been updated by User:(null) from Interface:
Jan 16 20:16:23 192.0.2.5 AUDIT[2257]: Local User Management (strong password policy :disabled) by User:(null) from Interface:
Jan 16 20:16:23 192.0.2.5 AUDIT[2257]: Password Expiration Parameters (password_history:5,password_expiry_state:disabled,password_expiry_duration:0,notification_period:15,grace_period:0) has been updated by User:(null) from Interface:
Jan 16 20:16:26 192.0.2.5 AUDIT[2257]: Local Syslog Severity changed to "Debug" by User:(null) from Interface:
Jan 16 20:16:27 192.0.2.5 AUDIT[2257]: Secured Remote Syslog with(serverId =1, secure_enabled =0) by User:(null) from Interface:

Troubleshoot

A packet capture can be performed on the Fabric Interconnects to confirm if the Syslog packets were forwarded correctly. Change the minimum severity to report to debug. Ensure Syslog reports as much information as possible.

From the command line interface, start a packet capture on the management port and filter by port 514 (Syslog port):

FI-6536-A# connect nxos
FI-6536-A(nx-os)# ethanalyzer local interface mgmt capture-filter "port 514" limit-captured-frames 0 
Capturing on mgmt0

In this example, a server port on Fabric Interconnect A was flapped to generate Syslog traffic.

1. Navigate to Fabric Interconnects > Inventory.
2. Click the checkbox for the desired port, open the ellipsis menu on the right, and choose disable.
   Shut down an interface on an Fabric Interconnect to generate syslog traffic for testing
3. The console on the Fabric Interconnect must capture the Syslog packet:
   

   FI-6536-A(nx-os)# ethanalyzer local interface mgmt capture-filter "port 514" limit-captured-frames 0 
   Capturing on mgmt0
   2025-01-16 22:17:40.676560 192.0.2.3 -> 192.0.2.2 Syslog LOCAL7.NOTICE: : 2025 Jan 16 22:17:40 UTC: %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/3 is down (Transceiver Absent)

4. The message must be logged in your remote server:

[root@alma jormarqu]# tail -n 1 /var/log/remote/msg/192.0.2.3/_.log 
Jan 16 17:15:03 192.0.2.3 : 2025 Jan 16 22:17:40 UTC: %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/3 is down (Transceiver Absent)

The same test can be run on servers:

Note: This procedure only works for servers with out-of-band configuration on their IMC Access Policy. If Inband is in use, perform the packet capture on the remote Syslog server instead, or reach out to TAC to perform it with internal debug commands.

Verify the configuration on the IMC access policy

In this example, the LED locator on a C220 M5 Integrated Server was enabled. This does not require downtime.

1. Verify which Fabric Interconnect sends out-of-band traffic for your server. The server IP is 192.0.2.5, so Fabric Interconnect A forwards its management traffic ("secondary route" means that the Fabric Interconnect acts as a proxy for the server management traffic):
   

   FI-6536-A(nx-os)# show ip interface mgmt 0
   
   IP Interface Status for VRF "management"(2)
   mgmt0, Interface status: protocol-up/link-up/admin-up, iod: 2,
   IP address: 192.0.2.3, IP subnet: 192.0.2.0/24 route-preference: 0, tag: 0 
   IP address: 192.0.2.5, IP subnet: 192.0.2.0/24 secondary route-preference: 0, tag: 0 

2. Start a packet capture on the appropriate Fabric Interconnect:
   

   FI-6536-A(nx-os)# ethanalyzer local interface mgmt capture-filter "port 514" limit-captured-frames 0
   Capturing on mgmt0

3. Navigate to Servers > Actions > System and choose Turn On Locator:
   Turn on LED locator in a Server
4. The console on the Fabric Interconnect must show the Syslog packet captured:
   

   FI-6536-A(nx-os)# ethanalyzer local interface mgmt capture-filter "port 514" limit-captured-frames 0 
   Capturing on mgmt0
   2025-01-16 22:34:27.552020 192.0.2.5 -> 192.0.2.2 Syslog AUTH.NOTICE: Jan 16 22:38:38 AUDIT[2257]: 192.0.2.5 CIMC Locator LED is modified to "ON" by User:(null) from Interface
   :redfish Remote IP:

5. The Syslog message must be logged in your remote server AUDIT.log file:
   

   root@alma jormarqu]# tail -n 1 /var/log/remote/msg/192.0.2.5/AUDIT.log
   Jan 16 22:38:38 192.0.2.5 AUDIT[2257]: CIMC Locator LED is modified to "ON" by User:(null) from Interface:

If Syslog packets were generated by UCS, but the Syslog server did not log them:

1. Confirm that the packets arrived at the remote Syslog server with a packet capture.
2. Verify the configuration of your remote Syslog server (including but not limited to: configured syslog port and firewall settings).

Related Information

• RFC 5424 - The Syslog Protocol
• Intersight IMM Expert Series - Syslog Policy
• Cisco Intersight Help Center - Configure UCS Domain Profile Policies
• Cisco Intersight Help Center - Configure Server Policies

[Thu Jan 16 23:12:10 root@C220-WZP22460WCD:~]$tcpdump -i bond0 port 514 -v 
tcpdump: listening on bond0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:12:39.817814 IP (tos 0x0, ttl 64, id 24151, offset 0, flags [DF], proto UDP (17), length 173)
  192.168.70.25.49218 > 10.31.123.134.514: Syslog, length: 145
    Facility auth (4), Severity notice (5)
Msg: Jan 16 23:12:39 C220-WZP22460WCD AUDIT[2257]: CIMC Locator LED is modified to "OFF" by User:(null) from Interface:redfish Remote IP:127.0.0.1