Ultra-M: Troubleshoot and Monitor Provisioning IP Address Assignment from OSPD

Introduction

This document describes how to monitor and troubleshoot the IP address assignment from OpenStack Platform Director (OSPD) to openstack compute nodes in Ultra-M.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• StarOs
• Ultra-M base architecture

Components Used

The information in this document is based on Ultra 5.1.x release. 

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

For the Ultra-M solution in place, Cisco UCS-C servers are being used.

In Ultra-M setup, during the under-cloud install process, the OSPD assigns the IP address to the compute node provisioning interface.

This interface is used for Pre-Boot Execution Environment  (during the introspection phase).

Abbreviations

These abbreviations are used in this article

Problem

In Cisco's Ultra-M solution, during the under-cloud install process, the OSPD assigns the IP address to the compute node provisioning interface.  This interface is used for PXE boot (during the introspection phase). If this interface is not reachable or Ip address is not properly assigned, the introspection fails and overall under-cloud install also fails.

Solution

If the UCS is stuck in booting while waiting for the DHCP offer from the DHCP server, follow these steps to troubleshoot.

Step 1. Identify on which catalyst port the UCS is connected and verify port is up and running (show int gig x/x/x -> check it is UP).

Step 2. While introspection runs, verify that the arp entry on the switches is correct as well as that the number of the packets increases.

show int gig x/x/x | grep packet -> verify that the input/output packet increase (that means UCS is sending and receiving packets). 

Step 3. If the output from step 2 is fine, check the config on the OSPD. The pool is configured in the undercloud.conf for 5.1 based release.

dhcp_start = 192.x.y.101
dhcp_end = 192.x.y.150 

As of 6.0 releases, the configuration for the dhcp is part of the vim-orch configuration file.

provisioning-network dhcp-ip-range start 192.x.y.101
provisioning-network dhcp-ip-range end 192.x.y.150
provisioning-network inspection-ip-range start 192.x.y.201
provisioning-network inspection-ip-range end 192.x.y.250

Step 4. Now, you can check further in the /var/log/messages if there are any failures.

For example, if so, you could see something like this:

"Dec 20 13:33:51 dnucs001-ospd dnsmasq-dhcp[5632]: DHCPDISCOVER(tap96912f54-41) 38:0e:4d:9c:ba:36 no address available"

This means that OSPD failed to allocate the IP to the UCS (and the introspection process would not be completed). 

Note that OSPD expects the DHCPDISCOVER as the first message. If you see in the traces that only BOOTP is received, then OSPD would not proceed further in allocating the address. This would be the case if there is a router in between the UCS Servers and the OSPD (non-standard UltraM design) and there is no IP helper-address OSPD-IP configured on the interfaces facing the UCS servers.

C3750#show run inter vlan x
Building configuration...

Current configuration : 140 bytes
!
interface Vlan3
 description CIMC/IPMI & OC External/VirtIO subnets
 ip address 192.x.y.254 255.255.0.0
 ip helper-address 192.x.y.1 <<<<<<<<<<< ADD THIS ONE POINTING AT OSPD's IP ADDRESS IN PROV/DHCP VLAN/SUBNET
end

Step 5. Now, further check if the dhcp process is fully operational and running and if there is actually available IP's.

(be aware of the RH bug: 1301659 )

[stack@bru-ospd-ultram-1 ~]$ systemctl list-units | grep dns  openstack-ironic-inspector-dnsmasq.service                                                                              loaded active running   PXE boot dnsmasq service for Ironic Inspector

And in details - to see actual IP addresses assigned to the specific MACs:

[stack@bru-ospd-ultram-1 ~]$ sudo systemctl status openstack-ironic-inspector-dnsmasq.service
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
 Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
 Active: active (running) since Thu 2017-12-14 13:43:08 CET; 6 days ago
 Process: 17511 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=0/SUCCESS)
 Main PID: 17513 (dnsmasq)
 CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service
 └─17513 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf

Dec 14 15:31:53 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.207 38:0e:4d:9c:97:c2
Dec 14 15:31:53 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.207 38:0e:4d:9c:97:c2
Dec 14 15:32:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.206 38:0e:4d:9c:a4:46
Dec 14 15:32:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.206 38:0e:4d:9c:a4:46
Dec 14 15:32:17 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.209 38:0e:4d:9c:9c:42
Dec 14 15:32:17 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.209 38:0e:4d:9c:9c:42
Dec 14 15:32:34 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.208 38:0e:4d:9c:a3:6e
Dec 14 15:32:34 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.208 38:0e:4d:9c:a3:6e
Dec 14 15:32:39 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.207 38:0e:4d:9c:97:c2
Dec 14 15:32:39 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.207 38:0e:4d:9c:97:c2

Same output in a different format can be seen also with sudo journalctl -u openstack-ironic-inspector-dnsmasq.

(as per the RH docs: Troubleshoot director issues )

[stack@bru-ospd-ultram-1 ~]$ sudo journalctl -u openstack-ironic-inspector-dnsmasq
-- Logs begin at Mon 2017-12-04 10:26:05 CET, end at Thu 2017-12-21 10:30:36 CET. --
Dec 12 18:02:19 bru-ospd-ultram-1.cisco.com systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Dec 12 18:02:19 bru-ospd-ultram-1.cisco.com dnsmasq[1105]: dnsmasq: unknown interface br-ctlplane
Dec 12 18:02:19 bru-ospd-ultram-1.cisco.com systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=
Dec 12 18:02:19 bru-ospd-ultram-1.cisco.com systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Dec 12 18:02:19 bru-ospd-ultram-1.cisco.com systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Dec 12 18:02:19 bru-ospd-ultram-1.cisco.com systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.
-- Reboot --
Dec 13 17:56:31 bru-ospd-ultram-1.cisco.com systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Dec 13 17:56:31 bru-ospd-ultram-1.cisco.com dnsmasq[1108]: dnsmasq: unknown interface br-ctlplane
Dec 13 17:56:31 bru-ospd-ultram-1.cisco.com systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=
Dec 13 17:56:31 bru-ospd-ultram-1.cisco.com systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Dec 13 17:56:31 bru-ospd-ultram-1.cisco.com systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Dec 13 17:56:31 bru-ospd-ultram-1.cisco.com systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.
Dec 14 13:43:08 bru-ospd-ultram-1.cisco.com systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Dec 14 13:43:08 bru-ospd-ultram-1.cisco.com dnsmasq[17513]: started, version 2.66 DNS disabled
Dec 14 13:43:08 bru-ospd-ultram-1.cisco.com dnsmasq[17513]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no
Dec 14 13:43:08 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCP, IP range 192.x.y.201 -- 192.x.y.250, lease time 2m
Dec 14 13:43:08 bru-ospd-ultram-1.cisco.com systemd[1]: Started PXE boot dnsmasq service for Ironic Inspector.
Dec 14 15:26:56 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPDISCOVER(br-ctlplane) 84:3d:c6:99:2e:de
Dec 14 15:26:56 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPOFFER(br-ctlplane) 192.x.y.201 84:3d:c6:99:2e:de
Dec 14 15:26:59 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPDISCOVER(br-ctlplane) 84:3d:c6:98:d3:78
Dec 14 15:26:59 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPOFFER(br-ctlplane) 192.x.y.202 84:3d:c6:98:d3:78
Dec 14 15:26:59 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.201 84:3d:c6:99:2e:de
Dec 14 15:26:59 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.201 84:3d:c6:99:2e:de
Dec 14 15:27:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPDISCOVER(br-ctlplane) 84:3d:c6:10:25:92
Dec 14 15:27:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPOFFER(br-ctlplane) 192.x.y.203 84:3d:c6:10:25:92
Dec 14 15:27:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.202 84:3d:c6:98:d3:78
Dec 14 15:27:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.202 84:3d:c6:98:d3:78
Dec 14 15:27:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPREQUEST(br-ctlplane) 192.x.y.203 84:3d:c6:10:25:92
Dec 14 15:27:02 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPACK(br-ctlplane) 192.x.y.203 84:3d:c6:10:25:92
Dec 14 15:27:06 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPDISCOVER(br-ctlplane) 84:3d:c6:98:cd:24
Dec 14 15:27:06 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPOFFER(br-ctlplane) 192.x.y.204 84:3d:c6:98:cd:24
Dec 14 15:27:09 bru-ospd-ultram-1.cisco.com dnsmasq-dhcp[17513]: DHCPDISCOVER(br-ctlplane) 84:3d:c6:98:d0:f0
[stack@bru-ospd-ultram-1 ~]$