OpenShift 4 on FlashStack User Provisioned Infrastructure

OpenShift 4 on FlashStack User Provisioned Infrastructure

Deployment Guide for OpenShift on FlashStack with Cisco UCS 6400 Fabric Interconnect and Pure Storage

FlashArray//X70 R2

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Published: August 2020

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

In partnership with:

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2020 Cisco Systems, Inc. All rights reserved.

 

Table of Contents

Executive Summary. 4

Introduction. 4

Audience. 4

Purpose of this Document 4

Solution Summary. 5

Deployment Hardware and Software. 7

Software Revisions. 7

Configuration Guidelines. 8

Create OpenShift User Provisioned Infrastructure. 10

Prepare OpenShift User Provisioned Infrastructure Deployment 10

Create vSphere Cluster 10

Create Deployment Node. 17

Create and Prepare Installation Files. 24

Install and Create the Ignition Configuration Files on Mgmt-host 27

Configure External Dependencies. 29

Configure DNS. 30

Configure Load Balancer 30

Prepare OCP Nodes. 32

Create Template. 32

Create Bootstrap Node. 40

Deploy and Configure OpenShift Container Platform Cluster 57

Power on Nodes. 58

Appendix. 69

DNS Entries. 69

About the Authors. 74

Acknowledgements. 74

 

 

Cisco Validated Designs (CVDs) consist of systems and solutions that are designed, tested, and documented to facilitate and improve customer deployments. These designs incorporate a wide range of technologies and products into a portfolio of solutions that have been developed to address the business needs of our customers.

This document details the design described in the Design Guide for OpenShift 4.3 on FlashStack User Provisioned Infrastructure, which showed a validated converged infrastructure jointly developed by Cisco and Pure Storage.  In this solution we will walk through the deployment of a predesigned, best-practice data center architecture with OpenShift 4.3 deployed on VMware vSphere built on the Cisco Unified Computing System (Cisco UCS), the Cisco Nexus® 9000 family of switches, and Pure Storage FlashArray//X R2 all flash storage configured for iSCSI based storage access.

When deployed, the architecture presents a robust infrastructure viable for a wide range of application workloads implemented as containers.
Solution Overview

Introduction

In the current industry there is a trend for pre-engineered solutions which standardize the data center infrastructure, offering the business operational efficiencies, agility, and scale to address cloud, bi-modal IT, and their business.  Their challenge is complexity, diverse application support, efficiency, and risk; all these are met by FlashStack with:

·       Reduced complexity and automatable infrastructure and easily deployed resources

·       Robust components capable of supporting high performance and high bandwidth virtualized applications

·       Efficiency through optimization of network bandwidth and in-line storage compression with de-duplication

·       Risk reduction at each level of the design with resiliency built into each touch point throughout

Cisco and Pure Storage have partnered to deliver this Cisco Validated Design, combining storage, server, and network components to serve as the foundation for virtualized workloads, enabling efficient architectural designs that can be quickly and confidently deployed. 

Audience

The intended audience for this document includes, but is not limited to, DevOps managers, IT infrastructure managers, application development leaders, business digital transformation leaders, storage and data management managers, sales engineer and architects working with hybrid and private clouds, and other parties that are looking for a tested, market-proven CI solution that offers flexibility and simplicity in support of their cloud native and application modernization needs along with their digital transformation journey.

Purpose of this Document

This document details a step-by-step configuration and implementation guide for deploying OpenShift on the FlashStack solution.  This will cover the provisioning on the deployment, bootstrap, master, and work nodes. This will also cover the configuration of DCHP, DNS, and Load Balancer entries required to support this solution but will not cover the overall deployment of these external resources. The details for deploying the underlying FlashStack solution, including configuration for the Cisco UCS, Cisco Nexus, Pure Storage FlashArray//X70 R2, and VMware vSphere can be found in FlashStack Virtual Server Infrastructure with iSCSI Storage for VMware vSphere 6.7 available here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flashstack_vsi_iscsi_vm67_u1.html

Solution Summary

The FlashStack Virtual Server Infrastructure is a validated reference architecture, collaborated on by Cisco and Pure Storage, built to serve enterprise data centers.  The solution is built to deliver a VMware vSphere based environment, leveraging the Cisco Unified Computing System (Cisco UCS), Cisco Nexus switches, and Pure Storage FlashArray.

The architecture brings together a simple, wire once solution that is SAN booted from iSCSI and is highly resilient at each layer of the design. This creates an infrastructure that is ideal for a variety of virtualized and containerized application deployments that can reliably scale when growth is needed.

Figure 1 shows the base physical architecture used in FlashStack Virtual Server Infrastructure.

The reference hardware configuration includes:

·       Two Cisco Nexus 9336C-FX2 Switches

·       Two Cisco UCS 6454 Fabric Interconnects

·       Cisco UCS 5108 Chassis with two Cisco UCS 2408 Fabric Extenders

·       Three Cisco UCS B200 M5 Blade Servers

·       One Pure Storage FlashArray//X70 R2

The virtual environment this supports is within VMware vSphere 6.7 U3 and includes virtual management and automation components from Cisco and Pure Storage built into the solution, or as optional add-ons.

This document assumes that this environment is deployed based on the FlashStack Virtual Server Infrastructure with iSCSI Storage deployment guide.

Figure 2 shows the base logical computing architecture used to deploy the OpenShift Container Platform (OCP) on the User Provisioned FlashStack VSI.  It shows the Bootstrap, Management, and Work nodes deployed as virtual machines on the physical environment displayed in Figure 1.

Software Revisions

Table 1 lists the software versions for hardware and virtual components used in this solution.  Each of these versions have been used have been certified within interoperability matrixes supported by Cisco, Pure Storage, and VMware.  For more current supported version information, consult the following sources:

·       FlashStack Compatibility Matrix: https://support.purestorage.com/FlashStack/Product_Information/FlashStack_Compatibility_Matrix

·       Cisco UCS Hardware and Software Interoperability Tool: http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html

·       Pure Storage Interoperability(note, this interoperability list will require a support login form Pure):  https://support.purestorage.com/FlashArray/Getting_Started/Compatibility_Matrix

·       VMware Compatibility Guide: http://www.vmware.com/resources/compatibility/search.php

Additionally, it is also strongly suggested to align FlashStack deployments with the recommended release for the Cisco Nexus 9000 switches used in the architecture:

·       Cisco Nexus: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/recommended_release/b_Minimum_and_Recommended_Cisco_NX-OS_Releases_for_Cisco_Nexus_9000_Series_Switches.html

Table 1    Software Revisions

Configuration Guidelines

This document details the step-by-step configuration of a fully redundant and highly available Kubernetes Container Infrastructure built on Cisco, Pure Storage, and RedHat components.  References are made to which component is being configured with each step, either 01, 02, and so on. For example, Master-01 and Master-02 are used to identify the two of the OCP Master Nodes within the cluster that is provisioned with this document, and Cisco Nexus A or Cisco Nexus B identifies the pair of Cisco Nexus switches that are configured. Additionally, this document details the steps for provisioning multiple Cisco UCS hosts, and these examples are identified as: OCP-VM-Host-iSCSI-01, OCP-VM-Host-iSCSI-02, etc to represent iSCSI booted infrastructure and production hosts deployed to the fabric interconnects in this document. Finally, to indicate that you should include information pertinent to your environment in a given step, <<text>> appears as part of the command structure. See the following example during a configuration step for both Cisco Nexus switches:

AA12-9336C-A&B (config)# ntp server <<var_oob_ntp>> use-vrf management

This document is intended to enable you to fully configure the customer environment. In this process, various steps require you to insert customer-specific naming conventions, IP addresses, and VLAN schemes, as well as to record appropriate MAC addresses. Table 2 lists the VLANs necessary for deployment as outlined in this guide, and Table 3 lists the virtual machines (VMs) necessary for deployment as outlined in this guide.

Table 2    Required VLANs

Table 3    Infrastructure Servers and FlashStack Components

Table 4    OpenShift Variables

Prepare OpenShift User Provisioned Infrastructure Deployment

To prepare the Cisco UCS, Cisco Nexus, Pure Storage FlashArray//X R2, and VMware vSphere environment as a User Provisioned Infrastructure for this deployment, follow the FlashStack for vSphere 6.7 deployment guide located here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flashstack_vsi_iscsi_vm67_u1.html

Create vSphere Cluster

Create vSphere Data Center

To create a vSphere data center, follow these steps:

1.     Log into the vCenter Web Console.

2.     Select Host and Clusters.

3.     Right-click the vCenter icon and select New Datacenter… from the drop-down list.

4.     From the New Datacenter pop-up dialogue enter in a Datacenter name and click OK.

Create vSphere Cluster

To create a vSphere cluster, follow these steps:

1.     Right-click the Datacenter icon and select New Cluster… from the drop-down list.

2.     From the New Cluster pop-up dialogue enter in a Cluster name, enabled DRS and HA, and click OK.

Add ESXi Host

To add an ESXi host, follow these steps:

1.     From the Cluster context select Configuration > Quick Start and click Add Host.

2.     Enter the host IP/FQDN, username, and password. Click Next. Multiple host can be added at once.

3.     Verify the SHA1 Thumbprint and click OK.

4.     Confirm host Summary and click Next.

5.     Click Finish.

The vSwitch, virtual distributed switch, iSCSI, and other host configurations are identical to those found in the FlashStack Virtual Server Infrastructure with iSCSI Storage for VMware vSphere 6.7 U1.

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flashstack_vsi_iscsi_vm67_u1.html#_Toc16591962

Create Infrastructure Datastore (vCenter Plug-in)

A shared datastore will need to be created to be used by the BootStrap, Master, and Worker Virtual Machines. This will need to be configured to be used by the ESXi host used for this cluster.

1.     In the vCenter Web Client click Host and Clusters.

2.     Right-click the OCP Cluster and click Pure Storage.

3.     Right-click the OCP Cluster and click Pure Storage -> Create Datastore.

4.     Set type to VMFS 6, Datastore name to <<var_shared_ds_name>>, and Cluster to the OCP Cluster.

5.     Select Create.

Create Deployment Node

Before deploying OpenShift Container Platform, you need to download the installation files to a locate computer. This must be Linux or macOS and requires 500 MB of free disk space.

Create RHEL VM (optional)

To create a RHEL virtual machine, follow these steps:

1.     Click Actions > New Virtual Machine…

2.     Click Create a new virtual machine and click Next.

3.     Click the compute cluster you wish to deploy this VM.

4.     Click the Datastore you wish to use.

5.     Set Compatibility to ESXi 6.7.

6.     Click the hardware for this VM.

7.     Review the settings and click Finish.

Install OCP Installation Program

To install OCP installation program, follow these steps:

1.     Open a web browser to the OpenShift download portal: cloud.redhat.com/openshift/install

2.     Click Run on VMware vSphere.

3.     Select the appropriate OS for your installation computer and click Download.

4.     Copy your pull secret for later.

5.     Download RHCOS if you do not have it.

6.     Download the OpenShift command-line tool.

Create and Prepare Installation Files

For installations of OpenShift Container Platform that use user-provisioned infrastructure, we must manually generate the installation configuration file after the OCP installation program and the access token for the cluster are obtained.

To create installation configuration file, follow these steps:

1.     Create an installation directory on the management host to store your required installation assets in:

2.     Customize the install-config.yaml file template and save it in the <installation_directory>.

3.     Manually create a file and name this configuration file install-config.yaml.

4.     For the install-config.yaml, if required, enter the following input:

·       base domain

·       OCP cluster id

·       OCP pull secret

·       ssh public key (~/.ssh/id_rsa.pub)

·       vCenter host

·       vCenter user

·       vCenter password

·       vCenter datacenter

·       vCenter datastore

Sample install-config.yaml File for VMware vSphere

Customize the install-config.yaml file to specify more details about your OpenShift Container Platform cluster’s platform or modify the values of the required parameters shown in in red font below to suit your environment.

Example File

File Used for the deployment:

The required parameters displayed in red in the above file are described below simultaneously in the same order as they are listed in the file.

·       baseDomain: The base domain of the cluster. All DNS records must be sub-domains of this base and include the cluster name.

·       hyperthreading: Whether to enable or disable simultaneous multithreading, or hyperthreading. By default, simultaneous multithreading is enabled to increase the performance of your machines' cores.

·       replicas: You must set the value of the replica’s parameter to 0. This parameter controls the number of workers that the cluster creates and manages for you, which are functions that the cluster does not perform when you use user-provisioned infrastructure. We will manually deploy worker machines for the cluster to use before you finish installing OpenShift Container Platform.

·       replicas: The number of control plane machines that you add to the cluster. Because the cluster uses this value as the number of etcd endpoints in the cluster, the value must match the number of control plane machines that you deploy

·       name: cluster name that you specified in your DNS records.

·       vcenter: The fully qualified host name or IP address of the vCenter server.

·       username: The name of the user for accessing the server. This user must have at least the roles and privileges that are required for static or dynamic persistent volume provisioning in vSphere.

·       password: The password associated with the vSphere user.

·       datacenter: The vSphere datacenter.

·       defaultDatastore: The default vSphere datastore to use.

·       fips: Whether to enable or disable FIPS mode. By default, FIPS mode is not enabled. If FIPS mode is enabled, the Red Hat Enterprise Linux CoreOS (RHCOS) machines that OpenShift Container Platform runs on bypass the default Kubernetes cryptography suite and use the cryptography modules that are provided with RHCOS instead.

·       pullSecret: The pull secret that is obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

·       sshKey: The public portion of the default SSH key for the core user in Red Hat Enterprise Linux CoreOS (RHCOS)

1.     Back up the install-config.yaml file so that it can be used to install multiple clusters.

2.     The install-config.yaml file is consumed during the next step of the installation process. The file can be backed up now using the following command.

Install and Create the Ignition Configuration Files on Mgmt-host

The openshift-installer obtained from OpenShift Infrastructure Providers was run to create the Ignition configuration files. The openshift-installer expects the YAML formatted file that was created in the above step (install-config.yaml) in order to generate the cluster configuration information.

To prepare the OCP Cluster installation, follow these steps:

Creating the Kubernetes manifest and Ignition config files

Since we must modify some cluster definition files and manually start the cluster machines, we must generate the Kubernetes manifest and Ignition config files that the cluster needs to make its machines.

1.     Generate the Kubernetes manifests for the cluster which defines the objects bootstrap nodes will have to create initially:

2.     For <<var_installation_directory>>, specify the installation directory that contains the install-config.yaml file that was created, else change into the directory.

3.     Modify the manifests/cluster-scheduler-02-config.yml Kubernetes manifest file to prevent Pods from being scheduled on the control plane machines:

a.     Open the manifests/cluster-scheduler-02-config.yml file.

b.     Locate the “masters Schedulable” parameter and set its value to “False”.

c.     Save and exit the file.

4.     Create the Ignition config files. Ignition is the utility that is used by RHCOS to manipulate disks during initial configuration. It completes common disk tasks, including partitioning disks, formatting partitions, writing files, and configuring users. On first boot, Ignition reads its configuration from the installation media or the location specified and applies the configuration to the machines.

5.     The following files are generated in the directory:

Copy Bootstrap Ignition File to the HTTP Server

To copy the bootstrap ignition file to the HTTP server, follow these steps:

1.     Change permissions and copy the generated bootstrap.ign file to HTTP server, ensure that the file can be downloaded with http:

2.     To verify the download is successful from your http server, run:

3.     Save the following secondary Ignition config file for your bootstrap node to your computer as <installation_directory>/append-bootstrap.ign.

4.     Convert the master, worker, and secondary bootstrap Ignition config files to Base64 encoding.

Configure External Dependencies

OpenShift Container Platform requires several DNS and a Layer 4 Load Balancer to be configured outside of the cluster to be used.  This section explains the DNS and load balancer entries required to deploy the cluster.

Configure DNS

A domain name service (DNS) is required for access to the cluster as discussed earlier. This should use existing domain name servers for production deployments. A Windows DNS server was used in the validated environment.  The configuration for this is shown in the Appendix.

Table 5    Required DNS Entries

Configure Load Balancer

Before you install OpenShift Container Platform, you must provision two layer-4 load balancers. The API requires one load balancer and the default Ingress Controller needs the second load balancer to provide ingress to applications.

The validated environment used an external load balancer running HAproxy to offer a single-entry point for the many Red Hat OpenShift Container Platform components. Organizations can provide their own currently deployed load balancers if the service already exists.

The load balancer (haproxy) available with the RHEL distribution was used to create the haproxy server. The configuration files used for creating the haproxy server are listed in the Appendix.

To configure the load balancer, follow these steps:

1.     Install the haproxy operating system package using the yum or rpm command:

2.     Update the configuration files (/etc/haproxy/haproxy.cfg), as listed in the Appendix.

3.     Start or Restart the haproxy service:

4.     Add the following firewall rules to allow clients connection to access HAProxy server:

5.     Optionally, the Firewall can be stopped and disabled for system startup using the following commands:

6.     The output of the following commands should display the status as “Active: active (running)”, without any errors and the load balancer needs to be configured with the values as follows:

7.     The Status of all the previously installed services can be verified using the following commands before proceeding with the OpenShift Container Platform cluster installation:

Prepare OCP Nodes

Create Template

Prior to installing the OCP cluster on VMware vSphere, you need to create RHCOS machines on the vSphere hosts.

To create the VM template using RHCOS OVA, follow these steps:

1.     Obtain the RHCOS OVA image from the Product Downloads page on the Red Hat customer portal or the RHCOS image mirror page, https://cloud.Red Hat.com/openshift/install/vsphere/user-provisioned

2.     The file name contains the OpenShift Container Platform version number in the format rhcos-<version>-vmware.<architecture>.ova. (for example: rhcos-4.3.8-x86_64-vmware.x86_64.ova)

3.     In the vSphere Client, create a template for the OVA image.

In the following steps, use the same template for all cluster virtual machines when you provision:

1.     From the Hosts and Clusters tab, right-click your cluster’s name <vCenter_Cluster> and click Deploy OVF Template.

2.     From the Select an OVF tab, specify the name of the RHCOS OVA file that you downloaded.

3.     From the Select a name and folder tab, set a Virtual machine name, such as RHCOS, click the name of your vSphere cluster.

4.     From the Select a compute resource tab, click the name of your vSphere cluster.

5.     Click Next.

6.     From the Select storage tab, configure the storage options for your VM. Select Thin Provision and the datastore <OCP_Infra_1> that you specified in your install-config.yaml file.

7.     From the Select network tab, specify the network <OCC-VLAN> previously configured for the OCP cluster.

8.     Since you will use the same template for all cluster machine types, do not specify the values on the Customize template tab.

9.     Review the details and select FINISH if correct.

10.  In the vSphere Client, create a folder in your datacenter to store your VMs.

11.  Click the VMs and Templates view.

12.  Right-click the name of your data center.

13.  Click New Folder, then click New VM and Template Folder.

 

14.  In the window that is displayed, enter the folder name. The folder name must match the cluster name that you specified in the install-config.yaml file.

Create Bootstrap Node

To create the bootstrap node, follow these steps:

1.     Right-click the template’s name and click Clone, then click Clone to Virtual Machine.

2.     On the Select a name and folder tab, specify a name for the VM as bootstrap. Select the name of the folder that you created for the cluster.

3.     On the Select a compute resource tab, select the name of a host in your data center.

4.     On Select storage, select the datastore specified in the install-config.yaml.

5.     On the Select clone options, select Customize this virtual machine’s hardware.

6.     Customize the Virtual Hardware for 4 vCPU, 16 GB RAM, and 120 GB Hard disk 1.

7.     On the Customize hardware tab, click VM Options, then click Advanced.

8.     Click Edit Configuration then from the Configuration Parameters window, click Add Configuration Params. Define the following parameter names and values:

-      guestinfo.ignition.config.data: Paste the contents of the base64-encoded append-bootstrap.64 Ignition config.

-      guestinfo.ignition.config.data.encoding: Specify base64.

-      disk.EnableUUID: Specify TRUE.

9.     Click OK to return to Customize hardware.

10.  Click Next to move to Customize vApp properties.

11.  Click Next to move to Ready to Complete.

12.  Click Finish.

Create Master Node

To create the master node, follow these steps:

1.     Right-click the template’s name and click Clone, then click Clone to Virtual Machine.

 

2.     On the Select a name and folder tab, specify a name for the VM as master<n>. Select the name of the folder that you created for the cluster.

3.     On the Select a compute resource tab, select the name of a host in your data center.

4.     On Select storage, select the datastore specified in the install-config.yaml.

5.     On the Select clone options, select Customize this virtual machine’s hardware.

 

6.     Customize the Virtual Hardware for 4 vCPU, 16 GB RAM, and 120 GB Hard disk 1.

7.     On the Customize hardware tab, click VM Options, then click Advanced.

 

8.     Click Edit Configuration, and on the Configuration Parameters window, click Add Configuration Params. Define the following parameter names and values:

-      guestinfo.ignition.config.data: Paste the contents of the base64-encoded master.64 Ignition config.

-      guestinfo.ignition.config.data.encoding: Specify base64.

-      disk.EnableUUID: Specify TRUE.

9.     Click OK to return to Customize hardware.

10.  Click Next to move to Customize vApp properties.

11.  Click Next.

12.  Click Finish.

13.  Repeat steps 1-12 for Master nodes 0, 1, and 2.

Create Worker Node

To create a worker node, follow these steps:

1.     Right-click the template’s name and click Clone and then click Clone to Virtual Machine.

2.     From the Select a name and folder tab, specify a name for the VM as worker0. Select the name of the folder that you created for the cluster.

3.     From the Select a compute resource tab, select the name of a host in your data center.

4.     On Select storage, select the datastore specified in the install-config.yaml.

5.     On the Select clone options, select Customize this virtual machine’s hardware.

6.     Customize the Virtual Hardware for 2 vCPU, 8 GB RAM, and 120 GB Hard disk 1.

7.     On the Customize hardware tab, click VM Options and then click Advanced.

8.     Click Edit Configuration and from the Configuration Parameters window, click Add Configuration Params. Define the following parameter names and values:

-      guestinfo.ignition.config.data: Paste the contents of the base64-encoded worker.64 Ignition config.

-      guestinfo.ignition.config.data.encoding: Specify base64.

-      disk.EnableUUID: Specify TRUE.

9.     Click OK to return to Customize hardware.

10.  Click Next to move to Customize vApp properties.

11.  Click Next to move to Ready to Complete.

12.  Click Finish.

13.  Repeat steps 1-13 for worker nodes 0, 1, 2, and 3.

Update DHCP Records

Bootstrap, Master, and Worker nodes need to be assigned the correct IP addresses on boot. DHCP reservations need to be configured to match the MAC addresses of these nodes.  To obtain the MAC address for each VM, follow these steps:

1.     Select the Virtual Machine.

2.     Under VM Hardware, expand Network Adapter 1.

3.     Record the value of the MAC Address and update the DHCP record for this host.

4.     Repeat steps 1-3 for all Bootstrap, Master, and Worker nodes.

Configure Host Rules

A Host rule will be created to ensure that Master nodes are running on different physical host.  This is done to ensure that the high availability provided by using three (3) master nodes is also provided at the hardware layer.

To configure the host rules, follow these steps:

1.     Select the OCP Cluster.

2.     Select Configure.

3.     Select VM/Host Rules.

4.     Select Add.

5.     Provide a Name, set type to Separate Virtual Machines, select Add.

6.     Add Master Nodes 0-2.

7.     Click OK.

8.     Click OK.

Deploy and Configure OpenShift Container Platform Cluster

This section explains the deployment of the OpenShift Container Platform cluster and the post deployment configuration to install Pure Service Orchestrator to provide Persistent Volumes (PV) and Persistent Volume Claims (PVC).

Power on Nodes

To power on nodes, follow these steps:

1.     Select the bootstrap VM.  Click Actions > Power > Power On. Repeat this step for all master and worker nodes.

2.     Monitor cluster creation.

Log into the Cluster

You can log into your cluster as a default system user by exporting the cluster kubeconfig file. The kubeconfig file contains information about the cluster that is used by the CLI to connect a client to the correct cluster and API server. The file is specific to a cluster and is created during OpenShift Container Platform installation.

To log into the cluster, follow these steps:

1.     Export the kubeadmin credentials:

2.     Verify you can run oc commands:

Complete OCP Installation

To complete the OCP installation, follow these steps:

1.     Verify the cluster components are online:

2.     Confirm Kubernetes API server is communicating with the Pods:

Add iSCSI Network Adapters to Worker Nodes

To add iSCSI network adapter on each worker node, follow these steps:

1.     Access FlashSTack vCenter.

2.     Right-click a worker node in the inventory and select Edit Settings.

3.     Click ADD NEW DEVICE.

4.     Select Network Adapter from the drop-down list.

The new network adapter appears at the bottom of the device list.

5.     Expand New Network and check the boxes against both Connected and Connected at power on.

6.     From the drop-down list next to the New Network label, select the iSCSI-A port group.

7.     Click OK.

8.     Click ADD NEW DEVICE again.

9.     Select Network Adapter from the drop-down list.

10.  The new network adapter appears at the bottom of the device list.

11.  Expand New Network and check the boxes against Connected and Connected at power on.

12.  From the drop-down list next to the New Network label, select the iSCSI-B port group.

13.  Repeat steps 1-12 for all the worker nodes.

14.  Note down the MAC addresses of the newly created adapters on all the worker nodes. The MAC addresses will be used to assign IP addresses from storage VLAN to the adapters using machine config file.

To configure the new network adapters through defining new machineconfig, follow these steps:

1.     Create a new ifcfg text file which defines a HWADDR which corresponds to the MAC address of the adapter to be configured. Create one file for each adapter on all worker nodes.

2.     Run the following command to base64 encode the ifcfg file(s). 

3.     Create a new machineconfig yaml file which contains the base64 encoded ifcfg files.

4.     Append the base64 content after data:text/plain;charset=utf-8;base64,

5.     Create a new file object for each adapter which needs to be configured.

6.     The following is an example of how to configure two network adapters on each worker node in a cluster consisting of four worker nodes. A file was created with the name the-machine-config and the contents were updated as follows:

7.     Apply the machineconfig by running the following command:

8.     Wait for all impacted nodes to restart.

iSCSI Connectivity Configuration

To configure the iSCSI connectivity, follow these steps for each worker node:

1.     Obtain the iSCSI initiator name in the /etc/iscsi/initiatorname.iscsi file:

2.     Add the worker node as a host on the Pure Storage FlashArray//X via the Web console.

3.     Select Storage -> Host.

4.     Under host select + to create host.

5.     Enter a host name and click Create.

6.     Select the newly created host for the Hosts list.

7.     Select Configured IQNs… under Host Ports.

8.     Enter the IQN from the previous step and click Add.

9.     For the iSCSI initd script startup, set a session to automatic in /etc/iscsi/iscsid.conf: “node.startup = automatic”

10.  Enable multipathd:

Obtain FlashArray//X API Token

To obtain the FlashArray//X API token, follow these steps:

1.     Log into FlashArray//X Web Console.

2.     Click Settings > Users.

3.     Click the gear icon from your admin user and select Show API Token…

4.     Record the API Token:

Install Pure Service Orchestrator (PSO)

Pure Service Orchestrator installation must be run from a node that has the OpenShift Command-line interface installed. To install PSO, follow these steps:

1.     Clone the PSO installation files:

2.     Configure values.yaml located in ../helm-charts/operator-csi-plugin/ to match your FlashArray settings:

3.     Install PSO:

4.     Configured RBAC rules for PSO Operator:

Set Pure-Block as the Default Storage Class (Optional)

To set the Pure-Block as the default storage class, follow these steps:

1.     Log into OpenShift Cluster Web Console.

2.     Select Administrator.

3.     Click Storage >Storage Class.

4.     Select the Edit Annotations for Storage Class thin.

5.     Set storageclass.kubernetes.io/is-default-class to false.

6.     Click Edit Annotations for Storage Class pure-block.

7.     Set storageclass.kubernetes.io/is-default-class to true.

DNS Entries

The following are screenshots showing the DNS entries used in the validation of this deployment

Load Balancer Configuration

The following is the /etc/haproxy/haproxy.cfg file used in the validation of this deployment:

Pure Service Orchestrator Values Entries

The following is the /helm-charts/operator-csi-plugin/values.yaml file used in the validation of this deployment:

Allen Clark, Technical Marketing Engineer, Cisco Systems, Inc.

Allen Clark has over 16 years of experience working with enterprise storage and data center technologies.  As a member of various organizations within Cisco, Allen has worked with hundreds of customers on implementation and support of compute and storage products. Allen holds a bachelor’s degree in Computer Science from North Carolina State University and is a dual Cisco Certified Internetwork Expert (CCIE 39519, Storage Networking and Data Center)

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank:

·       Sreeni Edula, Technical Marketing Engineer, Cisco Systems, Inc.

·       Brian Everitt, Technical Marketing Engineer, Cisco Systems, Inc.

·       Craig Waters, Solutions Architecture / Product Management, Pure Storage, Inc.

·       Simon Dodsley, Principle Field Solutions Architect, Pure Storage, Inc.