FlexPod Datacenter with Oracle RAC Databases on Cisco UCS and NetApp AFF A-Series

Deployment Guide for Oracle RAC 12cR2 Databases on Cisco Unified Computing System and NetApp AFF A-700s Storage using Fibre Channel

FlexPod Datacenter with Oracle RAC Databases on Cisco UCS and NetApp AFF A-Series PDF

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Last Updated: January 10, 2019

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, visit:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2019 Cisco Systems, Inc. All rights reserved.

 

Table of Contents

Executive Summary. 6

Solution Overview.. 7

Introduction. 7

Audience. 7

Purpose of this Document. 7

FlexPod System Overview.. 8

What’s New in this Release?. 9

Solution Components. 11

Cisco UCS 5108 Blade Server Chassis. 11

Cisco UCS 2304 Fabric Extender. 11

Cisco UCS B200 M5 Blade Servers. 12

Cisco UCS Virtual Interface Card (VIC) 1340. 12

Cisco UCS 6332-16UP Fabric Interconnect. 12

Cisco UCS MDS 9148S Fabric Switch. 13

Cisco Nexus 93180LC-EX Switch. 13

NetApp AFF All Flash Array A700s. 13

Solution Design. 16

Physical Infrastructure. 16

Physical Topology. 16

Design Topology. 18

Solution Configuration. 21

Configure Cisco Nexus 93180LC-EX Switch. 21

Set Up Initial Configuration. 21

Configure Global Settings for Cisco Nexus A and Cisco Nexus B.. 23

Configure VLANs for Cisco Nexus A and Cisco Nexus B Switches. 24

Virtual Port Channel (vPC) Summary for Data and Storage Network. 24

Create vPC Peer-Link Between the Two Nexus Switches. 25

Create vPC Configuration Between Nexus 9372PX-E and Fabric Interconnects. 26

Verify All vPC Status Is Up on Both the Nexus Switches. 30

Cisco UCS Configuration Overview.. 33

Perform Initial Setup of Cisco UCS 6332-16UP Fabric Interconnects for a Cluster Setup. 33

Log in to Cisco UCS Manager. 35

Upgrade Cisco UCS Manager Software to Version 3.2. 36

Anonymous Reporting. 36

Configure Cisco UCS Call Home. 37

High Level Steps to Configure Base Cisco UCS.. 37

Set Fabric Interconnects to Fibre Channel End Host Mode. 38

Synchronize Cisco UCS to NTP.. 39

Configure Fabric Interconnect for Chassis and Blade Discovery. 39

Configure LAN and SAN on Cisco UCS Manager. 42

Configure IP, UUID, Server, MAC, WWNN and WWPN Pools. 49

Set Jumbo Frames in both the Cisco Fabric Interconnect. 54

Configure Server BIOS Policy. 55

Create Adapter Policy. 56

Configure Update Default Maintenance Policy. 58

Configure vNIC and vHBA Template. 58

Create Server Boot Policy for SAN Boot. 62

Configure and Create a Service Profile Template. 69

Create Service Profile Template. 69

Create Service Profiles from Template and Associate to Servers. 80

Create Service Profiles from Template. 80

Associate Service Profiles to the Servers. 81

Configure Cisco MDS 9148S Switch. 83

Configure feature for MDS Switch A and MDS Switch B.. 84

Configure VSANs for MDS Switch A and MDS Switch B.. 85

Create and Configure Fibre Channel Zoning. 86

Create Zoning. 88

Verify FC Ports on MDS Switch. 91

Configure NetApp AFF A700s Storage. 93

NetApp Storage Connectivity. 93

Operating System and Database Deployment. 96

Operating System Configuration. 96

Operating System Prerequisites for Oracle Software Installation. 98

Configure BIOS.. 98

Prerequisites Automatic Installation. 99

Additional Prerequisites Setup. 100

Configure Multipath Setup. 100

Oracle Database 12c GRID Infrastructure Deployment. 103

Install and Configure Oracle Database Grid Infrastructure Software. 105

Install Oracle Database Software. 113

Oracle Flex ASM... 117

Scalability Test and Results. 120

Hardware Calibration Test using FIO.. 120

4k and 8k Random Read/Write IOPs Tests. 120

Bandwidth Tests. 121

Database Creation with DBCA.. 123

SLOB Test. 124

Swingbench Test. 128

Resiliency and Failure Tests. 140

Summary. 151

Appendix. 152

References. 179

Cisco Unified Computing System... 179

Cisco UCS Data Center Design Guides. 179

FlexPod Converged Infrastructure. 179

Cisco UCS B200 M5 Servers. 179

Oracle Database 12c Release 2. 179

NetApp AFF A-Series All Flash Storage. 179

NetApp Support. 179

About the Authors. 180

Acknowledgements. 180

 

 

The Cisco Unified Computing System™ (Cisco UCS®) is a next-generation data center platform that unites computing, network, storage access, and virtualization into a single cohesive system. Cisco UCS is an ideal platform for the architecture of mission critical database workloads such as Oracle RAC. The combination of Cisco UCS, NetApp and Oracle Real Application Cluster Database architecture can accelerate your IT transformation by enabling faster deployments, greater flexibility of choice, efficiency, high availability and lower risk.

Cisco Validated Designs include systems and solutions that are designed, tested, and documented to facilitate and improve customer deployments. These designs incorporate a wide range of technologies and products into a portfolio of solutions that have been developed to address the business needs of customers. Cisco and NetApp have partnered to deliver FlexPod, which serves as the foundation for a variety of workloads and enables efficient architectural designs that are based on customer requirements. A FlexPod solution is a validated approach for deploying Cisco and NetApp technologies as a shared cloud infrastructure.

The FlexPod Datacenter with NetApp All Flash AFF system is a converged infrastructure platform that combines best-of breed technologies from Cisco and NetApp into a powerful converged platform for enterprise applications. Cisco and NetApp works closely with Oracle to support the most demanding transactional and response-time-sensitive databases required by today’s businesses.

This Cisco Validated Design (CVD) describes the reference FlexPod Datacenter architecture using Cisco UCS and NetApp® All Flash AFF Storage for deploying a highly available Oracle RAC Database environment. This document shows the hardware and software configuration of the components involved, results of various tests and offers implementation and best practices guidance using Cisco UCS Compute Servers, Cisco Fabric Interconnect Switches, Cisco MDS Switches, Cisco Nexus Switches, NetApp AFF Storage and Oracle RAC Database.  

Introduction

Data powers essentially every operation in a modern enterprise, from keeping the supply chain operating efficiently to managing relationships with customers. Modern data centers face increasing demands for agile, high-performance service delivery. Digital transformations are driving an increased number of new applications, with more sources of data. Organizations of all kinds rely on their relational databases for both transaction processing (OLTP) and analytics (OLAP), but many still have challenges in meeting their goals of high availability, security, and performance. Applications must be able to move quickly from development to a reliable, scalable platform. An ideal solution integrates best-in-class components that can scale compute and storage independently to meet the needs of dynamic business requirements.

Like all the FlexPod Systems, the FlexPod Datacenter with NetApp All Flash AFF is comprised of compute (database, application and management servers from Cisco), network (three-layer network and SAN technologies from Cisco), and storage (NetApp All Flash AFF storage systems).

This CVD describes how the Cisco Unified Computing System™ (Cisco UCS®) can be used in conjunction with NetApp® AFF storage systems to implement a mission-critical applications such as an Oracle Real Application Clusters (RAC) 12cR2 database solution. This CVD documents validation of the real world performance, ease of management, and agility of the FlexPod Datacenter with Cisco UCS and All Flash AFF in high-performance Oracle RAC Databases environments.

Audience

The intended audience for this document includes, but is not limited to, sales engineers, field consultants, database administrators, IT managers, oracle database architects, and customers who want to deploy Oracle RAC 12cR2 database solution on FlexPod Converged Infrastructure with NetApp clustered Data ONTAP® and the Cisco UCS platform. A working knowledge of Oracle RAC Database, Linux, Storage technology, and Network is assumed but is not a prerequisite to read this document.

Purpose of this Document

Oracle RAC databases deployments are extremely complicated in nature and customers face enormous challenges in maintaining these landscapes in terms of time, efforts and cost.  Oracle RAC databases often manage the mission critical components of a customer’s IT department.  Ensuring availability while also lowering the IT TCO is always their top priority. This FlexPod solution for Oracle databases delivers industry-leading storage, unprecedented scalability, continuous data access, and automated data management for immediate responses to business opportunities.

The goal of this document is to determine the Oracle database server read latency, peak sustained throughput and IOPS of this FlexPod reference architecture system while running the Oracle OLTP and OLAP workloads.

This document provides a step-by-step configuration and implementation guide for the FlexPod Datacenter with Cisco UCS Compute Servers, Cisco Fabric Interconnect Switches, Cisco MDS Switches, Cisco Nexus Switches and NetApp AFF Storage to deploy an Oracle RAC Database solution. Here are the objectives we would like to accomplish in this reference document

1.        Provide reference FlexPod architecture design guidelines for the Oracle RAC Databases solution.

2.       Demonstrate simplicity and agility with the software-driven architecture and high performance of Cisco UCS compute Servers.

3.        Build, validate and predict performance of Servers, Network and Storage platform on a per workload basis.

FlexPod System Overview

Built on innovative technology from NetApp and Cisco, the FlexPod converged infrastructure platform meets and exceeds the challenges of simplifying deployments for best-in-class data center infrastructure. FlexPod is a defined set of hardware and software that serves as an integrated foundation for both virtualized and non-virtualized solutions. Composed of pre-validated storage, networking, and server technologies, FlexPod is designed to increase IT responsiveness to organizational needs and reduce the cost of computing with maximum uptime and minimal risk. Simplifying the delivery of data center platforms gives enterprises an advantage in delivering new services and applications.

FlexPod provides the following differentiators:

1.        Flexible design with a broad range of reference architectures and validated designs.

4.       Elimination of costly, disruptive downtime through Cisco UCS and NetApp® ONTAP®.

5.        Leverage a pre-validated platform to minimize business disruption and improve IT agility and reduce deployment time from months to weeks.

6.       Cisco Validated Designs (CVDs) and NetApp Validated Architectures (NVAs) covering a variety of use cases.

Cisco and NetApp have carefully validated and verified the FlexPod solution architecture and its many use cases while creating a portfolio of detailed documentation, information, and references to assist customers in transforming their data centers to this shared infrastructure model.

Figure 1  FlexPod System Overview

FlexPod datacenter architecture includes three components:

·         Cisco Unified Computing System (Cisco UCS)

·         Cisco MDS and Nexus Switches

·         NetApp AFF Storage Systems

One benefit of the FlexPod architecture is the ability to customize or "flex" the environment to suit a customer's requirements. A FlexPod can easily be scaled as requirements and demand change. The unit can be scaled both up (adding resources to a FlexPod unit) and out (adding more FlexPod units). This document highlights the resiliency, cost benefit, and ease of deployment of a Fibre Channel storage solution to deploy Oracle RAC Database environments on FlexPod Infrastructure.

What’s New in this Release?

This version of FlexPod CVD introduces new hardware with NetApp A-Series All Flash Storage AFF A700s along with Cisco UCS B200 M5 Blade Server featuring Intel Xeon Scalable Family of CPUs.

It incorporates the following features:

·         Support for the Cisco UCS 3.2 unified software release and Cisco UCS B200 M5 Blade Servers

·         Support for the latest release of NetApp ONTAP® 9.3

·         Fibre channel storage design

·         Validation of Oracle RAC Database 12c Release 2

Cisco UCS 5108 Blade Server Chassis

Cisco UCS 5108 Blade Server Chassis, is six rack units (6RU) high, can mount in an industry-standard 19-inch rack, and uses standard front-to-back cooling. A chassis can accommodate up to eight half-width or four full-width Cisco UCS B-Series Blade Servers form factors within the same chassis.

By incorporating unified fabric and fabric-extender technology, Cisco Unified Computing System eliminates the need for dedicated chassis management and blade switches, reduces cabling, and allowing scalability to 20 chassis without adding complexity. The Cisco UCS 5108 Blade Server Chassis is a critical component in delivering the simplicity and IT responsiveness for the data center as part of Cisco Unified Computing System.

Cisco UCS 2304 Fabric Extender

Cisco UCS 2304 Fabric Extender brings the unified fabric into the blade server enclosure, providing multiple 40 Gigabit Ethernet connections between blade servers and the fabric interconnect, simplifying diagnostics, cabling, and management.

The Cisco UCS 2304 connects the I/O fabric between the Cisco UCS 6300 Series Fabric Interconnects and the Cisco UCS 5100 Series Blade Server Chassis, enabling a lossless and deterministic Fibre Channel over Ethernet (FCoE) fabric to connect all blades and chassis together.

Cisco UCS B200 M5 Blade Servers

The Cisco UCS B200 M5 Blade Server delivers performance, flexibility, and optimization for deployments in data centers, in the cloud, and at remote sites. This enterprise-class server offers market-leading performance, versatility, and density without compromise for workloads including Virtual Desktop Infrastructure (VDI), web infrastructure, distributed databases, converged infrastructure, and enterprise applications such as Oracle Databases.

The Cisco UCS B200 M5 server can quickly deploy stateless physical and virtual workloads through programmable, easy-to-use Cisco UCS Manager Software and simplified server access through Cisco Single-Connect technology.

Cisco UCS Virtual Interface Card (VIC) 1340

The Cisco UCS Virtual Interface Card (VIC) 1340 is a 2-port, 40 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE)-capable modular LAN on motherboard (mLOM) mezzanine adapter.

Cisco UCS 1340 VIC delivers 80 Gbps throughput to the Server and helps reduce TCO by consolidating the overall number of NICs, HBAs, cables, and switches; LAN and SAN traffic runs over the same mezzanine card and fabric.

Cisco UCS 6332-16UP Fabric Interconnect

The 6332-16UP Fabric Interconnect is the management and communication backbone for Cisco UCS B-Series Blade Servers, C-Series Rack Servers, and 5100 Series Blade Server Chassis. It implements 20x40 Gigabit Ethernet and Fibre Channel over Ethernet ports, with additional support for 16 unified ports that can be configured to 1 or 10 Gbps Ethernet, or 4/8/16 Gbps Fibre Channel.

The Fabric Interconnect provides high-speed upstream connectivity to the network, or converged traffic to servers through its 40 Gbps ports, but also allows for Fibre Channel connectivity to SAN switches like the MDS, or alternately directly attached Fibre Channel to storage arrays.

Cisco UCS MDS 9148S Fabric Switch

The Cisco® MDS 9148S 16G Multilayer Fabric Switch is the next generation of highly reliable, flexible and low-cost Cisco MDS 9100 Series Switches. It provides up to 48 auto-sensing Fibre Channel ports, which are capable of speeds of 2, 4, 8, and 16 Gbps, with 16 Gbps of dedicated bandwidth for each port.

In all, the Cisco MDS 9148S is a powerful and flexible switch that delivers high performance and comprehensive Enterprise-class features at an affordable price.

Cisco Nexus 93180LC-EX Switch

The Cisco Nexus 93180LC-EX Switch is the industry’s first 50-Gbps-capable 1RU switch that supports 3.6 Tbps of bandwidth and over 2.6 bpps across up to 32 fixed 40- and 50-Gbps QSFP+ ports or up to 18 fixed 100-Gbps ports (Figure 3). The switch can support up to 72 10-Gbps ports using breakout cables. A variety of flexible port configurations are supported using templates.

NetApp AFF All Flash Array A700s

This solution includes the NetApp All flash fabric-attached storage AF 700s unified scale-out storage system for Oracle Database 12cR2.

Built on ONTAP software, AFF speeds up the operations required to meet your business requirements, without compromising efficiency or reliability, while providing great flexibility and scalability. As true enterprise-class, all-flash arrays, these systems accelerate, manage, and protect business-critical data, now and in the future.  

NetApp AFF provides industry-leading performance while continuing to provide a full suite of enterprise-grade data management and data protection features. Powered by NetApp clustered Data ONTAP, the AF 700s series unifies the SAN and NAS storage infrastructure.  Systems architects can choose from a range of models representing a spectrum of cost-versus-performance points. Every model, however, provides the following core benefits:

·         HA and fault tolerance. Non-disruptive operations are achieved through clustering, HA pairing of controllers, hot-swappable components, NetApp RAID-TEC® and disk protection (allowing two independent disk failures without data loss), and network interface redundancy.

·         Data Protection. Create thousands of instantaneous backups with NetApp Snapshot® copies, replication of both active data and snapshot backups with NetApp SnapMirror® software, and application backup integration with NetApp SnapCenter® storage management software.

·         Storage efficiency. Users can store more data with less physical media. This is achieved with thin provisioning (unused space is shared among volumes), NetApp Snapshot® copies (zero-storage, read-only replicas of data over time), NetApp FlexClone® volumes, and LUNs (read/write copies of data in which only changes are stored), deduplication (dynamic detection and removal of redundant data), and data compression.

·         Unified Data Management. Every model runs the same software (clustered Data ONTAP); supports all popular storage protocols (CIFS, NFS, iSCSI, FCP, and FCoE). This allows freedom of choice in upgrades and expansions, without the need to redesign the solution or retraining operations personnel.

·         Advanced clustering. Storage controllers are grouped into clusters for both availability and performance pooling. Workloads can be moved between controllers, permitting dynamic load balancing and zero-downtime maintenance, upgrades, and even complete hardware refreshes. Physical media and storage controllers can be added as needed to support growing demand without downtime.

This solution utilizes the NetApp AFF A700s as seen in the above figure. This controller provides the high-performance benefits of 40GbE and all flash SSDs.

Combined with the 3 disk shelfs and 72 1TB Solid State disks (SSD), this solution can provide over ample horsepower and over 60TB of raw capacity, all while taking up minimum valuable rack space. This makes it an ideal controller for a shared workload converged infrastructure. For situations where more performance is needed, the A700s would be an ideal fit. ONTAP 9.3 is used with the AFF A700s storage platform in our design.

ONTAP data management software offers unified storage for applications that read and write data over block or file-access protocols in storage configurations that range from high-speed flash to lower-priced spinning media or cloud-based object storage. The ONTAP platform used on the AFF systems also delivers interconnection with a larger ONTAP-based storage environment. In addition to AFF hardware systems, ONTAP is also available on commodity hardware (ONTAP Select), and in private, public, or hybrid clouds (NetApp Private Storage and Cloud Volumes ONTAP).

Together these implementations form the basic framework of the NetApp Data Fabric, with a common software-defined approach to data management and fast, efficient replication across platforms. FlexPod and ONTAP can serve as the foundation for both hybrid cloud and private cloud designs

Starting with ONTAP 9, NetApp guarantees that the use of NetApp storage efficiency technologies on AFF systems reduce the total logical capacity used to store customer data by 75 percent, a data reduction ratio of 4:1. This space reduction is enabled by a combination of several different technologies, including deduplication, compression, and compaction.

Beginning with ONTAP 9.1, NetApp has extended the encryption capabilities further with NetApp Volume Encryption (NVE), a software-based mechanism for encrypting data. It allows a user to encrypt data at the per-volume level instead of requiring encryption of all data in the cluster, thereby providing more flexibility and granularity to the ONTAP administrators. This encryption extends to Snapshot copies and NetApp FlexClone® volumes that are created in the cluster. One benefit of NVE is that it executes after the implementation of the storage efficiency features, and, therefore, it does not interfere with the ability of ONTAP to create space savings.

Physical Infrastructure

FlexPod is a defined set of hardware and software that serves as an integrated foundation for both virtualized and non-virtualized solutions. FlexPod components are connected and configured according to best practices of both Cisco and NetApp to provide the ideal platform for running a variety of enterprise workloads with confidence. This solution provides an end-to-end architecture with Cisco Unified Computing System, Oracle, and NetApp technologies and demonstrates the FlexPod configuration benefits for running Oracle RAC Databases 12cR2 workloads with high availability and server redundancy.

The reference FlexPod architecture covered in this document is built on the NetApp All Flash AFF A700s for Storage, Cisco B200 M5 Blade Servers for Compute, Cisco Nexus 93180LC-EX Switches, Cisco MDS 9148S 16G Multilayer Fabric Switches and Cisco Fabric Interconnects 6332-16UP Fabric Interconnects for System Management in a single package. The design is flexible enough that the networking, computing, and storage can fit in one data center rack or be deployed according to a customer's data center design.

The reference architecture reinforces the "wire-once" strategy, because as additional storage is added to the architecture, no re-cabling is required from the hosts to the Cisco UCS fabric interconnect.

Physical Topology

This section describes the design considerations for the Oracle RAC Database 12c Release 2 on FlexPod deployment. In this solution design, we have used two Cisco UCS Blade Server Chassis with 4 identical Intel Xeon CPU based Cisco UCS B-Series B200 M5 Blade Servers for hosting the 4-Node Oracle RAC Databases. The Cisco UCS B200 M5 Server has Virtual Interface Card (VIC) 1340 with port expander and they were connected two ports from each Cisco Fabric extender of the Cisco UCS Chassis to the Cisco Fabric Interconnects, which were in turn connected to the Cisco MDS Switches for upstream connectivity to access the NetApp AFF Storage A700s.

Figure 2 shows the architecture diagram of the FlexPod components and the network connections to deploy a four-node Oracle RAC 12cR2 Databases solution. This reference design is a typical network configuration that can be deployed in a customer's environments.

Figure 2  FlexPod Architecture Design

Figure 2 details the cable connections used in the validation lab for the 40Gb end-to-end with Fibre Channel topology based on the Cisco UCS 6332-16UP fabric interconnect. As shown in above architecture, a pair of Cisco UCS 6332-16UP Fabric Interconnects (FI) carries both storage and network traffic from the Cisco UCS B200 M5 server blades with the help of Cisco Nexus 93180LC-EX and Cisco MDS 9148S switches. Both the Fabric Interconnects and the Cisco Nexus switches are clustered with the peer link between them to provide high availability. Two virtual Port-Channels (vPCs) are configured to provide public network and private network paths for the server blades to northbound switches. Each vPC has VLANs created for application network data and management data paths.

As illustrated in the above architecture, four (2 x 40G link per chassis) links go to Fabric Interconnect – A. Similarly, four (2 x 40G link per chassis) links go to Fabric Interconnect – B. Fabric Interconnect – A links are used for Oracle Public network traffic shown as green lines. Fabric Interconnect – B links are used for Oracle private interconnect traffic shown as red lines. FC Storage access from Fabric Interconnect – A and Fabric Interconnect – B shown as orange lines.

Four 16Gb uplinks are connected from Cisco UCS Fabric Interconnect A to MDS switch A. Similarly, four 16Gb uplinks are connected from Cisco UCS Fabric Interconnect B to MDS switch B. The NetApp Storage AFF A700s have eight active FC connection goes to the Cisco MDS switches. Four FC ports are connected to Cisco MDS-A, and other four FC ports are connected to Cisco MDS-B Switches. The SAN Ports FC-Port-0-Slot-2 and FC-Port-0-Slot-3 of NetApp AFF A700s Controller – 1 are connected to Cisco MDS Switch A and FC-Port-1-Slot-2 and FC-Port-1-Slot-3 are connected to Cisco MDS Switch B. Similarly, the SAN Ports FC-Port-0-Slot-2 and FC-Port-0-Slot-3 of NetApp AFF A700s Controller – 2 are connected to Cisco MDS Switch A and FC-Port-1-Slot-2 and FC-Port-1-Slot-3 are connected to Cisco MDS Switch B.

Additional 1Gb management connections will be needed for an out-of-band network switch that sits apart from the FlexPod infrastructure.  Each Cisco UCS fabric interconnect and Cisco Nexus switch is connected to the out-of-band network switch, and each AFF controller has two connections to the out-of-band network switch.

The reference architecture includes the following hardware:

·         Two Cisco UCS 5108 Blade Server Chassis

·         Four Cisco UCS B200 M5 Blade Servers with Cisco Virtual Interface Cards (VIC)

·         Two Cisco UCS 6332-16UP Fabric Interconnects

·         Two Cisco MDS 9148S Multilayer Fabric Switch

·         Two Cisco Nexus 93180LC-EX Switch

·         One NetApp AFF A700s (HA pair) running ONTAP with Disk shelves and Solid State Drives (SSD)

Although this is the base design, each of the components can be scaled easily to support specific business requirements. For example, more servers or even blade chassis can be deployed to increase compute capacity, additional disk shelves can be deployed to improve I/O capability and throughput, and special hardware or software features can be added to introduce new features. This document guides you through the low-level steps for deploying the base architecture, as shown in Figure 2. These procedures cover everything from physical cabling to network, compute and storage device configurations.

Design Topology

This section describes the physical and logical high-level design considerations for this Oracle RAC 12cR2 Databases on FlexPod deployment.

The inventory of the components used in this Solution stack is listed in Table 1  .

Table 1          Hardware Components

Table 2  lists the Cisco UCS B200 M5 Blade Server Configuration.

Table 2         Cisco UCS B200 M5 Blade Server Configuration

For this FlexPod solution design, we configured two VLAN and two VSAN as listed below:

Table 3          VLAN and VSAN Configurations

This FlexPod solution consists of NetApp All-Flash AFF-Series Storage as listed in Table 4  .

Table 4         NetApp AFF A700s Storage Configuration

For this FlexPod solution, we used the following versions of the software and firmware releases.

Table 5          Software and Firmware Revisions

Configure Cisco Nexus 93180LC-EX Switch

The following procedures describe how to configure the Cisco Nexus switches for use in a base FlexPod environment.  This procedure assumes the use of Cisco Nexus 93180LC-EX 7.0(3)I6(1) switches deployed with the 40Gb end-to-end topology.

Set Up Initial Configuration

Cisco Nexus A

To set up the initial configuration for the Cisco Nexus A switch on <nexus-A-hostname>, complete the following steps:

1.        Configure the switch.

Abort Power on Auto Provisioning and continue with normal setup? (yes/no) [n]: yes

Do you want to enforce secure password standard (yes/no) [y]: Enter

Enter the password for "admin": <password>

Confirm the password for "admin": <password>

Would you like to enter the basic configuration dialog (yes/no): yes

Create another login account (yes/no) [n]: Enter

Configure read-only SNMP community string (yes/no) [n]: Enter

Configure read-write SNMP community string (yes/no) [n]: Enter

Enter the switch name: <nexus-A-hostname>

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: Enter

Mgmt0 IPv4 address: <nexus-A-mgmt0-ip>

Mgmt0 IPv4 netmask: <nexus-A-mgmt0-netmask>

Configure the default gateway? (yes/no) [y]: Enter

IPv4 address of the default gateway: <nexus-A-mgmt0-gw>

Configure advanced IP options? (yes/no) [n]: Enter

Enable the telnet service? (yes/no) [n]: Enter

Enable the ssh service? (yes/no) [y]: Enter

Type of ssh key you would like to generate (dsa/rsa) [rsa]: Enter

Number of rsa key bits <1024-2048> [1024]: Enter

Configure the ntp server? (yes/no) [n]: y

NTP server IPv4 address: <global-ntp-server-ip>

Configure default interface layer (L3/L2) [L3]: L2

Configure default switchport interface state (shut/noshut) [noshut]: Enter

Configure CoPP system profile (strict/moderate/lenient/dense/skip) [strict]: Enter

Would you like to edit the configuration? (yes/no) [n]: Enter

2.       Review the configuration summary before enabling the configuration.

Use this configuration and save it? (yes/no) [y]: Enter

Cisco Nexus B

To set up the initial configuration for the Cisco Nexus B switch on <nexus-B-hostname>, complete the following steps:

1.        Configure the switch.

Abort Power on Auto Provisioning and continue with normal setup? (yes/no) [n]: yes

Do you want to enforce secure password standard (yes/no) [y]: Enter

Enter the password for "admin": <password>

Confirm the password for "admin": <password>

Would you like to enter the basic configuration dialog (yes/no): yes

Create another login account (yes/no) [n]: Enter

Configure read-only SNMP community string (yes/no) [n]: Enter

Configure read-write SNMP community string (yes/no) [n]: Enter

Enter the switch name: <nexus-B-hostname>

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: Enter

Mgmt0 IPv4 address: <nexus-B-mgmt0-ip>

Mgmt0 IPv4 netmask: <nexus-B-mgmt0-netmask>

Configure the default gateway? (yes/no) [y]: Enter

IPv4 address of the default gateway: <nexus-B-mgmt0-gw>

Configure advanced IP options? (yes/no) [n]: Enter

Enable the telnet service? (yes/no) [n]: Enter

Enable the ssh service? (yes/no) [y]: Enter

Type of ssh key you would like to generate (dsa/rsa) [rsa]: Enter

Number of rsa key bits <1024-2048> [1024]: Enter

Configure the ntp server? (yes/no) [n]: y

NTP server IPv4 address: <global-ntp-server-ip>

Configure default interface layer (L3/L2) [L3]: L2

Configure default switchport interface state (shut/noshut) [noshut]: Enter

Configure CoPP system profile (strict/moderate/lenient/dense/skip) [strict]: Enter

Would you like to edit the configuration? (yes/no) [n]: Enter

2.       Review the configuration summary before enabling the configuration.

Use this configuration and save it? (yes/no) [y]: Enter

Configure Global Settings for Cisco Nexus A and Cisco Nexus B

To set global configuration, follow these steps on both the nexus switches:

1.        Log in as admin user into the Nexus Switch A and run the following commands to set global configurations:

configure terminal

feature interface-vlan

feature hsrp

feature lacp

feature vpc

spanning-tree port type network default

spanning-tree port type edge bpduguard default

port-channel load-balance src-dst l4port

policy-map type network-qos jumbo

  class type network-qos class-default

    mtu 9216

system qos

  service-policy type network-qos jumbo

vrf context management

  ip route 0.0.0.0/0 10.29.135.1

copy run start

2.       Log in as admin user into the Nexus Switch B and run the same above commands to set global configurations.

Configure VLANs for Cisco Nexus A and Cisco Nexus B Switches

To create the necessary virtual local area networks (VLANs), follow these steps on both Nexus switches:

1.        Log in as admin user into the Nexus Switch A.

2.       Create VLAN 135 for Public Network Traffic and VLAN 10 for Private Network Traffic.

configure terminal

vlan 135

name Oracle_RAC_Public_Traffic

no shutdown

vlan 10

name Oracle_RAC_Private_Traffic

no shutdown

copy run start

3.        Log in as admin user into the Nexus Switch B and create VLAN 135 for Public Network Traffic and VLAN 10 for Private Network Traffic.

Virtual Port Channel (vPC) Summary for Data and Storage Network

In the Cisco Nexus 93180LC-EX switch topology, a single vPC feature is enabled to provide HA, faster convergence in the event of a failure, and greater throughput. Cisco Nexus vPC configurations with the vPC domains and corresponding vPC names and IDs for Oracle Database Servers is shown in Table 6 

Table 6         vPC Summary

As listed in Table 6  , a single vPC domain with Domain ID 1 is created across two Cisco Nexus 93180LC-EX member switches to define vPC members to carry specific VLAN network traffic. In this topology, we defined a total number of 3 vPCs.

vPC ID 1 is defined as Peer link communication between two Nexus switches in Fabric A and B.

To define vPC IDs 51 and 52 for public and private network traffic from Cisco UCS fabric interconnects, complete the steps in the following sections.

Create vPC Peer-Link Between the Two Nexus Switches

1.        Login as admin user into the Nexus Switch A.

2.       Create the necessary port channels between devices on both Nexus Switches:

configure terminal

vpc domain 1

peer-keepalive destination 10.29.135.104 source 10.29.135.103

auto-recovery

 

interface port-channel1

description vPC peer-link

switchport mode trunk

switchport trunk allowed vlan 1,10,135

spanning-tree port type network

vpc peer-link

exit

 

interface Ethernet1/1

description Peer link connected to N9K-B-Eth1/1

switchport mode trunk

switchport trunk allowed vlan 1,10,135

channel-group 1 mode active

no shutdown

interface Ethernet1/2

description Peer link connected to N9K-B-Eth1/2

switchport mode trunk

switchport trunk allowed vlan 1,10,135

channel-group 1 mode active

no shutdown

 

interface Ethernet1/29

description connect to uplink switch

switchport access vlan 135

speed 1000

exit

copy run start

3.        Log in as admin user into the Nexus Switch B and repeat the above steps to configure second nexus switch.

Create vPC Configuration Between Nexus 9372PX-E and Fabric Interconnects

Next, you will create and configure vPC 51 and 52 for Data network between Nexus switches and Fabric Interconnects.

Table 7  lists the vPC IDs, allowed VLAN IDs, and Ethernet uplink ports.

Table 7          vPC IDs and VLAN IDs

To create the necessary port channels between devices, follow these steps on both the Nexus Switches:

1.        Log in as admin user into the Nexus Switch A and follow these steps:

configure terminal

interface port-channel51

description Port-Channel FI-A

switchport mode trunk

switchport trunk allowed vlan 1,10,135

spanning-tree port type edge trunk

mtu 9216

vpc 51

no shutdown

 

interface port-channel52

description Port-Channel FI-B

switchport mode trunk

switchport trunk allowed vlan 1,10,135

spanning-tree port type edge trunk

mtu 9216

vpc 52

no shutdown

 

interface Ethernet1/21

  description Fabric-Interconnect-A-31

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 51 mode active

  no shutdown

 

interface Ethernet1/22

  description Fabric-Interconnect-A-32

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 51 mode active

  no shutdown

 

interface Ethernet1/23

  description Fabric-Interconnect-B-31

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 52 mode active

  no shutdown

 

interface Ethernet1/24

  description Fabric-Interconnect-B-32

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 52 mode active

  no shutdown

 

copy run start

2.       Log in as admin user into the Nexus Switch B and run the following commands to complete second switch configuration:

configure terminal

interface port-channel51

description Port-Channel FI-A

switchport mode trunk

switchport trunk allowed vlan 1,10,135

spanning-tree port type edge trunk

mtu 9216

vpc 51

no shutdown

 

interface port-channel52

description Port-Channel FI-B

switchport mode trunk

switchport trunk allowed vlan 1,10,135

spanning-tree port type edge trunk

mtu 9216

vpc 52

no shutdown

 

interface Ethernet1/21

  description Fabric-Interconnect-A-33

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 51 mode active

  no shutdown

 

interface Ethernet1/22

  description Fabric-Interconnect-A-34

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 51 mode active

  no shutdown

 

interface Ethernet1/23

  description Fabric-Interconnect-B-33

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 52 mode active

  no shutdown

 

interface Ethernet1/24

  description Fabric-Interconnect-B-34

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 52 mode active

  no shutdown

 

copy run start

Verify All vPC Status Is Up on Both the Nexus Switches

1.        Verify the port-channel summary of Nexus Switch A as shown below:

2.       Verify the port-channel summary of Nexus Switch B as shown below:

3.        Verify the vPC summary of Nexus Switch A as shown below:

4.       Verify the vPC summary of Nexus Switch B as shown below:

Cisco UCS Configuration Overview

This section details the Cisco UCS configuration that was completed as part of the infrastructure build out. The racking, power, and installation of the chassis are described in the installation guide (see www.cisco.com/c/en/us/support/servers-unified-computing/ucs-manager/products-installation-guides-list.html). It is beyond the scope of this document to cover detailed information about the UCS infrastructure setup and connectivity. The documentation guides and examples are available at http://www.cisco.com/en/US/products/ps10281/products_installation_and_configuration_guides_list.html

We will list all the tasks to configure Cisco UCS system but only include some of the screenshots in this document.

Perform Initial Setup of Cisco UCS 6332-16UP Fabric Interconnects for a Cluster Setup

This section provides detailed procedures for configuring the Cisco Unified Computing System (Cisco UCS) for use in a FlexPod environment. The steps are necessary to provision the Cisco UCS B-Series and C-Series servers and should be followed precisely to avoid improper configuration.

Cisco UCS Fabric Interconnect A and Fabric Interconnect B

To configure the Cisco UCS Fabric Interconnects, complete the following steps:

1.        Verify the following physical connections on the fabric interconnect:

a.        The management Ethernet port (mgmt0) is connected to an external hub, switch, or router

b.       The L1 ports on both fabric interconnects are directly connected to each other

c.        The L2 ports on both fabric interconnects are directly connected to each other

2.       Connect to the console port on the first Fabric Interconnect.

3.        Review the settings printed to the console. Answer yes to apply and save the configuration.

4.       Wait for the login prompt to make the configuration has been saved to Fabric Interconnect A.

5.        Connect the console port on the second Fabric Interconnect and do the following:

6.       Review the settings printed to the console. Answer yes to apply and save the configuration.

7.        Wait for the login prompt to make the configuration has been saved to Fabric Interconnect B.

Log in to Cisco UCS Manager

To log in to the Cisco Unified Computing System (UCS) environment, complete the following steps:

1.        Open a web browser and navigate to the Cisco UCS fabric interconnect cluster address.

2.       Click the Launch UCS Manager link under HTML to launch Cisco UCS Manager.

3.        If prompted to accept security certificates, accept as necessary.

4.       When prompted, enter admin as the user name and enter the administrative password.

5.        Click Login to log into Cisco UCS Manager.

Upgrade Cisco UCS Manager Software to Version 3.2

This document assumes the use of Cisco UCS 3.2(3c). To upgrade the Cisco UCS Manager software and the Cisco UCS Fabric Interconnect software to version 3.2(3c), refer to Cisco UCS Manager Install and Upgrade Guides.

Anonymous Reporting

To create anonymous reporting, complete the following step:

1.        In the Anonymous Reporting window, select whether to send anonymous data to Cisco for improving future products. If you select Yes, enter the IP address of your SMTP Server.  Click OK.

Configure Cisco UCS Call Home

It is highly recommended by Cisco to configure Call Home in Cisco UCS Manager.  Configuring Call Home will accelerate resolution of support cases.  To configure Call Home, follow these steps:

1.        In Cisco UCS Manager, click Admin on the left.

2.       Select All > Communication Management > Call Home.

3.        Change the State to On.

4.       Fill in all the fields according to your Management preferences and click Save Changes and OK to complete configuring Call Home.

High Level Steps to Configure Base Cisco UCS

Using logical servers that are disassociated from the physical hardware removes many limiting constraints around how servers are provisioned. Cisco UCS Service Profiles contain values for a server's property settings, including virtual network interface cards (vNICs), MAC addresses, boot policies, firmware policies, fabric connectivity, external management, and HA information. The service profiles represent all the attributes of a logical server in Cisco UCS model. By abstracting these settings from the physical server into a Cisco Service Profile, the Service Profile can then be deployed to any physical compute hardware within the Cisco UCS domain. Furthermore, Service Profiles can, at any time, be migrated from one physical server to another. Furthermore, Cisco is the only hardware provider to offer a truly unified management platform, with Cisco UCS Service Profiles and hardware abstraction capabilities extending to both blade and rack servers.

The following are the high-level steps for configuring the Cisco UCS for this FlexPod solution.

1.        Set Fabric Interconnect to Fibre Channel End Host Mode.

2.       Synchronize Cisco UCS to NTP.

3.        Configure Fabric Interconnect for Chassis and Blade Discovery.

a.        Configure Global Policies

b.       Configure Server Ports

4.       Configure LAN and SAN on Cisco UCS Manager

a.        Configure Ethernet LAN Uplink Ports

b.       Create Uplink Port Channels to Cisco Nexus Switches

c.        Configure FC SAN Uplink Ports

d.       Configure VLAN

e.       Configure VSAN

5.        Configure IP, UUID, Server, MAC, WWNN and WWPN Pools

a.        IP Pool Creation

b.       UUID Suffix Pool Creation

c.        Server Pool Creation

d.       MAC Pool Creation

e.       WWNN and WWPN Pool Creation

6.       Set Jumbo Frames in both the Cisco Fabric Interconnect

7.        Configure Server BIOS Policy

8.       Create Adapter Policy

9.       Configure Update Default Maintenance Policy

10.    Configure vNIC and vHBA Template

a.        Create Public and Private vNIC Template

b.       Create Storage vHBA Template

11.     Create Server Boot Policy for SAN Boot

Details for each step are discussed in the following sections.

Set Fabric Interconnects to Fibre Channel End Host Mode

To set the Fabric Interconnects to the Fibre Channel End Host Mode, complete the following steps:

1.        On the Equipment tab, expand the Fabric Interconnects node and click Fabric Interconnect A.

2.       On the General tab in the Actions pane, click Set FC End Host mode.

3.        Follow the dialogs to complete the change.

Synchronize Cisco UCS to NTP

To synchronize the Cisco UCS environment to the NTP server, complete the following steps:

1.        In Cisco UCS Manager, in the navigation pane, click the Admin tab.

2.       Select All > Time zone Management.

3.        In the Properties pane, select the appropriate time zone in the Time zone menu.

4.       Click Save Changes and then click OK.

5.        Click Add NTP Server.

6.       Enter the NTP server IP address and click OK.

7.        Click OK to finish.

Configure Fabric Interconnect for Chassis and Blade Discovery

Cisco UCS 6332-16UP Fabric Interconnects are configured for redundancy. It provides resiliency in case of failures. The first step to establish connectivity between blades and Fabric Interconnects.

Configure Global Policies

The chassis discovery policy determines how the system reacts when you add a new chassis. We recommend using the platform max value as shown. Using platform max insures that Cisco UCS Manager uses the maximum number of IOM uplinks available.

To configure Global Policies, follow these steps:

1.        Go to Equipment > Policies (right pane) > Global Policies > Chassis/FEX Discovery Policies.

2.       Select Action as “Platform Max” from the drop down list and set Link Grouping to Port Channel as shown below. Click Save Changes and then click OK.

The difference between Discrete mode vs Port Channel mode is shown below:

Configure Server Ports

Configure Server Ports to initiate Chassis and Blade discovery. To configure server ports, follow these steps:

1.        Go to Equipment > Fabric Interconnects > Fabric Interconnect A > Fixed Module > Ethernet Ports.

2.       Select the ports (for this solution ports are 17-20) which are connected to the Cisco IO Modules of the two B-Series 5108 Chassis.

3.        Right-click and select “Configure as Server Port.”

4.       Click Yes to confirm and click OK.

5.        Repeat the same task for Fabric Interconnect B.

6.       After configuring Server Ports, acknowledge both the Chassis. Go to Equipment >Chassis > Chassis 1 > General > Actions > select “Acknowledge Chassis”. Similarly, acknowledge the chassis 2.

7.        After acknowledging both the chassis, Re-acknowledge all the servers placed in the chassis. Go to Equipment > Chassis 1 > Servers > Server 1 > General > Actions > select “Server Maintenance” > select option “Re-acknowledge” and click OK. Similarly, repeat the process to Re-acknowledge all the eight Servers.

8.       When the acknowledgement of the Servers completed, verify “Port-channel” of Internal LAN. Go to tab LAN > Internal LAN > Internal Fabric A > Port Channels as shown below.

9.       Verify the same for Internal Fabric B.

Configure LAN and SAN on Cisco UCS Manager

Configure Ethernet Uplink Ports and Fibre Channel (FC) Storage ports on Cisco UCS as explained below.

Configure Ethernet LAN Uplink Ports

To configure network ports used to uplink the Fabric Interconnects to the Cisco Nexus switches, follow these steps:

1.        In Cisco UCS Manager, in the navigation pane, click the Equipment tab.

2.       Select Equipment > Fabric Interconnects > Fabric Interconnect A > Fixed Module.

3.        Expand Ethernet Ports.

4.       Select ports (for this solution ports are 31-34) that are connected to the Nexus switches, right-click them, and select Configure as Network Port.

5.        Click Yes to confirm ports and click OK.

6.       Verify the Ports connected to Cisco Nexus upstream switches are now configured as network ports.

7.        Repeat the above steps for Fabric Interconnect B. The screenshot shows the network uplink ports for Fabric A.

We created four uplink ports on each Fabric Interconnect as shown above. These ports will be used to create Virtual Port Channel in the next section.

Create Uplink Port Channels to Cisco Nexus Switches

In this procedure, two port channels were created: one from Fabric A to both Cisco Nexus switch and one from Fabric B to both Cisco Nexus switch. To configure the necessary port channels in the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.       Under LAN > LAN Cloud, expand node Fabric A tree:

3.        Right-click Port Channels.

4.       Select Create Port Channel.

5.        Enter 51 as the unique ID of the port channel.

6.       Enter FI-A as the name of the port channel.

7.        Click Next.

8.       Select Ethernet ports 31-34 for the port channel.

9.       Click >> to add the ports to the port channel.

10.    Click Finish to create the port channel and then click OK.

11.     Repeat steps 1-10 for Fabric Interconnect B, substituting 52 for the port channel number and FI-B for the name. The resulting configuration should look like the screenshot shown above.

Configure FC SAN Uplink Ports

The fibre channel port selection options for the 6332-16UP are from the first 16 ports starting from the first port on the left, and configured in increments of the first 6, 12, or all 16 of the unified ports.

To enable the Fibre channel ports, follow these steps for the 6332-16UP:

1.        In Cisco UCS Manager, click Equipment on the left.

2.       Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary).

3.        Select Configure Unified Ports.

4.       Click Yes on the pop-up window warning that changes to the fixed module will require a reboot of the fabric interconnect and changes to the expansion module will require a reboot of that module.

5.        Within the Configured Fixed Ports pop-up window move the gray slider bar from the left to the right to select either 6, 12, or 16 ports to be set as FC Uplinks.

6.       For this solution, we configured the first six ports on the FI as FC Uplink ports. Click OK, then click Yes, then click OK to continue

7.        Select Equipment > Fabric Interconnects > Fabric Interconnect B (primary).

8.       Select Configure Unified Ports.

9.       Click Yes on the pop-up window warning that changes to the fixed module will require a reboot of the fabric interconnect and changes to the expansion module will require a reboot of that module.

10.    Within the Configured Fixed Ports pop-up window move the gray slider bar from the left to the right to select either 6, 12, or 16 ports to be set as FC Uplinks.

11.     Click OK, then Yes, then OK to continue.

12.    Wait for both Fabric Interconnects to reboot.

13.     Log back into Cisco UCS Manager.

Configure VLAN

To configure the necessary virtual local area networks (VLANs) for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the LAN tab in the navigation pane.

 

2.       Select LAN > LAN Cloud.

3.        Right-click VLANs.

4.       Select Create VLANs

5.        Enter Public_Traffic as the name of the VLAN to be used for Public Network Traffic.

6.       Keep the Common/Global option selected for the scope of the VLAN.

7.        Enter 135 as the ID of the VLAN ID.

8.       Keep the Sharing Type as None.

9.       Click OK and then click OK again.

Configure VSAN

To configure the necessary virtual storage area networks (VSANs) for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the SAN tab in the navigation pane.

2.       Select SAN > SAN Cloud.

3.        Under VSANs, right-click VSANs.

4.       Select Create VSANs.

5.        Enter ORA-VSAN-A as the name of the VSAN.

6.       Leave FC Zoning set at Disabled.

7.        Select Fabric A for the scope of the VSAN.

8.       Enter 101 as the ID of the VSAN.

9.       Click OK and then click OK again.

10.    Repeat these steps to create the VSANs necessary for this solution. VSAN 101 and 102 are configured as shown below:

Configure IP, UUID, Server, MAC, WWNN and WWPN Pools

IP Pool Creation

An IP address pool on the out of band management network must be created to facilitate KVM access to each compute node in the Cisco UCS domain. To create a block of IP addresses for server KVM access in the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, in the navigation pane, click the LAN tab.

2.       Select Pools > root > IP Pools >Click Create IP Pool.

3.        We have named IP Pool as ORA-IP-Pool for this solution.

4.       Select option Sequential to assign IP in sequential order then click next.

5.        Click Add IPv4 Block.

6.       Enter the starting IP address of the block and the number of IP addresses required, and the subnet and gateway information as shown below.

7.        Click Next and then click Finish to create the IP block.

UUID Suffix Pool Creation

To configure the necessary universally unique identifier (UUID) suffix pool for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.       Select Pools > root.

3.        Right-click UUID Suffix Pools and then select Create UUID Suffix Pool.

4.       Enter ORA-UUID-Pool as the name of the UUID name.

5.        Optional: Enter a description for the UUID pool.

6.       Keep the prefix at the derived option and select Sequential in as Assignment Order then click Next.

7.        Click Add to add a block of UUIDs

8.       Create a starting point UUID as per your environment.

9.       Specify a size for the UUID block that is sufficient to support the available blade or server resources.

Server Pool Creation

To configure the necessary server pool for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.       Select Pools > root > Right-click Server Pools > Select Create Server Pool.

3.        Enter ORA-Pool as the name of the server pool.

4.       Optional: Enter a description for the server pool then click Next

5.        Select all the eight servers to be used for the Oracle RAC management and click > to add them to the server pool.

6.       Click Finish and then click OK

MAC Pool Creation

To configure the necessary MAC address pools for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.       Select Pools > root > right-click MAC Pools under the root organization.

3.        Select Create MAC Pool to create the MAC address pool.

4.       Enter ORA-MAC-A as the name for MAC pool.

5.        Enter the seed MAC address and provide the number of MAC addresses to be provisioned.

6.       Click OK and then click Finish.

7.        In the confirmation message, click OK.

8.       Create MAC Pool B and assign unique MAC Addresses as shown below.

WWNN and WWPN Pool Creation

To configure the necessary WWNN pools for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the SAN tab in the navigation pane.

2.       Select Pools > Root > WWNN Pools > right click WWNN Pools > Select Create WWNN Pool.

3.        Assign name and Assignment Order as sequential as shown below.

4.       Click Next and then click Add to add block of Ports.

5.        Enter Block for WWN and size of WWNN Pool as shown below.

6.       Click OK and then click Finish.

To configure the necessary WWPN pools for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the SAN tab in the navigation pane.

2.       Select Pools > Root > WWPN Pools > right-click WWPN Pools > Select Create WWPN Pool.

3.        Assign name as ORA-WWPN-A and Assignment Order as sequential.

4.       Click Next and then click Add to add block of ports.

5.        Enter Block for WWN and size.

6.       Click OK and then Finish.

7.        Configure the ORA-WWPN-B Pool and assign the unique block IDs as shown below.

Set Jumbo Frames in both the Cisco Fabric Interconnect

To configure jumbo frames and enable quality of service in the Cisco UCS fabric, follow these steps:

1.        In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.       Select LAN > LAN Cloud > QoS System Class.

3.        In the right pane, click the General tab.

4.       On the Best Effort row, enter 9216 in the box under the MTU column.

5.        Click Save Changes in the bottom of the window.

6.       Click OK.

Configure Server BIOS Policy

To create a server BIOS policy for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click Servers on the left.

2.       Select Policies > root.

3.        Right-click BIOS Policies.

4.       Select Create BIOS Policy.

5.        Enter OLTP_BIOS as the BIOS policy name

6.       Select and click the newly created BIOS Policy.

7.        Click the Advanced tab, leaving the Processor tab selected within the Advanced tab.

8.       Set the following within the Processor tab:

9.       Click Save Changes and then click OK

 

 

Create Adapter Policy

To create an Adapter Policy for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.       Select Policies > root > right-click Adapter Policies.

3.        Select Create Ethernet Adapter Policy.

4.       Provide a name for the Ethernet adapter policy. Change the following fields and click Save Changes when you are finished:

a.        Resources

                                 i.      Transmit Queues: 8

                                ii.      Ring Size: 4096

                              iii.      Receive Queues: 8

                              iv.      Ring Size: 4096

                                v.      Completion Queues: 16

                              vi.      Interrupts: 32

b.       Options

                                 i.            Receive Side Scaling (RSS): Enabled

5.        Configure adapter policy as shown below.

6.       Click OK to finish.

Configure Update Default Maintenance Policy

To update the default Maintenance Policy, follow these steps:

1.        In Cisco UCS Manager, click the Servers tab in the navigation pane

2.       Select Policies > root > Maintenance Policies > Default.

3.        Change the Reboot Policy to User Ack.

4.       Click Save Changes.

5.        Click OK to accept the changes.

Configure vNIC and vHBA Template

For this solution, we created two vNIC template for Public Network and Private Network Traffic.

Create Public and Private vNIC Template

To create vNIC (virtual network interface card) template for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.       Select Policies > root > vNIC Templates > right-click to vNIC Template and Select "Create vNIC Template"

3.        Enter ORA-vNIC-A as the vNIC template name and keep Fabric A selected.

4.       Select the Enable Failover checkbox for high availability of the vNIC.

5.        Select Template Type  as Updating Template.

6.       Under VLANs, select the checkboxes default and Public_Traffic and set Native-VLAN as the Public_Traffic.

7.        Keep MTU value 1500 for Public Network Traffic.

8.       In the MAC Pool list, select ORA-MAC-A.

9.       Click OK to create the vNIC template as shown below.

10.    Click OK to finish.

11.     Create another vNIC template for Private Network Traffic with few changes:

a.        Enter ORA-vNIC-B as the vNIC template name for Private Network Traffic.

b.       Select the Fabric B and Enable Failover for Fabric ID options.

c.        Select Template Type  as Updating Template.

d.       Under VLANs, select the checkboxes default and Private_Traffic and set Native-VLAN as the Private_Traffic.

e.       Set MTU value to 9000 and MAC Pool as ORA-MAC-B.

12.    Click OK to create the vNIC template as shown below.

Create Storage vHBA Template

To create multiple virtual host bus adapter (vHBA) templates for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the SAN tab in the navigation pane

2.       Select Policies > root > Right Click vHBA Templates > Select “Create vHBA Template” to create vHBAs.

3.        Enter name as ORA-vHBA-A and keep Fabric A selected.

4.       Select VSAN as ORA-VSAN-A and template type to Updating Template.

5.        Select WWPN Pool as ORA-WWPN-A from the drop-down list as shown below.

6.       Enter name as ORA-vHBA-B and select Fabric B. Select WWPN Pool for ORA-vHBA-B as “ORA-WWPN-B” as shown below.

Create Server Boot Policy for SAN Boot

All Oracle nodes were set to boot from SAN for the Cisco Validated Design as part of the Service Profile template. The benefits of booting from SAN are numerous; disaster recovery, lower cooling and power requirements for each server since a local drive is not required, and better performance.

This process applies to a Cisco UCS environment in which the storage SAN ports are configured as detailed in the following sections.

Create Local Disk Configuration Policy

A Local disk configuration for the Cisco UCS is necessary if the servers in the environments have a local disk.

To configure Local disk policy, follow these steps:

1.        Go to tab Servers > Policies > root > right-click Local Disk Configuration Policy > enter “SAN-Boot” as the local disk configuration policy name and change the mode to “No Local Storage.”

2.       Click OK to create the policy as shown below.

Create SAN Boot Policy

The screenshot below shows the network interface, WWPN and ports connectivity configured for NetApp AFF A700s controller. Four Fibre Channel logical interfaces (LIFs) are created on storage controller cluster node 1 (node1_lif02a, node1_lif02b, node1_lif03a and node1_lif03b) and four Fibre Channel LIFs are created on storage controller cluster node 2 (node2_lif02a, node2_lif02b, node2_lif03a and node2_lif03b).

The SAN boot policy configures the SAN Primary's primary-target to be network interface node1_lif02a on NetApp storage cluster and SAN Primary's secondary-target to be network interface node1_lif03a on NetApp storage cluster. Similarly, the SAN Secondary’s primary-target to be network interface node2_lif02b on NetApp storage cluster and SAN Secondary's secondary-target to be network interface node2_lif03b on NetApp storage cluster. Login into NetApp storage controller and verify all the port information is correct. This information can be found in the NetApp Storage GUI under Network > Network Interfaces.

To create Boot Policies for the Cisco UCS environments follow these steps:

1.        Go to UCS Manager and then go to tab Servers > Policies > root > Boot Policies.

2.       Right-click and select Create Boot Policy. Enter SAN_Boot as the name of the boot policy as shown below.

7.        Expand the Local Devices drop-down menu and Choose Add CD/DVD. Expand the vHBAs drop-down menu and select Add SAN Boot.

8.       In the Add SAN Boot dialog box, select Type as “Primary” and name vHBA as “hba0”. Click OK to add SAN Boot.

9.       Select add SAN Boot Target to enter WWPN address of storage LIF. Keep 0 as the value for Boot Target LUN. Enter the WWPN of NetApp Storage cluster interface node1_lif02a and add SAN Boot Primary Target.

10.    Add secondary SAN Boot target into same hba0, enter boot target LUN as 0 and WWPN of NetApp Storage cluster interface node1_lif03a and add SAN Boot Secondary Target.

11.     From the vHBA drop-down list and Choose Add SAN Boot. In the Add SAN Boot dialog box, enter "hba1" in the vHBA field.

12.    Click OK to SAN Boot, then choose add SAN Boot Target.

13.     Enter 0 as the value for Boot Target LUN. Enter the WWPN of NetApp Storage cluster interface node2_lif02b and add SAN Boot Primary Target.

14.    Add secondary SAN Boot target into same hba1, and enter boot target LUN as 0 and WWPN of NetApp Storage cluster interface node2_lif03b and add SAN Boot Secondary Target.

15.     After creating the FC boot policies, you can view the boot order in the UCS Manager GUI. To view the boot order, navigate to Servers > Policies > Boot Policies. Click Boot Policy SAN-Boot-A to view the boot order in the right pane of the Cisco UCS Manager as shown below.

Configure and Create a Service Profile Template

Service profile templates enable policy based server management that helps ensure consistent server resource provisioning suitable to meet predefined workload needs.

The Cisco UCS service profiles with SAN boot policy provides the following benefits:

·         Scalability - Rapid deployment of new servers to the environment in a very few steps.

·         Manageability - Enables seamless hardware maintenance and upgrades without any restrictions.

·         Flexibility - Easy to repurpose physical servers for different applications and services as needed.

·         Availability - Hardware failures are not impactful and critical.  In rare case of a server failure, it is easier to associate the logical service profile to another healthy physical server to reduce the impact.

You will create one Service Profile Template “ORA_FLEXPOD” using boot policy created earlier to utilize four LIF ports from NetApp Storage for high-availability in case of any FC links go down.

The following sections detail how to create ORA_FLEXPOD.

Create Service Profile Template

To create a service profile template, follow these steps:

1.        In the Cisco UCS Manager, go to Servers > Service Profile Templates > root and right-click to “Create Service Profile Template” as shown below.

2.       Enter the Service Profile Template name, select the UUID Pool that was created earlier and click Next.

3.        Select Local Disk Configuration Policy to SAN-Boot as no Local Storage.

4.       In the networking window, select “Expert” and click “Add” to create vNICs. Add one or more vNICs that the server should use to connect to the LAN.

5.        Select vNIC Template as ORA-vNIC-A and Adapter Policy as ORA_Linux_Tuning which was created earlier for vNIC “eth0”.

6.       Select vNIC Template as ORA-vNIC-B and Adapter Policy as ORA_Linux_Tuning for vNIC “eth1”. eth0 and eth1 vNICs are created so that Servers should use to connect to the LAN.

7.        When the vNICs are created, you need to create vHBAs. Click Next.

8.       In the SAN Connectivity menu, select “Expert” to configure as SAN connectivity. Select WWNN (World Wide Node Name) pool, created previously. Click “Add” to add vHBAs as shown below. The following four HBA have been created:

a.        hba0 using vHBA Template Oracle-HBA-A

b.       hba1 using vHBA Template Oracle-HBA-B

c.        hba2 using vHBA Template Oracle-HBA-A

d.       hba3 using vHBA Template Oracle-HBA-B

9.       In vNIC/vHBA Placement menu, keep option as Let System Perform Placement.

10.    For this solution, do not configure any vMedia Policy. Click Next.

11.     In the Server Boot Order, select “SAN_Boot” as Boot Policy created previously.

12.    The maintenance policy and server assignment options were left as default in the configuration.

13.     In operational policies menu, select OLTP_BIOS as BIOS Policy, created previously.

14.    The rest of the configuration is left as default in the configuration. However, it may vary from site-to-site depending on workloads, best practices, and policies.

15.     Click Finish to create service profile template as “ORA_FLEXPOD”. This service profile template will be used to create four service profiles for four oracle RAC nodes(flex1, flex2, flex3 and flex4).

Now you have one Service profile template as “ORA_FLEXPOD” having four vHBAs and two vNICs.

Create Service Profiles from Template and Associate to Servers

Create Service Profiles from Template

For all four Oracle RAC Nodes (flex1, flex2, flex3 and flex4), you will create four Service Profiles as FLEX1, FLEX2, FLEX3 and FLEX4 from Template “ORA_FLEXPOD”.

To create Service Profiles from Template, follow these steps:

1.        Go to tab Servers > Service Profiles > root > and right-click “Create Service Profiles from Template”.

2.       Select the Service profile template as “ORA_FLEXPOD” which we created earlier and name the service profile as “FLEX”.

3.        To create four service profiles, enter “Number of Instances” as 4 as shown below. This process creates service profiles as “FLEX1”, “FLEX2”, “FLEX3” and “FLEX4”.

When the service profiles are created, associate them to the servers as described below.

Associate Service Profiles to the Servers

To associate service profiles to the servers, follow these steps:

1.        Under the Servers tab, select the desired service profile, and select change service profile association.

2.       Right-click the name of service profile you want to associate with the server and select the option "Change Service Profile Association".

3.        In the Change Service Profile Association page, from the Server Assignment drop-down list, select the existing server that you want to assign and click OK.

4.       Assign service profiles FLEX1 to Chassis 1 Server 1 and service profile FLEX2 to Chassis 1 Server 2. Similarly, we will assign, service profiles FLEX3 to Chassis 2 Server 1 and service profile FLEX4 to Chassis 2 Server 2.

5.        As shown above, make sure all the server nodes have no major or critical faults and all are in an operable state.

This completes the configuration required for Cisco UCS Manager Setup.

Configure Cisco MDS 9148S Switch

This section provides a detailed procedure for configuring the Cisco MDS 9148S Switches.

We connected MDS Switches to Fabric Interconnects and NetApp AFF A700s Storage System as shown below.

For this solution, we connected four ports (ports 1 to 4) of MDS Switch A to Fabric Interconnect A (ports 1-4). We also connected four ports (ports 1 to 4) of MDS Switch B to Fabric Interconnect B (ports 1-4). All ports carry 16 Gb/s FC Traffic. Table 8  lists port connectivity of Cisco MDS Switches to the Fabric Interconnects.

Table 8         MDS Switch Connectivity to the Fabric Interconnects

For these solution, we have connected four ports (ports 9 to 12) of MDS Switch A to the NetApp AFF A700s Storage controller. Similarly, we have connected four ports (ports 9 to 12) of MDS Switch B to the NetApp AFF A700s Storage controller. All ports carry 16 Gb/s FC Traffic. Table 9  lists port connectivity of Cisco MDS Switches to NetApp AFF A700s Controller.

Table 9         MDS Switch Connectivity to the NetApp AFF A700s Storage

Configure feature for MDS Switch A and MDS Switch B

To configure feature on MDS Switches, follow these steps:

1.        Log in as admin user into MDS Switch A:

FLEXPOD-MDS-A# config terminal

FLEXPOD-MDS-A(config)# feature npiv

FLEXPOD-MDS-A(config)# feature telnet

FLEXPOD-MDS-A(config)# switchname FLEXPOD-MDS-A

FLEXPOD-MDS-A(config)# copy running-config startup-config

2.       Login as admin user into MDS Switch B:

FLEXPOD-MDS-B# config terminal

FLEXPOD-MDS-B(config)# feature npiv

FLEXPOD-MDS-B(config)# feature telnet

FLEXPOD-MDS-B(config)# switchname FLEXPOD-MDS-B

FLEXPOD-MDS-B(config)# copy running-config startup-config

Configure VSANs for MDS Switch A and MDS Switch B

To create VSANs, follow these steps:

1.        Log in as admin user into MDS Switch A.

2.       Create VSAN 101 for Storage Traffic:

FLEXPOD-MDS-A # config terminal

FLEXPOD-MDS-A(config)# VSAN database

FLEXPOD-MDS-A(config-vsan-db)# vsan 101

FLEXPOD-MDS-A(config-vsan-db)# vsan 101 interface fc 1/1-12

FLEXPOD-MDS-A(config-vsan-db)# exit

 

FLEXPOD-MDS-A(config)# interface fc 1/1-12

FLEXPOD-MDS-A(config-if)# switchport trunk allowed vsan 101

FLEXPOD-MDS-A(config-if)# switchport trunk mode off

FLEXPOD-MDS-A(config-if)# port-license acquire

FLEXPOD-MDS-A(config-if)# no shutdown

FLEXPOD-MDS-A(config-if)# exit

FLEXPOD-MDS-A(config)# copy running-config startup-config

3.        Login as admin user into MDS Switch B.

4.       Create VSAN 102 for Storage Traffic:

FLEXPOD-MDS-B # config terminal

FLEXPOD-MDS-B(config)# VSAN database

FLEXPOD-MDS-B(config-vsan-db)# vsan 102

FLEXPOD-MDS-B(config-vsan-db)# vsan 102 interface fc 1/1-12

FLEXPOD-MDS-B(config-vsan-db)# exit

 

FLEXPOD-MDS-B(config)# interface fc 1/1-12

FLEXPOD-MDS-B(config-if)# switchport trunk allowed vsan 102

FLEXPOD-MDS-B(config-if)# switchport trunk mode off

FLEXPOD-MDS-B(config-if)# port-license acquire

FLEXPOD-MDS-B(config-if)# no shutdown

FLEXPOD-MDS-B(config-if)# exit

FLEXPOD-MDS-B(config)# copy running-config startup-config

Create and Configure Fibre Channel Zoning

This procedure sets up the Fibre Channel connections between the Cisco MDS 9148S switches, the Cisco UCS Fabric Interconnects, and the NetApp AFF Storage systems.

Before you configure the zoning details, decide how many paths are needed for each LUN and extract the WWPN numbers for each of the HBAs from each server. We used 4 HBAs for each Server. Two HBAs (HBA0 and HBA2) are connected to MDS Switch-A and other two HBAs (HBA1 and HBA3) are connected to MDS Switch-B.

To create and configure the Fibre channel zoning, follow these steps:

1.        Log into the Cisco UCS Manager > Equipment > Chassis > Servers and select the desired server. On the right-hand menu, click the Inventory tab and HBA's sub-tab to get the WWPN of HBA's as shown below.

2.       Log into NetApp storage controller and extract the WWPN of FC LIFs configured and verify all the port information is correct. This information can be found in the NetApp Storage GUI under Network > Network Interfaces.

The screenshot below shows the network interface, WWPN and ports connectivity configured for NetApp AFF A700s controller. Four Fibre Channel logical interfaces (LIFs) are created on storage controller cluster node 1 (node1_lif02a, node1_lif02b, node1_lif03a and node1_lif03b) and four Fibre Channel LIFs are created on storage controller cluster node 2 (node2_lif02a, node2_lif02b, node2_lif03a and node2_lif03b).

The NetApp Storage AFF A700s have eight active FC connection goes to the Cisco MDS switches. Four FC ports are connected to Cisco MDS-A, and other four FC ports are connected to Cisco MDS-B Switches. The SAN Ports FC-Port-0-Slot-2 and FC-Port-0-Slot-3 of NetApp AFF A700s Controller – 1 are connected to Cisco MDS Switch A and FC-Port-1-Slot-2 and FC-Port-1-Slot-3 are connected to Cisco MDS Switch B. Similarly, the SAN Ports FC-Port-0-Slot-2 and FC-Port-0-Slot-3 of NetApp AFF A700s Controller – 2 are connected to Cisco MDS Switch A and FC-Port-1-Slot-2 and FC-Port-1-Slot-3 are connected to Cisco MDS Switch B.

Create Device Aliases for Fibre Channel Zoning on Cisco MDS Switch A

To configure device aliases and zones for the SAN boot paths as well as datapaths of MDS switch A, follow these steps:

1.        Log in as admin user and run the following commands:

configure terminal

device-alias database

  device-alias name flex1-hba0 pwwn 20:00:00:25:b5:8a:a0:00

  device-alias name flex1-hba2 pwwn 20:00:00:25:b5:8a:a0:01

  device-alias name flex2-hba0 pwwn 20:00:00:25:b5:8a:a0:02

  device-alias name flex2-hba2 pwwn 20:00:00:25:b5:8a:a0:03

  device-alias name flex3-hba0 pwwn 20:00:00:25:b5:8a:a0:04

  device-alias name flex3-hba2 pwwn 20:00:00:25:b5:8a:a0:05

  device-alias name flex4-hba0 pwwn 20:00:00:25:b5:8a:a0:06

  device-alias name flex4-hba2 pwwn 20:00:00:25:b5:8a:a0:07

  device-alias name NetApp-A700s-01-2A pwwn 20:06:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-01-3A pwwn 20:07:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-02-2A pwwn 20:09:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-02-3A pwwn 20:0a:00:a0:98:af:7c:5b

  device-alias commit

copy run start

Create Device Aliases for Fibre Channel Zoning on Cisco MDS Switch B

To configure device aliases and zones for the SAN boot paths as well as datapaths of MDS switch B, follow these steps:

1.        Log in as admin user and run the following commands:

configure terminal

device-alias database

  device-alias name flex1-hba1 pwwn 20:00:00:25:b5:8b:b0:00

  device-alias name flex1-hba3 pwwn 20:00:00:25:b5:8b:b0:01

  device-alias name flex2-hba1 pwwn 20:00:00:25:b5:8b:b0:02

  device-alias name flex2-hba3 pwwn 20:00:00:25:b5:8b:b0:03

  device-alias name flex3-hba1 pwwn 20:00:00:25:b5:8b:b0:04

  device-alias name flex3-hba3 pwwn 20:00:00:25:b5:8b:b0:05

  device-alias name flex4-hba1 pwwn 20:00:00:25:b5:8b:b0:06

  device-alias name flex4-hba3 pwwn 20:00:00:25:b5:8b:b0:07

  device-alias name NetApp-A700s-01-2B pwwn 20:0c:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-01-3B pwwn 20:08:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-02-2B pwwn 20:0d:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-02-3B pwwn 20:0b:00:a0:98:af:7c:5b 

device-alias commit

copy run start

Create Zoning

Cisco MDS Switch A

To configure zones for the Cisco MDS Switch A, follow these steps:

1.        Create a zone for each service profile as shown below.

2.       Log in as admin user into MDS Switch A and run these commands to create the zone:

configure terminal

zone name flex1 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:00

    member pwwn 20:00:00:25:b5:8a:a0:01

    member pwwn 20:06:00:a0:98:af:7c:5b

    member pwwn 20:07:00:a0:98:af:7c:5b

    member pwwn 20:09:00:a0:98:af:7c:5b

    member pwwn 20:0a:00:a0:98:af:7c:5b

 

zone name flex2 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:02

    member pwwn 20:00:00:25:b5:8a:a0:03

    member pwwn 20:06:00:a0:98:af:7c:5b

    member pwwn 20:07:00:a0:98:af:7c:5b

    member pwwn 20:09:00:a0:98:af:7c:5b

    member pwwn 20:0a:00:a0:98:af:7c:5b

 

zone name flex3 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:04

    member pwwn 20:00:00:25:b5:8a:a0:05

    member pwwn 20:06:00:a0:98:af:7c:5b

    member pwwn 20:07:00:a0:98:af:7c:5b

    member pwwn 20:09:00:a0:98:af:7c:5b

    member pwwn 20:0a:00:a0:98:af:7c:5b

 

zone name flex4 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:06

    member pwwn 20:00:00:25:b5:8a:a0:07

    member pwwn 20:06:00:a0:98:af:7c:5b

    member pwwn 20:07:00:a0:98:af:7c:5b

    member pwwn 20:09:00:a0:98:af:7c:5b

    member pwwn 20:0a:00:a0:98:af:7c:5b

3.        After the zone for the service profile has been created, create the zone set and add the necessary members.

zoneset name flex vsan 101

    member flex1

    member flex2

    member flex3

    member flex4

4.       Activate the zone set by running following commands

zoneset activate name flex vsan 101

copy run start

Cisco MDS Switch B

To configure zones for the Cisco MDS Switch B, follow these steps:

1.        Create a zone for each service profile.

2.       Log in as admin user into MDS Switch B and run the below commands to create the zone:

configure terminal

zone name flex1 vsan 102

    member pwwn 20:00:00:25:b5:8b:b0:00

    member pwwn 20:00:00:25:b5:8b:b0:01

    member pwwn 20:0c:00:a0:98:af:7c:5b

    member pwwn 20:08:00:a0:98:af:7c:5b

    member pwwn 20:0d:00:a0:98:af:7c:5b

    member pwwn 20:0b:00:a0:98:af:7c:5b

 

zone name flex2 vsan 102

    member pwwn 20:00:00:25:b5:8b:b0:02

    member pwwn 20:00:00:25:b5:8b:b0:03

    member pwwn 20:0c:00:a0:98:af:7c:5b

    member pwwn 20:08:00:a0:98:af:7c:5b

    member pwwn 20:0d:00:a0:98:af:7c:5b

    member pwwn 20:0b:00:a0:98:af:7c:5b

 

zone name flex3 vsan 102

    member pwwn 20:00:00:25:b5:8b:b0:04

    member pwwn 20:00:00:25:b5:8b:b0:05

    member pwwn 20:0c:00:a0:98:af:7c:5b

    member pwwn 20:08:00:a0:98:af:7c:5b

    member pwwn 20:0d:00:a0:98:af:7c:5b

    member pwwn 20:0b:00:a0:98:af:7c:5b

 

zone name flex4 vsan 102

    member pwwn 20:00:00:25:b5:8b:b0:06

    member pwwn 20:00:00:25:b5:8b:b0:07

    member pwwn 20:0c:00:a0:98:af:7c:5b

    member pwwn 20:08:00:a0:98:af:7c:5b

    member pwwn 20:0d:00:a0:98:af:7c:5b

    member pwwn 20:0b:00:a0:98:af:7c:5b

3.        After the zone for the service profile has been created, create the zone set and add the necessary members:

zoneset name flex vsan 102

    member flex1

    member flex2

    member flex3

    member flex4

4.       Activate the zone set by running following commands:

zoneset activate name flex vsan 102

copy run start

Verify FC Ports on MDS Switch

To verify FC ports on the MDS switch, follow these steps:

1.        Log in as admin user into MDS Switch A and run the “show flogi database vsan 101” to verify all FC ports.

2.       Log in as admin user into MDS Switch B and run the “show flogi database vsan 101” to verify all FC ports.

Configure NetApp AFF A700s Storage

NetApp Storage Connectivity

It is beyond the scope of this document to describe all the detailed information about NetApp storage connectivity and infrastructure configuration. Follow the link below for the installation and setup instructions for the NetApp AFF A700s System:

https://library.netapp.com/ecm/ecm_download_file/ECMLP2619982

For more technical information, refer to the following Cisco site:

https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/flexpod-design-guides.html

Please refer to the following technical reports of “Oracle Databases on ONTAP” and “Database Data Protection: Backup, Recovery, Replication, and DR” from NetApp for the best practices:

https://www.netapp.com/us/media/tr-3633.pdf

https://www.netapp.com/us/media/tr-4591.pdf

This section describes the storage layout and design considerations for the database deployment.

A NetApp ONTAP cluster serves data through at least one and possibly multiple storage virtual machines (SVMs; formerly called Vservers). An SVM is a logical abstraction that represents the set of physical resources of the cluster. Data volumes and network logical interfaces (LIFs) are created and assigned to an SVM and might reside on any node in the cluster to which the SVM has been given access. An SVM might own resources on multiple nodes concurrently, and those resources can be moved non-disruptively from one node in the storage cluster to another. For example, a NetApp FlexVol® flexible volume can be non-disruptively moved to a new node, or a data LIF can be transparently reassigned to a different physical network port. The SVM abstracts the cluster hardware, and thus it is not tied to any specific physical hardware.

As shown above, for both the storage controller nodes (FlexPod-a700s-01 and FlexPod-a700s-02), we used ports 2a, 2b and 3a, 3b to configure LIFs. WWPN of these LIFs are used for zoning into the MDS switches for storage to MDS connectivity. The screenshot below shows the network interface configuration for this solution.

For all the database deployment, we configured two aggregates (one aggregate on each storage node) into a single SVM (ORA12C_SVM) and each aggregate contains 35 SSD (960GB Each) drives that were subdivided into RAID DP groups, plus one spare drive as shown below.

For each RAC database deployment, we distributed equal number of volumes and LUNs on both the storage node by placing those volumes and LUNs into both the aggregate.

The design goal of the reference architecture was to best represent a real-world environment as closely as possible. The approach included features of Cisco UCS to rapidly deploy stateless servers and use NetApp AFF Storage A700s boot LUNs to provision the O.S on top it. Zoning was performed on the Cisco MDS 9148S switches to enable the initiators discover the targets during boot process.

A Service Profile was created within Cisco UCS Manager to deploy the 4 servers quickly with a standard configuration. SAN boot volumes for these servers were hosted on the NetApp AFF Storage A700s Cluster. Once the stateless servers were provisioned, following process was performed to enable Rapid deployment of 4 RAC nodes.

Each Server node has dedicated single LUN to install operating system and all the four server node was booted off SAN. For this solution, we have installed Oracle Linux 7.5(4.1.12-124.16.4.el7uek.x86_64) on this LUNs and performed all the pre-requisite packages for Oracle Database 12cR2 to create four node Oracle RAC database solution.

Operating System Configuration

1.        Download Oracle Linux 7.5 OS image from https://edelivery.oracle.com/linux.

2.       Launch KVM console on desired server by going to tab Equipment > Chassis > Chassis 1 > Servers > Server 1 > from right side windows General > and select KVM Console to open KVM.

3.        Click Accept security and open KVM. Enable virtual media, map the Oracle Linux ISO image and reset the server.

4.       When the Server starts booting, it will detect the NetApp Storage active FC paths as shown below. If you see the following message in the KVM console while the server is rebooting along with the target WWPNs, it confirms the setup is done correctly and boot from SAN will be successful.

5.        During server boot order, it will detect the virtual media connected as Oracle Linux cd. It should launch the Oracle Linux installer. Select language and assign the Installation destination as NetApp Storage LUN. Apply hostname and click “Configure Network” to configure all network interfaces. Alternatively, you can only configure “Public Network” in this step. You can configure additional interfaces as part of post install steps.

6.       After the OS install, reboot the server, complete appropriate registration steps. You can choose to synchronize the time with NTP server. Alternatively, you can choose to use Oracle RAC cluster synchronization daemon (OCSSD). Both NTP and OCSSD are mutually exclusive and OCSSD will be setup during GRID install if NTP is not configured.

Operating System Prerequisites for Oracle Software Installation

Configure BIOS

This section describes how to optimize the BIOS settings to meet requirements for the best performance and energy efficiency for the Cisco UCS M5 generation of blade servers.

Configure BIOS for OLTP Workloads

OLTP systems are often decentralized to avoid single points of failure. Spreading the work over multiple servers can also support greater transaction processing volume and reduce response time. Make sure to disable Intel IDLE driver in the OS configuration section. When Intel idle driver is disabled, the OS uses acpi_idle driver to control the c-states.

The following options are recommended for optimizing OLTP workloads on Cisco UCS M5 platforms managed by Cisco UCS Manager.

For more information about BIOS settings, refer to: https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-b-series-blade-servers/whitepaper_c11-740098.pdf

If the CPU gets into a deeper C-state and not able to get out to deliver full performance quickly. The result is unwanted latency spikes for workloads. To address this, it is recommended to disable C states in the BIOS and in addition, Oracle recommends disabling it from OS level as well by modifying grub entries. For this solution, we have configured BIOS options by modifying in /etc/default/grub file as shown below:

[root@flex1 ~]# cat /etc/default/grub

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

GRUB_DEFAULT=saved

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL_OUTPUT="console"

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never biosdevname=0 net.ifnames=0 intel_idle.max_cstate=0 processor.max_cstate=0 rdloaddriver=scsi_dh_alua"

GRUB_DISABLE_RECOVERY="true"

Prerequisites Automatic Installation

After installing Oracle Linux 7.5(4.1.12-124.16.4.el7uek.x86_64) on all the server nodes (flex1, flex2, flex3 and flex4), you have to configure operating system pre-requisites on all the four nodes to successfully install Oracle RAC Database 12cR2.

To configure operating system pre-requisite for Oracle 12cR2 software on all four nodes, follow these steps:

To configure the prerequisites on all the four nodes, follow these steps:

For this solution, we have configured the prerequisites automatically by installing the “oracle-database-server-12cR2-preinstall" rpm package. You can also download the required packages from http://public-yum.oracle.com/oracle-linux-7.html. If you plan to use the “oracle-database-server-12cR2-preinstall" rpm package to perform all your prerequisite setup automatically, then log in as root user and issue the following command:

[root@flex1 ~]# yum install oracle-database-server-12cR2-preinstall

Additional Prerequisites Setup

After configuring automatic or manual prerequisites steps, you have to configure a few additional steps to complete the prerequisites for the Oracle database software installations on all the four nodes as described in the following sections.

Disable SELinux

Since most Organizations might be running hardware-based firewalls to protect their corporate networks, we disabled Security Enhanced Linux (SELinux) and the firewalls at the server level for this reference architecture.

You can set secure Linux to permissive by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follows.

SELINUX= permissive

Disable Firewall

Check the status of the firewall by running following commands. (The status displays as active (running) or inactive (dead)). If the firewall is active / running, enter this command below to stop it.

systemctl status firewalld.service

systemctl stop firewalld.service

Also, to completely disable the firewalld service, so it does not reload when you restart the host machine, run the following command:

systemctl disable firewalld.service

Set the User Passwords

Run the following commands to change the password for Oracle and Grid Users:

passwd oracle

passwd grid

Configure Multipath Setup

 

You can configure DM-Multipath for use in multipathing in environments that use native Linux solutions. With DM-Multipath, you can configure multiple I/O paths between a host and storage controllers into a single device. If one path fails, DM-Multipath reroutes I/Os to the remaining paths. Configure multipaths to access the LUNs presented from NetApp Storage to the nodes as shown below.

Add or modify “/etc/multipath.conf” file accordingly to give the alias name of each LUN id presented from NetApp Storage as given below into all eight nodes:

Run “multipath –ll” command to view all the LUN id:

[root@oraracx1 ~]# cat /etc/multipath.conf

defaults {

        find_multipaths yes

        user_friendly_names no

}

multipaths {

        multipath {

                wwid            3600a098038304173475d4c766a49744e

                alias           node1_os

        }

        multipath {

                wwid            3600a098038304173475d4c766a49754a

                alias           ocrvote_1

        }

}

 

Configure UDEV Rules

You need to configure UDEV rules to assign permission in all the Oracle RAC nodes to access NetApp Storage LUNs. This includes the device details along with required permissions to enable grid and oracle user to have read/write privileges on these devices. Configure UDEV rules on all the Oracle Nodes as shown below:

Create a new file named “/etc/udev/rules.d/99-oracleasm.rules” with the following entries on all nodes:

 [root@flex1 ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules

#All LUNs which starts with dg_orarac_* #

ENV{DM_NAME}=="ocrvote_*", OWNER:="grid", GROUP:="oinstall", MODE:="660"

#All LUNs which starts with dg_oradata_* #

ENV{DM_NAME}=="oradata_*", OWNER:="oracle", GROUP:="oinstall", MODE:="660"

#All LUNs which starts with dg_oraredo_* #

ENV{DM_NAME}=="oraredo_*", OWNER:="oracle", GROUP:="oinstall", MODE:="660"

#All LUNs which starts with dg_oraarchive_* #

ENV{DM_NAME}=="oraarchive_*", OWNER:="oracle", GROUP:="oinstall", MODE:="660"

Configure Public and Private NICs on Each RAC Node

If you have not configured network settings during OS installation, then configure it now. Each node must have at least two network interface cards (NICs), or network adapters. One adapter is for the public network traffic and the other adapter is for the private network traffic (the node interconnects).

Log in as a root user on each node and go to “/etc/sysconfig/network-scripts” and configure Public network and Private network IP Address. Configure the private and public NICs with the appropriate IP addresses across all the Oracle RAC nodes.

Configure “/etc/hosts” on Each RAC Node

Log in as a root user into node and edit “/etc/hosts” file. Provide the details for Public IP Address, Private IP Address, SCAN IP Address and Virtual IP Address for all nodes. Configure these settings on each Oracle RAC Nodes as shown below:

[root@flex1 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

 

##### Public IP #####

10.29.135.121   flex1.cisco.com flex1

10.29.135.122   flex2.cisco.com flex2

10.29.135.123   flex3.cisco.com flex3

10.29.135.124   flex4.cisco.com flex4

##### Virtual IP #####

10.29.135.125   flex1-vip       flex1-vip.cisco.com

10.29.135.126   flex2-vip       flex2-vip.cisco.com

10.29.135.127   flex3-vip       flex3-vip.cisco.com

10.29.135.128   flex4-vip       flex4-vip.cisco.com

##### Private IP #####

10.10.10.121  flex1-priv        flex1-priv.cisco.com

10.10.10.122  flex2-priv        flex2-priv.cisco.com

10.10.10.123  flex3-priv        flex3-priv.cisco.com

10.10.10.124  flex4-priv        flex4-priv.cisco.com

#SCAN IP

10.29.135.129   flex-cluster    flex-cluster.cisco.com

10.29.135.130   flex-cluster    flex-cluster.cisco.com

10.29.135.131   flex-cluster    flex-cluster.cisco.com

·         A Public IP Address for each node

·         A Virtual IP address for each node

·         Three single client access name (SCAN) address for the oracle database cluster

These steps complete the prerequisite for Oracle Database 12cR2 Installation at OS level on Oracle RAC Nodes.

 

 

When the O.S level prerequisites are completed, you are ready to install the Oracle Grid Infrastructure as a grid user. Download Oracle Database 12c Release 2 (12.2.0.1.0) for Linux x86-64 and Oracle Database 12c Release 2 Grid Infrastructure (12.2.0.1.0) for Linux x86-64 software from Oracle Software site. Copy these software binaries to Oracle RAC Node 1 and unzip all files into appropriate directories.

Oracle Database 12c GRID Infrastructure Deployment

For this Oracle Database FlexPod solution, you will install the Oracle Grid and Database software on all four nodes (flex1, flex2, flex3 and flex4). The installation guides you through the gathering of all node information and configuring the ASM devices and all the prerequisite validations for GI. 

This section describes the high-level steps for Oracle Database 12c R2 RAC install. Prior to GRID and database install, verify all the prerequisites are completed. As an alternative, you can install Oracle validated RPM that will make sure all prerequisites are meet before Oracle grid install.

For the Oracle Database 12c Release 2 install and upgrade guide, click this link: https://docs.oracle.com/en/database/oracle/oracle-database/12.2/install-and-upgrade.html

Create Directory Structure

 

1.        Log in as a root user and create the following directory structure.

mkdir -p /u01/app/grid

mkdir -p /u01/app/12.2.0/grid

mkdir -p /u01/app/oracle/product/12.2.0/dbhome_1

 

chown -R grid:oinstall /u01/app/grid/

chown -R grid:oinstall /u01/app/12.2.0/grid/

chown -R oracle:oinstall /u01/app/oracle/

2.       Log in as a grid user, download the Oracle Grid Infrastructure image files and unzip the files into the Grid home:

cd /u01/app/12.2.0/grid

unzip -q download_location/linuxx64_12201_grid_home

Run Cluster Verification Utility

This step verifies that all the prerequisites are meet to install Oracle Grid Infrastructure Software. Oracle Grid Infrastructure ships with the Cluster Verification Utility (CVU) that can run to validate pre and post installation configurations. To run this utility, log in as Grid User in Oracle RAC Node 1 and go to the directory where oracle grid software binaries are located. Run script named as “runcluvfy.sh” as follows:

./runcluvfy.sh stage -pre crsinst -n flex1,flex2,flex3,flex4 –verbose

Configure HugePages

HugePages is a method to have a larger page size that is useful for working with a very large memory. For Oracle Databases, using HugePages reduces the operating system maintenance of page states and increases Translation Lookaside Buffer (TLB) hit ratio.

Advantages of HugePages:

·         HugePages are not swappable so there is no page-in/page-out mechanism overhead.

·         HugePages uses fewer pages to cover the physical address space, so the size of "book keeping"(mapping from the virtual to the physical address) decreases, so it requiring fewer entries in the TLB and so TLB hit ratio improves.

·         HugePages reduces page table overhead. Also, HugePages eliminated page table lookup overhead: Since the pages are not subject to replacement, page table lookups are not required.

·         Faster overall memory performance: On virtual memory systems each memory operation is actually two abstract memory operations. Since there are fewer pages to work on, the possible bottleneck on page table access is clearly avoided.

Please refer to the Oracle support for HugePages configuration details: https://docs.oracle.com/en/database/oracle/oracle-database/12.2/unxar/administering-oracle-database-on-linux.html#GUID-CC72CEDC-58AA-4065-AC7D-FD4735E14416

Install and Configure Oracle Database Grid Infrastructure Software

 

To install Oracle Database Grid Infrastructure Software, follow these steps:

1.        Go to grid home where the Oracle 12c R2 Grid Infrastructure software binaries are located and launch the installer as the "grid" user.

2.       Start the Oracle Grid Infrastructure installer by running the following command:

./gridSetup.sh

3.        Select option “Configure Oracle Grid Infrastructure for a New Cluster” as shown below, then click Next.

4.       Select cluster configuration options “Configure an Oracle Standalone Cluster”, then click Next.

5.        In next window, enter the Cluster Name and SCAN Name fields.

6.       In Cluster node information window, click the "Add" button to add all four nodes Public Hostname and Virtual Hostname as shown below:

7.        As shown above, you will see all nodes listed in the table of cluster nodes. Make sure the Role column is set to HUB for all four nodes. Click the SSH Connectivity button at the bottom of the window. Enter the operating system user name and password for the Oracle software owner (grid). Click Setup.

8.       A message window appears, indicating that it might take several minutes to configure SSH connectivity between the nodes. After sometime, another message window appears indicating that password-less SSH connectivity has been established between the cluster nodes. Click OK to continue.

9.       In Network Interface Usage screen, select the usage type for each network interface displayed as shown below:

10.    Select the Oracle ASM storage configuration option as “Configure ASM using block devices.” Click Next and Choose “No” option into separate ASM disk group for the Grid Infrastructure Management Repository data, and then click Next.

11.     In the Create ASM Disk Group window, select the OCRVOTE LUNs assigned from NetApp Storage to store OCR and Voting disk files. Enter the name of disk group as OCRVOTE and select appropriate redundancy options as show below.

12.    Choose the password for the Oracle ASM SYS and ASMSNMP account, then click Next.

13.     Select the option “Do not use Intelligent Platform Management Interface (IPMI)”, then click Next. You can configure to have this instance of Oracle Grid Infrastructure and Oracle Automatic Storage Management to be managed by Enterprise Manager Cloud Control. Specify the details of the Cloud Control configuration to perform and click Next.

14.    Select the appropriate operating system group names for Oracle ASM according to your environments.

15.     Specify the directory to use for the Oracle base for the Oracle Grid Infrastructure installation and then click Next. The Oracle base directory must be different from the Oracle home directory. Click Next and choose Inventory Directory according to your setup.

16.    Click Automatically run configuration scripts to run scripts automatically and enter the relevant root user credentials. Click Next.

17.     Wait while the prerequisite checks complete. If you have any issues, use the "Fix & Check Again" button.

18.    Review the contents of the Summary window and then click Install. The installer displays a progress indicator enabling you to monitor the installation process.

19.    Wait for the grid installer configuration assistants to complete.

20.    When the configuration complete successfully, click Close to finish and exit the grid installer.

21.    When GRID install is successful, login to each of the nodes and perform minimum health checks to make sure that Cluster state is healthy. After your Oracle Grid Infrastructure installation is complete, you can install Oracle Database on a cluster node for high availability, or install Oracle RAC.

Install Oracle Database Software

 

To install Oracle Database Software, follow these steps:

1.        Start the runInstaller command from the Oracle Database 12c Release 2 (12.2) installation media where Oracle database software is located.

2.       Select option “Install database software only” into Select Installation Option.

3.        Select option "Oracle Real Application Clusters database installation" and click Next.

4.       Select nodes in the cluster where installer should install Oracle RAC. For this setup, you will install software on all nodes as shown below.

5.        Click the "SSH Connectivity..." button and enter the password for the "oracle" user. Click the "Setup" button to configure passwordless SSH connectivity, and the "Test" button to test it when it is complete. When the test is complete, click Next.

6.       Select Database Edition Options according to your environments and then click Next.

7.        Enter Oracle Base as "/u01/app/oracle" and "/u01/app/oracle/product/12.2.0/dbhome_1" as the software location, then click Next.

8.       Select the desired operating system groups and then click Next.

9.       Wait for the prerequisite check to complete. If there are, any problems either click the "Fix & Check Again" button or try to fix those by checking and manually installing required packages. Click Next.

10.    Verify the Oracle Database summary information, click Install.

11.     When prompted, run the configuration script on each node. When the scripts run successfully on each node then click OK.

12.    After the installation of Oracle Database finish successful, click Close to exit of the installer.

Oracle Flex ASM

Oracle Flex ASM enables an Oracle ASM instance to run on a separate physical server from the database servers. With this deployment, larger clusters of Oracle ASM instances can support more database clients while reducing the Oracle ASM footprint for the overall system.

When using Oracle Flex ASM, Oracle ASM clients are configured with direct access to storage. With Oracle Flex ASM, you can consolidate all the storage requirements into a single set of disk groups. All these disk groups are mounted by and managed by a small set of Oracle ASM instances running in a single cluster. You can specify the number of Oracle ASM instances with a cardinality setting. The default is three instances.

Prior to Oracle 12c, if ASM instance on one of the RAC nodes crashes, all the instances running on that node will crash too. This issue has been addressed in Flex ASM; Flex ASM can be used even if all the nodes are hub nodes. However, GNS configuration is mandatory for enabling Flex ASM. We can check what instances are connected with a simple query as shown below.

As you can see from the above query, instance1 (flex1), instance2 (flex2) and instance4 (flex4) are connected to +ASM. Also, the screenshot below shows a few more commands to check the cluster and FLEX ASM details.

Before configuring a database for workload tests, it is extremely important to validate that this is indeed a balanced configuration that is capable of delivering expected performance. In this solution, you will test and validate node and user scalability on all 4 node Oracle RAC Databases with various database benchmarking tools as explained below.

Hardware Calibration Test using FIO

FIO is short for Flexible IO, a versatile IO workload generator. FIO is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. For our solution, we use FIO to measure the performance of a NetApp storage device over a given period of time. For the FIO Tests, we created 16 volumes and each volume has one LUN. These 16 LUNs were shared across all the four nodes for read/write IO operations.

We run various FIO tests for measuring IOPS, Latency and Throughput performance of this solution by changing block size parameter into the FIO test. For each FIO test, we also changed read/write ratio as 0/100 percent read/write, 50/50 percent read/write, 70/30 percent read/write, 90/10 percent read/write and 100/0 percent read/write to scale the performance of the system. We also ran the tests for at least 4 hours to help ensure that this configuration is capable of sustaining this type of load for longer period of time.

4k and 8k Random Read/Write IOPs Tests

The following chart shows results for the random read/write FIO test for the 4k block size.

For the 100/0 percent read/write test, we achieved about 860k IOPS with the read latency about 1.1 millisecond. Similarly, for the 90/10 percent read/write test, we achieved about 778k IOPS with the read latency about 1.1 millisecond and the write latency about 1.7 millisecond. For the 70/30 percent read/write test, we achieved 732k IOPS with the read latency about 1 millisecond and the write latency about 1.9 millisecond. For the 50/50 percent read/write test, we achieved about 592k IOPS with the read latency about 0.4 millisecond and the write latency about 2.8 millisecond. For the 0/100 percent read/write test, we achieved 180k IOPS with the write latency about 5.3 millisecond.

The chart below shows results for the random read/write FIO test for the 8k block size representing OLTP type of workloads.

For the 100/0 percent read/write test, we achieved about 891k IOPS with the read latency about 0.8 millisecond. Similarly, for the 90/10 percent read/write test, we achieved about 800k IOPS with the read latency about 0.9 millisecond and the write latency about 1.3 millisecond. For the 70/30 percent read/write test, we achieved about 732k IOPS with the read latency about 0.7 millisecond and the write latency about 1.7 millisecond. For the 50/50 percent read/write test, we achieved about 496k IOPS with the read latency about 0.4 millisecond and the write latency about 2.6 millisecond. For the 0/100 percent read/write test, we achieved about 156k IOPS with the write latency about 4.9 millisecond.

Bandwidth Tests

The bandwidth tests are carried out with 512k and 1MB IO Size and represents the DSS database type workloads. The chart below shows results for the sequential read/write FIO test for the 512k block size.

For the 100/0 percent read/write test, we achieved about 11.3 GB/s throughput with the read latency about 2.9 millisecond. Similarly, for the 90/10 percent read/write test, we achieved about 11.5 GB/s throughput with the read latency about 2.8 millisecond and the write latency about 3.5 millisecond. For the 70/30 percent read/write test, we achieved about 8.3 GB/s throughput with the read latency about 2.3 millisecond and the write latency about 6.3 millisecond. For the 50/50 percent read/write test, we achieved about 7.1 GB/s throughput with the read latency about 3.1 millisecond and the write latency about 8 millisecond. For the 0/100 percent read/write test, we achieved about 4.1 GB/s throughput with the write latency about 8.1 millisecond.

The chart below shows results for the sequential read/write FIO test for the 1MB block size.

For the 100/0 percent read/write test, we achieved around 8.8 GB/s throughput with the read latency around 7.6 millisecond. Similarly, for the 90/10 percent read/write test, we achieved about 10.2 GB/s throughput with the read latency about 6.5 millisecond and the write latency about 6.7 millisecond. For the 70/30 percent read/write test, we achieved about 10.3 GB/s throughput with the read latency about 5.5 millisecond and the write latency about 8.8 millisecond. For the 50/50 percent read/write test, we achieved about 7.1 GB/s throughput with the read latency about 6.6 millisecond and the write latency about 7.1 millisecond. For the 0/100 percent read/write test, we achieved about 4.1 GB/s throughput with the write latency about 16.1 millisecond.

The slight decrease in bandwidth between the 90 percent read workload and 100 percent read workload is a result of ONTAP's extremely efficient write processing. Inbound write data is journaled into mirrored NVRAM. At this point, the write operation is complete and on durable media, and the IO is then acknowledged to the client. NVRAM is not cache, it is a journal that is only used for recovery of an interrupted write operation. The actual write operation to the SSD drives is a direct memory-to-drive transfer. The bandwidth and latency characteristics of NVRAM exceeds even that of SSD drives, so any tests with write IO will result in an increase in overall bandwidth compared to 100percent read tests where all IO must be drawn from slower Flash-based media.

We did not see any performance dips or degradation over the period of run time. It is also important to note that this is not a benchmarking exercise and the numbers presented are not the peak numbers where there is hardware resource saturation. These are practical and out of box test numbers that can be easily reproduced by any one. At this time, we are ready to create OLTP database(s) and continue with database tests.

Database Creation with DBCA

We used Oracle Database Configuration Assistant (DBCA) to create three OLTP (SLOB, SOE and OLTP) and one DSS (DSS) databases for SLOB and Swingbench test calibration. Alternatively, you can use Database creation scripts to create the databases as well.

For all the database deployment, we have configured two aggregates (one aggregate on each storage node) into a single SVM and each aggregate contains 35 SSD (960GB Each) drives that were subdivided into RAID DP groups, plus one spare drive as explained earlier in the storage configuration section.

For each RAC database, we have created total number of 18 volumes and each volume contains one LUN. We distributed equal number of volumes and LUNs on both the storage node by placing those volumes and LUNs into both the aggregates. All the databases files were also spread evenly across the 2 nodes of the storage system so that each storage node served data for the databases. The screenshot below shows the storage layout of all the volumes and LUN configuration for all the databases.

As shown above, for each database, we created total 18 volumes and each volume contains one LUN. On these 18 LUNs, we created two disk groups to store the data and redolog files for the database. We used 16 LUNs to create Oracle ASM “Data” disk group and 2 LUNs to create Oracle ASM “redolog” disk group for each database.

We used a widely adopted SLOB and Swingbench database performance test tools to test and validate throughput, IOPS, and latency for various test scenarios as explained below.

SLOB Test

The Silly Little Oracle Benchmark (SLOB) is a toolkit for generating and testing I/O through an Oracle database. SLOB is very effective to test the I/O subsystem with genuine Oracle SGA-buffered physical I/O. SLOB supports testing physical random single-block reads (db file sequential read) and random single block writes (DBWR flushing capability). SLOB issues single block reads for the read workload that are generally 8K (as the database block size was 8K).

To test the SLOB workload, we created one database as SLOB. For SLOB database, we created total 18 volumes and each volume contains one LUN. On these 18 LUNs, we created two disk groups to store the data and redolog files for the SLOB database. First disk-group “DATASLOB” was created with 16 LUNs (300 GB each) while second disk-group “REDOSLOB” was created with two LUNs (100 GB each) as shown below.

Those ASM disk groups provided the storage required to create the tablespaces for the SLOB Database. We loaded SLOB schema on “DATASLOB” disk-group of up to 3 TB in size.

We used SLOB2 to generate our OLTP workload. Each database server applied the workload to Oracle database, log, and temp files. The following tests were performed and various metrics like IOPS and latency were captured along with Oracle AWR reports for each test scenario.

User Scalability Test

SLOB2 was configured to run against all the four RAC nodes and the concurrent users were equally spread across all the nodes. We tested the environment by increasing the number of Oracle users in database from a minimum of 32 users up to a maximum of 512 users across all the four nodes. At each load point, we verified that the storage system and the server nodes could maintain steady-state behavior without failure. We also made sure that there were no bottlenecks across servers or networking systems.

We performed User Scalability test with 32, 64, 128, 192, 256 and 512 users on 4 Oracle RAC nodes by varying read/write ratio as explained below:

·         Varying workloads

-        100% read (0% update)

-        90% read (10% update)

-        70% read (30% update)

-        50% read (50% update)

The following table illustrate total number of IOPS (both read and write) for user scalability test when run with 32, 64, 128, 192, 256 and 512 Users on the SLOB database.

Table 10      Total IOPS for SLOB User Scalability Tests

The following graphs demonstrate total number of IOPS while running SLOB workload for various concurrent users for each test scenario.

The above graph shows the linear scalability with increased users and similar IOPS from 32 users to 512 users with 100% read, 90% read, 70% read and 50% read.

The AWR screenshot below was captured from a 100% Read (0% update) Test scenario while running SLOB test for 512 users. The snapshot shows a section from the Oracle AWR report from the run that highlights Physical Reads/Sec and Physical Writes/Sec for each instance.

The screenshot above highlights that IO load is distributed across all the cluster nodes performing workload operations. Due to variations in workload randomness, we conducted multiple runs to help ensure consistency in behavior and test results.

The screenshot below was captured from a 70% Read (30% update) Test scenario while running SLOB test for 512 users. The snapshot shows a section from AWR report from the run that highlights Physical Reads/Sec and Physical Writes/Sec for each instance.

The following graph illustrates the latency exhibited by the NetApp AFF A700s Storage across different workloads. All the workloads experienced less than 1 millisecond latency and it varies based on the workloads. As expected, the 50% read (50% update) test exhibited higher latencies as the user counts increases.

The following screenshot was captured from 100 % Read (0% Update) Test scenario while running SLOB test for 512 users. The screenshot shows a section of AWR report from the run that highlights top timed Events.

Swingbench Test

We used Swingbench for OLTP and DSS workload testing. Swingbench is a simple to use, free, Java-based tool to generate database workload and perform stress testing using different benchmarks in Oracle database environments. Swingbench can be used to demonstrate and test technologies such as Real Application Clusters, Online table rebuilds, Standby databases, online backup and recovery, etc.

Swingbench provides four separate benchmarks, namely Order Entry, Sales History, Calling Circle, and Stress Test. For the tests described in this solution, Swingbench Order Entry benchmark was used for OLTP workload testing and the Sales History benchmark was used for the DSS workload testing.

The Order Entry benchmark is based on SOE schema and is TPC-C like by types of transactions. The workload uses a very balanced read/write ratio about 60/40 and can be designed to run continuously and test the performance of a typical Order Entry workload against a small set of tables, producing contention for database resources.

The Sales History benchmark is based on the SH schema and is TPC-H like. The workload is query (read) centric and is designed to test the performance of queries against large tables.

Typically encountered in the real-world deployments, we tested a combination of scalability and stress related scenarios that ran on all the 4-node Oracle RAC cluster configuration.

·         OLTP database user scalability representing small and random transactions

·         DSS database workload representing larger transactions

·         Mixed workload featuring OLTP and DSS database workloads running simultaneously for 24 hours

For Swingbench workload, we created two OLTP (OLTP and SOE) and one DSS (DSS) database to demonstrate database multi-tenancy capability, performance and sustainability.  We created approximately 3 TB of OLTP Database, 4 TB of SOE Database and 6 TB of DSS database to perform Order Entry and Sales Entry swingbench workload testing.

The first step after the databases creation is calibration; about the number of concurrent users, nodes, throughput, IOPS and latency for database optimization. For this FlexPod solution, we have tested system performance with different databases running at a time and capture the results as explained in the following sections.

One (SOE) Database Performance

For one OLTP database workload featuring Order Entry schema, we used one SOE database. For the SOE database (4 TB), we used 128GB size of System Global Area (SGA). We also made sure that HugePages were in use. The OLTP Database scalability test was run for at least 12 hours and made sure that results are consistent for the duration of the full run.

We ran the SwingBench scripts on each node to start SOE database and generate AWR reports for each scenario as shown below.

User Scalability

Table 11  lists the Transaction Per Minutes (TPM), IOPS and System Utilization for SOE Database while running swingbench workloads from 100 users to 800 users.

Table 11       TPM, TPS, IOPS and System Utilization for SOE Database User Scale Tests

The chart below shows the IOPS and Latency of SOE database while running swingbench workload for 100 users to 800 users on all 4 RAC nodes.

The chart below shows TPM and System Utilization of the system while running swingbench workload for 100 user to 800 users.

The screenshot below was captured from the 800 User Scale Test scenario while running Swingbench workload on SOE database. The snapshot shows a section from 4-hour window of AWR Global report from the run that highlights Physical Reads/Sec and Physical Writes/Sec for each instance. Notice that IO load is distributed across all the cluster nodes performing workload operations.

The AWR screenshot below shows top timed events for the same 800 User Scale Test while Swingbench test was running.

The Oracle Enterprise Manager screenshot below shows All Wait Events, IO Requests Per Second, CPU Utilization for the same 800 User Scale Test while Swingbench test was running for 24 hours.

Two (SOE and OLTP) Database Performance

For two OLTP database workload featuring Order Entry schema, we used SOE and OLTP database. For both the databases, we used 128GB of System Global Area (SGA). We also made sure that HugePages were in use all the time while databases were running. The SOE + OLTP Database scalability test were run for at least 12 hours and ensured that results are consistent for the duration of the full run.

Notice that SOE database is of approximately 4 TB in size, while OLTP database is 3 TB in size. While running below tests on both the database together, we scaled more users on the SOE database compared to the OLTP database. We ran the Swingbench scripts on each node to start SOE and OLTP database and generate AWR reports for each scenario as shown below.

User Scalability

Table 12  lists the IOPS and System Utilization for SOE + OLTP databases running together Swingbench workloads from 200 users to 800 users across all the four RAC nodes:

Table 12      Total IOPS for SLOB + OLTP Database for User Scalability Tests

The graph below demonstrates the total IOPS for both SOE and OLTP Database while running Swingbench workload together from 200 users to 800 users.

Table 13  lists the TPM and System Utilization for SOE + OLTP databases running Swingbench workloads together from 100 users to 800 users.

Table 13       Total TPM for SLOB + OLTP Database for User Scalability Tests

The graph below illustrates the total IOPS for both SOE and OLTP Database while running Swingbench workload together from 200 users to 800 users.

The results were in line with prior assessments where the user scalability was almost linear till 600 users and beyond 600 users the rate of IOPS increase slowed down.

The screenshot shown below was captured from the 800 Users Scale Test scenario while running Swingbench workload on two database at the same time for 24 hours. The snapshot shows a section from 24-hour window of AWR Global report from the run that highlights Physical Reads/Sec, Physical Writes/Sec and Transactions per Seconds for each instance for both the databases. Notice that IO load is distributed across all the cluster nodes performing workload operations.

The screenshot below shows top timed events for SOE database while running the same above Swingbench test for 24 hours.

The screenshot below shows top timed events for OLTP database while running the same above Swingbench test for 24 hours.

The screenshots shown below were captured from the Oracle Enterprise Manager while running 800 users Swingbench workload test for 24 hours.

Figure 3  Wait Events, IO Requests per Second, and CPU Utilization for the SOE Database

Figure 4  Wait Events, IO Requests per Second and CPU Utilization for the OLTP Database

One (DSS) Database Performance

DSS database workloads are generally sequential in nature, read intensive and exercise large IO size. DSS database workload runs a small number of users that typically exercise extremely complex queries that run for hours. We configured 6 TB of DSS database by loading Swingbench sh schema into Datafile Tablespace.

DSS Database activity is captured using Oracle Enterprise Manager for 24 hour Swingbench workload test as shown below.

For 24 hour DSS workload test, we observed the total sustained IO bandwidth average was about 10.5 GB/sec after the initial ramp up workload. As shown in above screenshot, the IO was consistent throughout the run and we did not observe any significant dips in performance for complete period of time.

The screenshot shown below was captured from Oracle AWR report while running Swingbench SH workload on DSS database for 24 hours.

The screenshot below shows Wait Class events for each Instance of DSS Database while running Swingbench workload test for 24 hours.

The screenshot below shows IO throughput and CPU utilization for each instance of DSS Database while running Swingbench workload test for 24 hours.

All Three (SOE, OLTP and DSS) Database Performance

In this test, we ran both OLTP (SOE+OLTP) and DSS (DSS) Database Swingbench workload at the same time to measure the system performance on small random queries presented via OLTP databases as well as large and sequential transactions submitted via DSS database workload.

The screenshots shown below were captured from Oracle AWR reports while running the Swingbench workload tests on all three database at the same time for 24 hours. We achieved about 128k IOPS for SOE database, 68k IOPS for OLTP database and 4.8 GB/s throughput for DSS database.

SOE Database AWR snapshot that highlights Physical Reads/Sec, Physical Writes/Sec and Transactions per Seconds.

OLTP Database AWR snapshot that highlights Physical Reads/Sec, Physical Writes/Sec and Transactions per Seconds.

DSS Database AWR snapshot that highlights database MB per Seconds.

The screenshots shown below were captured from the Oracle Enterprise Manager while running the Swingbench workload tests on all three database together for 24 hours.

Figure 5  SOE Database Wait Events per Instance, IO Requests per Second and CPU Utilization

Figure 6  OLTP Database Wait Events per Instance, IO Requests per Second and CPU Utilization

Figure 7  DSS Database Wait Events per Instance, IO Bytes and CPU Utilization

 

 

The goal of these tests is to help ensure that reference architecture withstands commonly occurring failures due to either unexpected crashes, hardware failures or human errors. We conduct many hardware (disconnect power), software (process kills) and OS specific failures that simulate real world scenarios under stress condition. In the destructive testing, we also demonstrate unique failover capabilities of Cisco UCS components.

Table 14      Hardware Failover Tests

Figure 8  The FlexPod Solution Infrastructure Diagram Under Normal Operating Conditions

The screenshot below shows a complete infrastructure details of MAC address and VLAN information for Cisco UCS Fabric Interconnect – A switch before failover test.

Log into Cisco Fabric Interconnect – A and “connect nxos a” then type “show mac address-table” to see all VLAN connection on Fabric Interconnect – A as shown below:

As shown in the screenshot above, Fabric Interconnect – A carry Oracle Public Network traffic on VLAN 135 under normal operating conditions before failover test.

Log in to Cisco Fabric Interconnect – B and “connect nxos b” then type “show mac address-table” to see all VLAN connection on Fabric – B as shown in the screenshot below:

As shown in the above screenshot, Fabric Interconnect – B carry Oracle Private Network traffic on VLAN 10 under normal operating conditions before failover test.

Test 1 – Cisco UCS 6332-16UP Fabric Interconnect – A Failure Test

We conducted a hardware failure test on Fabric Interconnect – A by disconnecting power cable to the switch as explained below.

The figure below illustrates how during Fabric Interconnect – A switch failure, the respective blades (flex1 and flex2) on chassis 1 and (flex3 and flex4) on chassis 2 will fail over the public network interface MAC addresses and its VLAN network traffic to fabric interconnect – B.

Unplug the power cable from Fabric Interconnect – A, and check the MAC address and VLAN information on Cisco UCS Fabric Interconnect – B.

We noticed (as shown in the screenshot above), when the Fabric Interconnect – A failed, it routed all the Public Network traffic of VLAN 135 to Fabric Interconnect – B. Also, we observed some performance impact on databases workload as shown in the screenshots below.

When we disconnected the power from Fabric Interconnect – A, some performance issues on the total IOPS, latency on SOE and OLTP database, as well as throughput on DSS database, as shown in the screenshots (above). However, we did not see any interruption in any Private, Public and Storage Network Traffic.

We noticed this behavior because each server node can failover vNICs from one fabric interconnect switch to another fabric interconnect switch, but there is no vHBAs failover from one fabric interconnect switch to another fabric interconnect switch. Therefore, if one fabric interconnect failure occurred, we would lose half of the number of vHBAs and consequently experience some performance impact on the databases, as shown in the screenshot (above).

After plugging the power cable into the Fabric Interconnect – A Switch, the respective blades (flex1 and flex2) on chassis 1 and (flex3 and flex4) on chassis 2 will route back the MAC addresses and its VLAN traffic to Fabric Interconnect – A. In normal operating conditions, the operating system level multipath configuration will bring back all the failed storage path back to active and database performance will resume to peak performance as previously shown.

The figure below shows details of MAC address, VLAN information and Server connections for Cisco UCS Fabric Interconnect – A switch under normal operating condition.

The figure below shows details of MAC address, VLAN information and Server connections for Cisco UCS Fabric Interconnect – B switch under normal operating condition.

Test 2 – Cisco UCS 6332-16UP Fabric Interconnect – B Failure Test

As similar to the test above, we conducted a hardware failure test on Fabric Interconnect – B by disconnecting power cable to the switch.

The figure below illustrates how during Fabric Interconnect – B switch failure, the respective blades (flex1 and flex2) on chassis 1 and (flex3 and flex4) on chassis 2 will fail over the private network interface MAC addresses and its VLAN network traffic to fabric interconnect – A.

Unplug power cable from Fabric Interconnect – B, and check the MAC address and VLAN information on Cisco UCS Fabric Interconnect – A.

As shown in the figure above, when the Fabric Interconnect – B failed, it routed all the Private Network traffic of VLAN 10 to Fabric Interconnect – A.

When we disconnected power from Fabric Interconnect – B, some performance issues occurred on total IOPS, latency on SOE and OLTP Database as well as throughput on DSS database similar as Fabric Interconnect A failure test. Notice that, we did not see any interruption in any Private, Public and Storage Network Traffic.

We noticed this behavior because each server node can failover vNICs from one fabric interconnect switch to another fabric interconnect switch but there is no vHBAs failover from one fabric interconnect switch to another fabric interconnect switch. Therefore, in case of any one fabric interconnect failure, we would lose half of the number of vHBAs and consequently some performance impact on the databases.

After plug back power cable to Fabric Interconnect – B Switch, the respective blades (flex1 and flex2) on chassis 1 and (flex3 and flex4) on chassis 2 will route back the MAC addresses and its VLAN traffic to Fabric Interconnect – B. In normal operating conditions, the operating system level multipath configuration will bring back all the failed storage path back to active and database performance will resume to peak performance as previously shown.

Test 3 and 4 – Cisco Nexus Switch Failure Test

We conducted a hardware failure test on Nexus Switch – A by disconnecting power cable to the switch and checking the MAC address and VLAN information on Cisco UCS Nexus Switch – B. During Nexus Switch – A failure, it routed all the Private Network and Public Network Traffic of VLAN 10 and VLAN 134 to Nexus Switch – B. So, Nexus Switch – A Failover did not cause any disruption to Private, Public and Storage Network Traffic.

Similarly, we conducted a hardware failure test on Nexus Switch – B by disconnecting power cable to the switch and checking the MAC address and VLAN information on Cisco UCS Nexus Switch – A. During Nexus Switch – B failure, it routed all the Private Network and Public Network Traffic of VLAN 10 and VLAN 134 to Nexus Switch – A.

Test 5 and 6 – Cisco MDS Switch Failure Test

We conducted hardware failure test on MDS Switch – A by disconnecting power cable to the Switch and checking the storage network traffic on MDS Switch – B. As we expected, we observed some impact on all three databases performance as we lost half of the VSAN (VSAN-A 101) traffic. While VSAN-A (101) stays locally into the switch and only carry storage traffic through the MDS switch A, VSAN-A doesn’t failover to MDS Switch B therefore we reduced server to storage connectivity into half during MDS Switch A failure.

As a result, during one MDS Switch failure test, we observed similar performance impact on all the databases as Fabric Interconnect Switch failure tests.

Test 7 – Cisco UCS Chassis 1 and 2 IOM Links Failure

We conducted a Cisco UCS Chassis 1 and Chassis 2 IOM Link Failure test by disconnecting one of the server port link cable from the Chassis as explained below.

Unplug one server port cable from Chassis 1 and Chassis 2 and check the MAC address and VLAN traffic information on both UCS Fabric Interconnects. The screenshot below shows network traffic on Fabric Interconnect A when one link from Chassis 1 and one link from Chassis 2 IOM Failed.

The screenshot below shows network traffic on Fabric Interconnect B when one link from Chassis 1 and one link from Chassis 2 IOM Failed.

We noticed no disruption in public and private network traffic, even though one failed traffic link from both the Chassis because of the port-channel feature.

The Cisco Unified Computing System™ (Cisco UCS®) is a next-generation data center platform that unites computing, network, storage access, and virtualization into a single cohesive system. Cisco UCS is an ideal platform for the architecture of mission critical database workloads such as Oracle RAC. The FlexPod Datacenter with NetApp All Flash AFF system is a converged infrastructure platform that combines best-of breed technologies from Cisco and NetApp into a powerful converged platform for enterprise applications. The pre-validated FlexPod architecture delivers proven value, agility, and performance that drive higher productivity, faster decision making, and greater opportunities for growth.

An essential feature for Oracle databases deployed on shared enterprise system is the ability to deliver consistent and dependable high performance. High performance must be coupled with non-disruptive operations, high availability, scalability, and storage efficiency. Customers can depend on Cisco UCS and NetApp Clustered Data ONTAP Storage to provide these essential elements. Built on clustered Data ONTAP unified scale-out architecture, AFF consistently meets or exceeds the high performance demands of Oracle databases. It also provides rich data management capabilities, such as integrated data protection and non-disruptive upgrades and data migration. These features allow customers to eliminate performance silos and seamlessly integrate AFF into a shared infrastructure.

Clustered Data ONTAP 9.3 delivers an enhanced inline compression capability that significantly reduces the amount of flash storage required and carries near-zero effects on system performance. The combination of Cisco UCS, NetApp and Oracle Real Application Cluster Database architecture can provide the following benefits to accelerate your IT transformation:

·         Cisco UCS stateless computing architecture provided by the Service Profile capability of Cisco UCS allows fast, non-disruptive workload changes to be executed simply and seamlessly across the integrated UCS infrastructure and Cisco x86 servers.

·         A single platform built from unified compute, fabric, and storage technologies, allowing you to scale to large-scale data centers without architectural changes.

·         Enabling faster deployments, greater flexibility of choice, efficiency, high availability and lower risk.

Cisco Nexus 93180 LC-EX Configuration

FLEXPOD-NEXUS-A# show running-config

!Command: show running-config

!Time: Tue Nov 20 21:01:25 2018

version 7.0(3)I6(1)

switchname FLEXPOD-NEXUS-A

policy-map type network-qos jumbo

  class type network-qos class-default

    mtu 9216

vdc FLEXPOD-NEXUS-A id 1

  limit-resource vlan minimum 16 maximum 4094

  limit-resource vrf minimum 2 maximum 4096

  limit-resource port-channel minimum 0 maximum 511

  limit-resource u4route-mem minimum 248 maximum 248

  limit-resource u6route-mem minimum 96 maximum 96

  limit-resource m4route-mem minimum 58 maximum 58

  limit-resource m6route-mem minimum 8 maximum 8

 

cfs eth distribute

feature interface-vlan

feature hsrp

feature lacp

feature vpc

 

no password strength-check

username admin password 5 $5$FWM47Q8G$RxxMd920jZrjYX5GXLRCkfWgDtka29dQV1TiP/k4VmD  role network-admin

ip domain-lookup

system default switchport

system qos

  service-policy type network-qos jumbo

copp profile strict

snmp-server user admin network-admin auth md5 0x54fc9b6e68ba6a06acf4165853e18078 priv 0x54fc9b6e68ba6a06acf4165853e18078 localizedkey

rmon event 1 description FATAL(1) owner PMON@FATAL

rmon event 2 description CRITICAL(2) owner PMON@CRITICAL

rmon event 3 description ERROR(3) owner PMON@ERROR

rmon event 4 description WARNING(4) owner PMON@WARNING

rmon event 5 description INFORMATION(5) owner PMON@INFO

ntp server 72.163.32.44 use-vrf default

 

vlan 1,10,135

vlan 10

  name Oracle_RAC_Private_Traffic

vlan 135

  name Oracle_RAC_Public_Traffic

 

spanning-tree port type edge bpduguard default

spanning-tree port type network default

vrf context management

  ip route 0.0.0.0/0 10.29.135.1

port-channel load-balance src-dst l4port

vpc domain 1

  peer-keepalive destination 10.29.135.104 source 10.29.135.103

  auto-recovery

 

interface Vlan1

 

interface port-channel1

  description vPC peer-link

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type network

  vpc peer-link

 

interface port-channel51

  description Port-Channel FI-A

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  vpc 51

 

interface port-channel52

  description Port-Channel FI-B

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  vpc 52

 

interface Ethernet1/1

  description Peer link connected to N9K-B-Eth1/1

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  channel-group 1 mode active

 

interface Ethernet1/2

  description Peer link connected to N9K-B-Eth1/2

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  channel-group 1 mode active

 

interface Ethernet1/3

...........

...........

interface Ethernet1/20

 

interface Ethernet1/21

  description Fabric-Interconnect-A-31

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 51 mode active

 

interface Ethernet1/22

  description Fabric-Interconnect-A-32

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 51 mode active

 

interface Ethernet1/23

  description Fabric-Interconnect-B-31

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 52 mode active

 

interface Ethernet1/24

  description Fabric-Interconnect-B-32

  switchport mode trunk

  switchport trunk allowed vlan 1,10,135

  spanning-tree port type edge trunk

  mtu 9216

  channel-group 52 mode active

 

interface Ethernet1/25

  description NetApp Controller 1 e5e

  switchport access vlan 135

 

interface Ethernet1/26

...........

interface Ethernet1/28

 

interface Ethernet1/29

  description connect to uplink switch

  switchport access vlan 135

  speed 1000

 

interface Ethernet1/30

...........

...........

interface Ethernet1/32

 

interface mgmt0

  vrf member management

  ip address 10.29.135.103/24

line console

line vty

boot nxos bootflash:/nxos.7.0.3.I6.1.bin

no system default switchport shutdown

MDS 9148S FC Switch Configuration

FLEXPOD-MDS-A# show running-config

!Command: show running-config

!Time: Tue Nov 20 21:19:36 2018

version 7.3(0)D1(1)

power redundancy-mode redundant

feature npiv

no feature http-server

role name default-role

  description This is a system defined role and applies to all users.

  rule 5 permit show feature environment

  rule 4 permit show feature hardware

  rule 3 permit show feature module

  rule 2 permit show feature snmp

  rule 1 permit show feature system

username admin password 5 $5$lC5R4cI9$rfRLLbkccOM1lwxALFaHFjx1XtCse5pPjAk3i/HJi0/  role network-admin

no password strength-check

ip domain-lookup

ip host FLEXPOD-MDS-A  10.29.135.105

aaa group server radius radius

snmp-server user admin network-admin auth md5 0xe2b9f6bd292aa43dc651e566873540cf priv 0xe2b9f6bd292aa43dc651e566873540cf localizedkey

rmon event 1 log description FATAL(1) owner PMON@FATAL

rmon event 2 log description CRITICAL(2) owner PMON@CRITICAL

rmon event 3 log description ERROR(3) owner PMON@ERROR

rmon event 4 log description WARNING(4) owner PMON@WARNING

rmon event 5 log description INFORMATION(5) owner PMON@INFO

ntp server 72.163.32.44

vsan database

  vsan 101

device-alias database

  device-alias name flex1-hba0 pwwn 20:00:00:25:b5:8a:a0:00

  device-alias name flex1-hba2 pwwn 20:00:00:25:b5:8a:a0:01

  device-alias name flex2-hba0 pwwn 20:00:00:25:b5:8a:a0:02

  device-alias name flex2-hba2 pwwn 20:00:00:25:b5:8a:a0:03

  device-alias name flex3-hba0 pwwn 20:00:00:25:b5:8a:a0:04

  device-alias name flex3-hba2 pwwn 20:00:00:25:b5:8a:a0:05

  device-alias name flex4-hba0 pwwn 20:00:00:25:b5:8a:a0:06

  device-alias name flex4-hba2 pwwn 20:00:00:25:b5:8a:a0:07

  device-alias name NetApp-A700s-01-2A pwwn 20:06:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-01-3A pwwn 20:07:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-02-2A pwwn 20:09:00:a0:98:af:7c:5b

  device-alias name NetApp-A700s-02-3A pwwn 20:0a:00:a0:98:af:7c:5b

device-alias commit

fcdomain fcid database

  vsan 1 wwn 20:06:8c:60:4f:bd:31:80 fcid 0x940000 dynamic

  vsan 1 wwn 20:05:8c:60:4f:bd:31:80 fcid 0x940100 dynamic

  vsan 1 wwn 20:01:00:de:fb:92:99:00 fcid 0x940300 dynamic

  vsan 1 wwn 20:02:00:de:fb:92:99:00 fcid 0x940400 dynamic

  vsan 1 wwn 20:03:00:de:fb:92:99:00 fcid 0x940500 dynamic

  vsan 1 wwn 20:04:00:de:fb:92:99:00 fcid 0x940600 dynamic

  vsan 1 wwn 50:0a:09:82:80:12:f8:47 fcid 0x940700 dynamic

  vsan 1 wwn 50:0a:09:84:80:12:f8:47 fcid 0x940800 dynamic

  vsan 1 wwn 50:0a:09:81:80:12:f8:47 fcid 0x940900 dynamic

  vsan 1 wwn 50:0a:09:83:80:12:f8:47 fcid 0x940a00 dynamic

  vsan 1 wwn 50:0a:09:81:80:12:f8:3d fcid 0x940b00 dynamic

  vsan 1 wwn 50:0a:09:83:80:12:f8:3d fcid 0x940c00 dynamic

  vsan 101 wwn 50:0a:09:81:80:12:f8:47 fcid 0xa20000 dynamic

  vsan 101 wwn 50:0a:09:83:80:12:f8:47 fcid 0xa20100 dynamic

  vsan 101 wwn 50:0a:09:81:80:12:f8:3d fcid 0xa20200 dynamic

  vsan 101 wwn 50:0a:09:83:80:12:f8:3d fcid 0xa20300 dynamic

  vsan 101 wwn 20:01:00:de:fb:92:99:00 fcid 0xa20400 dynamic

  vsan 101 wwn 20:00:00:25:b5:8a:a0:00 fcid 0xa20702 dynamic

!              [flex1-hba0]

  vsan 101 wwn 20:00:00:25:b5:8a:a0:04 fcid 0xa20501 dynamic

!              [flex3-hba0]

  vsan 1 wwn 20:4d:54:7f:ee:76:cd:80 fcid 0x940200 dynamic

  vsan 101 wwn 20:00:00:25:b5:8a:a0:06 fcid 0xa20502 dynamic

!              [flex4-hba0]

  vsan 101 wwn 20:00:00:25:b5:8a:a0:02 fcid 0xa20701 dynamic

!              [flex2-hba0]

  vsan 101 wwn 20:02:00:de:fb:92:99:00 fcid 0xa20500 dynamic

  vsan 101 wwn 20:03:00:de:fb:92:99:00 fcid 0xa20600 dynamic

  vsan 101 wwn 20:04:00:de:fb:92:99:00 fcid 0xa20700 dynamic

  vsan 101 wwn 20:02:00:a0:98:af:7c:5b fcid 0xa20001 dynamic

  vsan 101 wwn 20:04:00:a0:98:af:7c:5b fcid 0xa20101 dynamic

  vsan 101 wwn 20:06:00:a0:98:af:7c:5b fcid 0xa20002 dynamic

!              [NetApp-A700s-01-2A]

  vsan 101 wwn 20:07:00:a0:98:af:7c:5b fcid 0xa20102 dynamic

!              [NetApp-A700s-01-3A]

  vsan 101 wwn 20:09:00:a0:98:af:7c:5b fcid 0xa20201 dynamic

!              [NetApp-A700s-02-2A]

  vsan 101 wwn 20:0a:00:a0:98:af:7c:5b fcid 0xa20301 dynamic

!              [NetApp-A700s-02-3A]

  vsan 101 wwn 20:00:00:25:b5:8a:a0:01 fcid 0xa20601 dynamic

!              [flex1-hba2]

  vsan 101 wwn 20:00:00:25:b5:8a:a0:03 fcid 0xa20404 dynamic

!              [flex2-hba2]

  vsan 101 wwn 20:00:00:25:b5:8a:a0:05 fcid 0xa20401 dynamic

!              [flex3-hba2]

  vsan 101 wwn 20:00:00:25:b5:8a:a0:07 fcid 0xa20603 dynamic

!              [flex4-hba2]

!Active Zone Database Section for vsan 101

zone name flex1 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:00

!               [flex1-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:01

!               [flex1-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zone name flex2 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:02

!               [flex2-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:03

!               [flex2-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zone name flex3 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:04

!               [flex3-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:05

!               [flex3-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zone name flex4 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:06

!               [flex4-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:07

!               [flex4-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zoneset name flex vsan 101

    member flex1

    member flex2

    member flex3

    member flex4

 

zoneset activate name flex vsan 101

do clear zone database vsan 101

!Full Zone Database Section for vsan 101

zone name flex1 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:00

!               [flex1-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:01

!               [flex1-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zone name flex2 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:02

!               [flex2-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:03

!               [flex2-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zone name flex3 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:04

!               [flex3-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:05

!               [flex3-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zone name flex4 vsan 101

    member pwwn 20:00:00:25:b5:8a:a0:06

!               [flex4-hba0]

    member pwwn 20:00:00:25:b5:8a:a0:07

!               [flex4-hba2]

    member pwwn 20:06:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-2A]

    member pwwn 20:07:00:a0:98:af:7c:5b

!               [NetApp-A700s-01-3A]

    member pwwn 20:09:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-2A]

    member pwwn 20:0a:00:a0:98:af:7c:5b

!               [NetApp-A700s-02-3A]

 

zoneset name flex vsan 101

    member flex1

    member flex2

    member flex3

    member flex4

 

interface mgmt0

  ip address 10.29.135.105 255.255.255.0

vsan database

  vsan 101 interface fc1/1

  vsan 101 interface fc1/2

  vsan 101 interface fc1/3

  vsan 101 interface fc1/4

  vsan 101 interface fc1/5

  vsan 101 interface fc1/6

  vsan 101 interface fc1/7

  vsan 101 interface fc1/8

  vsan 101 interface fc1/9

  vsan 101 interface fc1/10

  vsan 101 interface fc1/11

  vsan 101 interface fc1/12

switchname FLEXPOD-MDS-A

line console

line vty

boot kickstart bootflash:/m9100-s5ek9-kickstart-mz-npe.7.3.0.D1.1.bin

boot system bootflash:/m9100-s5ek9-mz-npe.7.3.0.D1.1.bin

interface fc1/1

……………

……………

interface fc1/48

interface fc1/1

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/2

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/3

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/4

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/5

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/6

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/7

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/8

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/9

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/10

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/11

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/12

  switchport trunk allowed vsan 101

  switchport trunk mode off

  port-license acquire

  no shutdown

 

interface fc1/13

  port-license acquire

……………

……………

interface fc1/48

  port-license acquire

 

no system default switchport shutdown

ip default-gateway 10.29.135.1

Multipath Configuration “/etc/multipath.conf”

defaults {

        find_multipaths yes

        user_friendly_names no

}

multipaths {

        multipath {

                wwid            3600a098038304173475d4c766a49744e

                alias           node1_os

        }

        multipath {

                wwid            3600a098038304173475d4c766a49754a

                alias           ocrvote_1

        }

        multipath {

                wwid            3600a098038304173452b4c7959614634

                alias           ocrvote_2

        }

        multipath {

                wwid            3600a098038304173475d4c766a49754b

                alias           oradata_slob_1

        }

        multipath {

                wwid            3600a098038304173452b4c7959614635

                alias           oradata_slob_2

        }

        multipath {

                wwid            3600a098038304173475d4c766a49754c

                alias           oradata_slob_3

        }

        multipath {

                wwid            3600a098038304173452b4c7959614636

                alias           oradata_slob_4

        }

        multipath {

                wwid            3600a098038304173475d4c766a49754d

                alias           oradata_slob_5

        }

        multipath {

                wwid            3600a098038304173452b4c7959614637

                alias           oradata_slob_6

        }

        multipath {

                wwid            3600a098038304173475d4c766a49754e

                alias           oradata_slob_7

        }

        multipath {

                wwid            3600a098038304173452b4c7959614638

                alias           oradata_slob_8

        }

        multipath {

                wwid            3600a098038304173475d4c766a49754f

                alias           oradata_slob_9

        }

        multipath {

                wwid            3600a098038304173452b4c7959614639

                alias           oradata_slob_10

        }

        multipath {

                wwid            3600a098038304173475d4c766a497550

                alias           oradata_slob_11

        }

        multipath {

                wwid            3600a098038304173452b4c795961462d

                alias           oradata_slob_12

        }

        multipath {

                wwid            3600a098038304173475d4c766a497551

                alias           oradata_slob_13

        }

        multipath {

                wwid            3600a098038304173452b4c7959614641

                alias           oradata_slob_14

        }

        multipath {

                wwid            3600a098038304173475d4c766a497552

                alias           oradata_slob_15

        }

        multipath {

                wwid            3600a098038304173452b4c7959614642

                alias           oradata_slob_16

        }

        multipath {

                wwid            3600a098038304173475d4c766a497553

                alias           oraredo_slob_1

        }

        multipath {

                wwid            3600a098038304173452b4c7959614643

                alias           oraredo_slob_2

        }

        multipath {

                wwid            3600a098038304173475d4c766a497563

                alias           oradata_oltp_1

        }

       multipath {

                wwid            3600a098038304173452b4c795961464e

                alias           oradata_oltp_2

        }

       multipath {

                wwid            3600a098038304173475d4c766a497564

                alias           oradata_oltp_3

        }

       multipath {

                wwid            3600a098038304173452b4c795961464f

                alias           oradata_oltp_4

        }

       multipath {

                wwid            3600a098038304173475d4c766a497565

                alias           oradata_oltp_5

        }

       multipath {

                wwid            3600a098038304173452b4c7959614650

                alias           oradata_oltp_6

        }

       multipath {

                wwid            3600a098038304173475d4c766a497566

                alias           oradata_oltp_7

        }

       multipath {

                wwid            3600a098038304173452b4c7959614651

                alias           oradata_oltp_8

        }

       multipath {

                wwid            3600a098038304173475d4c766a497567

                alias           oradata_oltp_9

        }

       multipath {

                wwid            3600a098038304173452b4c7959614652

                alias           oradata_oltp_10

        }

       multipath {

                wwid            3600a098038304173475d4c766a497568

                alias           oradata_oltp_11

        }

       multipath {

                wwid            3600a098038304173452b4c7959614653

                alias           oradata_oltp_12

        }

       multipath {

                wwid            3600a098038304173475d4c766a497569

                alias           oradata_oltp_13

        }

       multipath {

                wwid            3600a098038304173452b4c7959614654

                alias           oradata_oltp_14

        }

       multipath {

                wwid            3600a098038304173475d4c766a49756a

                alias           oradata_oltp_15

        }

       multipath {

                wwid            3600a098038304173452b4c7959614655

                alias           oradata_oltp_16

        }

        multipath {

                wwid            3600a098038304173475d4c766a49756b

                alias           oraredo_oltp_1

        }

        multipath {

                wwid            3600a098038304173452b4c7959614656

                alias           oraredo_oltp_2

        }

        multipath {

                wwid            3600a098038304173475d4c766a49756c

                alias           oradata_soe_1

        }

        multipath {

                wwid            3600a098038304173452b4c7959614657

                alias           oradata_soe_2

        }

        multipath {

                wwid            3600a098038304173475d4c766a49756d

                alias           oradata_soe_3

        }

        multipath {

                wwid            3600a098038304173452b4c7959614658

                alias           oradata_soe_4

        }

        multipath {

                wwid            3600a098038304173475d4c766a49756e

                alias           oradata_soe_5

        }

        multipath {

                wwid            3600a098038304173452b4c7959614659

                alias           oradata_soe_6

        }

        multipath {

                wwid            3600a098038304173475d4c766a49756f

                alias           oradata_soe_7

        }

        multipath {

                wwid            3600a098038304173452b4c795961465a

                alias           oradata_soe_8

        }

        multipath {

                wwid            3600a098038304173475d4c766a497570

                alias           oradata_soe_9

        }

        multipath {

                wwid            3600a098038304173452b4c795961462f

                alias           oradata_soe_10

        }

        multipath {

                wwid            3600a098038304173475d4c766a497571

                alias           oradata_soe_11

        }

        multipath {

                wwid            3600a098038304173452b4c7959614661

                alias           oradata_soe_12

        }

        multipath {

                wwid            3600a098038304173475d4c766a497572

                alias           oradata_soe_13

        }

        multipath {

                wwid            3600a098038304173452b4c7959614662

                alias           oradata_soe_14

        }

        multipath {

                wwid            3600a098038304173475d4c766a497573

                alias           oradata_soe_15

        }

        multipath {

                wwid            3600a098038304173452b4c7959614663

                alias           oradata_soe_16

        }

        multipath {

                wwid            3600a098038304173475d4c766a497574

                alias           oraredo_soe_1

        }

        multipath {

                wwid            3600a098038304173452b4c7959614664

                alias           oraredo_soe_2

        }

        multipath {

                wwid            3600a098038304173475d4c766a497575

                alias           oradata_dss_1

        }

        multipath {

                wwid            3600a098038304173452b4c7959614665

                alias           oradata_dss_2

        }

        multipath {

                wwid            3600a098038304173475d4c766a497576

                alias           oradata_dss_3

        }

        multipath {

                wwid            3600a098038304173452b4c7959614666

                alias           oradata_dss_4

        }

        multipath {

                wwid            3600a098038304173475d4c766a497577

                alias           oradata_dss_5

        }

        multipath {

                wwid            3600a098038304173452b4c7959614667

                alias           oradata_dss_6

        }

        multipath {

                wwid            3600a098038304173475d4c766a497578

                alias           oradata_dss_7

        }

        multipath {

                wwid            3600a098038304173452b4c7959614668

                alias           oradata_dss_8

        }

        multipath {

                wwid            3600a098038304173475d4c766a497579

                alias           oradata_dss_9

        }

        multipath {

                wwid            3600a098038304173452b4c7959614669

                alias           oradata_dss_10

        }

        multipath {

                wwid            3600a098038304173475d4c766a49757a

                alias           oradata_dss_11

        }

        multipath {

                wwid            3600a098038304173452b4c795961466a

                alias           oradata_dss_12

        }

        multipath {

                wwid            3600a098038304173475d4c766a497630

                alias           oradata_dss_13

        }

        multipath {

                wwid            3600a098038304173452b4c795961466b

                alias           oradata_dss_14

        }

        multipath {

                wwid            3600a098038304173475d4c766a497631

                alias           oradata_dss_15

        }

        multipath {

                wwid            3600a098038304173452b4c795961466c

                alias           oradata_dss_16

        }

        multipath {

                wwid            3600a098038304173475d4c766a497632

                alias           oraredo_dss_1

        }

        multipath {

                wwid            3600a098038304173452b4c795961466d

                alias           oraredo_dss_2

        }

}

Configuration of “/etc/sysctl.conf”

# oracle-database-server-12cR2-preinstall setting for fs.file-max is 6815744

fs.file-max = 6815744

# oracle-database-server-12cR2-preinstall setting for kernel.sem is '250 32000 100 128'

kernel.sem = 250 32000 100 128

# oracle-database-server-12cR2-preinstall setting for kernel.shmmni is 4096

kernel.shmmni = 4096

# oracle-database-server-12cR2-preinstall setting for kernel.shmall is 1073741824 on x86_64

kernel.shmall = 1073741824

# oracle-database-server-12cR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_64

kernel.shmmax = 4398046511104

# oracle-database-server-12cR2-preinstall setting for kernel.panic_on_oops is 1 per Orabug 19212317

kernel.panic_on_oops = 1

# oracle-database-server-12cR2-preinstall setting for net.core.rmem_default is 262144

net.core.rmem_default = 262144

# oracle-database-server-12cR2-preinstall setting for net.core.rmem_max is 4194304

net.core.rmem_max = 4194304

# oracle-database-server-12cR2-preinstall setting for net.core.wmem_default is 262144

net.core.wmem_default = 262144

# oracle-database-server-12cR2-preinstall setting for net.core.wmem_max is 1048576

net.core.wmem_max = 1048576

# oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.all.rp_filter is 2

net.ipv4.conf.all.rp_filter = 2

# oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.default.rp_filter is 2

net.ipv4.conf.default.rp_filter = 2

# oracle-database-server-12cR2-preinstall setting for fs.aio-max-nr is 1048576

fs.aio-max-nr = 1048576

# oracle-database-server-12cR2-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500

net.ipv4.ip_local_port_range = 9000 65500

vm.nr_hugepages=180000

Configuration of “/etc/security/limits.d/oracle-database-server-12cR2-preinstall.conf”

# oracle-database-server-12cR2-preinstall setting for nofile soft limit is 1024

oracle   soft   nofile    1024

# oracle-database-server-12cR2-preinstall setting for nofile hard limit is 65536

oracle   hard   nofile    65536

# oracle-database-server-12cR2-preinstall setting for nproc soft limit is 16384

# refer orabug15971421 for more info.

oracle   soft   nproc    16384

# oracle-database-server-12cR2-preinstall setting for nproc hard limit is 16384

oracle   hard   nproc    16384

# oracle-database-server-12cR2-preinstall setting for stack soft limit is 10240KB

oracle   soft   stack    10240

# oracle-database-server-12cR2-preinstall setting for stack hard limit is 32768KB

oracle   hard   stack    32768

# oracle-database-server-12cR2-preinstall setting for memlock hard limit is maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM

oracle   hard   memlock    473985890

# oracle-database-server-12cR2-preinstall setting for memlock soft limit is maximum of 128GB on x86_64 or 3GB on x86 OR 90% of RAM

oracle   soft   memlock    473985890

Configuration of “/etc/udev/rules.d/99-oracle-asmdevices.rules”

#All LUNs which starts with dg_orarac_* #

ENV{DM_NAME}=="ocrvote_*", OWNER:="grid", GROUP:="oinstall", MODE:="660"

#All LUNs which starts with dg_oradata_* #

ENV{DM_NAME}=="oradata_*", OWNER:="oracle", GROUP:="oinstall", MODE:="660"

#All LUNs which starts with dg_oraredo_* #

ENV{DM_NAME}=="oraredo_*", OWNER:="oracle", GROUP:="oinstall", MODE:="660"

#All LUNs which starts with dg_oraarchive_* #

ENV{DM_NAME}=="oraarchive_*", OWNER:="oracle", GROUP:="oinstall", MODE:="660"

The following references were used in preparing this document.

Cisco Unified Computing System

https://www.cisco.com/c/en/us/products/servers-unified-computing/index.html

Cisco UCS Data Center Design Guides

https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/data-center-design-guides-all.html#Hyperconverged

FlexPod Converged Infrastructure

https://www.cisco.com/c/en/us/solutions/data-center-virtualization/flexpod/index.html#~tab-resources

https://www.netapp.com/us/products/converged-systems/flexpod-converged-infrastructure.aspx

Cisco UCS B200 M5 Servers

https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-b-series-blade-servers/datasheet-c78-739296.html

Oracle Database 12c Release 2

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/index.html

NetApp AFF A-Series All Flash Storage

https://www.netapp.com/us/products/storage-systems/all-flash-array/aff-a-series.aspx

https://www.netapp.com/us/media/ds-3582_tcm10-121337.pdf

https://hwu.netapp.com/Controller/Index

https://www.netapp.com/us/products/storage-systems/disk-shelves-and-storage-media/index.aspx#tech-specs

NetApp Support

https://mysupport.netapp.com/

For a comprehensive list of storage hardware specifications, including supported media type and capacities, see the NetApp Hardware Universe.

 

Tushar Patel, Principal Engineer, CSPG UCS Product Management and Data Center Solutions Engineering Group, Cisco Systems, Inc.

Tushar Patel is a Principal Engineer in the Cisco Systems CSPG UCS Product Management and Data Center Solutions Engineering Group and a specialist in flash storage technologies and Oracle RAC RDBMS. Tushar has over 23 years of experience in Flash Storage architecture, database architecture, design and performance. Tushar also has strong background in Intel X86 architecture, hyper converged systems, storage technologies and virtualization. He has worked with large number of enterprise customers, evaluate and deploy mission critical database solutions. Tushar has presented to both internal and external audiences at various conferences and customer events.

Hardikkumar Vyas, Solution Engineer, CSPG UCS Product Management and Data Center Solutions Engineering Group, Cisco Systems, Inc.

Hardikkumar Vyas is a Solution Engineer in the Cisco Systems CSPG UCS Product Management and Data Center Solutions Engineering Group for configuring, implementing and validating infrastructure best practices for highly available Oracle RAC databases solutions on Cisco UCS Servers, Cisco Nexus Products and various storage technologies. Hardikkumar Vyas holds a Master’s degree in Electrical Engineering and has over 5 years of experience working with Oracle RAC Databases and associated applications. Hardikkumar Vyas’s main focus is developing database solutions on different platforms, perform benchmarks, prepare reference architectures and write technical documents for Oracle RAC Databases on Cisco UCS Platforms.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank:

James Bradshaw, NetAppNOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).