New and Changed Information

The following table provides an overview of the significant changes up to this current release. The table does not provide an exhaustive list of all changes or of the new features up to this release.

Cisco APIC Release Version

Feature

Description

5.1(1)

Rancher support with Cisco Application Centric Infrastructure (ACI) Container Network Interface (CNI)

You can integrate Rancher with Cisco ACI to simplify the management of Kubernetes clusters.

Cisco ACI and Rancher Integration

Rancher is a software stack that simplifies the management of containers. Beginning in Cisco Application Policy Infrastructure Controller (APIC) Release 5.1(1), you can integrate Rancher with the Cisco Application Centric Infrastructure (ACI) Container Network Interface (CNI) plug-in to install and manage Kubernetes clusters.

Rancher includes the Rancher Kubernetes Engine (RKE), a certified Kubernetes distribution. It operates within platform-independent Docker containers, simplifying installation and providing consistent management of Kubernetes across all platforms. You can install and manage Kubernetes on VMware clusters, bare-metal servers, and on VMs in clouds—regardless of whether the clouds support Kubernetes service.

This document provides an introduction to Rancher and instructions for integrating it with Cisco ACI. It is assumed that you are familiar with the Cisco ACI CNI plug-in and Kubernetes.


Note
The Cisco APIC Release 5.1(1) supports Rancher 2.4 and Rancher 2.5 as an imported cluster. The RKE minimum version is 1.2.2. The first supported integrated versions of Rancher and RKE are 2.5.4 and 1.2.4.

Rancher Architecture

Rancher contains several key components that enable you to install and maintain Kubernetes clusters. They include Rancher Kubernetes Engine (RKE), Rancher server, and cluster and node agents.

The following illustration shows Rancher 2.5 high-level architecture. The figure shows a Rancher server installation that managers two Kubernetes clusters. One is created by RKE, and the other is created by Google Kubernetes Engine (GKE). This document provides instructions for using RKE.

Figure 1. Rancher 2.5 High-Level Technical Architecture



Note
The preceding illustration is from Rancher 2.5: Technical Architecture. Used with permission.

The rest of this section describes key components of Rancher. For details, see Rancher 2.5: Technical Architecture, Rancher 2.4: Technical Architecture, and other content on the Rancher website.

Rancher Kubernetes Engine

RKE is Kubernetes distribution certified by the Cloud Native Computing Foundation (CNCF). It runs entirely within Docker containers. You use RKE to install Kubernetes on VMware clusters, bare metal servers, and VMs on clouds, including clouds that do not support Kubernetes.

Rancher server

The Rancher server contains several components that manage the entire Rancher deployment:

  • Rancher API server: The Rancher API server is the foundation for all Rancher server controllers. All Rancher-specific resources that are created using Rancher API are translated to custom resources definition (CRD) objects. Rancher controllers manage the lifecycle of the objects.

  • Management controller: Management controllers perform the activities that occur at the Rancher server level that are not specific to an individual cluster. Such activities include configuring access policies to clusters and projects and managing the global-level catalog.

  • User cluster controller: User cluster controllers perform activities specific to a cluster. For horizontal scaling, user cluster controllers are distributed across the Rancher server pods. Cluster-specific activities include managing workloads and applying roles and bindings that are defined in global policies.

  • Authentication proxy:

    The authentication proxy proxies all Kubernetes API calls. It integrates with local authentication, Active Directory, and GitHub. For every Kubernetes API call, the proxy authenticates the caller and sets the correct Kubernetes impersonation headers. It forwards the call to the Kubernetes master nodes.

Rancher agents

Rancher deploys agents into Kubernetes nodes and clusters:

  • Cluster agent: Rancher deploys one cluster agent for each managed Kubernetes cluster. The cluster agent opens a web-socket tunnel to the Rancher server so the user cluster controllers and authentication proxy can communicate with the user cluster Kubernetes API server.

  • Node agent: RKE uses node agents primarily to deploy components during initial installation and subsequent upgrades. The node agents are installed also on cloud Kubernetes clusters to provide extra functions, such as fallback for cluster agents and proxy for the kubectl shell.

Prerequisites for Integrating Rancher

Complete the following tasks before integrating Rancher, following the referenced procedures on the Rancher and Cisco websites.

  1. Configure the following:

    • A Cisco Application Centric Infrastructure (ACI) tenant

    • An attachable entity profile (AEP)

    • A VRF

    • A Layer 3 outside connection (L3Out)

    • A Layer 3 external network for the cluster you plan to provision with RKE


    Note

    The VRF and L3Out in Cisco ACI that you use to provide outside connectivity to Kubernetes external services can be in any tenant. The VRF and L3Out are usually in the common tenant or in a tenant that you dedicate to the Kubernetes cluster. You can also have separate VRFs, one for the Kubernetes bridge domains and one for the L3Out, and you can configure route leaking between them.

  2. Install the Cisco ACI CNI tools from the Cisco Download site for the specific Linux distribution on a tools server that you will use to provision the cluster.

    This server must have SSH access and reachability to all nodes in the cluster that you will provision.

  3. Prepare the nodes for Cisco ACI Container Network Interface (CNI) plug-in installation.

    Follow the procedures in Cisco ACI and Kubernetes Integration on Cisco.com until the end of the section "Preparing the Kubernetes Nodes." Use the flavor RKE-1.2.3 instead of kubernetes-version for acc-provision. Perform the steps for the cluster and every node in the cluster.


    Note

    ACI CNI in nested mode is only supported with VMM-integrated VMware (with dVS).

  4. Prepare the nodes for the Rancher server nodes.

    Fulfill the hardware, software, and networking requirements for the nodes where you will install the Rancher server. See the page Installation Requirements for Rancher 2.x on the Rancher website.

  5. Prepare the nodes in the user cluster.

    Fulfill the requirements for the nodes where you will install apps and services. See the page Node Requirements for User Clusters or Rancher 2.x on the Rancher website.

Integrating Rancher

We recommended that you install Rancher server in a separate cluster, which is called a Rancher server cluster. This cluster creates and manages other clusters, which are called user clusters.

You typically install the Rancher server cluster using Rancher Kubernetes Engine (RKE). In Cisco Application Policy Infrastructure Controller (APIC) Release 5.1(1), we support creating a custom cluster on an on-premises deployment and creating one cluster per tenant.


Note
The Cisco Application Centric Infrastructure (ACI) Container Network Interface (CNI) does not support Rancher-launched Kubernetes. However, RKE-installed Kubernetes is supported, and RKE-installed clusters can be imported into Rancher. To see the differences, see Setting up Kubernetes Clusters in Rancher for Rancher 2.2 on the Rancher website.

This section includes prerequisites and instructions for integrating Rancher.

Install Kubernetes Using RKE

Before you begin

Fulfill the requirements in the section Prerequisites for Integrating Rancher.

Procedure

Step 1

Install the RKE binary file on the tools server.

Follow the procedure "Download the RKE Binary" on the RKE Kubernetes Installation page on the Rancher.com website.

Step 2

Install Docker on all the cluster nodes.

Follow the guidelines and procedure in "Requirements" in the "RKE" section of the Rancher.com website.
Step 3

Run acc_provision to generate the network provider section of the cluster.yml file that the RKE requires.

See the section "Provisioning Cisco ACI to Work with Kubernetes" in Cisco ACI and Kubernetes Integration on Cisco.com.

Example:

acc-provision -f RKE-1.2.3 --sample input-file
Step 4

Edit the input file that was generated by the previous step to match your cluster:

Example:

acc-provision -f RKE-1.2.3 -c input-file -u apic-user-name -p apic-password -o network_provider_cluster,yml.

The command generates the network-provider file that use in step 6.
Step 5

Use RKE to generate a sample cluster.yml file and edit the file to add the cluster nodes.

Alternatively, you can use an existing cluster.yml file.

Step 6

Replace the network provider section in the cluster.yml file that you generated with the acc_provision command.

The following text provides an example of a network-provider file:

Example:

kubernetes_version: v1.18.12-rancher1-2
network:
  plugin: "aci"
  aci_network_provider:
    system_id: "rancher"
    apic_hosts: ["\"172.28.184.110\""]
    token: "2f8cc816-21e6-49d3-b18e-00d1bf38f37d"
    apic_user_name: "rancher"
    apic_user_key: "(user-key as generated by acc-provision)"
    apic_user_crt: "(user-crt as generated by acc-provision)"
    encap_type: "vxlan"
    mcast_range_start: "225.10.1.1"
    mcast_range_end: "225.10.255.255"
    aep: "sauto-aep"
    vrf_name: "sauto_l3out-1_vrf"
    vrf_tenant: "common"
    l3out: "sauto_l3out-1"
    node_subnet: "10.100.1.1/24"
    l3out_external_networks: ["\"sauto_l3out-1_epg\""]
    extern_dynamic: "10.3.0.1/24"
    extern_static: "10.4.0.1/24"
    node_svc_subnet: "10.5.0.1/24"
    kube_api_vlan: "213"
    service_vlan: "214"
    infra_vlan: "4081"
    service_monitor_interval: "5"

The following configuration parameters are the same as in the acc_provision input file. The pod subnet is a main configuration option, called cluster_cidr, in the cluster.yml file. 

system_images:
  aci_cni_deploy_container: noiro/cnideploy:5.1.1.0.1ae238a
  aci_host_container: noiro/aci-containers-host:5.1.1.0.1ae238a
  aci_opflex_container: noiro/opflex:5.1.1.0.1ae238a
  aci_mcast_container: noiro/opflex:5.1.1.0.1ae238a
  aci_ovs_container: noiro/openvswitch:5.1.1.0.1ae238a
  aci_controller_container: noiro/aci-containers-controller:5.1.1.0.1ae238a

There is mapping between a chosen Rancher version and the set of network provider images. See the rancher kontainer-driver-metadata GitHub repository for up-to-date mapping. You can define system images to use based on the Cisco ACI release. If you encounter an error saying that the image cannot be found, specify the image versions explicitly.

Note 
The aci_gbp_server_container and aci_opflex_server_container images are not required for on-premises deployment, which is the only profile that this release supports.
Step 7

Enter the following command: rke up.

The command internally generates the Cisco ACI Container Network Interface (CNI) .yaml file and applies it. The resulting .yml file is stored as configmap in kube-system/rke-network-plugin.

What to do next

User can import the RKE cluster into the Rancher server. Follow the procedure Importing Existing Clusters into Rancher for Rancher 2.x on the Rancher website.

Upgrade Kubernetes Using RKE

You can use Rancher Kubernetes Engine (RKE) to upgrade Kubernetes.

Before you begin

  • See the Cisco APIC Container Plug-in Release 5.1(1), Release Notes on Cisco.com for the version of Kubernetes that you want to upgrade to.

  • Ensure that the version of Kubernetes that you want to upgrade to is supported for your environment. See the Cisco ACI Virtualization Compatibility Matrix and All Supported Versions on the Rancher website.

  • Take a backup of cluster.rkestate and cluster.yml and a snapshot of etcd.

    Follow the instructions in One-time Snapshots for RKE on the Rancher website.


    Note
    If upgrade fails for any reason, you can use the snapshot and backed-up files to roll back to the previous version of Kubernetes. See Restoring from Backup for RKE on the Rancher website.

Procedure

Step 1

Update the cluster.yml file with the desired version of Kubernetes by revising the line kubernetes_version: (rancher-image-version) as needed.

Step 2

Enter the following command: rke up.

The command internally generates the Cisco ACI Container Network Interface (CNI) .yaml file and applies it. The resulting .yml file is stored as configmap in kube-system/rke-network-plugin.

Related Documentation

In addition to the Rancher integration information in this document, you may want to consult the following documentation: