The system requires that each CF VM be configured with:

• 8 vCPUs

• 20 GB RAM (Release 21.4 and higher); 16 GB RAM (Releases prior to 21.4)

• First vNIC is the DI Network.

• Second vNIC is the management port.

• First vHDD is for boot image and config storage (/flash, non-RAID, 4GB recommended).

• Second vHDD is for record storage [optional] (hd-local1, RAID, 16GB minimum recommended).

Note	Both CF VMs must be identically configured.

The system requires that each SF VM be configured with:

• 12 or more vCPUs (see vCPU and vRAM Options).

  Note	An SF VM will not boot and will report the following error if this minimum vCPU requirements is not met. Found hardware disparate in minimum number of cores, found n cores, minimum expected cores are 12.

• 32 GB or more vRAM (see vCPU and vRAM Options).

• First vNIC is the DI Network.

• Second and subsequent vNICs are service ports. The system supports up to 12 vNICs, but this number may be limited by the hypervisor.

• vHDD is for boot image, 2 GB recommended.

Note	All SF VMs must be identically configured. Refer to VM Hardware Verification for information on monitoring the VM hardware configuration.

A CPU is a single physical computer chip that can have more than one physical CPU core that is fully capable of running the entire system and applications by itself. Virtual core technology supports multiple logical processors (vCPUs) per physical core. The total number of vCPUs supported on a specific platform varies based on the number of available physical cores and the type of virtual core technology implemented in each core.

CF and SF run within VMs that are assigned a number of vCPUs, each supporting one thread (sequence of instructions). The number of available vCPUs supported by the platform CPU may exceed the maximum number of vCPUs that can be assigned to the VM via the hypervisor.

Note	The number vCPUs per VM should never exceed the maximum number of vCPUs supported by the platform CPU.

To maximize performance, it may be desirable to adjust the number of vCPUs or vRAM to align with the underlying hardware. SF supports varied vCPU and vRAM combinations, however all SFs must share the same combination within an instance.

Software will determine the optimal number of SESSMGR tasks per SF on startup of the SF based on the number of vCPUs and amount of vRAM on that SF.

In this release the supported vNIC options include:

• VIRTIO—Paravirtual NIC for KMV

• ixgbe—Intel 10 Gigabit NIC virtual function

• enic—Cisco UCS NIC

• SR-IOV—Single root I/O virtualization ixgbe and enic interfaces

The system implements a Virtio front-end based on a DPDK-based user application which can interact with both vhost-net and vhost-user based back-end mechanisms. Vhost-user and vhost-net provide an implementation of the vhost paradigm of using shared memory based, event, and interrupt descriptors. The DPDK-based front-end driver in combination with vhost-net and vhost-user provide a higher performance data path compared to a Linux bridge-based data path.

• Vhost-user provides packet handling completely in the user space, which improves performance. The system implements the front-end in a DPDK-based user space application, while a host user space application implements the back-end based on the vhost-user interface.

• Vhost-net provides the kernel-level back-end for Virtio networking that reduces virtualization overhead by moving Virtio packet processing tasks out of the user space (the QEMU process) and into the kernel (the vhost-net driver). This allows device emulation code to directly call into kernel subsystems, instead of performing system calls from the user space.

  The system supports single queue in vhost-user.

In additional to the mandatory /flash (non-RAID) drive, the system supports RAID1 under a virtual machine (VM). For each VM, Virtual SCSI disks can be created, on CF only, matching the SCSI ID shown in this table. The minimum disk size must be greater than 16 GB.

For record storage (CDRs and UDRs) the CF VM should be provisioned with a second vHDD sized to meet anticipated record requirements (minimum 16GB). Records will be written to /records on the second vHDD.

In order for the VMs within a UGP instance to communicate with each other, each instance must have a private L2 network that interconnects the VMs. This network should utilize a VLAN within the IaaS/virtualization infrastructure and be exposed untagged to each VM as the first vNIC.

The DI network must be for the exclusive use of a single UGP instance. No other devices may be connected to this network.

Note	If more than one instance is instantiated within the same datacenter, each instance must have its own DI network.

All the VMs within an instance must be physically located in the same site, ideally in the same few racks with minimal interconnecting devices. The reliability of the DI network is important for the stability of the UGP instance. Using L2 tunneling protocols across a WAN or congested links is highly discouraged.

Figure 2. DI Network

This network is for CF/SF and VNFM (ESC) communication. Specifically VNFM, CF and SF are attached to this network.

This is a dedicated network to provide service interfaces (for instance S11, S10, and so on). All SFs will provide service using this network segment. Individual SF can be configured to have up to 4 ports as maximum to connect to Service network. SF ports are used to receive and transmit bearer and signaling packets. To simplify network settings and address usage, only VLANs for high-bandwidth (bearer) packets need to be connected to all SFs. Low-bandwidth interfaces (signaling) can be connected to just two SFs.

SF ports are used to receive and transmit bearer and signaling packets. To simplify network settings and address usage, only VLANs for high-bandwidth (bearer) packets need to be connected to all SFs. Low-bandwidth interfaces (signaling) can be connected to just two SFs. In the diagrams below, the bearer VLANs are connected to all SFs, while signaling and other VLANs are only connected to the first two SFs.

Note	This asymmetric arrangement means that fewer interfaces are needed, however careful consideration should be paid to failures since the loss of two VMs results in loss of services.

ECMP does hashing based on a hash and can send traffic to any SF VM.

On ingress, the SFs perform flow lookups and direct packets to the specific SESSMGR task on a specific SF. Some of this ingress traffic is processed by local SESSMGR tasks, otherwise it is relayed via the DI network to the correct SF. On egress, each SF sends out packets from its local port (provided ECMP is used). In most cases, the number of VMs that packets traverse is less than two. However, ACLs and tunneling may increase the number of hops for specific flows depending on the EPC configuration.

The Intel Data Plane Development Kit (DPDK) is an integral part of the VPC architecture and is used to enhance system performance. The DPDK Internal Forwarder (IFTASK) is a software component that is responsible for packet input and output operations and provides a fast path for packet processing in the user space by bypassing the Linux kernel. It is required for system operation. Upon CF or SF instantiation, DPDK allocates a certain proportion of the CPU cores to IFTASK depending on the total number of CPU cores. The remaining CPU cores are allocated to applications.

To determine which CPU cores are used by IFTASK and view their utilization, use the show npu utilization table command as shown here:

[local]mySystem# show npu utilization table
 
Wednesday July 06 10:53:55 PDT 2017
          -------iftask-------
   lcore     now   5min  15min
--------  ------ ------ ------
  01/0/1     38%    53%    52%
  01/0/2     51%    55%    55%
  02/0/1     66%    72%    68%
  02/0/2     66%    63%    67%
  03/0/1     57%    55%    55%
  03/0/2     51%    47%    45%
  03/0/3     89%    89%    89%
  03/0/4     88%    88%    89%
  04/0/1     67%    59%    58%
  04/0/2     54%    40%    48%
  04/0/3     89%    89%    90%
  04/0/4     90%    89%    89%
  05/0/1     55%    55%    56%
  05/0/2     68%    45%    45%
  05/0/3     90%    89%    89%
  05/0/4     90%    89%    89%
  06/0/1     50%    58%    58%
  06/0/2     24%    24%    25%
  06/0/3     89%    90%    90%
  06/0/4     91%    90%    90%
  

To view CPU utilization for the VM without the IFTASK cores, use the show cpu info command. For more detailed information use the verbose keyword.

[local]mySystem# show cpu info card 6 
Tuesday July 05 10:39:52 PDT 2017
Card 6, CPU 0:
  Status                : Active, Kernel Running, Tasks Running
  Load Average          : 7.74, 7.62, 7.54 (9.44 max)
  Total Memory          : 49152M
  Kernel Uptime         : 4D 5H 7M
  Last Reading:
    CPU Usage           : 25.4% user, 7.8% sys, 0.0% io, 0.1% irq, 66.7% idle
    Poll CPUs           : 4 (1, 2, 3, 4)
    Processes / Tasks   : 177 processes / 35 tasks
    Network             : 164.717 kpps rx, 1025.315 mbps rx, 164.541 kpps tx, 1002.149 mbps tx
    File Usage          : 8256 open files, 4941592 available
    Memory Usage        : 21116M 43.0% used
  Maximum/Minimum:
    CPU Usage           : 32.9% user, 8.9% sys, 0.0% io, 0.4% irq, 59.1% idle
    Poll CPUs           : 4 (1, 2, 3, 4)
    Processes / Tasks   : 184 processes / 36 tasks
    Network             : 178.388 kpps rx, 1270.977 mbps rx, 178.736 kpps tx, 1168.999 mbps tx
    File Usage          : 8576 open files, 4941272 available
    Memory Usage        : 21190M 43.1% used
  

Modeling of bandwidth requirements on the L2 switches that host a UGP instance is required for each operator deployment.

In addition to the predominant bearer traffic, the DI network also passes session signaling and internal control data between the VMs.

Internal control traffic will be heavy during redundancy operations, but significantly less under normal operation. Heavy use of control traffic occurs during:

• Migrations of tasks from an active SF VM to the standby SF

• Startup or restart of a standby SF

• Startup or restart of an SF

• Startup or restart of an SF or standby CF

• Heavy signaling traffic (high Call Events per Second [CEP] rate)

• Significant CLI and/or Bulkstats usage

Depending on the CEPS rate, configuration, and management operations, each VM places a load on its DI network interface regardless of bearer throughput. This load is expected to be highly variable, but average less than 1 Gbps per VM with some VMs having higher usage than others.

When a UGP instance is deployed, there are several expectations of the environment on which UGP is running that are beyond the control of StarOS. Most of these fall into requirement of the Orchestration System.

• Provisioning of UGP VMs including install and parameters assignment: configuration, connectivity, and persistent block storage for each VM.

• L2 provisioning of the DI network to ensure that the DI network meets reliability requirements.

• Policy enforcement of network separation, if applicable.

• Physical placement of VMs that enforce redundancy rules.

• Providing useful monitoring tools for physical resources, such as CPU, RAM, NIC, etc.

If an orchestration system is not used to deploy a UGP instance, these requirements must still be maintained. However, they should be enforced manually or through other means. Refer to VM Hardware Verification for information on monitoring the VM hardware configuration.

Provisioning of a UGP instance has two phases:

• VMs and network interconnections are created and linked.

• UGP instance is configured for services.

IaaS administrators set up and interconnect the servers and use hypervisor VM templates or orchestration software to create a set of VMs, the DI network, and the redundancy configuration to meet Service Level Agreement (SLA) requirements.

Deploying a UGP instance requires a detailed configuration plan that addresses the operator's deployment requirements.

Sizing a UGP instance requires modeling of the expected call model.

Initial software versions support up to 2 CFs and 30 SFs.

Many service types require more resources than others. Packet size, throughput per session, CEPS (Call Events per Second) rate, IPSec usage (site-to-site, subscriber, LI), contention with other VMs, and the underlying hardware type (CPU speed, number of vCPUs) will further limit the effective number of maximum subscribers. Qualification of a call model on equivalent hardware and hypervisor configuration is required.

Software-based transmit batching greatly enhances the system performance.

Because UGP runs within VMs, no hardware diagnostics or monitoring are provided. Retrieval of hardware sensor data (temperature, voltage, memory errors) are accomplished via the hypervisor and external monitoring systems. To determine the configuration of the underlying VMs, refer to VM Hardware Verification.

UGP monitors and exports vCPU, vRAM, and vNIC usage per VM through existing mechanisms including CLI show commands, bulkstats and MIB traps. However, an operator may find that monitoring physical CPU, RAM, and NIC values per host in the hypervisor is more useful.

Because vNICs have a variable max throughput (not defined as 1 Gbps or 10 Gbps for example), counters and bulkstats that export utilization as a percentage of throughput may have little value. Absolute values (bps) can be obtained from the VM, but where possible physical infrastructure utilization should be obtained from the hypervisor. This would not apply to pass-through PF NICs, as those have a fixed maximum throughput.

The Cisco Prime for Mobility suite of analytics provides scalable management of a UGP instance.

Cisco Prime for Mobility supports the following:

• Integrated operator workflows across the Radio Access Network (RAN) backhaul and packet core

• Centralized network visibility and advanced troubleshooting and diagnostics

• Pre-integrated network management software components that reduce time and resources required for integration

For additional information, contact your Cisco account representative.

The following StarOS build filename types are associated with UGP:

• .qvpc-di-<version>.iso initial installation or startover ISO file.

• .qvpc-di-<version>.bin update, upgrade or recovery file for a system that is already running.

• .qvpc-di-template-libvirt-kvm-<version>.tgz KVM libvirt template plus ssi_install.sh.

• .qvpc-di.qcow2.tgz KVM QCOW2 disk template.

• .qvpc-di-template-vmware.tgz VMware files.

For additional information on supported operating system and hypervisor packages, as well as platform configurations, please contact your Cisco representative. The Cisco Advanced Services (AS) group offer consultation, installation and network deployment services for the UGP product.