Mobile Cloud Network (MCN) is a network infrastructure that includes Infrastructure as a Service (IaaS), the orchestration mechanisms, analytics mechanisms etc., upon which the VPC-SI as well as other services are deployed.

This figure below shows basic L2/L3 interconnection as supported by VPC-SI.

In the figure above, a virtual switch is embedded within the hypervisor to support SDN L2 capabilities across the data center. The virtual switch is interconnected to other virtual switches using 802.1Q trunks (VLANs). Typically, the virtual switch is a dynamically loaded kernel module.

VPC-SI is essentially StarOS running within a Virtual Machine (VM) on a COTS platform. It can be used as a stand-alone single VM within an enterprise, remote site, or customer data center. Alternatively, VPC-SI can be integrated as a part of a larger service provider orchestration solution.

The Single Instance architecture is best suited for low capacity scenarios. Scaling the VPC-SI Virtual Network Function (VNF) requires significant network level configurations for certain VNF types (such as, P-GW, S-GW, MME, PCRF, Firewall and NAT). For example, if a new VPC-SI P-GW is added or removed, various Diameter peers must be configured with this information DNS is provisioned or de-provisioned with this information.

VPC-SI only interacts with supported hypervisors KVM (Kernel-based Virtual Machine) and VMware ESXi. It has little or no knowledge of physical devices.

Typically, VPC-SI should be deployed in Interchassis Session Recovery (ICSR) pairs to provide physical redundancy in case of hardware or hypervisor failure.

Each VPC-SI VM takes on the roles of an entire StarOS system. The only interfaces exposed outside the VM are those for external management and service traffic. Each VM is managed independently.

Each VPC-SI VM performs the following StarOS functions:

• Controller tasks
• Out-of-band management for CLI and Logging (vSerial and vKVM)
• Local context vpnmgr
• Local context management (vNICs)
• System boot image and configuration storage on vHDD
• Record storage on vHDD
• NPU simulation via fastpath and slowpath
• Non-local context (vNICs, 1 to 12).
• Demux and vpnmgr for session processing
• Crypto processing

The VM is represented as a virtual card with a single CPU subsystem. This makes many CLI commands, logs, and functions work similarly to StarOS running on ASR 5500 platform.

Applications written for StarOS see VPC-SI as just another platform with a one-slot virtual chassis supporting a single virtual card.

StarOS concepts of contexts, services, pools, interfaces, cards, and ports exist on VPC-SI just as on existing platforms.

When the VM boots, the vNICs configured in the VM profile are detected and an equivalent number of 'Virtual Ethernet' type ports appear in the StarOS CLI.

Note

You cannot be guaranteed that the order of the vNICs as listed in the hypervisor CLI/GUI is the same as how the hypervisor offers them to VPC-SI. On initial setup you must use the show hardware CLI command to walk through the MAC addresses shown on the hypervisor's vNIC configuration and match them up with the MAC addresses learned by VPC-SI. This will confirm that the VPC-SI interfaces are connected to the intended BR group/Vswitch.

VMs within a VPC-SI instance perform software-based encryption and tunneling of packets (as opposed to the higher-throughput hardware-based services). Call models that make heavy use of encryption for bearer packets or have significant PKI (Public Key Infrastructure) key generation rates may require significant compute resources.

Security of external traffic including tunneling, encryption, Access Control Lists (ACLs), context separation, and user authentication function as on existing StarOS platforms. User ports and interfaces on the CFs and SFs are protected through StarOS CLI configuration.

The virtual system adds additional security concerns on the customer because network communication travel over the DI network on datacenter equipment.

The DI network must be isolated from other hosts within the datacenter by limiting membership in the system network's VLAN to VMs within that specific VPC-SI instance. Unauthorized access to the DI network through other hosts being inadvertently added to that network or the compromise of a router, switch or hypervisor could disrupt or circumvent the security measures of StarOS. Such disruptions can result in failures, loss of service, and/or exposure of control and bearer packets. Properly securing access to the DI network is beyond the control of StarOS.

Communication between DI network component (e.g. CF and SF) VMs is now only possibley via authentication over externally supplied SSH keys. In addition, the system enforces public/private key-based SSH authentication for logins within the DI network. No passwords, keys or LI information are stored or sent in clear text.

If an operator requires physical separation of networks, such as management versus bearer versus LI (Lawful Intercept), then physical separation of the DI network should also be done since it carries sensitive data. In a virtualized environment, the physical separation of networks may not be possible or practical. Operators that have these requirements may need to qualify their hypervisor and infrastructure to confirm that it will provide sufficient protection for their needs.

The virtual system relies on the underlying hardware and hypervisor for overall resource redundancy and availability. The StarOS handles the system redundancy on top of that.

The hardware and hypervisor should provide:

• Redundant hardware components where practical (such as power supplies, disks)

• Redundant network paths (dual fabric/NICs, with automatic failover)

• Redundant network uplinks (switches, routers, etc.)

High availability can only be achieved if the underlying infrastructure (hosts, hypervisor, and network) can provide availability and reliability that exceeds expected values. The system is only as reliable as the environment on which it runs.

Interchassis Session Recovery (ICSR) is also recommended to improve availability and recovery time in the case of a non-redundant hardware failure (such as CPU, memory, motherboard, hypervisor software). ICSR provides redundancy at the session level for gateways only.

VPC-SI supports ICSR between two instances for services that support ICSR in the StarOS software release. When more than one service type is in use, only those services that support ICSR will be able to use ICSR.

Note	The two VPC-DI instances must be on a different DI-Network.

ICSR supports redundancy for site/row/rack/host outages, and major software faults. To do so, the two instances should be run on non-overlapping hosts and network interconnects. ICSR is supported only between like-configured instances. ICSR between a VPC-SI instance and another type of platform is not supported.

L3 ICSR is supported.

For additional information, refer to the Interchassis Session Recovery chapter in this guide.

VPC-SI requires that the VM be configured with:

• X vCPUs (see vCPU and vRAM Options )

• Y vRAM (see vCPU and vRAM Options )

• First vNIC is the management port (see vNIC Options)

• Second and subsequent vNICs are service ports; one vNIC is required and up to 12 are supported by the VPC, but this number may be limited by the hypervisor

• First vHDD is for boot image and configuration storage (4 GB recommended)

• Second vHDD is for record storage [optional] (16 GB minimum)

A CPU is a single physical computer chip that can have more than one physical CPU core that is fully capable of running the entire system and applications by itself. Virtual core technology supports multiple logical processors (vCPUs) per physical core. The total number of vCPUs supported on a specific platform varies based on the number of available physical cores and the type of virtual core technology implemented in each core.

It is best practice to avoid spanning VM across NUMA nodes. The vCPU shall align with the underlying CPU. A VM should not span across NUMA node for performance optimization.

CF and SF run within VMs that are assigned a number of vCPUs, each supporting one thread (sequence of instructions). The number of available vCPUs supported by the platform CPU may exceed the maximum number of vCPUs that can be assigned to the VM via the hypervisor.

Note	The number vCPUs per VM should never exceed the maximum number of vCPUs supported by the platform CPU.

To maximize performance, it may be desirable to adjust the number of vCPUs or vRAM to align with the underlying hardware. SF supports varied vCPU and vRAM combinations, however all SFs must share the same combination within an instance.

Software will determine the optimal number of SESSMGR tasks per SF on startup of the SF based on the number of vCPUs and amount of vRAM on that SF.

In this release the supported vNIC options include:

• VMXNET3—Paravirtual NIC for VMware

• VIRTIO—Paravirtual NIC for KVM

• ixgbe—Intel 10 Gigabit NIC virtual function

• enic—Cisco UCS NIC

In additional to the mandatory /flash (non-RAID) drive, the system supports RAID1 under a virtual machine (VM). For each VM, Virtual SCSI disks can be created, on CF only, matching the SCSI ID shown in this table. The minimum disk size must be greater than 16 GB.

For record storage (CDRs and UDRs) the CF VM should be provisioned with a second vHDD sized to meet anticipated record requirements (minimum 16GB). Records will be written to /records on the second vHDD.

The structure of the boot parameters file is:

VARIABLE_NAME = VALUE

Specify one variable per line with a newline as the end of the line terminator (UNIX text file format). Variable names and values are case insensitive. Invalid values are ignored and an error indication is displayed on the VM console. If there are duplicate values for a variable (two different values specified for the same variable name), the last value defined is used.

Numeric values do not need to be zero padded. For example a PCI_ID of 0:1:1.0 is treated the same as 0000:01:01.0.

VPC-SI assigns the vNIC interfaces in the order offered by the hypervisor. You cannot be guaranteed that the order of the vNICs as listed in the hypervisor CLI/GUI is the same as how the hypervisor offers them to the VM.

The order that VPC-SI finds the vNICs is subject to the PCI bus enumeration order and even paravirtual devices are represented on the PCI bus. The PCI bus is enumerated in a depth first manner where bridges are explored before additional devices at the same level. If all the network interfaces are of the same type then knowing the PCI topology is sufficient to get the vNIC order correct. If the network interfaces are of different types, then the order is dependent on the PCI topology plus the device driver load order inside the VM. The device driver load order is not guaranteed to be the same from software release to release but in general paravirtual devices are prior to pass-through devices.

There are several methods available to identify NICs.

• MAC address: MAC address of the interface

• Virtual PCI ID

• Bonded interfaces: When using network device bonding, network interfaces are identified to serve as the slave interface role. The slave interfaces in the bond are identified using MAC, PCI ID, or Interface type.

• Interface type and instance number.

Virtual PCI ID

Devices on a PCI bus are identified by a unique tuple known as the domain, bus, device, and function numbers. These identifiers can be identified in several ways.

Inside the guest, the lspci utility shows the bus configuration:

# lspci

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer
00:04.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
00:06.0 Ethernet controller: Red Hat, Inc Virtio network device
  

The domain, bus, device, and function numbers for this virtual bus are shown here:

For libvirt-based virtual machines, you can get the virtual PCI bus topology from the virsh dumpxml command. Note that the libvirt schema uses the term slot for the device number. This is a snippet of the xml description of the virtual machine used in the previous example:

    <interface type='bridge'>
      <mac address='52:54:00:c2:d0:5f'/>
      <source bridge='br3043'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <driver name='vhost' queues='8'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:c3:60:eb'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
  

Interface Type and Instance Number

Here the NIC is identified by its type using its Linux device driver name (virtio_net, vmxnet3, ixgbe, i40e, etc) and its instance number. The instance number is based on PCI enumeration order for that type of interface starting at instance number 1. The interface type is available to identify both paravirtual types as well as pass-through interfaces and SR-IOV virtual functions. The PCI enumeration order of devices on the PCI bus can be seen from the lspci utility, which is on the host OS.

For example, a CF with the following guest PCI topology indicates that virtio_net interface number1 is the Ethernet controller at 00:05.0 and virtio_net interface number 2 is the Ethernet Controller at 00:06.0. The output is from the lspci command executed in the guest:

# lspci

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer
00:04.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
00:06.0 Ethernet controller: Red Hat, Inc Virtio network device
  

Here is the complete list of the supported Linux drivers:

** Note: A known issue exists where MAC address assignment does not occur dynamically for SRIOV VFs created on the host when using the i40e driver. MAC address assignment is necessary to boot the StarOS VM. As a workaround, MAC address assignment must be configured from the host. Refer to the following link for more information:https://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.pdf

VPC-SI VMs have one interface configured to connect to the management network. This interface is typically configured in StarOS and should be part of the Day 0 configuration. The management interface supports static address assignment through the main StarOS configuration file.

An additional 0 to 4 network interfaces serve as service ports. These interfaces are configured by StarOS. Typically these ports are configured as trunk ports in the VNF infrastructure (VNFI).

VPC-SI VMs have the option of having a network interface that is connected to the virtual network function (VNF) manager (VNFM) if it exists. This interface can be configured via DHCP or static IP assignment and is used to talk to a VNFM or higher level orchestrator. This interface is enabled before the main application starts.

Example params.cfg

MGMT_INTERFACE=MAC:00:01:02:03:04:05
SERVICE1_INTERFACE=PCI_ID:0000:01:02.0
SERVICE2_INTERFACE=PCI_ID:0000:01:03.0
VNFM_INTERFACE=MAC:00:01:02:03:04:AA
VNFM_IPV4_ENABLE=true
VNFM_IPV4_DHCP_ENABLE=false
VNFM_IPV4_ADDRESS=10.1.1.100
VNFM_IPV4_NETMASK=255.255.255.0
VNFM_IPV4_GATEWAY=10.1.1.1
IFTASK_MCDMA_CORES=20
CHASSIS_ID=2sg9x1wqbj
IFTASK_SERVICE_TYPE=0

Use the StarOS command show cloud hardware iftask to verify that the iftask boot parameters took effect.

Note	These configuration options are optional.

The virtual network functions manager (VNFM) interface is designed to communicate between each VM and a VNFM. The VNFM interface initializes before the main application and only boot parameters can configure the interface.

The VNFM interface is disabled by default.

Enable VNFM IPv4 Interface

The default value is False (disabled).

Configure IPv4 DHCP Client

Configure IPv4 Static IP

Note	If IPv4 DHCP client is enabled, static configuration parameters are ignored.

Enable VNFM IPv6 Interface.

Enable IPv6 Static IP Configuration

If set to true, static IP parameters configuration applies to the interface as shown in the following section. If set to false, the interface attempts to use both stateless autoconfiguration (RFC4862) and DHCPv6 to configure the address of the interface.

Configure IPv6 Static IP

Note	If the "VNFM_IPV6_ENABLE" parameter value is set to false, the static configuration parameters are ignored. The IPv6 address field should conform to RFC 5952. Prefix is fixed at /64.

The following sections list the parameters that are applicable only when the FORWARDER_TYPE selected is VPP. These parameters enable a fine-grained control over the CPU of the VPP and the interface configuration.

Note	Before overriding any of the VPP configuration parameters, ensure that you contact your Cisco account representative to help identify the override values.

VPP CPU Assignment

VPP workers are real-time threads that consume an entire CPU core. While the VPP main thread does not consume an entire core, it can be busy. Therefore, assign it to avoid conflicts.

The following table lists the VPP-CPU parameters.

VPP_CPU_MAIN=1

VPP_CPU_WORKER_CNT=3

VPP_CPU_WORKER_LIST=2,9,10

Default DPDK Configuration

The following parameters that are listed in the table configure DPDK in general or set the interface defaults.

VPP_DPDK_BUFFERS=200000

VPP_DPDK_RX_QUEUES=2

VPP_DPDK_TX_QUEUES=4

VPP_DPDK_RX_DESCS=256

VPP_DPDK_TX_DESCS=512

Interface-Specific Configuration

The following parameters that are listed in the table refine individual interfaces. These parameters also override the default DPDK configuration, where applicable.

SERVICE1_INTERFACE_VPP_RX_QUEUES=2

SERVICE1_INTERFACE_VPP_RX_DESCS=1024

SERVICE1_INTERFACE_VPP_WORKER_LIST=2,3